Skip to content

Next

  • Projects
  • Groups
  • Snippets
  • Help
  • Sign in / Register
Minds Backend - Engine
Minds Backend - Engine
  • Project overview
  • Repository
  • Issues 298
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 39
  • CI / CD
  • Security & Compliance
  • Packages
  • Wiki
  • Snippets
  • Members
  • Collapse sidebar
  • Graph
  • Charts
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • Minds
  • Minds Backend - EngineMinds Backend - Engine
  • Issues
  • #1120

Closed
Open
Opened 4 hours ago by Guy Thouret@gthouret
Report abuse New issue

Thumbnail request causing OOM in PHP

Sentry Issue: ENGINE-HF

ErrorException: Allowed memory size of 134217728 bytes exhausted (tried to allocate 116526568 bytes)
  File "/lib/elgglib.php", line 947, in fatalErrorShutdownHandler
    \Sentry\captureLastError();
  File "[internal]", line 0

Related issues
...

    • Discussion 10
    • Designs 0
    • Guy Thouret @gthouret added Priority::1 - High Regression::Production Sprint::11/06 - Rolling Rabbit Squad::Blue Status::InProgress Type::Bug scoped labels 4 hours ago

      added scoped labels

    • Guy Thouret
      Guy Thouret @gthouret · 4 hours ago
      Developer

      Example request: https://www.minds.com/fs/v1/thumbnail/1026508015016828928/xlarge

      Link to media page: https://www.minds.com/media/1026508015016828928

    • Guy Thouret
      Guy Thouret @gthouret · 4 hours ago
      Developer

      Offending section of code in /api/v1/thumbnail.php:

                  try {
                      $finfo = new \finfo(FILEINFO_MIME);
                      $contentType = $finfo->buffer($contents) ?: 'image/jpeg';
                  } catch (\Exception $e) {
                      error_log($e);
                      $contentType = 'image/jpeg';
                  }

      Issue appears related to finfo under PHP 7.3.

      Running a referenced test script show the clear performance degradation between our 7.1 and 7.3 builds when calling finfo.

    • Guy Thouret
      Guy Thouret @gthouret · 4 hours ago
      Developer

      While I'm not able to reproduce the OOM condition in our sandbox, I am able to demonstrate the performance degradation.

    • Guy Thouret
      Guy Thouret @gthouret · 4 hours ago
      Developer

      PHP 7.1

      /var/www/Minds/engine # ITERS=100 php test.php
      Took: 0.094574928283691
      /var/www/Minds/engine # ITERS=1000 php test.php
      Took: 0.91444492340088

      PHP 7.3

      /var/www/Minds/engine # ITERS=100 php test.php
      Took: 0.42086601257324
      /var/www/Minds/engine # ITERS=1000 php test.php
      Took: 4.215017080307
    • Guy Thouret
      Guy Thouret @gthouret · 4 hours ago
      Developer

      Likely this is a good place to start looking for the OOM issue.

    • Guy Thouret
      Guy Thouret @gthouret · 2 hours ago
      Developer

      The referenced performance relates to finfo::file() whereas we are using finfo::buffer().

      Using finfo::buffer() does not cause a slow down.

      Continuing by instrumenting some mock code to show memory usage before and after finfo::buffer()

    • Guy Thouret
      Guy Thouret @gthouret · 2 hours ago
      Developer

      There is also the possibility that something prior to this call is using too much memory and this final call tips it over the limit where it would not have done previously.

    • Guy Thouret
      Guy Thouret @gthouret · 1 hour ago
      Developer

      The use of $finfo->buffer() to get the mime type has an impact on the memory overhead of the script. The difference between PHP 7.1 and PHP 7.3 is minimal.

      I suspect the real issue is that the entire image file is read into memory from S3 and then passed around before being written to the HTTP response.

      After a couple of requests these values are stabilised:

      finfo usage: 3585240
      after usage: 3782552
      total usage: 3782552

      finfo is the memory delta just before and after the finfo calls. total is the delta from the start of request to the end after is the memory usage at the end of the script

      finfo call accounts for most of the total memory usage of the script though is probably just when the memory is assigned for the file.

    • Guy Thouret
      Guy Thouret @gthouret · 1 hour ago
      Developer

      With and S3 backend, we already set the mime type to the metadata using the same finfo call when the file was written so we shouldn't need to repeat this process again.

      Looking to see if we can extend the S3 calls to fetch the metadata...

    • Guy Thouret
      Guy Thouret @gthouret · 1 hour ago
      Developer

      Sentry issue: ENGINE-HF

    Please register or sign in to reply
    Assignee
    Guy Thouret's avatar
    Guy Thouret @gthouret
    none
    Epic
    None
    None
    Milestone
    None
    Time tracking
    No estimate or time spent
    None
    Due date
    None
    6
    Labels
    Priority::1 - High Regression::Production Sprint::11/06 - Rolling Rabbit Squad::Blue Status::InProgress Type::Bug
    None
    Weight
    None
    Confidentiality
    Not confidential
    Lock issue
    Unlocked
    1
    1 participant
    user avatar
    Guy Thouret
    Reference: minds/engine#1120