Skip to content

Next

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
    • Help
    • Submit feedback
    • Contribute to GitLab
    • Switch to GitLab Next
  • Sign in / Register
Minds Backend - Engine
Minds Backend - Engine
  • Project
    • Project
    • Details
    • Activity
    • Releases
    • Cycle Analytics
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Charts
    • Locked Files
  • Issues 138
    • Issues 138
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 29
    • Merge Requests 29
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Charts
  • Registry
    • Registry
  • Packages
    • Packages
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Charts
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • Minds
  • Minds Backend - EngineMinds Backend - Engine
  • Merge Requests
  • !159

Open
Opened 3 weeks ago by Brian Hatchet@brianhatchet:speech_balloon:
  • Report abuse
Report abuse

WIP: Firehose api

New firehose admin endpoints for getting an unsorted list of activites from minds_badger and an admin endpoint that takes an activity guid and will mark it as moderated.

New managers and repositories for talking to elastic search Redis lock system with some super basic hash keying. These records shouldn't get very big, so we might want to KISS for now. Else, there are distributed locking libraries we should look into

It stores a key in moderation:entity_id:user_id format where the entity is the activity being moderated and user is the admin who "checked it out". These are set to a TTL of 15 minutes for now. When we query elastic search, we look for the opposite set (any locks that DON'T belong to the user) and for any activity that has not already been moderated. Requires adding two new fields to badger (in postman)

Edited 2 weeks ago by Brian Hatchet

Check out, review, and merge locally

Step 1. Fetch and check out the branch for this merge request

git fetch https://gitlab.com/brianhatchet/engine.git admin_firehose
git checkout -b brianhatchet/engine-admin_firehose FETCH_HEAD

Step 2. Review the changes locally

Step 3. Merge the branch and fix any conflicts that come up

git fetch origin
git checkout origin/master
git merge --no-ff brianhatchet/engine-admin_firehose

Step 4. Push the result of the merge to GitLab

git push origin master

Note that pushing to GitLab requires write access to this repository.

Tip: You can also checkout merge requests locally by following these guidelines.

Request to merge brianhatchet:admin_firehose into master
Open in Web IDE
  • Email patches
  • Plain diff
Pipeline #63781138 failed for 3cecc956 on brianhatchet:admin_firehose
      Requires 2 more approvals.
      This is a Work in Progress
      Allows commits from members who can merge to the target branch

      Deletes source branch

      You can merge this merge request manually using the
      • Discussion 9
      • Commits 20
      • Pipelines 19
      • Changes 19
      2/8 discussions resolved
      • Loading...
      • Brian Hatchet :speech_balloon: @brianhatchet changed milestone to %sprint: Dapper Dingo 3 weeks ago

        changed milestone to %sprint: Dapper Dingo

      • Brian Hatchet :speech_balloon: @brianhatchet added 7 commits 3 weeks ago

        added 7 commits

        • 269df1ee...1eaf8c06 - 5 commits from branch minds:epic/ReportingAndModeration
        • 50fabf98 - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose
        • a1c2f3a6 - Fixing previous tests

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 3 weeks ago

        added 1 commit

        • e881f09a - Refactored to use the redis cache

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 2 weeks ago

        added 1 commit

        • 64942ca3 - Fixing unit tests by not overriding DI config in features. Thanks, Mark <3

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 37 commits 2 weeks ago

        added 37 commits

        • 64942ca3...5f29f688 - 36 commits from branch minds:epic/ReportingAndModeration
        • feb5a3b0 - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose

        Compare with previous version

      • Mark Harding
        Mark Harding @markeharding started a discussion on the diff 2 weeks ago
        Resolved by Mark Harding 54 minutes ago
      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 2 weeks ago

        added 1 commit

        • efa38fcc - Updated tested to reflect the new elastic search query

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 3 commits 2 weeks ago

        added 3 commits

        • efa38fcc...9dbd95e5 - 2 commits from branch minds:epic/ReportingAndModeration
        • 8a3c9f0a - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 2 weeks ago

        added 1 commit

        • 96127343 - MR fixes. Performant ES query. Mapping fixes. Moved reconstitution to the manager

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 2 weeks ago

        added 1 commit

        • 66977958 - Moving save to the manager

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 2 weeks ago

        added 1 commit

        • 77c99287 - firehose manager reporting

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 2 weeks ago

        added 1 commit

        • b1701f83 - Bug fixes

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet unmarked as a Work In Progress 2 weeks ago

        unmarked as a Work In Progress

      • Brian Hatchet :speech_balloon: @brianhatchet changed title from WIP firehose api to Firehose api 2 weeks ago

        changed title from WIP firehose api to Firehose api

      • Brian Hatchet :speech_balloon: @brianhatchet changed the description 2 weeks ago

        changed the description

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 2 weeks ago

        added 1 commit

        • c5cf7221 - Merging upstream

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 16 commits 1 week ago

        added 16 commits

        • c5cf7221...3927173f - 15 commits from branch minds:epic/ReportingAndModeration
        • 2dcea399 - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 51 commits 6 days ago

        added 51 commits

        • 2dcea399...eadf4ebd - 50 commits from branch minds:epic/ReportingAndModeration
        • 74c34afb - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 2 commits 6 days ago

        added 2 commits

        • 9dc49442 - 1 commit from branch minds:epic/ReportingAndModeration
        • e3a25163 - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 6 commits 6 days ago

        added 6 commits

        • e3a25163...dd117ff0 - 5 commits from branch minds:epic/ReportingAndModeration
        • a3dcc56b - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 24 commits 1 day ago

        added 24 commits

        • bd6e7d55 - (feat) add report reasons to config endpoint
        • 25b05ce9 - (chore): ban spam domain
        • d07486fc - (chore): ban spammy email
        • b4a60db8 - (fix): Re-sync ES if expired boost was already completed (#448 (closed))
        • e9491d70 - Merge branch 'hotfix/es-boosts-resync' into 'master'
        • 56b60f52 - Merge branch 'sprint/ElevatedEagle.feat.report-reasons.added.minds-config.endpoint' into 'master'
        • cb33fe87 - Merge branch 'epic/ReportingAndModeration'
        • e0ced6e2 - (chore): content policy
        • a593916e - (chore): if channel reported for spam then do immediate ban
        • 1dc7c4b9 - [Sprint/FunnyFrog])(bug): Forcing NSFW reasons array into integers.
        • fcfd028f - Merge branch 'fix-nsfw-remind-inheritance-bug' into 'master'
        • 35a80e27 - (chore): change SEO to The Jury System
        • ef9255df - (fix): remove revoked from stats
        • 8634a913 - CORS enabled
        • c710bdef - Merge branch 'checkout-oauth' into 'master'
        • 8d0ccfc1 - Sprint/ElevatedEagle: Forced querystring params into lower case
        • 8bd9d70c - Merge branch 'force-lower-case-search-query-and-hashtags' into 'master'
        • 32b1f6ff - (chore): email on a friday
        • cf10fc88 - (chore); email on a friday...
        • e1ec640d - (fix): typos in email
        • 0c777244 - (chore): loop email iterator if no entities returned
        • 7546fd46 - (chore): dev code for promotions email
        • 827ae8e9 - (chore): email promotions to use markdown and include template key in hash
        • e605f46b - Merge remote-tracking branch 'upstream/master' into admin_firehose

        Compare with previous version

        Toggle commit list
      • Brian Hatchet :speech_balloon: @brianhatchet added 8 commits 3 hours ago

        added 8 commits

        • f75596ab - (chore): ban more domains
        • 01e1a432 - (chore): bring back channel level spam check for new channels
        • 85de3a3c - (chore): add another blocked domain
        • e40d62d3 - (chore): spam check comments
        • 23f0c499 - (chore): Appeals loop and awaiting TTL (#463)
        • 5b0ef158 - Merge branch 'hotfix/appeal-summons-tweaks-1' into 'master'
        • 294a8bc2 - (feat): support onchain boost label
        • f3d0b5ee - Merge remote-tracking branch 'upstream/master' into admin_firehose

        Compare with previous version

        Toggle commit list
      • Mark Harding @markeharding changed target branch from epic/ReportingAndModeration to master 56 minutes ago

        changed target branch from epic/ReportingAndModeration to master

      • Mark Harding
        Mark Harding @markeharding started a discussion on the diff 55 minutes ago
        Controllers/api/v2/admin/firehose.php
        26
        27 return Factory::response([
        28 'status' => 'success',
        29 'entities' => Exportable::_($activities),
        30 ]);
        31 }
        32
        33 public function post($pages)
        34 {
        35 if (!is_numeric($pages[0])) {
        36 header('X-Minds-Exception: activity guid required');
        37 http_response_code(400);
        38 return Factory::response([]);
        39 }
        40
        41 $activity = new Activity($pages[0]);
        • Mark Harding
          Mark Harding @markeharding · 55 minutes ago
          Owner

          I don't think we can assume it will be an activity. EntitiesBuilder->single() should provide the lookup for you with the correct entity type object

        Please register or sign in to reply
      • Mark Harding
        Mark Harding @markeharding started a discussion on an old version of the diff 55 minutes ago
        Last updated by Brian Hatchet 41 minutes ago
        Controllers/api/v2/admin/firehose.php
        10 use Minds\Entities\Activity;
        11
        12 class firehose implements Interfaces\Api, Interfaces\ApiAdminPam
        13 {
        14 /**
        15 * Gets a list of entities sorted for admin approval.
        16 *
        17 * @param array $pages
        18 *
        19 * @throws \Exception
        20 */
        21 public function get($pages)
        22 {
        23 /** @var Core\Feeds\Firehose\Manager $manager */
        24 $manager = Di::_()->get('Feeds\Firehose\Manager');
        25 $activities = $manager->getList(Session::getLoggedinUser());
        • Mark Harding
          Mark Harding @markeharding · 55 minutes ago
          Owner

          I think an array here would bring it in line with other manager ->getList functions.

          eg:

          ->getList([
            'user' => Session::getLoggedinUser(),
          ])
        • Brian Hatchet :speech_balloon: @brianhatchet changed this line in version 20 of the diff 41 minutes ago

          changed this line in version 20 of the diff

        Please register or sign in to reply
      • Mark Harding
        Mark Harding @markeharding started a discussion on the diff 53 minutes ago
        Core/Feeds/Firehose/Manager.php
        24 Repository $repository = null,
        25 EntitiesBuilder $entitiesBuilder = null,
        26 UserReportManager $reportManager = null
        27 ) {
        28 $this->repository = $repository ?: new Repository();
        29 $this->entitiesBuilder = $entitiesBuilder ?: new EntitiesBuilder();
        30 $this->reportManager = $reportManager ?: Di::_()->get('Moderation\Reports\Manager');
        31 }
        32
        33 /**
        34 * @param User $user the user who own the cached keys. No user, no caching
        35 * @param array $opts filtering options
        36 *
        37 * @return array entities that don't contain moderator_guids
        38 */
        39 public function getList(User $user = null, array $opts = [])
        • Mark Harding
          Mark Harding @markeharding · 53 minutes ago
          Owner

          As above, maybe merge in $user to $opts

        Please register or sign in to reply
      • Mark Harding
        Mark Harding @markeharding started a discussion on the diff 52 minutes ago
        Core/Feeds/Firehose/Repository.php
        25 Config $config = null,
        26 ModerationCache $cache = null)
        27 {
        28 $this->client = $client ?: Di::_()->get('Database\ElasticSearch');
        29 $this->config = $config ?: Di::_()->get('Config');
        30 $this->cache = $cache ?: new ModerationCache();
        31 $this->index = $this->config->get('elasticsearch')['index'];
        32 }
        33
        34 /**
        35 * @param User $user the user who own the cached keys. No user, no caching
        36 * @param array $opts filtering options
        37 *
        38 * @return Response $reponse Response object containing activity guids
        39 */
        40 public function getList(User $user = null, array $opts = [])
        • Mark Harding
          Mark Harding @markeharding · 52 minutes ago
          Owner

          Ideally Repository would be an interface that would catch this, but I think we should just send an array as a single param

        Please register or sign in to reply
      • Mark Harding
        Mark Harding @markeharding started a discussion on the diff 51 minutes ago
        Resolved by Mark Harding 24 minutes ago
      • Mark Harding
        Mark Harding @markeharding started a discussion on the diff 51 minutes ago
        Entities/Activity.php
        691 697 return $this->ownerObj;
        692 698 }
        693 699
        700 public function setModeratorGuid($moderatorGuid) {
        701 $this->moderator_guid = $moderatorGuid;
        702 }
        • Mark Harding
          Mark Harding @markeharding · 51 minutes ago
          Owner

          spacing is little off

        Please register or sign in to reply
      • Mark Harding
        Mark Harding @markeharding started a discussion on the diff 50 minutes ago
        Entities/Activity.php
        691 697 return $this->ownerObj;
        692 698 }
        693 699
        700 public function setModeratorGuid($moderatorGuid) {
        • Mark Harding
          Mark Harding @markeharding · 50 minutes ago
          Owner

          Can we get some phpdocs in ?

        Please register or sign in to reply
      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 41 minutes ago

        added 1 commit

        • 3cecc956 - WIP using top feed manager

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet marked as a Work In Progress from brianhatchet/engine@3cecc956 41 minutes ago

        marked as a Work In Progress from brianhatchet/engine@3cecc956

      • You're only seeing other activity in the feed. To add a comment, switch to one of the following options.
      Please register or sign in to reply
      Assignee
      Brian Hatchet's avatar Brian Hatchet @brianhatchet
      Assign to
      sprint: Dapper Dingo
      Milestone
      sprint: Dapper Dingo
      Assign milestone
      None
      Time tracking
      No estimate or time spent
      0
      Labels
      None
      Assign labels
      • View project labels
      Lock merge request
      Unlocked
      2
      2 participants
      user avatar
      Brian Hatchet
      user avatar
      Mark Harding
      Reference: minds/engine!159