Skip to content

Next

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
    • Help
    • Submit feedback
    • Contribute to GitLab
    • Switch to GitLab Next
  • Sign in / Register
Minds Backend - Engine
Minds Backend - Engine
  • Project
    • Project
    • Details
    • Activity
    • Releases
    • Cycle Analytics
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Charts
    • Locked Files
  • Issues 146
    • Issues 146
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 37
    • Merge Requests 37
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Charts
  • Registry
    • Registry
  • Packages
    • Packages
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Charts
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • Minds
  • Minds Backend - EngineMinds Backend - Engine
  • Merge Requests
  • !159

Open
Opened 1 month ago by Brian Hatchet@brianhatchet:speech_balloon:
  • Report abuse
Report abuse

Firehose api

New firehose endpoint uses the top feed algorithms to deliver reserved entities for users to moderate.

New managers and repositories for talking to elastic search via the top feed managers Redis lock system with some super fast hash keying. These records shouldn't get very big, so we might want to KISS for now. Else, there are distributed locking libraries we should look into

It stores a key in as set entity_id:user_id format where the entity is the activity being moderated and user is the admin who "checked it out". These are set to a TTL of 10 minutes for now. When we query elastic search, we look for the opposite set (any locks that DON'T belong to the user) and for any activity that has not already been moderated. Requires adding two new fields to badger (in postman)

Edited 1 week ago by Brian Hatchet

Check out, review, and merge locally

Step 1. Fetch and check out the branch for this merge request

git fetch https://gitlab.com/brianhatchet/engine.git admin_firehose
git checkout -b brianhatchet/engine-admin_firehose FETCH_HEAD

Step 2. Review the changes locally

Step 3. Merge the branch and fix any conflicts that come up

git fetch origin
git checkout origin/master
git merge --no-ff brianhatchet/engine-admin_firehose

Step 4. Push the result of the merge to GitLab

git push origin master

Note that pushing to GitLab requires write access to this repository.

Tip: You can also checkout merge requests locally by following these guidelines.

Request to merge brianhatchet:admin_firehose into master
The source branch is 8 commits behind the target branch
Open in Web IDE
  • Email patches
  • Plain diff
Pipeline #64494935 (#391) passed for 1a2d4ea5 on brianhatchet:admin_firehose
      Requires approval. Approved by
      Mark Harding
      Mark Harding
      Ready to be merged automatically. Ask someone with write access to this repository to merge this request

      Allows commits from members who can merge to the target branch

      Deletes source branch

      • Discussion 15
      • Commits 30
      • Pipelines 28
      • Changes 22
      7/8 discussions resolved
      • Loading...
      • Brian Hatchet :speech_balloon: @brianhatchet changed milestone to %sprint: Dapper Dingo 1 month ago

        changed milestone to %sprint: Dapper Dingo

      • Brian Hatchet :speech_balloon: @brianhatchet added 7 commits 1 month ago

        added 7 commits

        • 269df1ee...1eaf8c06 - 5 commits from branch minds:epic/ReportingAndModeration
        • 50fabf98 - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose
        • a1c2f3a6 - Fixing previous tests

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 1 month ago

        added 1 commit

        • e881f09a - Refactored to use the redis cache

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 1 month ago

        added 1 commit

        • 64942ca3 - Fixing unit tests by not overriding DI config in features. Thanks, Mark <3

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 37 commits 1 month ago

        added 37 commits

        • 64942ca3...5f29f688 - 36 commits from branch minds:epic/ReportingAndModeration
        • feb5a3b0 - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose

        Compare with previous version

      • Mark Harding
        Mark Harding @markeharding started a discussion on the diff 1 month ago
        Resolved by Mark Harding 1 week ago
      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 1 month ago

        added 1 commit

        • efa38fcc - Updated tested to reflect the new elastic search query

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 3 commits 1 month ago

        added 3 commits

        • efa38fcc...9dbd95e5 - 2 commits from branch minds:epic/ReportingAndModeration
        • 8a3c9f0a - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 1 month ago

        added 1 commit

        • 96127343 - MR fixes. Performant ES query. Mapping fixes. Moved reconstitution to the manager

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 1 month ago

        added 1 commit

        • 66977958 - Moving save to the manager

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 4 weeks ago

        added 1 commit

        • 77c99287 - firehose manager reporting

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 4 weeks ago

        added 1 commit

        • b1701f83 - Bug fixes

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet unmarked as a Work In Progress 3 weeks ago

        unmarked as a Work In Progress

      • Brian Hatchet :speech_balloon: @brianhatchet changed title from WIP firehose api to Firehose api 3 weeks ago

        changed title from WIP firehose api to Firehose api

      • Brian Hatchet :speech_balloon: @brianhatchet changed the description 3 weeks ago

        changed the description

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 3 weeks ago

        added 1 commit

        • c5cf7221 - Merging upstream

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 16 commits 3 weeks ago

        added 16 commits

        • c5cf7221...3927173f - 15 commits from branch minds:epic/ReportingAndModeration
        • 2dcea399 - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 51 commits 2 weeks ago

        added 51 commits

        • 2dcea399...eadf4ebd - 50 commits from branch minds:epic/ReportingAndModeration
        • 74c34afb - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 2 commits 2 weeks ago

        added 2 commits

        • 9dc49442 - 1 commit from branch minds:epic/ReportingAndModeration
        • e3a25163 - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 6 commits 2 weeks ago

        added 6 commits

        • e3a25163...dd117ff0 - 5 commits from branch minds:epic/ReportingAndModeration
        • a3dcc56b - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 24 commits 2 weeks ago

        added 24 commits

        • bd6e7d55 - (feat) add report reasons to config endpoint
        • 25b05ce9 - (chore): ban spam domain
        • d07486fc - (chore): ban spammy email
        • b4a60db8 - (fix): Re-sync ES if expired boost was already completed (#448 (closed))
        • e9491d70 - Merge branch 'hotfix/es-boosts-resync' into 'master'
        • 56b60f52 - Merge branch 'sprint/ElevatedEagle.feat.report-reasons.added.minds-config.endpoint' into 'master'
        • cb33fe87 - Merge branch 'epic/ReportingAndModeration'
        • e0ced6e2 - (chore): content policy
        • a593916e - (chore): if channel reported for spam then do immediate ban
        • 1dc7c4b9 - [Sprint/FunnyFrog])(bug): Forcing NSFW reasons array into integers.
        • fcfd028f - Merge branch 'fix-nsfw-remind-inheritance-bug' into 'master'
        • 35a80e27 - (chore): change SEO to The Jury System
        • ef9255df - (fix): remove revoked from stats
        • 8634a913 - CORS enabled
        • c710bdef - Merge branch 'checkout-oauth' into 'master'
        • 8d0ccfc1 - Sprint/ElevatedEagle: Forced querystring params into lower case
        • 8bd9d70c - Merge branch 'force-lower-case-search-query-and-hashtags' into 'master'
        • 32b1f6ff - (chore): email on a friday
        • cf10fc88 - (chore); email on a friday...
        • e1ec640d - (fix): typos in email
        • 0c777244 - (chore): loop email iterator if no entities returned
        • 7546fd46 - (chore): dev code for promotions email
        • 827ae8e9 - (chore): email promotions to use markdown and include template key in hash
        • e605f46b - Merge remote-tracking branch 'upstream/master' into admin_firehose

        Compare with previous version

        Toggle commit list
      • Brian Hatchet :speech_balloon: @brianhatchet added 8 commits 1 week ago

        added 8 commits

        • f75596ab - (chore): ban more domains
        • 01e1a432 - (chore): bring back channel level spam check for new channels
        • 85de3a3c - (chore): add another blocked domain
        • e40d62d3 - (chore): spam check comments
        • 23f0c499 - (chore): Appeals loop and awaiting TTL (#463 (closed))
        • 5b0ef158 - Merge branch 'hotfix/appeal-summons-tweaks-1' into 'master'
        • 294a8bc2 - (feat): support onchain boost label
        • f3d0b5ee - Merge remote-tracking branch 'upstream/master' into admin_firehose

        Compare with previous version

        Toggle commit list
      • Mark Harding @markeharding changed target branch from epic/ReportingAndModeration to master 1 week ago

        changed target branch from epic/ReportingAndModeration to master

      • Mark Harding
        Mark Harding @markeharding started a discussion on an old version of the diff 1 week ago
        Resolved by Brian Hatchet 1 week ago
      • Mark Harding
        Mark Harding @markeharding started a discussion on an old version of the diff 1 week ago
        Resolved by Brian Hatchet 1 week ago
      • Mark Harding
        Mark Harding @markeharding started a discussion on an old version of the diff 1 week ago
        Resolved by Brian Hatchet 1 week ago
      • Mark Harding
        Mark Harding @markeharding started a discussion on an old version of the diff 1 week ago
        Resolved by Brian Hatchet 1 week ago
      • Mark Harding
        Mark Harding @markeharding started a discussion on an old version of the diff 1 week ago
        Resolved by Mark Harding 1 week ago
      • Mark Harding
        Mark Harding @markeharding started a discussion on an old version of the diff 1 week ago
        Resolved by Brian Hatchet 1 week ago
      • Mark Harding
        Mark Harding @markeharding started a discussion on an old version of the diff 1 week ago
        Last updated by Brian Hatchet 1 week ago
        Entities/Activity.php
        691 697 return $this->ownerObj;
        692 698 }
        693 699
        700 public function setModeratorGuid($moderatorGuid) {
        • Mark Harding
          Mark Harding @markeharding · 1 week ago
          Owner

          Can we get some phpdocs in ?

        • Brian Hatchet :speech_balloon: @brianhatchet changed this line in version 21 of the diff 1 week ago

          changed this line in version 21 of the diff

        Please register or sign in to reply
      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 1 week ago

        added 1 commit

        • 3cecc956 - WIP using top feed manager

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet marked as a Work In Progress from brianhatchet/engine@3cecc956 1 week ago

        marked as a Work In Progress from brianhatchet/engine@3cecc956

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 1 week ago

        added 1 commit

        • cc861b6b - MR changes and everything working edition

        Compare with previous version

      • Mark Harding @markeharding added 1 commit 1 week ago

        added 1 commit

        • 0962e34f - (chore): remove un-needed changes

        Compare with previous version

      • Mark Harding @markeharding added 1 commit 1 week ago

        added 1 commit

        • 9ad3dc0d - (chore): remove un-needed changes

        Compare with previous version

      • Mark Harding @markeharding added 1 commit 1 week ago

        added 1 commit

        • 84be2d49 - (chore): remove cassandra log

        Compare with previous version

      • Mark Harding @markeharding added 1 commit 1 week ago

        added 1 commit

        • eabf1086 - (fix): avoid full db scan on pattern

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 5 commits 1 week ago

        added 5 commits

        • eabf1086...9b4c685a - 2 commits from branch minds:master
        • 3a70ba3f - Merge branch 'master' of gitlab.com:minds/engine into admin_firehose
        • b5505f87 - Merge remote-tracking branch 'origin/admin_firehose' into admin_firehose
        • 3d8cdeff - Everything working and complete.

        Compare with previous version

        Toggle commit list
      • Brian Hatchet :speech_balloon: @brianhatchet unmarked as a Work In Progress 1 week ago

        unmarked as a Work In Progress

      • Brian Hatchet :speech_balloon: @brianhatchet changed the description 1 week ago

        changed the description

      • Brian Hatchet :speech_balloon: @brianhatchet changed milestone to %sprint: Funny Frog 1 week ago

        changed milestone to %sprint: Funny Frog

      • Brian Hatchet :speech_balloon: @brianhatchet added 1 commit 1 week ago

        added 1 commit

        • a878510b - Actually filing decisions with the jury manager

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 9 commits 1 week ago

        added 9 commits

        • a878510b...678abba9 - 8 commits from branch minds:master
        • 43c9b71f - Merge remote-tracking branch 'upstream/master' into admin_firehose

        Compare with previous version

      • Brian Hatchet :speech_balloon: @brianhatchet added 8 commits 1 week ago

        added 8 commits

        • 43c9b71f...75d495b6 - 7 commits from branch minds:master
        • 1a2d4ea5 - Merge remote-tracking branch 'upstream/master' into admin_firehose

        Compare with previous version

      • Mark Harding @markeharding approved this merge request 52 minutes ago

        approved this merge request

      • Mark Harding
        Mark Harding @markeharding · 52 minutes ago
        Owner

        @all please approve

      • You're only seeing other activity in the feed. To add a comment, switch to one of the following options.
      Please register or sign in to reply
      Assignee
      Brian Hatchet's avatar Brian Hatchet @brianhatchet
      Assign to
      sprint: Funny Frog
      Milestone
      sprint: Funny Frog
      Assign milestone
      None
      Time tracking
      No estimate or time spent
      0
      Labels
      None
      Assign labels
      • View project labels
      Lock merge request
      Unlocked
      15
      15 participants
      user avatar
      Brian Hatchet
      user avatar
      Mark Harding
      user avatar
      Olivia Madrid
      user avatar
      Faizan Khan
      user avatar
      Martin Santangelo
      user avatar
      Marcelo Rivera
      user avatar
      Bill Ottman
      Reference: minds/engine!159