WIP: Firehose api
New firehose admin endpoints for getting an unsorted list of activites from minds_badger and an admin endpoint that takes an activity guid and will mark it as moderated.
New managers and repositories for talking to elastic search Redis lock system with some super basic hash keying. These records shouldn't get very big, so we might want to KISS for now. Else, there are distributed locking libraries we should look into
It stores a key in moderation:entity_id:user_id format where the entity is the activity being moderated and user is the admin who "checked it out". These are set to a TTL of 15 minutes for now. When we query elastic search, we look for the opposite set (any locks that DON'T belong to the user) and for any activity that has not already been moderated. Requires adding two new fields to badger (in postman)
changed milestone to %sprint: Dapper Dingo
added 7 commits
- 269df1ee...1eaf8c06 - 5 commits from branch
minds:epic/ReportingAndModeration
- 50fabf98 - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose
- a1c2f3a6 - Fixing previous tests
- 269df1ee...1eaf8c06 - 5 commits from branch
added 1 commit
- 64942ca3 - Fixing unit tests by not overriding DI config in features. Thanks, Mark <3
added 37 commits
- 64942ca3...5f29f688 - 36 commits from branch
minds:epic/ReportingAndModeration
- feb5a3b0 - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose
- 64942ca3...5f29f688 - 36 commits from branch
- Resolved by Mark Harding
added 1 commit
- efa38fcc - Updated tested to reflect the new elastic search query
added 3 commits
- efa38fcc...9dbd95e5 - 2 commits from branch
minds:epic/ReportingAndModeration
- 8a3c9f0a - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose
- efa38fcc...9dbd95e5 - 2 commits from branch
added 1 commit
- 96127343 - MR fixes. Performant ES query. Mapping fixes. Moved reconstitution to the manager
unmarked as a Work In Progress
changed title from WIP firehose api to Firehose api
changed the description
added 16 commits
- c5cf7221...3927173f - 15 commits from branch
minds:epic/ReportingAndModeration
- 2dcea399 - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose
- c5cf7221...3927173f - 15 commits from branch
added 51 commits
- 2dcea399...eadf4ebd - 50 commits from branch
minds:epic/ReportingAndModeration
- 74c34afb - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose
- 2dcea399...eadf4ebd - 50 commits from branch
added 6 commits
- e3a25163...dd117ff0 - 5 commits from branch
minds:epic/ReportingAndModeration
- a3dcc56b - Merge remote-tracking branch 'upstream/epic/ReportingAndModeration' into admin_firehose
- e3a25163...dd117ff0 - 5 commits from branch
added 24 commits
- bd6e7d55 - (feat) add report reasons to config endpoint
- 25b05ce9 - (chore): ban spam domain
- d07486fc - (chore): ban spammy email
- b4a60db8 - (fix): Re-sync ES if expired boost was already completed (#448 (closed))
- e9491d70 - Merge branch 'hotfix/es-boosts-resync' into 'master'
- 56b60f52 - Merge branch 'sprint/ElevatedEagle.feat.report-reasons.added.minds-config.endpoint' into 'master'
- cb33fe87 - Merge branch 'epic/ReportingAndModeration'
- e0ced6e2 - (chore): content policy
- a593916e - (chore): if channel reported for spam then do immediate ban
- 1dc7c4b9 - [Sprint/FunnyFrog])(bug): Forcing NSFW reasons array into integers.
- fcfd028f - Merge branch 'fix-nsfw-remind-inheritance-bug' into 'master'
- 35a80e27 - (chore): change SEO to The Jury System
- ef9255df - (fix): remove revoked from stats
- 8634a913 - CORS enabled
- c710bdef - Merge branch 'checkout-oauth' into 'master'
- 8d0ccfc1 - Sprint/ElevatedEagle: Forced querystring params into lower case
- 8bd9d70c - Merge branch 'force-lower-case-search-query-and-hashtags' into 'master'
- 32b1f6ff - (chore): email on a friday
- cf10fc88 - (chore); email on a friday...
- e1ec640d - (fix): typos in email
- 0c777244 - (chore): loop email iterator if no entities returned
- 7546fd46 - (chore): dev code for promotions email
- 827ae8e9 - (chore): email promotions to use markdown and include template key in hash
- e605f46b - Merge remote-tracking branch 'upstream/master' into admin_firehose
Toggle commit listadded 8 commits
- f75596ab - (chore): ban more domains
- 01e1a432 - (chore): bring back channel level spam check for new channels
- 85de3a3c - (chore): add another blocked domain
- e40d62d3 - (chore): spam check comments
- 23f0c499 - (chore): Appeals loop and awaiting TTL (#463)
- 5b0ef158 - Merge branch 'hotfix/appeal-summons-tweaks-1' into 'master'
- 294a8bc2 - (feat): support onchain boost label
- f3d0b5ee - Merge remote-tracking branch 'upstream/master' into admin_firehose
Toggle commit listchanged target branch from
epic/ReportingAndModeration
tomaster
26 27 return Factory::response([ 28 'status' => 'success', 29 'entities' => Exportable::_($activities), 30 ]); 31 } 32 33 public function post($pages) 34 { 35 if (!is_numeric($pages[0])) { 36 header('X-Minds-Exception: activity guid required'); 37 http_response_code(400); 38 return Factory::response([]); 39 } 40 41 $activity = new Activity($pages[0]); - Owner
I don't think we can assume it will be an activity.
EntitiesBuilder->single()
should provide the lookup for you with the correct entity type object
- Last updated by Brian Hatchet
10 use Minds\Entities\Activity; 11 12 class firehose implements Interfaces\Api, Interfaces\ApiAdminPam 13 { 14 /** 15 * Gets a list of entities sorted for admin approval. 16 * 17 * @param array $pages 18 * 19 * @throws \Exception 20 */ 21 public function get($pages) 22 { 23 /** @var Core\Feeds\Firehose\Manager $manager */ 24 $manager = Di::_()->get('Feeds\Firehose\Manager'); 25 $activities = $manager->getList(Session::getLoggedinUser()); - Owner
I think an array here would bring it in line with other manager ->getList functions.
eg:
->getList([ 'user' => Session::getLoggedinUser(), ])
changed this line in version 20 of the diff
24 Repository $repository = null, 25 EntitiesBuilder $entitiesBuilder = null, 26 UserReportManager $reportManager = null 27 ) { 28 $this->repository = $repository ?: new Repository(); 29 $this->entitiesBuilder = $entitiesBuilder ?: new EntitiesBuilder(); 30 $this->reportManager = $reportManager ?: Di::_()->get('Moderation\Reports\Manager'); 31 } 32 33 /** 34 * @param User $user the user who own the cached keys. No user, no caching 35 * @param array $opts filtering options 36 * 37 * @return array entities that don't contain moderator_guids 38 */ 39 public function getList(User $user = null, array $opts = []) - Owner
As above, maybe merge in $user to $opts
25 Config $config = null, 26 ModerationCache $cache = null) 27 { 28 $this->client = $client ?: Di::_()->get('Database\ElasticSearch'); 29 $this->config = $config ?: Di::_()->get('Config'); 30 $this->cache = $cache ?: new ModerationCache(); 31 $this->index = $this->config->get('elasticsearch')['index']; 32 } 33 34 /** 35 * @param User $user the user who own the cached keys. No user, no caching 36 * @param array $opts filtering options 37 * 38 * @return Response $reponse Response object containing activity guids 39 */ 40 public function getList(User $user = null, array $opts = []) - Owner
Ideally Repository would be an interface that would catch this, but I think we should just send an array as a single param
16 16 public function __construct() 17 17 { 18 18 $this->mappings = array_merge($this->mappings, [ 19 'rating' => [ 'type' => 'integer', '$exportField' => 'rating' ], 20 'custom_type' => [ 'type' => 'text', '$exportField' => 'custom_type' ], 21 'entity_guid' => [ 'type' => 'text', '$exportField' => 'entity_guid' ], 19 'rating' => ['type' => 'integer', '$exportField' => 'rating'], 20 'custom_type' => ['type' => 'text', '$exportField' => 'custom_type'], - Owner
this may need moving to EntityMapping?
691 697 return $this->ownerObj; 692 698 } 693 699 700 public function setModeratorGuid($moderatorGuid) { 701 $this->moderator_guid = $moderatorGuid; 702 } - Owner
spacing is little off
691 697 return $this->ownerObj; 692 698 } 693 699 700 public function setModeratorGuid($moderatorGuid) { - Owner
Can we get some phpdocs in ?
marked as a Work In Progress from brianhatchet/engine@3cecc956