スタディサプリを支えるデータ分析基盤～設計の勘所と利活用事例～

40. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. id grade_id gender school_id AAA 001 female 0000X BBB 002 male 0000Y CCC 005 male 0000Z id label 001 002 003 id label 0000X X 0000Y Y 0000Z Z id grade gender school AAA female X BBB male Y CCC male Z

54. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. embulk-input-mongodb by @hakobera embulk-input-postgresql embulk-parser-json embulk-filter-expand_json embulk-filter-hash by @kamatama41 embulk-filter-mask by @beniyama embulk-output-td

60. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. class DailyVideoRatingReport(luigi_td.Query): type = 'hive' database = LuigiConfig().main_db report_database = LuigiConfig().report_db source = ‘queries/hive/daily/history/daily_video_rating_report.sql’ jst_date = luigi.DateParameter(default=DateTimeUtil.get_jst_date()) def requires(self): return [ DailyUserMasterCreate(self.jst_date), DailyContentsMasterCreate(self.jst_date) ]

61. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. def run(self): TdUtil.drop_partition( table=‘video_rating_report’, to_jst_date=self.jst_date ) super(DailyVideoRatingReport, self).run() def output(self): return luigi_td.ResultTarget( '{0}/DailyVideoRatingReport-{1}.txt'.format( LuigiConfig().output_path, self.jst_date.strftime('%Y-%m-%d') ), 'td://@/reports/video_rating_report?mode=append' )

63. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. +video_rating_report: +partial_delete: td_partial_delete>: video_rating_report database: reports from: ${moment(session_time).utc().add("days", -1).unix()} to: ${moment(session_time).utc().unix()} +insert_records: td>: queries/reporting/video_rating_report_insert.sql database: reports engine: presto insert_into: video_rating_report

64. (C) Recruit Marketing Partners Co.,Ltd. All rights reserved. +fetch_records: td>: queries/reporting/video_ratings_fetch.sql database: reports engine: presto store_last_results: true +slack_notification: http>: https://hooks.slack.com/services/XXXXXXXX method: POST content: text: “Average rating is ${td.last_results.average_ratings}” link_names: true content_format: json

スタディサプリを支えるデータ分析基盤 ～設計の勘所と利活用事例～

Tetsuo Yamabe