AWSの進化とSmartNewsの裏側

AWSの進化とSmartNewsの裏側

1. 1. Evolution of AWS Backend of SmartNews 2016/01/27 Classmethod x SmartNews AWS Night
2. 2. Self Introduction • Nobutoshi Ogata • Manager, Site Reliability Engineering • @nobu666 • ❤ Whiskey, Cat, Heavy Metal • Entrusted dev.(10y) ➡ Game platform infrastructure(3y) ➡ Some startup(1y) ➡ SmartNews(2015/05-)
3. 3. SmartNews
4. 4. 15,000,000+ downloads worldwide
5. 5. SmartNews 3.0
6. 6. LINE Stamp shipped today
7. 7. ! Channel!
8. 8. ! Channel!
9. 9. Questionnaire
10. 10. Agenda • Scale • Services • Changes in the configuration • Configuration management • Future plan
11. 11. Scale
12. 12. Scale • 300+ EC2 instances • 80 ELBs • 70 Auto Scaling Groups • 80+ S3 buckets (300+TB) • 40 RDS instances (MySQL & Aurora) • 100 Kinesis shards • 10 EMR clusters (Hive+Spark)
13. 13. Scale • 50+ ElastiCache clusters (Redis + Memcached) • 30+ CodeDeploy applications • 70,000,000 CloudSearch seachable documents • 8 Redshift nodes • 10+ Lambda functions • 80+ SQS queues • 20+ SNS topics
14. 14. Services
15. 15. Changes in the configuration
16. 16. Stone Age Analyzer Crawler Indexer On-memory DB API
17. 17. Monolithic Application
18. 18. Bronze Age Analyzer Indexer On-memory DB API Crawler Crawler Crawler Queue
19. 19. Split Crawler
20. 20. Iron Age API Indexer Analyzer Crawler
21. 21. Split more
22. 22. Present day Crawler Nationality detector Language detector Analyzer Facial recognition Indexer Ranking Search Recommend API Gateway Admin tool Validator Proxy Live Push Sexy detector Auth Proxy
23. 23. Micro Services?
24. 24. • SmartNews's journey into microservices • インフラ専任エンジニアが一人もいないSmartNewsにおける クラウド活用法 • SmartNews Ads • Next session about data platform
25. 25. Configuration management
26. 26. Pull style deploy
27. 27. ➡ ➡ • CodeDeploy was used to automate deployment tasks • Operating an AWS ELB • Fetching a new revision • Restarting services with our custom program • e.g. downloading machine learning model data, precompiling assets and deploying Docker containers
28. 28. Auto Scaling Group • There is an advantage even if you don't want to autoscaling • Easy to dispose • replace kernel, switch to other OS, ... • Easy A/B testing • change config only specific server • tag:Attributes • Instance Protection
29. 29. ASG ! InstanceName • We were stop using InstanceName • Using tag:Group and tag:Env
30. 30. Infrastructure as Code • Route53 ➡ winebarrel/roadworker • Security Group ➡ winebarrel/piculet • Elastic IP Addresses ➡ winebarrel/eipmap • Provisioning ➡ itamae-kitchen/itamae • Datadog ➡ airbnb/interferon
31. 31. DSL management is not perfect • Complicated automation • ChatOps ➡ change code ➡ git add && git commit && git push • Too much hassle... !
32. 32. Run Command • Secure • IAM, CloudTrail • Easy CI integration w/o SSH • be able to use EMR also
33. 33. Example aws ssm send-command --document-name "AWS-RunShellScript" --instance-ids $(aws emr list-instances --cluster-id $(aws emr list-clusters --active | jq -r '.Clusters[] | select (.Name=="FOOBAR") | .Id') | jq -r '.Instances[].Ec2InstanceId') --parameters '{"commands":["ls"],"executionTimeout":["10"]}' --timeout-seconds 10 --output-s3-bucket-name "ssm-logs" --output-s3-key-prefix "prefix-" --region ap-northeast-1
34. 34. Future plan
35. 35. ECS Spot Fleet Contaniner Registry ConsulAWS Blackbelt 2015 ECS p.98
36. 36. Containerize all services
37. 37. Scalable Safety Efficient Portable Measurable
38. 38. Aggregates events
39. 39. Events aggregator • Events • Deploy/Provision/Restart/Terminate etc. • All events goes to SNS or Kinesis. Users will describe and hook scripts • Visualize via Datadog • Notice to Slack • Auto healing
40. 40. Summary
41. 41. Summary• AWS release new service continuously • We ride on the shoulders of giants • We will never forget that OSS helped us
42. 42. We're hiring! Only two people on Site Reliability Engineering Team http://about.smartnews.com/en/careers/
43. 43. Credit https://flic.kr/p/hAcPRk taken by Niuton may https://flic.kr/p/9UnRWS taken by Alberto G. https://flic.kr/p/ydnEvw taken by ep_jhu https://flic.kr/p/pyfnTV taken by Kevin Dooley https://flic.kr/p/mtLpc4 taken by L1mey https://flic.kr/p/fgG226 taken by Linux Screenshots https://flic.kr/p/7DsuLz taken by Gloson Teh https://flic.kr/p/jfGpn7 taken by julochka