楽天が挑むDevOps

Transcript

• 1. 楽天が挑むDevOps July,29th 2013 YoheiSasaki PaaS Development& Operations Section Architecture CommitteeOffice DevelopmentUnit Rakuten,Inc.
• 2. About me • Yohei Sasaki • Lead of Rakuten Platform as a Service. • Technology Background: • IBM Tivoli / Ruby on Rails / Apache Hadoop / Apache CouchDB / Node.js / CloudFoundry / … • Job Background: • Technical Support Engineer / Application Developer / System Developer / System Administrator / Project Manager / … 万屋
• 3. About us • Rakuten, Inc. • Internet Service Company • Many businesses • 1000+ developers • 70+ teams • 1000+ projects インターネット企業
• 4. Go Global • Englishnization 英語公用語化
• 5. Agenda for Today • Why/What is RPaaS. • Practices in RPaaS. • Messages for DevOps people
• 6. WHY/WHAT IS RPAAS RPaaSについて
• 7. Business Objectives • Expand business services more rapidly • Reduce the cost 早い + 安い + 旨い
• 8. Rakuten Platform as a Service • Everything should be automated. • Developer’s IN Rakuten should focus on business implementation in the latest technologies. • Platform as a Service infrastructure for Developers in Rakuten, Inc. • Like Google App Engine, Heroku, ..etc • Abstraction Layer of On-Premise/Off-Premise プライベート クラウド?
• 9. Traditional Service Building Steps • Too many human to human workflows • Too many approvals. • Too many meetings. • Too many documents. Request Servers Wait for hardware/VM creation Setup Operating System Install Software Setup monitoring Deploy website Check / Test Add to Load Balancer ... Developer Operators 複雑 ステークホルダー多い 申請書たくさん
• 10. Steps in RPaaS Developer rpaas push myapp Grab a coffee • Make it simple • Deploy • rpaas push myapp • URL mapping • rpaas map myapp xxx.rakuten.co.jp • Manage scaling • rpaas instances myapp +10 • Bind database • rpaas bind-service db1 myapp • Set Monitoring • rpaas monitor myapp –cpu 90 –mem 80 コマンド1発
• 11. Tricks – DevOps Layering Service Devs System Operators Focus on Stable Infrastructure Focus on Development Ask Response 依頼
• 12. Tricks – DevOps Layering RPaaS API Service Service DevOps RPaaS DevOps Focus on Infrastructure Features Focus on Business App Ask Response Consult Consult 自動化された標準 一緒に 考える
• 13. PaaS Reference: RPaaS Architecture IaaS Network Hardware Operating System PaaS Kernel Data Services Application Runtime Operation API SDK RPaaS API Service 一緒に 考える場所
• 14. Key Concepts • Standard Document => Programmable API • Standard Infrastructure => Infrastructure with Library
• 15. Key Concepts • Standard Document => Programmable Standard • Standard Infrastructure => Infrastructure with Library 標準化 道具
• 16. Key Concepts • Standard Document => Programmable Standard • Standard Infrastructure => Infrastructure with Library 標準化 道具 早い 旨い
• 17. Key Concepts • Standard Document => Programmable Standard • Standard Infrastructure => Infrastructure with Library 標準化 道具 早い 旨い安い
• 18. Service Team DevOps rpaas stats rpaas instances rpaas log-storage git push rpaas push rpaas map デプロイサイクルが全部自動化 デバッグ用オペレーションもAPIで提供 サーバーのことは気にしない
• 19. PRACTICES IN RPAAS どうやってRPaaSを作っているのか
• 20. Change the business RPaaS DevOps Focus on Infrastructure Features System Operators Focus on Stable Infrastructure やり方を変える 安定だけではない 旨さを求める
• 21. Internal Competition • Objectives: • Provide multiple options for users PaaS IaaS Internal Customers Internal Market ユーザーを確保する ところからが仕事
• 22. Member Roles Definition • Objectives: • Clarify the responsibilities for each member. • A member has more than one role. Role In charge of: Release Manager • Conducting the release operations by cooperation with administrator. System Administrator • Handling Daily Operation. • Quick Fix on troubles. • Reporting system stats to Team Leader. System Developer • RPaaS functions from planning to operations. • Negotiation with Team Leader and Managers to decide the release line and scope. Team Leader • Coordination of system developers and related people • Evaluation for RPaaS. Manager • Budget Control • HR related things such as evaluation “兼任できる” ロールを明確にする
• 23. KPI Definition • Objectives: • Clarify the mission. • Evaluate the mission result. Role In charge of: System Administrator • # of alerts • Time to solve the alerts Release Manager • # of releases. • Time to release. System Developer • # of features newly implemented • # of solved issues Team Leader • # of launched business services ロールに目標をつけ、評価する
• 24. Release Note • Objectives: • Clarify impacts • Clarify who are customers of what you develop. リリースノートを 全員で最初に作る
• 25. Communication in Code • Objectives: • Do not use manual ops. • Anyone can do any ops 紙だけでコミュニケーションしない
• 26. Git-flow as Communication Protocol • Objectives: • Design & implementations in code repositories, not in meeting rooms. • Isolate each feature feature/* develop release/* master hotfix/* Administrator Release Manager Developer Admin/RM 独自のルールは極力作らない
• 27. Standard Release Procedure • Objectives: • Reduce manual operations. $ cd ~/rpaas-commander $ cap setup-summary $ cap setup-role -n $ cap setup-role $ cap service-in 手順書 = 動くコード
• 28. CI for infrastructure • Objectives: • Reduce the release tasks of which 80% are ‘task confirmation. 確認作業もコードで指示
• 29. Rolling Update • Objectives: • Easy to rollback • A/B test for infrastructure L7 Software LoadBalancer Virtual Machine (v1) Virtual Machine (v1) Virtual Machine (v1) Virtual Machine (v2) Virtual Machine (v2) Virtual Machine (v2) アップデートではなく新規構築
• 30. Eat our own dog food • Objectives: • Stands on customer side. • Understand pros and cons for our weapons. 自分の道具を自分で使う
• 31. Off topic?: Englishnization • Team members from 7 countries • Finding Specialists in the world • Source Code / Diagram > English?
• 32. MESSAGES FOR DEVOPS PEOPLE
• 33. Key Factors for DevOps • Everyone should have ‘developer role’. • Tool is a tool. • Everything should be automated.
• 34. Key Factors for DevOps • Everyone should have ‘administration role’. • Tool is a tool. • Everything should be automated.
• 35. Everyone should have ‘developer role’ • De-facto standard communication method. • Read source • Find a bug • Fix the bug • Create a new application. #!/usr/bin/env ruby puts “Hello Developers!” ソースコードいじりましょう
• 36. Tool is a tool • Chef does not solve the barrier between Devs and Ops. @@ -1,4 +1,4 @@ -package "apache" do +package "nginx" do action :install end I wanna use Nginx! Unacceptable. Pull Request Chef 使ってるかどうかは 問題ではない
• 37. Everything should be automated • Everything / Uncompromising • Automate after manual operations “手作業” を使っていいのは１ヶ月に１度 手順書が必要な作業を実施したら、 自動化するまでが仕事
• 38. Thank you!
• 39. And we are hiring! 楽天 PaaS開発エンジニア • Global Team • Emerging Technology • Trend words • Challenge Everyday