Providing CI / CD as a Service - AWS Summit Sydney April 2015

Transcript

• 1. PETER LESCHEV • TEAM LEAD • ATLASSIAN • @PETERLESCHEV Build Engineering @ Atlassian: Providing CI / CD as a Service AWS Summit Sydney - April 2015
• 2. SCALING THE BUILD GRID INTRODUCTION Introduction AWS ADVANTAGES FUTURE STATE
• 3. Build platform & services used internally within Atlassian to build, test & deliver software
• 4. Developers expect a reliable infrastructure & fast CI feedback
• 5. Build Engineering today @ Atlassian • 10 Bamboo Servers • maven.atlassian.com / 6 Nexus instances • Monitoring - opsview / graphite / statsd • 1000 build agents (own hardware + EC2 instances) • include SCM clients, JDKs, JVM build tools, databases, headless browser testing, python builds, NodeJS, installers & more • Maintain 20 AMIs of various build configurations • Nexus proxies
• 6. 3 years ago: 21kBuilds per month
• 7. Last month: 127kBuilds per month
• 8. Build Engineering today @ Atlassian
• 9. JIRA alone has 47kAutomated tests
• 10. SCALING THE BUILD GRID INTRODUCTION Scaling the Build Grid AWS ADVANTAGES FUTURE STATE
• 11. Build Agents
• 12. Own Hardware using KVM guests + EC2 instances
• 13. Differences between KVM guests & elastic build agents + the need for Functional Parity
• 14. 2 Problems
• 15. Problem 1: Network connectivity differences
• 16. Solution: AWS Direct Connect + VPC enabled EC2 instances
• 17. Problem 2: Configuration differences between KVM guests & elastic build agents
• 18. Solution: Removing tech debt using https://www.packer.io/ https://www.atlassian.com/software/bamboohttps://puppetlabs.com/ https://cukes.info/
• 19. SCALING THE BUILD GRID INTRODUCTION AWS Advantages AWS ADVANTAGES FUTURE STATE
• 20. Handle the waves of demand
• 21. Prices always decreasing Product range expanding https://aws.amazon.com/blogs/aws/category/price-reduction/
• 22. Hardware procurement API + Credit Card
• 23. ‘Do Not Resuscitate’ order on existing build agent hardware
• 24. Spot instances
• 25. Cheap!
• 26. Slower startup times
• 27. Not always available
• 28. Price Volatility
• 29. Expect Failure Embrace It
• 30. Prebake at AMI burn time rather than at EC2 instance startup time
• 31. Faster startup time
• 32. More reliable instance startup (+ retry on failure!)
• 33. Pre-caching of git repositories, docker images
• 34. “RequestLimitExceeded. The maximum request rate permitted by the Amazon EC2 APIs has been exceeded for your account”
• 35. Handle failure, retry with backoff & recover
• 36. “We currently do not have sufficient m3.large capacity in the Availability Zone you requested”
• 37. Multiple-AZ support in Bamboo 5.8
• 38. “we are experiencing increased error rates and latencies for the EC2 APIs”
• 39. SCALING THE BUILD GRID INTRODUCTION Future State AWS ADVANTAGES FUTURE STATE
• 40. Moving all Build Eng Infra to AWS
• 41. EC2 to ECS for build agents
• 42. Come help us!
• 43. Thank you! PETER LESCHEV • TEAM LEAD • ATLASSIAN • @PETER LESCHEV