Sysdig Monitorama Slides

Transcript

• 1. The Dark Art of Container Monitoring Loris Degioanni
• 2. Me Loris Degioanni sysdig’s creator and CEO Past: WinPcap, Wireshark @ldegio
• 3. Containers Are Great… • less overhead • faster deployments • reproducibility of environments • cost optimizations • Isolation • flexibility
• 4. …But Monitoring Them Is Not Easy Containers are: • Isolated • Self-Contained • Simple • Lightweight • Disposable
• 5. …But Monitoring Them Is Not Easy Containers are: • Isolated • Self-Contained • Simple • Lightweight • Disposable Is it possible to respect these properties and also have great monitoring?
• 6. • Containers • Overview of monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos In This Talk
• 7. • Containers • Overview of monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos In This Talk
• 8. In This Talk • Overview of container monitoring/troubleshooting options • command line • cAdvisor • Docker stats • sysdig • Examples and demos
• 9. In This Talk
• 10. Some Things We Want to Monitor • Resource usage (CPU/Memory/Disk) • Network activity • Application activity • File I/O activity • Errors/faults • Topology
• 11. Resource Usage (CPU/Memory/Disk) • ps/top/htop from the host • Pro: always there • Cons: Very limited container context (full cgroup string only), only work from the host • docker ps / docker top • cAdvisor • docker stats • sysdig
• 12. Installing cAdvisor sudo docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --name=cadvisor google/cadvisor:latest
• 13. cAdvisor Pros and Cons Pros • easy to install • works from a container • Nice local UI • API • Integrations (e.g. InfluxDB, Prometheus, Heapster) Cons: • (very) Limited set of metrics • Containers are black boxes
• 14. Docker Stats + API • docker stats mysql • echo -e "GET /containers/mysql/stats HTTP/1.1rn" | nc -U /var/run/docker.sock
• 15. Docker Stats + API • cpu_stats • network • memory_stats • blkio_stats http://docs.docker.com/reference/api/docker_remote_api_v1.17/#get- container-stats-based-on-resource-usage
• 16. Docker Stats Pros and Cons Pros • Part of Docker • Richer than cAdvisor • REST API • Integrations • docker-collectd-plugin github.com/cloudwatt/docker-collectd-plugin • commercial vendors Cons: • Docker only • Containers are black boxes
• 17. sysdig • Capture system events, filter them, run useful scripts • strace + tcpdump + lsof + Lua • Open Source
• 18. csysdig • Capture system events, filter them, run useful scripts • strace + tcpdump + lsof + htop + iftop + Lua • Open Source • Nice curses UI
• 19. Design Goals • Production-ready • Simple • lightweight • Rich data • Native support for containers • Runs in a container • Natural workflow
• 20. sysdig Architecture Kernel Container1 Docker Container2 Docker Container3 LXCAppApp
• 21. sysdig Architecture Kernel Container1 Docker Container2 Docker Container3 LXCAppApp Instrumentation through kernel module
• 22. sysdig Architecture Kernel Container1 Docker Container2 Docker Container3 LXCAppApp sysdig Docker Capture and analysis
• 23. sysdig Architecture Kernel Container1 Docker Container2 Docker Container3 LXCAppApp sysdig Docker (optionally) Save to a trace file foo.scap
• 24. Viewing Network/App Activity •cAdvisor •docker stats •iftop/tcpdump/tshark in the container •sysdig
• 25. Viewing File I/O Activity and Errors •iotop/lsof in the container •sysdig
• 26. Topology •Sysdig cloud
• 27. Thank You! www.sysdig.com @sysdig