Docker and Fluentd

Transcript of "Docker and Fluentd"

1. 1. and Docker meetup Tokyo #5
 Aug 19, 2015
2. 2. Who are you? • Masahiro Nakagawa • github: @repeatedly • Treasure Data Inc. • Fluentd / td-agent developer • https://jobs.lever.co/treasure-data • I love OSS :) • D Language, MessagePack, The organizer of several meetups, etc…
3. 3. Logging for containers • Storage: • should be outside of containers / hosts • Transferring: • should be over network • Aggregation: • should be done per container / per service
4. 4. Real world…
5. 5. Container-based world needs unified logging layer
6. 6. Structured logging Reliable forwarding Pluggable architecturehttp://fluentd.org/
7. 7. What’s Fluentd? • Data collector for unified logging layer • Streaming data transfer based on JSON • Written in Ruby • Gem based various plugins • http://www.fluentd.org/plugins • Working on lots of productions • http://www.fluentd.org/testimonials
8. 8. Before…
9. 9. After!
10. 10. M x N → M + N Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases buffering / processing /
11. 11. Core Plugins • Divide & Conquer • Buffering & Retrying • Error handling • Message routing • Parallelism • Read data • Parse data • Filter data • Format data • Write data
12. 12. Architecture EngineInput Filter Output Buffer > grep > record_transfomer > … > Forward > File tail > ... > Forward > File > ... Output > File > Memory not pluggable FormatterParser
13. 13. Aggregation patterns 1) 1-level aggregation 2) 2-level aggregation Pros: • simple configuration Cons: • fixed aggr, address • many connections • high load for aggr Pros: • less connections • lower load for aggr Cons: • more containers
14. 14. Aggregation patterns 1) 1-level aggregation 2) 2-level aggregation Apps sends logs
 over fluent-logger or … There are some patterns
 - via udp / tcp
 - tail file
 - etc…
15. 15. 2-level aggr w/ Fluentd 1. Transfer logs over network 2. Container logs & tail generated log files
16. 16. 1) Network transferring • Apps sends logs: • using fluent-logger-* • over inter-container network • Pros: • no performance penalty • Cons: • docker logs is out of scope • a bit complex for logging over TCP app
17. 17. 2) Tail container logs • Apps write logs to STDOUT • docker writes it to log file • fluentd container reads its file • Pros: • simple conf for apps & docker • logs include container logs • Cons: • in_tail performance penalty &
 a bit complex configuration log file app http://www.fluentd.org/guides/recipes/docker-logging
18. 18. Seems good? But we now have smarter way!
19. 19. Logging drivers • New from docker v1.6 • We can get docker logs directly via drivers • “fluentd" driver is coming from v1.8 • enabled by “—log-driver=fluentd” • contributed by @tagomoris
 https://github.com/docker/docker/pull/12876 • https://docs.docker.com/reference/logging/fluentd/
20. 20. 3) Logging driver • Apps write logs to STDOUT • Docker sends its logs to
 fluentd directly • Pros: • simple conf for apps & docker • Logs includes container logs • Cons: • Do you see any concerns? over TCP app
21. 21. Demo?
22. 22. Fluentd docker image • Official image by fluent organization • Main maintainer is @tagomoris • https://registry.hub.docker.com/u/fluent/fluentd/ • Or build your container! • https://github.com/fluent/fluentd-docker-image
23. 23. With Kubernetes • Kubernetes provides fluentd + Elasticsearch + Kibana image officially • https://hub.docker.com/r/kubernetes/fluentd- elasticsearch/ • http://www.dasblinkenlichten.com/logging-in- kubernetes-with-fluentd-and-elasticsearch/
24. 24. Cloud service for entire data pipeline!