Docker and Fluentd

Transcript

• 1. Docker and Fluentd 2015/06/01 Fluentd meetup 2015 Summer Satoshi Tagomori (@tagomoris)
• 2. @tagomoris Treasure Data, Inc. Fluentd, Norikra, Hadoop, ...
• 3. Announce: fluent-plugin-secure-forward Security issue: v0.2.x or earlier http://tagomoris.tumblr.com/post/120090873793/vulnerability-and-update-of-secure-forward-plugin http://tagomoris.hatenablog.com/entry/2015/05/28/182245 Please update to v0.3.2! v0.3.2: Updated w/ new config param set / code Incompatible protocol & configuration :(
• 4. Which more cute than another?
• 5. Docker and Fluentd Docker: > container platform what you love :D Fluentd: > logging platform what you love :D
• 6. Logging of container architecture Storage: > should be outside of container / hosts Transferring: > should be over network Aggregation: > should be done per container / per service
• 7. Aggregation Patterns (1) 1-level aggregation (2) 2-level aggregation
• 8. Aggregation Patterns (1) 1-level aggregation (2) 2-level aggregation pros: simple configuration cons: fixed aggr. address many connections high load for aggregator pros: less connections lower load for aggr. cons: a bit more container
• 9. Aggregation Patterns (1) 1-level aggregation (2) 2-level aggregation Apps sends logs over fluent-logger there're some patterns... or ...
• 10. Pattern of 2-level aggr. w/ Fluentd 1. Network transferring 2. Container log & tail
• 11. (1) Network transferring Apps sends logs: > using fluent-logger-* > over inter-container network Pros: > no performance penalty Cons: > logs of docker is out of scope > a bit complex apps for logging over TCP app
• 12. (2) Container logger & tail Apps write logs to STDOUT: > docker writes it to log file > fluentd container read file Pros: > simple conf for apps & docker > logs include container logs Cons: > in_tail performance penalty log file app http://www.fluentd.org/guides/recipes/docker-logging
• 13. Logging drivers New from docker v1.6! 'Add new Logging driver "fluentd"' > --log-driver=fluentd > https://github.com/docker/docker/pull/12876 > New for docker v1.7.0?
• 14. over TCP (3) Container logging driver "fluentd" Apps write logs to STDOUT: > docker send it to fluentd directly! Pros: > simple conf for apps & docker > logs include container logs Cons: > none? app Stay tune!
• 15. Fluentd docker image Official image by fluent organization https://registry.hub.docker.com/u/tagomori/fluentd/ (it should be "fluent/fluentd" ...) Use it as it is, or build your own container! https://github.com/fluent/fluentd-docker-image
• 16. ❤