Transcript

• 1. Masahiro Nakagawa Mar 14, 2015 Fossasia 2015 Fluentd Unified logging layer
• 2. Who am I > Masahiro Nakagawa > github: @repeatedly > Treasure Data, Inc. > Senior Software Engineer > Fluentd / td-agent developer > Living at OSS :) > D language - Phobos, a.k.a standard library, committer > Fluentd - Main maintainer > MessagePack / RPC - D and Python (only RPC) > The organizer of several meetups (Presto, DTM, etc…) > etc…
• 3. Structured logging ! Reliable forwarding ! Pluggable architecture http://fluentd.org/ github:fluent/fluentd
• 4. What’s Fluentd? > Data collector for unified logging layer > Streaming data transfer based on JSON > Simple core + plugins written in Ruby > Gem based various plugins > http://www.fluentd.org/plugins > List of users > http://www.fluentd.org/testimonials
• 5. Before ✓ duplicated code for error handling... ✓ messy code for retrying mechanism...
• 6. So painful!
• 7. After
• 8. Concept / Design
• 9. Core Plugins > Divide & Conquer
 > Buffering & Retrying
 > Error handling
 > Message routing
 > Parallelism > Read / receive data > Parse data > Filter data > Buffer data > Format data > Write / send data

• 10. Core Plugins > Divide & Conquer
 > Buffering & Retrying
 > Error handling
 > Message routing
 > Parallelism > Read / receive data > Parse data > Filter data > Buffer data > Format data > Write / send data
 Common Concerns Use Case Specific
• 11. > default second unit > from data source Event structure(log message) ✓ Time > for message routing > where is from? ✓ Tag > JSON format > MessagePack
 internally > schema-free ✓ Record
• 12. Reliable streaming data transfer error retry error retry retry retry Batch Stream Other stream (micro batch)
• 13. Nagios PostgreSQL Hadoop Alerting Amazon S3 Analysis Archiving Elasticsearch Apache Frontend Access logs syslogd App logs System logs Backend Databases buffering / retrying / routing M x N → M + N plugins
• 14. Use case
• 15. Simple forwarding
• 16. # logs from a file <source> type tail path /var/log/httpd.log pos_file /tmp/pos_file format apache2 tag backend.apache </source> ! # logs from client libraries <source> type forward port 24224 </source> ! # store logs to MongoDB <match backend.*> type mongo database fluent collection test </match>
• 17. Less Simple Forwarding - At-most-once / At-least-once
 - HA (failover) - Load-balancing
• 18. All data Near realtime and batch combo! Hot data
• 19. # logs from a file <source> type tail path /var/log/httpd.log pos_file /tmp/pos_file format apache2 tag web.access </source> ! # logs from client libraries <source> type forward port 24224 </source> ! # store logs to ES and HDFS <match web.*> type copy <store> type elasticsearch logstash_format true </store> <store> type webhdfs host namenode port 50070 path /path/on/hdfs/ </store> </match>
• 20. CEP for Stream Processing Norikra is a SQL based CEP engine: http://norikra.github.io/
• 21. Container Logging
• 22. > Kubernetes ! ! ! ! ! > Google Compute Engine > https://cloud.google.com/logging/docs/install/compute_install Fluentd on Kubernetes / GCE
• 23. Slideshare http://engineering.slideshare.net/2014/04/skynet-project-monitor-scale-and-auto-heal-a-system-in-the-cloud/
• 24. Log Analysis System And its designs in LINE Corp. 2014 early
• 25. Architecture
• 26. Internal Architecture Input Parser Buffer Output FormatterFilter OutputFormatter
• 27. Internal Architecture Input Parser Buffer Output FormatterFilter “input-ish” “output-ish”
• 28. Input plugins File tail (in_tail) Syslog (in_syslog) HTTP (in_http) HTTP/2 (in_http2 WIP) ... ✓ Receive logs ✓ Or pull logs from data sources ✓ non-blocking InpuInput
• 29. Parser plugins JSON Regexp Apache/Nginx/Syslog CSV/TSV
 etc. ✓ Parse into JSON ✓ Common formats out of the box ✓ Some inputs plugin depends on
 Parser plugin ✓ v0.10.46 and above ParseParser
• 30. Filter plugins grep record_transformer suppress … ✓ Filter / Mutate record ✓ Record level and Stream level ✓ v0.12 and above ParseParserFilter
• 31. Buffer plugins ✓ Improve performance ✓ Provide reliability ✓ Provide thread-safety Memory (buf_memory) File (buf_file) BuffeBuffer
• 32. Buffer internal ✓ Chunk = adjustable unit of data ✓ Buffer = Queue of chunks chunk chunk chunk output Input
• 33. Formatter plugins ✓ Format output ✓ Some plugins depends on
 Formatter plugins ✓ v0.10.46 and above JSON CSV/TSV “single value” msgpack FormattFormatter
• 34. Output plugins ✓ Write to external systems ✓ Buffered & Non-buffered ✓ 200+ plugins Outpu File (out_file) Amazon S3 (out_s3) MongoDB (out_mongo) ... Output
• 35. Roadmap > v0.10 (old stable) > v0.12 (current stable) > Filter / Label / At-least-once > v0.14 (spring, 2015) > New plugin APIs, ServerEngine, Time… > v1 (summer, 2015) > Fix new features / APIs https://github.com/fluent/fluentd/wiki/V1-Roadmap
• 36. Goodies
• 37. fluent-bit > Made for Embedded Linux > OpenEmbedded & Yocto Project > Intel Edison, RasPi & Beagle Black boards > https://github.com/fluent/fluent-bit > Standalone application or Library mode > Built-in plugins > input: cpu, kmsg, output: fluentd > First release at the end of Mar 2015
• 38. fluentd-ui > Manage Fluentd instance via Web UI > https://github.com/fluent/fluentd-ui
 
 
 
 
 

• 39. Treasure Agent (td-agent) > Treasure Data distribution of Fluentd > including Ruby and QA’ed plugins > Treasure Agent 2 is current stable > We recommend to use v2, not v1 > including fluentd-ui > Next release, 2.2.0, uses fluentd v0.12
• 40. Embulk > Bulk Loader version of Fluentd > Pluggable architecture > JRuby, JVM languages > High performance parallel processing > Share your script as a plugin > https://github.com/embulk http://www.slideshare.net/frsyuki/embuk-making-data-integration-works-relaxed
• 41. HDFS MySQL Amazon S3 Embulk CSV Files SequenceFile Salesforce.com Elasticsearch Cassandra Hive Redis ✓ Parallel execution ✓ Data validation ✓ Error recovery ✓ Deterministic behaviour ✓ Idempotent retrying Plugins Plugins bulk load
• 42. Check: treasuredata.com Cloud service for the entire data pipeline