Fluentd v0.12 master guide

Transcript

• 1. Masahiro Nakagawa June 1, 2015 Fluentd meetup 2015 Summer Fluentd- v0.12 master guide - #fluentdmeetup
• 2. Who are you? > Masahiro Nakagawa > github/twitter: @repeatedly > Treasure Data, Inc. > Senior Software Engineer > Fluentd / td-agent developer > I love OSS :) > D language - Phobos committer > Fluentd - Main maintainer > MessagePack / RPC - D and Python (only RPC) > The organizer of Presto Source Code Reading / meetup > etc…
• 3. Structured logging Reliable forwarding Pluggable architecture http://fluentd.org/
• 4. What’s Fluentd? > Data collector for unified logging layer > Streaming data transfer based on JSON > Written in Ruby > Gem based various plugins > http://www.fluentd.org/plugins > Working in production > http://www.fluentd.org/testimonials
• 5. v0.10 (old stable) > Mainly for log forwarding > with good performance > working in production > with td-agent 1 and td-agent 2.0 / 2.1 > Robust but not good for log processing
• 6. Architecture (v0.10) Buffer Output Input > Forward > HTTP > File tail > dstat > ... > Forward > File > MongoDB > ... > File > Memory Engine Output > rewrite > ... Pluggable Pluggable
• 7. v0.12 (current stable) > v1 configuration by default > Event handling improvement > Filter, Label, Error Stream > At-least-once semantics in forwarding > Add require_ack_response parameter > HTTP RPC based management > Latest release is v0.12.11
• 8. Architecture (v0.12 or later) EngineInput Filter Output Buffer > grep > record_transfomer > … > Forward > File tail > ... > Forward > File > ... Output > File > Memory not pluggable FormatterParser
• 9. v1 configuration > hash, array and enum types are added > hash and array are json > Embed Ruby code using "#{}", > easy to set variable values: "#{ENV['KEY']}" > Add :secret option to mask parameters > “@“ prefix for built-in parameters > @type, @id and @log_level
• 10. New v1 formats > Easy to write complex values > No trick or additional work for common cases
 
 
 
 
 
 <source> @type my_tail keys ["k1", "k2", "k3"] </source> <match **> @typo my_filter add_keys {"k1" : "v1"} </match> <filter **> @type my_filter env "#{ENV['KEY']}" </filter> Hash,Array, etc: Embedded Ruby code: • Socket.gethostname • `command` • etc...
• 11. :secret option > For masking sensitive parameters > In fluentd logs and in_monitor_agent
 
 
 
 
 
 
 
 2015-05-29 19:50:10 +0900 [info]: using configuration file: <ROOT> <source> @type forward </source> <match http.**> @type test sensitive_param xxxxxx </match> <ROOT> config_param :sensitive_param, :string, :secret => true
• 12. > Apply filtering routine to event stream > No more tag tricks and can’t modify tag
 
 
 
 
 
 <match access.**> type record_reformer tag reformed.${tag} </match> <match reformed.**> type growthforecast </match> <filter access.**> @type record_transformer … </filter> v0.10: v0.12: <match access.**> @type growthforecast </match> Filter
• 13. Processing pipeline comparison Output Engine Filter Output Output 1 transaction 1 transaction 1 transaction v0.10 v0.12
• 14. > Mutate events > http://docs.fluentd.org/articles/ filter_record_transformer
 
 
 
 
 
 <filter event.**> @type record_transformer <record> hostname "#{Socket.gethostname}" </record> </filter> <match event.**> @type mongodb </match> Filter: record_transformer
• 15. > Grep event streams > http://docs.fluentd.org/articles/filter_grep
 
 
 
 
 
 <filter event.**> @type grep regexp1 1 message cool regexp2 hostname ^webd+.example.com$ exclude1 message uncool </filter> <match event.**> @type mongodb </match> Filter: grep
• 16. > Print events to stdout > No need copy and stdout plugins combo! > http://docs.fluentd.org/articles/filter_stdout
 
 
 
 
 
 <filter event.**> @type stdout </filter> <match event.**> @type mongodb </match> Filter: stdout
• 17. > Override filter method
 
 
 
 
 
 
 module Fluent::AddTagFilter < Filter # Same as other plugins, initialize, configure, start, shudown # Define configurations by config_param utilities def filter(tag, time, record) # Process record record["tag"] = tag # Return processed record, # If return nil, that records are ignored record end end Filter: Plugin development 1
• 18. > Override filter_stream method
 
 
 
 
 
 
 module Fluent::AddTagFilter < Filter def filter_stream(tag, es) new_es = MultiEventStream.new es.each { |time, record| begin record["tag"] = tag new_es.add(time, record) rescue => e router.emit_error_event(tag, time, record, e) end } new_es end end Filter: Plugin development 2
• 19. > Internal event routing > Redirect events to another group > much easier to group and share plugins
 
 
 
 
 
 <source> @type forward </source> <match app1.**> @type s3 </match> … <source> @type forward @label @APP1 </source> <label @APP1> <match access.**> @type s3 </match> </label> v0.10: v0.12: Label
• 20. > Use router.emit instead of Engine.emit > Engine#emit API is deprecated
 
 
 
 
 
 tag = "" time = Engine.now record = {…} Engine.emit(tag, time, record) v0.10: v0.12: tag = "" time = Engine.now record = {…} router.emit(tag, time, record) Label : Need to update plugin
• 21. > Redirect events to another label
 
 
 
 
 
 
 <source> @type forward @label @RAW </source> Label: relabel output <label @RAW> <match **> @type copy <store> @type flowcounter </store> <store> @type relabel @label @MAIN </store> </match> </label> <label @MAIN> <match access.**> @type s3 </match> </label>
• 22. Error stream with Label > Can handle an error at each record level > router.emit_error_event(tag, time, record, error)
 
 
 
 
 
 
 ERROR! {"event":1, ...} {"event":2, ...} {"event":3, ...} chunk1 {"event":4, ...} {"event":5, ...} {"event":6, ...} chunk2 … Input OK ERROR! OK OK OK Output <label @ERROR> <match **> type file ... </match> </label> Error stream Built-in @ERROR is used when error occurred in “emit”
• 23. Support at-least-once semantics > Delivery guarantees in failure scenarios > At-most-once: messages may be lost > At-least-once: messages may be duplicated > Exactly-once: No lost and duplication > Fluentd supports at-most-once in v0.10 > Fluentd supports at-least-once since v.12! > set require_ack_response parameter
• 24. At-most-once and At-least-once <match app.**> @type forward require_ack_response </match> may be duplicated Error! <match app.**> @type forward </match> may be lost Error! × ×
• 25. HTTP RPC based management > Use HTTP/JSON API instead of signals > For Windows and JRuby support > RPC is based on HTTP RPC style, not REST > See https://api.slack.com/web#basics > Enabled by rpc_endpoint in <system> > Have a plan to add more APIs > stop input plugins, check plugins and etc
• 26. Supported RPCs > /api/processes.interruptWorkers > /api/processes.killWorkers > Same as SIGINT and SIGTERM > /api/plugins.flushBuffers > Same as SIGUSR1 > /api/config.reload > Same as SIGHUP
• 27. RPC example > Configuration
 
 
 
 > Curl
 
 <system> rpc_endpoint 127.0.0.1:24444 </system> $ curl http://127.0.0.1:24444/api/plugins.flushBuffers {"ok":true}
• 28. Ecosystem
• 29. Almost ecosystems are v0.12 based > Treasure Agent > v2.2 is shipped with v0.12 > docs.fluentd.org are now v0.12 > You can see v0.10 document via v0.10 prefix > http://docs.fluentd.org/v0.10/articles/quickstart > If your used plugins don’t use v0.12 feature,
 please contribute it!
• 30. Roadmap > v0.10 (old stable) > v0.12 (current stable) <- Now! > Filter / Label / At-least-once / HTTP RPC > v0.14 (summer, 2015) > New plugin APIs, ServerEngine, Time… > v1 (fall/winter, 2015) > Fix new features / APIs https://github.com/fluent/fluentd/wiki/V1-Roadmap
• 31. https://jobs.lever.co/treasure-data Cloud service for the entire data pipeline. We’re hiring!