18.
Problems...No unified method to collect logs> Too many bash/perl scripts Fragile for changes Less reliable> Mixed log formats Old-fashioned “Human-readable” text logs Not ready to analyze> High latency must wait a day for log rotation
23.
Input Plugins Output Plugins time tag 2012-02-04 01:33:51 JSON format myapp.buylog { “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } record
24.
Why Fluentd?> Extensibility - Plugin architecture collect logs from various systems forward logs to various systems> Unified log format - JSON format modern “Machine-readable” log format immediately ready to analyze> Reliable - HA configuration> Easy to install - RPM/deb packages deploy instantly to everywhere
25.
Comparision with other log collectors:> Scribe Less extensible No unified log format No longer developped?> Flume Less simple No unified log format Little information about Flume-NG
26.
0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd
27.
0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd
28.
NHN Japan COOKPAD NAVERCrocoshttp://www.quora.com/Who-uses-Fluentd-in-production
29.
0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd
30.
0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd
31.
Future of Fluentd> <filter>> <match> in <source>> <label>> MessagePack for Ruby v5> td-agent-lite> Pub/Sub & Monitoring API> New process model & Live restart> Backward compatibility
32.
<source> <match **> type tail type forward path /var/log/httpd.log host log.server format apache </match> tag not_filtered.apache</source> Mysterious tag<match not_filetered.**> type rewrite remove_prefix not_filtered tag operations <rule> key status pattern ^500$ ignore true </rule></match> Before
33.
<source> <match **> type tail type forward path /var/log/httpd.log host log.server format apache </match> tag apache</source><filter **> Filter plugins! type rewrite <rule> key status pattern ^500$ ignore true </rule></match> After (v11)
34.
<source> <match **> type tail type forward path /var/log/httpd.log host log.server format apache </match> tag apache <filter **> type rewrite <rule> key status pattern ^500$ <filter>/<match> in ignore true <source> </rule> </match></source> After (v11)
35.
<source> <match **> type tail type forward path /var/log/httpd.log host log.server tag apache </match></source> I want to add flowcounter here... Before
36.
<source> <store> type tail type forward path /var/log/httpd.log host log.server tag apache </store></source> </match><match flow.traffic> type forward host traffic.server</match> Nested!<match **> type copy <store> type flowcounter tag flow.traffic </store> Before
37.
<source> <match **> type tail type forward path /var/log/httpd.log host log.server tag apache </match></source><filter **> type copy <match> type flowcounter tag flow.traffic <match> type forward host traffic.server </match> </match> Filtering pipeline</match> After (v11)
38.
<source> # copy & label & forward type forward <filter **></source> type copy <match><filter **> type forward type copy label alert <match> host alerting.server type file </match> path /mnt/local_archive </filter> </match></filter> # copy & label & forward <filter **><label alert> type copy <match **> <match> ... type forward </match> label analysis</label> host analysis.server </match><label analysis> </filter> ...</label> After (v11)
40.
td-agent-lite> in_tail + out_forward in “single” binary statically linked ruby binary + scripts tied with the binary
41.
New process model & Live restartOld multiprocess model detached process fork() Supervisor Engine detached all data pass through process the central process
42.
New process model & Live restartNew multiprocess model detached process Process Supervisor Engine Manager detached process direct communication
43.
New process model & Live restartNew multiprocess model detached process Process Supervisor Engine Manager detached Live restart process Process Engine Manager
44.
Backward compatibilityFluentd v11 includes 2 namespaces: > Fluentd:: new code base > Fluent:: old code base + wrapper classesCheckout the repository for details: > http://github.com/frsyuki/fluentd-v11
45.
ConculutionFluentd makes logging better > Plugin architecture > JSON format > HA configuration > RPM/deb packageFluentd is under active developmentFluentd is suppored by many committers
Views
Actions
Embeds 0
Report content