12.
Core Plugins
12
• Divide & Conquer
• Buffering & Retries
• Error Handling
• Message Routing
• Parallelism
• Read Data
• Parse Data
• Buffer Data
• Write Data
• Format Data
13.
Core Plugins
13
• Divide & Conquer
• Buffering & Retries
• Error Handling
• Message Routing
• Parallelism
• Read Data
• Parse Data
• Buffer Data
• Write Data
• Format Data
Common
Concerns
Use Case
Specific
20.
M x N → M + N
Alerting
Nagios
Analysis
MongoDB
MySQL
Hadoop
Archiving
Amazon S3
Access logs
Apache
App logs
Frontend
Backend
System logs
syslogd
Databases
buffer/filter/route
23.
# logs from a file
<source>
type tail
path /var/log/httpd.log
format apache2
tag backend.apache
</source>
# logs from client libraries
<source>
type forward
port 24224
</source>
# store logs to ES and HDFS
<match backend.*>
type mongo
database fluent
collection test
</match>
26.
# logs from a file
<source>
type tail
path /var/log/httpd.log
format apache2
tag web.access
</source>
# logs from client libraries
<source>
type forward
port 24224
</source>
# store logs to ES and HDFS
<match backend.*>
type copy
<store>
type elasticsearch
logstash_format true
</store>
<store>
type webhdfs
host namenode
port 50070
path /path/on/hdfs/
</store>
</match>
33.
Input plugins
HTTP+JSON (in_http)
File tail (in_tail)
Syslog (in_syslog)
...
✓ Receive logs
✓ Or pull logs from data sources
✓ non-blocking
Input
34.
Input plugins
module Fluent
class NewTailInput < Input
Plugin.register_input('tail', self)
def initialize
super
@paths = []
@tails = {}
end
end
# Little more code
end
35.
Input plugins
module Fluent
class NewTailInput < Input
Plugin.register_input('tail', self)
def initialize
super
@paths = []
@tails = {}
end
config_param :path, :string
config_param :tag, :string
config_param :rotate_wait, :time, :default => 5
config_param :pos_file, :string, :default => nil
config_param :read_from_head, :bool, :default => false
config_param :refresh_interval, :time, :default => 60
attr_reader :paths
def configure(conf)
super
@paths = @path.split(',').map {|path| path.strip }
if @paths.empty?
raise ConfigError, "tail: 'path' parameter is required on tail input"
end
unless @pos_file
$log.warn "'pos_file PATH' parameter is not set to a 'tail' source."
$log.warn "this parameter is highly recommended to save the position to resume tailing."
end
configure_parser(conf)
configure_tag
@multiline_mode = conf['format'] == 'multiline'
@receive_handler = if @multiline_mode
method(:parse_multilines)
else
method(:parse_singleline)
end
end
def configure_parser(conf)
@parser = TextParser.new
@parser.configure(conf)
end
def configure_tag
if @tag.index('*')
@tag_prefix, @tag_suffix = @tag.split('*')
@tag_suffix ||= ''
else
@tag_prefix = nil
@tag_suffix = nil
end
end
def start
if @pos_file
@pf_file = File.open(@pos_file, File::RDWR|File::CREAT, DEFAULT_FILE_PERMISSION)
@pf_file.sync = true
@pf = PositionFile.parse(@pf_file)
end
@loop = Coolio::Loop.new
refresh_watchers
@refresh_trigger = TailWatcher::TimerWatcher.new(@refresh_interval, true, log, &method(:refresh_watchers))
@refresh_trigger.attach(@loop)
@thread = Thread.new(&method(:run))
end
def shutdown
@refresh_trigger.detach if @refresh_trigger && @refresh_trigger.attached?
stop_watchers(@tails.keys, true)
@loop.stop rescue nil # when all watchers are detached, `stop` raises RuntimeError. We can ignore this exception.
@thread.join
@pf_file.close if @pf_file
end
def expand_paths
date = Time.now
paths = []
@paths.each { |path|
path = date.strftime(path)
if path.include?('*')
paths += Dir.glob(path)
else
# When file is not created yet, Dir.glob returns an empty array. So just add when path is static.
paths << path
end
}
paths
end
# in_tail with '*' path doesn't check rotation file equality at refresh phase.
# So you should not use '*' path when your logs will be rotated by another tool.
# It will cause log duplication after updated watch files.
# In such case, you should separate log directory and specify two paths in path parameter.
# e.g. path /path/to/dir/*,/path/to/rotated_logs/target_file
def refresh_watchers
target_paths = expand_paths
existence_paths = @tails.keys
unwatched = existence_paths - target_paths
added = target_paths - existence_paths
700 lines!
36.
Input plugins
module Fluent
class TcpInput < SocketUtil::BaseInput
Plugin.register_input('tcp', self)
config_set_default :port, 5170
config_param :delimiter, :string, :default => "n" # syslog
family add "n" to each message and this seems only way to split
messages in tcp stream
def listen(callback)
log.debug "listening tcp socket on #{@bind}:#{@port}"
Coolio::TCPServer.new(@bind, @port,
SocketUtil::TcpHandler, log, @delimiter, callback)
end
end
end
37.
Input plugins
class BaseInput < Fluent::Input
# some code
def on_message(msg, addr)
@parser.parse(msg) { |time, record|
unless time && record
log.warn "pattern not match: #{msg.inspect}"
return
end
record[@source_host_key] = addr[3] if @source_host_key
Engine.emit(@tag, time, record)
}
# some code
end
38.
Input plugins
class BaseInput < Fluent::Input
# some code
def on_message(msg, addr)
@parser.parse(msg) { |time, record|
unless time && record
log.warn "pattern not match: #{msg.inspect}"
return
end
record[@source_host_key] = addr[3] if @source_host_key
Engine.emit(@tag, time, record)
}
# some code
end
39.
Parser plugins
JSON
Regexp
Apache/Nginx/Syslog
CSV/TSV, etc.
✓ Parse into JSON
✓ Common formats out of the box
✓ v0.10.46 and above
Parser
40.
Parser plugins
<source>
type tcp
tag tcp.data
format /^(?<field_1>d+) (?<field_2>w+)/
</source>
41.
Parser plugins
def call(text)
m = @regexp.match(text)
# some code
time = nil
record = {}
m.names.each {|name|
if value = m[name]
case name
when "time"
time = @mutex.synchronize { @time_parser.parse(value) }
else
record[name] = if @type_converters.nil?
value
else
convert_type(name, value)
end
end
end
}
# some code
end
42.
Buffer plugins
✓ Improve performance
✓ Provide reliability
Buffer
Memory (buf_memory) ✓ Provide thread-safety
File (buf_file)
43.
Buffer plugins
✓ Chunk = adjustable unit of data
✓ Buffer = Queue of chunks
chunk
chunk
chunk output
Input
45.
Output plugins
class FileOutput < TimeSlicedOutput
Plugin.register_output('file', self)
# some code
def write(chunk)
path = generate_path(chunk)
FileUtils.mkdir_p File.dirname(path)
case @compress
when nil
File.open(path, "a", DEFAULT_FILE_PERMISSION) {|f|
chunk.write_to(f)
}
when :gz
File.open(path, "a", DEFAULT_FILE_PERMISSION) {|f|
gz = Zlib::GzipWriter.new(f)
chunk.write_to(gz)
gz.close
}
end
return path # for test
end
# more code
46.
Formatter plugins
✓ Format output
✓ Only partially supported for now
Formatter
JSON ✓ v0.10.49 and above
CSV/TSV
“single value”
47.
Formatter plugins
class SingleValueFormatter
include Configurable
config_param :message_key, :string, :default => 'message'
config_param :add_newline, :bool, :default => true
def format(tag, time, record)
text = record[@message_key].to_s
text << "n" if @add_newline
text
end
end
53.
Treasure Agent
• Treasure Data distribution of Fluentd
• including Ruby, core libraries and
QA’ed 3rd party plugins
• rpm/deb/dmg
• 2.1.2 is released TODAY with fluentd-ui
53
54.
fluentd-forwarder
• Forwarding agent written in Go
• mainly for Windows support
• less mature than Fluentd
• Bundle TCP input/output and TD
output
• No plugin mechanism
54
Views
Actions
Embeds 35
Report content