IoT Data Connector Fluent Bit

1. Toru Takahashi June 17, 2015 IoTLT vol.4 IoT Data Connector - Fluent Bit #iotlt

2. About Me > Toru Takahashi > twitter: @nora96o > github: toru-takahashi > Treasure Data, Inc. > Technical Support Engineer > I love working in Maid café :) > Most of staying in 女中酒場幻橙館 2

3. Today’s Talk focus … on IoT mbed Device Server mbed OS Wi-Fi mbed OS Wi-Fi mbed OS Wi-Fi mbed Device Server mbed OS Wi-Fi mbed OS Wi-Fi mbed OS Wi-Fi mbed Device Server mbed OS Wi-Fi mbed OS Wi-Fi mbed OS Wi-Fi > Application Layer > Logging Layer 3

4. Logging Layer on Web

5. Before 5

6. Structured logging Reliable forwarding Pluggable architecture http://ﬂuentd.org/

7. After 7

8. What’s Fluentd? > Data collector for unified logging layer > Streaming data transfer based on JSON > Written in Ruby > Gem based various plugins > http://www.fluentd.org/plugins > Working in production > http://www.fluentd.org/testimonials 8

9. Logging Layer on IoT

10. Current ? 10

11. Future 12

12. Fluent Bit Open Source data collection tool for Embedded Linux > Sensors > Services > Signals / Radios > Operating System Info > Automotive / Telematics Committer: Eduardo Silva (@edsiper) • https://github.com/fluent/fluent-bit • http://fluentbit.io (official web site) 13

13. Features > Collection & Distribution > Support Custom Input / Output > Configurable (through file system files) > Built-in system metrics > C API for Developers (WIP) > Integration with third party services > Open Source / Apache License v2.0 Fluent Bit 14

14. Examples

15. CPU Usage $ ./fluent-bit -i cpu -o stdout [2015/06/14 12:13:35] [ info] Flush buf 120 bytes [0] {"time"=>1434284015, "cpu"=>12.000000} [1] {"time"=>1434284016, "cpu"=>100.000000} [2] {"time"=>1434284017, "cpu"=>100.000000} [3] {"time"=>1434284018, "cpu"=>100.000000} [4] {"time"=>1434284019, "cpu"=>82.000000} [4] {"time"=>1434284024, "cpu"=>0.000000} Built-in Metrics: 16

16. Kernel Log Message $ ./fluent-bit -i kmsg -o stdout [2015/06/14 12:19:00] [ info] starting engine [0] {"time"=>1434283478, "priority"=>6, "sequence"=>0, "sec"=>0, "usec"=>0, "msg"=>"Initializing cgroup subsys cpuset"} [1] {"time"=>1434283478, "priority"=>6, "sequence"=>1, "sec"=>0, "usec"=>0, "msg"=>"Initializing cgroup subsys cpu"} [2] {"time"=>1434283478, "priority"=>6, "sequence"=>2, "sec"=>0, "usec"=>0, "msg"=>"Initializing cgroup subsys cpuacct"} [3] {"time"=>1434283478, "priority"=>5, "sequence"=>3, "sec"=>0, "usec"=>0, "msg"=>"Linux version 3.14.35-28.38.amzn1.x86_64 (mockbuild@gobi- build-64012) (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Wed Mar 11 22:50:37 UTC 2015"} Built-in Metrics: 17

17. Conﬁg File [TD] # API # === # The TreasureData API key. To obtain this please log into your API SOME_API_KEY # Database # ======== # Specify the name of your database, it must exists. Database db_example # Table # ===== # Specify the database table name where the records will be stored Table table_example Output to TreasureData: 18

18. Roadmap • Library mode • Support a stock of sensors (inputs) • HTTP input • Release ﬁrst stable version • Documentation 19

19. IoT❤ https://github.com/ﬂuent/ﬂuent-bit

IoT Data Connector Fluent Bit

Toru Takahashi

Transcript