Modern Black Mages Fighting in the Real World

1. Modern Black Mages Fighting in the Real World Sep 9, 2016 in RubyKaigi 2016 @tagomoris Satoshi "Moris" Tagomori

2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, ... Treasure Data, Inc.

3. https://github.com/tagomoris/msgpack-inspect

4. http://docs.ﬂuentd.org/articles/logo http://www.ﬂuentd.org/

5. Fluentd • What is Fluentd? • Open Source Log Collector • Pluggable, Reliable, Less resource usage, Ease to use • Versions of Fluentd • v0.12: stable versions (2014/12 - Now) • v0.14: versions for next stable (2016/05 - Now)

6. http://docs.ﬂuentd.org/articles/logo

7. What's about this session? • Introduce some patterns of "Black Magic"s (a.k.a. meta programming) in Ruby • Show you some PRAGMATIC use of Black Magics

8. Fluentd v0.14 Release

9. Fluentd v0.14 API Update • Everything changed :) • Plugin namespace • before: Fluent::* (Top level classes even for plugins!) • after: Fluent::Plugin::* • Plugin base class for common methods • Inconsistent Output plugin hierarchy • Plugin must call `super` in common methods http://www.slideshare.net/tagomoris/ﬂuentd-v014-plugin-api-details

10. Classes hierarchy (v0.12) Fluent::Input F::Filter F::Output BufferedOutput Object Buffered Time Sliced Multi Output F::Buffer F::Parser F::Formatter 3rd party plugins

11. Classes hierarchy (v0.14) F::P::Input F::P::Filter F::P::Output Fluent::Plugin::Base F::P::Buffer F::P::Parser F::P::Formatter F::P::Storage both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin

12. diff v0.12 v0.14 F::P::Output Fluent::Plugin::Base both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin F::Output BufferedOutput Object Buffered Time Sliced Multi Output Super classes by how to buﬀer data All output plugins are just "Output"

13. Basic Black Magics: Class and Mixin in Ruby

14. Class and Subclass in Ruby class A #bar class B #bar super B.new.bar

15. class A #bar class B #bar super B.new.bar module M #bar Introducing Methods by Mixin

16. class A #bar class B #bar super B.new.bar module M #bar Singleton Class of Ruby #bar B.new.singleton_class

17. class A #bar class B #bar super b=B.new b.singleton_class.include M2 b.bar module M #bar Adding Methods on An Instance (1) B.new.singleton_class #bar M2 #bar

18. class A #bar class B #bar super b=B.new b.extend M2 b.bar module M #bar Adding Methods on An Instance (2) B.new.singleton_class #bar M2 #bar

19. Back to Fluentd code :)

20. diff v0.12 v0.14 F::P::Output Fluent::Plugin::Base both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin F::Output BufferedOutput Object Buffered Time Sliced Multi Output Super classes by how to buﬀer data All output plugins are just "Output"

21. Fluentd v0.12 Fluent::Output class Fluent::Output #emit(tag, es, chain) MyOutput Engine calls plugin.emit(tag, es, chain) @buﬀer

22. Fluentd v0.12 Fluent::BufferedOutput (1) class Fluent::Outputclass BuﬀeredOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buﬀer #emit(key, data, chain) #format(tag,time,record) #format_stream(tag,es)

23. Fluentd v0.12 Fluent::BufferedOutput (2) class Fluent::Outputclass BuﬀeredOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buﬀer #emit(key, data, chain) #format_stream(tag,es) #format_stream(tag,es) #format(tag,time,record)

24. Fluentd v0.12 Fluent::TimeSlicedOutput class Fluent::Outputclass BuﬀeredOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) Engine calls plugin.emit(tag, es, chain) @buﬀer #emit(key, data, chain) #emit(tag, es, chain) class TimeSlicedOutput #format(tag,time,record)

25. Fluentd v0.12 Fluent::ObjectBufferedOutput class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #emit(tag, es, chain) class ObjectBufferedOutput

26. Fluentd v0.12 Fluent::BufferedOutput class Fluent::Outputclass BuﬀeredOutputMyOutput @buffer calls #write in OutputThread @buﬀer chunk #write(chunk) OutputThread #pop

27. Fluentd v0.12 Fluent::TimeSlicedOutput class Fluent::Outputclass BuﬀeredOutput @buﬀer MyOutput class TimeSlicedOutput OutputThread #write(chunk) @buffer calls #write in OutputThread #write calls chunk.key chunk #pop

28. Fluentd v0.12 Fluent::ObjectBufferedOutput class Fluent::Outputclass BufferedOutput @buffer MyOutput class ObjectBufferedOutput OutputThread #write(chunk) #write(chunk) #write_object(chunk_key, chunk) @buffer calls #write in OutputThread chunk #pop

29. Fluentd v0.12 API Problems • Entry point method is implemented by Plugin subclasses • Fluentd core cannot add any processes • counting input events • hook arguments/return values to update API • Fluentd core didn't show ﬁxed API • Plugins have different call stacks • It's not clear what should be implemented for authors • It's not clear what interfaces are supported for arguments/return values

30. How can we solve this problem?

31. Fluent::Plugin::Output (v0.14)

32. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput #process(tag, es) Engine calls plugin.emit_events(tag, es) @buﬀer #write #emit_events(tag, es) #format(tag, time, record) #write(chunk) #try_write(chunk) #emit_sync(tag, es) #emit_buffered(tag, es)

33. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput Output calls plugin.write (or try_write) @buﬀer chunk #write(chunk) #try_write(chunk) ﬂush thread #process(tag, es) #format(tag, time, record)

34. Fluentd v0.14 Design Policy • Separate entry points from implementations • Methods in superclass control everything • Do NOT override these methods! • Methods in subclass do things only for themselves • not for data ﬂow, control ﬂow nor others • Plugins have simple/straightforward call stack • Easy to understand/maintain

35. ❤

36. How about existing v0.12 plugins?

37. Requirement: (Almost) All Existing Plugins SHOULD  Work Well WITHOUT ANY MODIFICATION

38. • Fluent::Compat namespace for compatibility layer v0.14 Plugins & Compat Layer F::P::Output F::P::Base v0.14 Plugins Fluent:: Compat:: Output F::C:: Buffered Output F::C:: TimeSliced Output F::C:: ObjectBuffered Output Fluent::Output F:: Buffered Output F:: TimeSliced Output F:: ObjectBuffered Output v0.12 Plugins

39. Double Decker Compat Layer? • Existing plugins inherits Fluent::Output or others • No more codes in Fluent top level :-( • Separate code into Fluent::Compat • and import it into Fluent top level

40. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput #process(tag, es) Engine calls plugin.emit_events(tag, es) @buﬀer #write #emit_events(tag, es) #format(tag, time, record) #write(chunk) #try_write(chunk) #emit_sync(tag, es) #emit_buffered(tag, es)

41. Fluentd v0.12 Fluent::BufferedOutput (2) class Fluent::Outputclass BuﬀeredOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buﬀer #emit(key, data, chain) #format_stream(tag,es) #format_stream(tag,es) #format(tag,time,record)

42. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) v0.12 Plugins via Compat Layer: Best case (virtual) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread

43. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) v0.12 Plugins via Compat Layer: Best case (real) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread

44. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #format_stream Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread

45. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #format_stream Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread default implementation for calling "super"

46. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread

47. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread When plugin overrides #emit

48. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread This call doesn't happen, in fact #emit doesn't return values! When plugin overrides #emit

49. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #emit calls @buffer.emit → NoMethodError !

51. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #emit 1. #emit calls @buffer.emit with data to be written in buffer 0. plugin calls @buffer.extend to add #emit 2. @buffer.emit stores arguments into plugin's attribute 3. get stored data 4. call @buffer.write with data

53. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Thinking about "chunk" instance ... Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #write may call "chunk.key", but v0.14 chunk doesn't have #key !

54. Fluent::Plugin::Outputclass MyOutput @buffer #write Compat::BufferedOutput #write(chunk) flush thread "chunk" has #metadata, and values of #key can be created via #metadata Let's "chunk.extend" ! Where to do so? ? Thinking about "chunk" instance ...

55. Fluent::Plugin::OutputMyOutput @buffer #write C::BufferedOutput #write(chunk) flush thread Thinking about "chunk" instance ... #write(chunk) BufferedChunkMixin plugin.extend BufferedChunkMixin in #configure

56. Similar hacks for TimeSlicedOutput and ObjectBufferedOutput ...

57. Controlling Plugin Lifecycle

58. Plugin Lifecycle Updated Methods(v0.12) • #configure • #start • #before_shutdown • #shutdown v0.12 Plugins often doesn't call "super"! Methods(v0.14) • #configure • #start • #stop • #before_shutdown • #shutdown • #after_shutdown • #close • #terminate In v0.14, these methods MUST call "super" • #configured? • #started? • #stopped? • #before_shutdown? • #shutdown? • #after_shutdown? • #closed? • #terminated?

59. For Example: shutdown compat plugins Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown It doesn't call "super"! We want to call this...

60. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown 1. call #shutdown anyway 0. Fluentd core calls #shutdown 2. call #shutdown? to check "super" is called or not 3. call #shutdown of superclass forcedly!

61. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown How to make this point?

62. One More Magic! Module#prepend

63. class A #bar class B #bar super B.new.bar Wrapping Methods on a Class (1) B.new.singleton_class #bar

64. class A #bar class B #bar super B.new.bar module M Wrapping Methods on a Class (2) B.new.singleton_class #bar #bar Using extend is powerful, but it should be done for all instances How about wrapping methods for all instances of the class?

65. class A #bar class B #bar super module M;def bar;super;end;end B.prepend M B.new.bar module M Wrapping Methods on a Class (3): Module#prepend B.new.singleton_class #bar #bar module M wraps B, and M#bar is called at ﬁrst

66. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown THIS ONE !!!

67. What We Got :-) Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown 1. call #shutdown anyway 0. prepend CallSuperMixin at ﬁrst 2. call #shutdown? to check "super" is called or not 3. if not, get method of superclass, bind self with it, then call it Thank you @unak -san!

68. Beating Test Code

69. Testing: Capturing return values by Test Driver OutputMyOutput @buﬀer #write #emit_events #format #emit_buffered Output Plugin Test Driver Create plugin instances Feed test data into plugin

70. Testing: Capturing return values by Test Driver OutputMyOutput @buﬀer #write #emit_events #format #emit_buffered We want to assert this return value! Output Plugin Test Driver Feed test data into plugin

71. Using #prepend to capture return values OutputMyOutput @buﬀer #write #emit_events #format #emit_buffered Output Plugin Test Driver Feed test data into plugin module M #format Store return value of "super"

72. Using #prepend ... doesn't work :-( OutputMyOutput @buﬀer #write #emit_events #format #emit_buffered Output Plugin Test Driver Feed test data into plugin #format Test code sometimes overwrites methods for many reasons :P singleton class #format 😱 module M

73. One Another Magic: Stronger Than Anything

74. class A #bar class B #bar super b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.bar Another Study: How To Wrap Singleton Method? B.new.singleton_class #bar module M #bar

75. class A #bar class B #bar super module P Another Study: How To Wrap Singleton Method? B.new.singleton_class #bar b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.singleton_class.prepend P b.bar #bar module M #bar Singleton class is a class, so it can be prepended :) It's actually done in Test Driver implementation...

76. Using #prepend on singleton_class: Yay! OutputMyOutput @buﬀer #write #emit_events #format #emit_buffered Output Plugin Test Driver Feed test data into plugin #format singleton class #format😃 module M Prepending modules on singleton_class overrides everything!

77. IS BUILT ON A TOP OF BUNCH OF BLACK MAGICS :P

78. Do Whatever You Can For Users! It Makes Everyone Happier! ... Except for Maintainers :(

Modern Black Mages Fighting in the Real World

SATOSHI TAGOMORI

Modern Black Mages Fighting in the Real World