Fighting API Compatibility On Fluentd Using "Black Magic"

1. Fighting API Compatibility On Fluentd Using "Black Magic" Jul 2 2016 in YAPC::Asia Hachioji #yapc8oji Satoshi "Moris" Tagomori (@tagomoris)

2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, ... Treasure Data, Inc.

3. http://docs.ﬂuentd.org/articles/logo

4. Fluentd v0.14 Release

5. Fluentd v0.14 API Update • Everything changed :) • Plugin namespace • before: Fluent::* (Top level classes even for plugins!) • after: Fluent::Plugin::* • Plugin base class for common methods • Inconsistent Output plugin hierarchy • Plugin must call `super` in common methods http://www.slideshare.net/tagomoris/ﬂuentd-v014-plugin-api-details

6. Classes hierarchy (v0.12) Fluent::Input F::Filter F::Output BufferedOutput Object Buffered Time Sliced Multi Output F::Buffer F::Parser F::Formatter 3rd party plugins

7. Classes hierarchy (v0.14) F::P::Input F::P::Filter F::P::Output Fluent::Plugin::Base F::P::Buffer F::P::Parser F::P::Formatter F::P::Storage both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin

8. diff v0.12 v0.14 F::P::Output Fluent::Plugin::Base both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin F::Output BufferedOutput Object Buffered Time Sliced Multi Output Super classes by how to buﬀer data All output plugins are just "Output"

9. Basic Weapons: Class and Mixin in Ruby

10. Class and Subclass in Ruby class A #bar class B #bar super B.new.bar

11. class A #bar class B #bar super B.new.bar module M #bar Introducing Methods by Mixin

12. class A #bar class B #bar super B.new.bar module M #bar Singleton Class of Ruby #bar B.new.singleton_class

13. class A #bar class B #bar super b=B.new b.singleton_class.include M2 b.bar module M #bar Adding Methods on An Instance (1) B.new.singleton_class #bar M2 #bar

14. class A #bar class B #bar super b=B.new b.extend M2 b.bar module M #bar Adding Methods on An Instance (2) B.new.singleton_class #bar M2 #bar

15. Back to Fluentd code :)

16. diff v0.12 v0.14 F::P::Output Fluent::Plugin::Base both of buffered/non-buffered F::P:: BareOutput (not for 3rd party plugins) F::P:: MultiOutput copy roundrobin F::Output BufferedOutput Object Buffered Time Sliced Multi Output Super classes by how to buﬀer data All output plugins are just "Output"

17. Fluentd v0.12 Fluent::Output class Fluent::Output #emit(tag, es, chain) MyOutput Engine calls plugin.emit(tag, es, chain) @buﬀer

18. Fluentd v0.12 Fluent::BufferedOutput (1) class Fluent::Outputclass BuﬀeredOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buﬀer #emit(key, data, chain) #format(tag,time,record) #format_stream(tag,es)

19. Fluentd v0.12 Fluent::BufferedOutput (2) class Fluent::Outputclass BuﬀeredOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buﬀer #emit(key, data, chain) #format_stream(tag,es) #format_stream(tag,es) #format(tag,time,record)

20. Fluentd v0.12 Fluent::TimeSlicedOutput class Fluent::Outputclass BuﬀeredOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) Engine calls plugin.emit(tag, es, chain) @buﬀer #emit(key, data, chain) #emit(tag, es, chain) class TimeSlicedOutput #format(tag,time,record)

21. Fluentd v0.12 Fluent::ObjectBufferedOutput class Fluent::Outputclass BufferedOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) Engine calls plugin.emit(tag, es, chain) @buffer #emit(key, data, chain) #emit(tag, es, chain) class ObjectBufferedOutput

22. Fluentd v0.12 Fluent::BufferedOutput class Fluent::Outputclass BuﬀeredOutputMyOutput @buffer calls #write in OutputThread @buﬀer chunk #write(chunk) OutputThread #pop

23. Fluentd v0.12 Fluent::TimeSlicedOutput class Fluent::Outputclass BuﬀeredOutput @buﬀer MyOutput class TimeSlicedOutput OutputThread #write(chunk) @buffer calls #write in OutputThread #write calls chunk.key chunk #pop

24. Fluentd v0.12 Fluent::ObjectBufferedOutput class Fluent::Outputclass BufferedOutput @buffer MyOutput class ObjectBufferedOutput OutputThread #write(chunk) #write(chunk) #write_object(chunk_key, chunk) @buffer calls #write in OutputThread chunk #pop

25. Fluentd v0.12 API Problems • Entry point method is implemented by Plugin subclasses • Fluentd core cannot add any processes • counting input events • hook arguments/return values to update API • Fluentd core didn't show ﬁxed API • Plugins have different call stacks • It's not clear what should be implemented for authors • It's not clear what interfaces are supported for arguments/return values

26. How can we solve this problem?

27. Fluent::Plugin::Output (v0.14)

28. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput #process(tag, es) Engine calls plugin.emit_events(tag, es) @buﬀer #write #emit_events(tag, es) #format(tag, time, record) #write(chunk) #try_write(chunk) #emit_sync(tag, es) #emit_buffered(tag, es)

29. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput Output calls plugin.write (or try_write) @buﬀer chunk #write(chunk) #try_write(chunk) ﬂush thread #process(tag, es) #format(tag, time, record)

30. Fluentd v0.14 Design Policy • Separate entry points from implementations • Methods in superclass control everything • Do NOT override these methods! • Methods in subclass do things only for themselves • not for data ﬂow, control ﬂow nor others • Plugins have simple/straightforward call stack • Easy to understand/maintain

31. ❤

32. How about existing v0.12 plugins?

33. Requirement: (Almost) All Existing Plugins SHOULD  Work Well WITHOUT ANY MODIFICATION

34. • Fluent::Compat namespace for compatibility layer v0.14 Plugins & Compat Layer F::P::Output F::P::Base v0.14 Plugins Fluent:: Compat:: Output F::C:: Buffered Output F::C:: TimeSliced Output F::C:: ObjectBuffered Output Fluent::Output F:: Buffered Output F:: TimeSliced Output F:: ObjectBuffered Output v0.12 Plugins

35. Double Decker Compat Layer? • Existing plugins inherits Fluent::Output or others • No more codes in Fluent top level :-( • Separate code into Fluent::Compat • and import it into Fluent top level

36. Fluentd v0.14 Fluent::Plugin::Output class Outputclass MyOutput #process(tag, es) Engine calls plugin.emit_events(tag, es) @buﬀer #write #emit_events(tag, es) #format(tag, time, record) #write(chunk) #try_write(chunk) #emit_sync(tag, es) #emit_buffered(tag, es)

37. Fluentd v0.12 Fluent::BufferedOutput (2) class Fluent::Outputclass BuﬀeredOutput #emit(tag, es, chain, key) MyOutput #emit(tag, es, chain) super(tag, es, chain, any_key) Engine calls plugin.emit(tag, es, chain) @buﬀer #emit(key, data, chain) #format_stream(tag,es) #format_stream(tag,es) #format(tag,time,record)

38. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) v0.12 Plugins via Compat Layer: Best case (virtual) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread

39. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) v0.12 Plugins via Compat Layer: Best case (real) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread

40. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #format_stream Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread

41. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #format_stream Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain, key) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread default implementation for calling "super"

42. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread

43. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread When plugin overrides #emit

44. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread This call doesn't happen, in fact #emit doesn't return values! When plugin overrides #emit

45. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #emit calls @buffer.emit → NoMethodError !

47. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) When plugin overrides #emit Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #emit 1. #emit calls @buffer.emit with data to be written in buffer 0. plugin calls @buffer.extend to add #emit 2. @buffer.emit stores arguments into plugin's attribute 3. get stored data 4. call @buffer.write with data

49. Fluent::Plugin::Outputclass MyOutput @buffer #write #emit_events(tag, es) Thinking about "chunk" instance ... Compat::BufferedOutput #emit_buffered(tag, es) #format(tag, time, record) #format_stream(tag,es) #handle_stream_* #handle_stream_simple #emit(tag, es, chain) #emit(tag, es, chain, key) #format_stream(tag,es) #write(chunk) flush thread #write may call "chunk.key", but v0.14 chunk doesn't have #key !

50. Fluent::Plugin::Outputclass MyOutput @buffer #write Compat::BufferedOutput #write(chunk) flush thread "chunk" has #metadata, and values of #key can be created via #metadata Let's "chunk.extend" ! Where to do so? ? Thinking about "chunk" instance ...

51. Fluent::Plugin::OutputMyOutput @buffer #write C::BufferedOutput #write(chunk) flush thread Thinking about "chunk" instance ... #write(chunk) BufferedChunkMixin plugin.extend BufferedChunkMixin in #configure

52. Similar hacks for TimeSlicedOutput and ObjectBufferedOutput ...

53. Controlling Plugin Lifecycle

54. Plugin Lifecycle Updated Methods(v0.12) • #configure • #start • #before_shutdown • #shutdown v0.12 Plugins often doesn't call "super"! Methods(v0.14) • #configure • #start • #stop • #before_shutdown • #shutdown • #after_shutdown • #close • #terminate In v0.14, these methods MUST call "super" • #configured? • #started? • #stopped? • #before_shutdown? • #shutdown? • #after_shutdown? • #closed? • #terminated?

55. For Example: shutdown compat plugins Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown It doesn't call "super"! We want to call this...

56. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown 1. call #shutdown anyway 0. Fluentd core calls #shutdown 2. call #shutdown? to check "super" is called or not 3. call #shutdown of superclass forcedly!

57. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown How to make this point?

58. More Weapon! Module#prepend

59. class A #bar class B #bar super B.new.bar Wrapping Methods on a Class (1) B.new.singleton_class #bar

60. class A #bar class B #bar super B.new.bar module M Wrapping Methods on a Class (2) B.new.singleton_class #bar #bar Using extend is powerful, but it should be done for all instances How about wrapping methods for all instances of the class?

61. class A #bar class B #bar super module M;def bar;super;end;end B.prepend M B.new.bar module M Wrapping Methods on a Class (3): Module#prepend B.new.singleton_class #bar #bar module M wraps B, and M#bar is called at ﬁrst

62. class A #bar class B #bar super b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.bar Another Study: How To Wrap Singleton Method? B.new.singleton_class #bar

63. class A #bar class B #bar super module M Another Study: How To Wrap Singleton Method? B.new.singleton_class #bar Singleton class is a class, so it can be prepended :) b=B.new b.singleton_class.module_eval{define_method(:bar){"1"}} b.singleton_class.prepend M b.bar #bar It's actually done in Test Driver implementation...

64. What We Want To Do: Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown THIS ONE !!!

65. What We Got :-) Fluent::Plugin::Base #shutdown F::P::Output super #shutdown? #shutdown F::C::Output #shutdown MyOutput #shutdown 1. call #shutdown anyway 0. prepend CallSuperMixin at ﬁrst 2. call #shutdown? to check "super" is called or not 3. if not, get method of superclass, bind self with it, then call it Thank you @unak -san!

66. IS BUILT ON A TOP OF BUNCH OF BLACK MAGICS :P

67. Do Whatever You Can For Users! It Makes Everyone Happier ... Except for Maintainers :(

Fighting API Compatibility On Fluentd Using "Black Magic"

SATOSHI TAGOMORI

Fighting API Compatibility On Fluentd Using "Black Magic"