Hijacking Ruby Syntax in Ruby

Hijacking Ruby Syntax in Ruby

1. 1. HIJACKING RUBY SYNTAX IN RUBY RubyKaigi 2018 Sendai #rubykaigi 2018/05/31 (Fri) @joker1007 & @tagomoris
2. 2. Self-Intro Joker ▸ id: joker1007 ▸ Repro inc. CTO ▸ I’m familiar with Ruby/Rails/Fluentd/ECS/Presto. ▸ I ♥ “Jojo’s Bizarre Adventure” so much. ▸ I’m very happy to talk in Sendai (Moriocho)
3. 3. Satoshi Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, Woothee, ... Treasure Data, Inc.
4. 4. Agenda ▸ Ruby Features ▸ Hacks ▸ finalist, overrider, abstriker ▸ binding_ninja ▸ with_resources, deferral
5. 5. Ruby Features: Binding Binding ▸ Context object, includes: ▸ Kernel#binding receiver ▸ local variables ▸ For template engines (?) ▸ Methods: ▸ #receiver, #eval,
 #local_variables,
 #local_variable_get,
 #local_variable_defined?,
 #local_variable_set
6. 6. テキスト
7. 7. テキスト
8. 8. テキスト
9. 9. Ruby Features: Binding Binding#local_variable_set ▸ Method to ▸ add a variable only in a binding instance ▸ overwrite values of existing variables in original context
10. 10. Ruby Features: TracePoint TracePoint ▸ Tracing events in VM ▸ For various events: ▸ :line , :raise ▸ :class , :end ▸ :call , :return , :c_call , :c_return , :b_call , :b_return ▸ :thread_begin , :thread_end , :fiber_switch ▸ "We can use TracePoint to gather information specifically for exceptions:" (from Ruby doc) ▸ This is COMPLETELY WRONG statement...
11. 11. Ruby Features: TracePoint TracePoint Methods ▸ Methods: ▸ Control: disable, enable, enabled? ▸ Event what/where: event, defined_class, path, lineno ▸ Method names: method_id, callee_id ▸ Event special: raised_exception, return_value ▸ And: binding ▸ So what? ▸ We can use TracePoint  ▸ to gather information ▸ to overwrite everything
12. 12. Ruby Features: TracePoint
13. 13. テキスト 10min Break (10min)
14. 14. Ruby Features: Refinements ▸ Refinements provide a way to extend a class locally ▸ Useful use case. (Safety monkey patching) Ruby Features: Refinements
15. 15. Another use case ▸ Super private method Ruby Features: Refinements
16. 16. Hacks 1 Method modifiers ▸ final (https://github.com/joker1007/finalist) ▸ forbid method override ▸ override (https://github.com/joker1007/overrider) ▸ enforce method has super method ▸ abstract (https://github.com/joker1007/abstriker) ▸ enforce method override These method modifiers work when Ruby defines class. It is actually runtime, but in most case, before main logic. Hacks: method modifiers
17. 17. Sample code: finalist Hacks: method modifiers
18. 18. Sample code: overrider Hacks: method modifiers
19. 19. Sample code: abstriker Hacks: method modifiers
20. 20. How to implement method modifiers I use so many hook methods. `included`, `extended`, `method_added`, and `TracePoint`. And I use `Ripper`. In other words, I use the power of many black magics !! Hacks: method modifiers
21. 21. Ruby Features: Method Hooks ▸ Ruby has six method hooks ▸ Module#method_added ▸ Module#method_removed ▸ Module#method_undefined ▸ BasicObject#singleton_method_added ▸ BasicObject#singleton_method_removed ▸ BasicObject#singleton_method_undefined Ruby Features: method hooks
22. 22. Sample: #method_added Ruby Features: method hooks
23. 23. Sample code: #method_removed Ruby Features: method hooks
24. 24. Sample code: #method_undefined Ruby Features: method hooks
25. 25. Use case in finalist (Simplified sample) Ruby Features: method hooks
26. 26. call method_added -> call verify_final_method
27. 27. Usage: Method Hook ▸ Method Hook provides a way to implement declarations (like protected, private) ▸ With class inheritance, Method Hook seems magic !! ▸ But it is not enough to implement “finalist” actually ▸ Ruby has so many cases of method definition ▸ `def` or `define_method` ▸ `include` module ▸ `extend`, `prepend ▸ Each case calls different hooks Ruby Features: method hooks
28. 28. “include” does not add method ▸ Because `include` insert module to method lookup hierarchy, it does not touch method entry table. Ruby Features: method hooks
29. 29. “include” changes only chain of method discovering Foo Object Bar Insert module to hierarchy It is different from to add method Class#ancestors displays class-module hierarchy Ruby Features: method hooks
30. 30. Hooks in finalist gem ▸ `method_added` to detect override by subclass ▸ `singleton_method_added` to detect override class method ▸ `included` to detect override by module include ▸ `extended` to detect override by module extend
31. 31. And I talk about TracePoint too
32. 32. overrider and abstriker uses TracePoint ▸ `inherited` and `included` to start `TracePoint`` call main logic call main logic Tracepoint in overrider, abstriker
33. 33. Tracepoint in overrider, abstriker Check abstract method existence in class hierarchy
34. 34. Why use TracePoint? ▸ In order to verify method existence at the end of class definition. ▸ Ruby interpreter needs to wait until the end of class definition to know a method absence. ▸ override and abstract cannot detect violation just when they are called. ▸ In ruby, The only way to detect the violation is `TracePoint`.
35. 35. Advanced TracePoint: Detect particular class end Advanced Tracepoint :end event cannot trace definition by `Class.new`. Use :c_return and return_value to detect particular class end
36. 36. Advanced TracePoint: Ripper combination Advanced Tracepoint Detect target Sexp node by TracePoint#linen Sexp node type expresses the style of method cal
37. 37. Ripper empowers TracePoint ▸ `Ripper.sexp` returns token position ▸ And TracePoint returns lineno an event occurs. ▸ Ripper provides more information about event context. ▸ ex. power_assert gem is implemented by this combination. Advanced Tracepoint
38. 38. Hacks 2 (binding_ninja) ▸ https://github.com/joker1007/binding_ninja ▸ It is method wrapper to pass binding of method caller implicitly. ▸ I talked about it at Rubykaigi 2017 LT. Hacks: binding_ninja
39. 39. Sample code: binding_ninja Hacks: binding_ninja Pass binding of caller implicitly
40. 40. How To Implement ▸ Binding is based on Ruby level control frame. ▸ When Ruby calls c-method, Ruby does not change Ruby level control frame. ▸ It is core logic of binding_ninja create binding in C level, binding keeps caller stack in Ruby level Hacks: binding_ninja
41. 41. Use case: binding_ninja ▸ I implement scala like implicit parameter in Ruby. ▸ https://github.com/ joker1007/ implicit_parameter Hacks: binding_ninja
42. 42. Hacks: binding_ninja Wrap method by `define_method` And wrap method by binding_ninja
43. 43. Black Magic is dangerous actually, but it is very fun, and it extends Ruby potential These gems is for proof of concept. But these work and decent practical.
44. 44. Ruby Quiz Break (25-30min) What the difference between: - #undef_method - #remove_method
45. 45. RUBY QUIZ class Foo def foo class Foo def foo class Bar < Foo def foo class Bar < Foo def foo Bar.new.foo()
46. 46. RUBY QUIZ class Foo def foo class Foo def foo class Bar < Foo def foo class Bar < Foo Bar.new.foo() undef_method(:foo) NoMethodError remove_method(:foo)
47. 47. Hack: with_resources Add "with" Statement in Ruby ▸ Safe resource allocate/release ▸ Ensure to release resources ▸ after a lexical scope ▸ in reverse order of allocation ▸ Idioms used very frequently ▸ Other languages: ▸ Java: try-with-resources ▸ Python: with ▸ C#: using
48. 48. Hack: with_resources Safe Resource Allocation/Release Statement in Ruby ▸ Open method with blocks ▸ File.open(path){|f| ... } ▸ Ruby way (?) ▸ More indentation ▸ Not implemented sometimes
 (e.g., TCPSocket) ▸ Simple begin-ensure ▸ Anomaly cases can't be
 handled
49. 49. Hack: with_resources with_resources.gem ▸ Safe resource allocation ▸ Top level "with" ▸ via Kernel refinement ▸ Resource allocation as lambda ▸ Multi statements to allocate resources ▸ to release first resource
 if second resource allocation raises exception ▸ Block to define scope for resources https://github.com/tagomoris/with_resources
50. 50. Hack: with_resources Implementing "with" in Ruby ▸ TracePoint ▸ "b_return": pass allocated resources to block arguments ▸ "line": identify allocated resources in lambda ▸ Binding ▸ detect newly defined
 local variables
 in allocation lambda ▸ Refinements ▸ introduce "with"
 in top-level without side effects https://github.com/tagomoris/with_resources
51. 51. Hack: deferral Alternative: defer? ▸ Multi-step resource
 allocation in a method ▸ using "with" ▸ Nesting! ▸ not so bad ▸ many nesting looks
 a bit messy :( ▸ Alternative? ▸ "defer" in golang
52. 52. Hack: deferral Making "defer" in Ruby ▸ Block based "defer" ▸ Should work ▸ Requires 1-level
 nesting *always* ▸ Defer.start, end (+ nesting)
 look too much (than golang) ▸ Abnormal case:
 reassigning variables
53. 53. Hack: deferral deferral.gem ▸ Safe resource release ▸ Top level "defer" ▸ via Kernel refinements ▸ Deferred processing to release resources ▸ at the end of scope (method, block) ▸ or exception raised
54. 54. Hack: deferral Implementing "defer" in Ruby ▸ #defer ▸ Enable TracePoint if not yet ▸ Initialize internal stack frame ▸ TracePoint ▸ Monitor method call stack ▸ Get the snapshot of local variables in defer block ▸ Call release blocks at the end of scope ▸ Binding ▸ save/restore local variables of release block ▸ Refinements ▸ introduce "defer" in top-level without side effects stack level 0 stack level 1
55. 55. The Hard Thing in Magical World: Debugging!
56. 56. "Are You Ready? I'm OK." Blono Bucciarati Thank You! @joker1007 & @tagomoris