shopify / rotoscope Goto Github PK
View Code? Open in Web Editor NEWHigh-performance logger of Ruby method invocations
License: MIT License
High-performance logger of Ruby method invocations
License: MIT License
Introduced in #69
Rotoscope#flatten_into keeps a single call stack, which I don't expect would work properly if multiple threads are running. I think we would need to have a per-thread stack, which would require having the thread id as part of the event trace.
When we define tests using the test
method, we pass a string that sometimes include quote characters. However, rotoscope just uses printf style formatting for the output with no escaping of quote characters (e.g. fprintf(stream, "\"%s\",\"%s\",%s,\"%s\",%d\n",
). This causes problems when parsing the CSV output.
tl;dr TracePoint logger is too slow to compute caller filepaths
Rotoscope is implemented in C using the Ruby TracePoint API. By default, the :call
and :return
Ruby trace events allow you to retrieve the filepath and line number (via rb_tracearg_lineno and rb_tracearg_path in C). While this is quite fast, it has the unfortunate side effect of pointing to the callee of the method (i.e. where the method is received), rather than the caller.
The path of the caller is highly important to maintain an accurate "blacklist/whitelist" option so we don't log every single method invocation inside of our framework or hot paths that we don't care to know the internals of. Introducing a blacklist dramatically improves the performance abilities of tracing method invocations.
In an attempt to remain performant and accurate, I examined a rudimentary implementation where I installed a :line
TracePoint (performant, hah) and kept the last read line in memory. This does technically work but it isn't easily possible to use this to determine the return of a method, since it will only know the deepest level of the stack, not when a frame pop occurs.
At the suggestion of @dylanahsmith, I tried another variation where I invoked Kernel.caller_locations
to compute the backtrace of the current trace, and extract the filepath/lineno. This does produce 100% correct results, but is incredibly slow in comparison to the other attempts.
Using the tracearg methods, I'm able to complete a Shopify/shopify CI run in ~10 minutes with Rotoscope wrapping all tests. Using caller_locations
and bumping up the CI timeouts lets most of the containers finish in 2.5 hours. Boo. This is the version currently in the master
branch.
I've compiled a table below where you can see the retreived caller results of the different attempts:
We can shortcut the complexity of Kernel.caller_locations
on C-invocations and instead use the default TracePoint methods, since the top-level stack won't be able to point to the method definition, and it falls back to the caller instead.
Examining how rb_tracearg_filepath
is implemented, we can see rb_vm_get_ruby_level_next_cfp
is invoked. Looking at the definition of the method, the logic is relatively simple: increment the frame pointer until a Ruby frame is found. In our case, we want to find a Ruby frame, and return the frame after that one. While the method is not static
(a miracle?!), it is unfortunately declared inside of internal.h
and uses a significant amount of internal data structures that are prohibitively expensive to copy the definitions of as a workaround (e.g. rb_control_frame_t
has a field for rb_iseq_t
, which itself holds a field for rb_iseq_constant_body
, and that struct looks like this.
At this point, it's not particularly obvious to me how I can get around this performance problem without forking Ruby to implement a method that returns the second Ruby VM frame, but I'd like to avoid that if possible.
I've added a comment to the already-open issue in the Ruby bugtracker that detailed this problem, but haven't received any traction on it. The original issue was opened over four years ago.
/cc @dylanahsmith @airhorns @nick-mcdonald @stephenminded
This is happening for a similar reason as #57
This is happening because rotoscope tries to keep information from the stack based by pushing a frame to this stack on call events and popping a stack from on a return event.
except that it doesn't keep track of calls to blocks.
Unlike that issue, this one is easier to fix with the current approach by also tracing block calls and returns so they can be included in the rotoscope stack.
As identified in #38.
When using dynamically defined methods via define_method
and define_singleton_method
, the call stack is inconsistent compared with how predefined Ruby methods look.
Failing test here:
rotoscope/test/rotoscope_test.rb
Lines 413 to 428 in 5155f39
On :call
events, the frame points to where the method is received in Ruby, i.e. the block passed to define_method
or define_singleton_method
. This is the expected behaviour for Ruby invocations:
[TracePoint] has the unfortunate side effect of pointing to the callee of the method (i.e. where the method is received), rather than the caller.
But, when the :return
event fires, instead of pointing to the end of the method definition like a normal ruby method, it points one frame up the stack, to where the original caller
occurred. Since we assume Ruby methods point one level deeper, Rotoscope pops the frame one level further, and points to where the caller's caller came from. This means the call
and return
events won't match up.
I tried to figure out how to get around this one (by using Ruby's bitmasks to determine if it was a dynamically defined method, etc.), but I've been unsuccessful thus far.
This problem is currently avoidable via flatten: true
, since unmatched returns are ignored.
I built this extenison, ruby-debug and debugger-ruby_code_sources with support for 1.8.7.
I am finally struck at missing debug.h dependencies, which seems to be not present as part of ruby sources 1.8.7 :-(.
Any help ?
See CI test failures (https://github.com/Shopify/rotoscope/runs/3954967334?check_suite_focus=true) from draft PR #86 changing CI to test on ruby 3
For example, if we have a method defined with
def create_applied_discount_from(discount, code: discount.try(:code))
the discount.try(:code)
call shows that line in the file path and line number for the call, but the entity and method name refer to the caller of create_applied_discount_from
which is inconsistent with the file path. Ideally, for the discount.try(:code)
call, the caller method would show up as create_applied_discount_from
and the caller entity would be the class of the object that method was called on.
This is happening because rotoscope tries to keep information from the stack based by pushing a frame to this stack on call events and popping a stack from on a return event. However, in this case the call event hasn't happened yet.
We're currently on v0.3.0.pre.4 (soon to be pre.5 probably), but the changes in v0.3.0 have been largely stable for a while. Should cut a proper release once #54 goes in.
The deduplication is done by using a hash, but it uses the hash of the line as the key rather than the line itself. Since different lines may hash to the same hash value, we could be omitting lines that aren't duplicates.
I'm not aware of specific places where we have gotten the wrong data, but I thought I would open this issue to document the possible issue.
Doing this properly would use a lot more memory. However, proper deduplication could be done outside of rotoscope using the simple awk script !seen[$0]++
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.