Accuracy of heap profile about async-profiler

[jrudolph]

In this particular case it seems to be a "sibling" method that is actually doing the allocations:

Counter: 9428816 (97.81%)
  [ 0] [Ljava/lang/Object;
  [ 1] akka.http.scaladsl.model.Uri$.create
  [ 2] akka.http.impl.model.parser.UriParser.parseHttpRequestTarget
  [ 3] akka.http.scaladsl.model.Uri$.parseHttpRequestTarget
  [ 4] akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.parseRequestTarget
  [ 5] akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.parseMessage
  [ 6] akka.http.impl.engine.parsing.HttpMessageParser$class.startNewMessage
  [ 7] akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.startNewMessage
  [ 8] akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$startNewMessage$1.apply

Using VisualVM I found out that it's not create but another call chain starting from UriParser.parseHttpRequestTarget which is actually doing the allocations.

[apangin]

This might be a bug of AsyncGetCallTrace. Does -XX:+DebugNonSafepoints make it better?
Do you have a small reproducer that you can share?

[retronym]

I've also run into seemingly impossible stack traces when CPU profiling the Scala compiler with flight recorder or async-profiler. For a while, I thought I was going crazy! It's comforting to know that the flaw might be elsewhere.

I've just trawled some open bugs on AsyncGetCallTrace. JDK-8022893 is a good one with a reproduction and has analysis of the root cause in the comments. It is plausibly the same problem as reported here.

[jrudolph]

@apangin in my case with -XX:+DebugNonSafepoints it is missing intermediate frames:

  [ 0] [Ljava/lang/Object;
  [ 1] akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.parseRequestTarget
  [ 2] akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.parseMessage
  [ 3] akka.http.impl.engine.parsing.HttpMessageParser$class.startNewMessage
  [ 4] akka.http.impl.engine.parsing.HttpRequestParser$$anon$1.startNewMessage
  [ 5] akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$startNewMessage$1.apply
  [ 6] akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$startNewMessage$1.apply
  [ 7] akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$continue$2.apply
  [ 8] akka.http.impl.engine.parsing.HttpMessageParser$$anonfun$continue$2.apply
  [ 9] akka.http.impl.engine.parsing.HttpMessageParser$class.run$1

while it should have three more frames on top of HttpRequestParser$$anon$1.parseRequestTarget.

I don't have a small reproducer right now, I guess I could build a bigger one sharing a ~30MB jar file and instructions.

[apangin]

OK, looks like I can reproduce the problem. Will investigate this.

[apangin]

After some investigation I can confirm that it is AsyncGetCallTraces that works wrong.
If I replace AGCT with the standard JVM TI GetStackTrace call, everything looks good. Fortunately, it is safe to do so for allocation profiler (but not for CPU profiler).

I think the right fix would be to check if current thread has _last_Java_sp set, and, if yes, call JVM TI GetStackTrace(), otherwise call AsyncGetCallTrace.

[jrudolph]

Might make sense to compare results with the output generated by grav: https://epickrram.blogspot.de/2017/09/heap-allocation-flamegraphs.html

[apangin]

Re-implemented Java stack traces with JVM TI for allocation profiling. This is going to be accurate now.

[apangin]

Unfortunately, calling JVM TI from VM context may sometimes trigger assertion in VM code (#64), so reverted back to AsyncGetCallTrace for now. Better inaccurate than unsafe.
Will fix that properly in #66.

[apangin]

Re-implemented allocation stack walking once again. Now the stacks should be accurate without a risk of internal JVM deadlocks or assertion failures.