timbunce / devel-nytprof Goto Github PK

==6497== Invalid read of size 1
==6497==    at 0x34C1FF: _platform_memcmp (in /usr/lib/system/libsystem_platform.dylib)
==6497==    by 0x1000CF588: Perl_hv_common (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x1000E1617: Perl_pp_helem (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x1000ACBA9: Perl_runops_debug (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x10002BD28: Perl_call_sv (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x73239D: load_perl_callback (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/darwin-thread-multi-2level/Devel/auto/Devel/NYTProf/NYTProf.bundle)
==6497==    by 0x72CCD2: load_profile_data_from_stream (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/darwin-thread-multi-2level/Devel/auto/Devel/NYTProf/NYTProf.bundle)
==6497==    by 0x72C358: XS_Devel__NYTProf__Data_load_profile_data_from_file (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/darwin-thread-multi-2level/Devel/auto/Devel/NYTProf/NYTProf.bundle)
==6497==    by 0x1000E5685: Perl_pp_entersub (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x1000ACBA9: Perl_runops_debug (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x10002B294: perl_run (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x10000110A: main (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==  Address 0x101097f2d is 0 bytes after a block of size 93 alloc'd
==6497==    at 0x70DB: malloc (in /usr/local/Cellar/valgrind/3.9.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==6497==    by 0x1000AD20B: Perl_safesysmalloc (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x1000CE45A: S_share_hek_flags (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x1000CF6F1: Perl_hv_common (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x1000E1617: Perl_pp_helem (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x1000ACBA9: Perl_runops_debug (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x10002BD28: Perl_call_sv (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x73239D: load_perl_callback (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/darwin-thread-multi-2level/Devel/auto/Devel/NYTProf/NYTProf.bundle)
==6497==    by 0x72CCD2: load_profile_data_from_stream (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/darwin-thread-multi-2level/Devel/auto/Devel/NYTProf/NYTProf.bundle)
==6497==    by 0x72C358: XS_Devel__NYTProf__Data_load_profile_data_from_file (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/darwin-thread-multi-2level/Devel/auto/Devel/NYTProf/NYTProf.bundle)
==6497==    by 0x1000E5685: Perl_pp_entersub (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)
==6497==    by 0x1000ACBA9: Perl_runops_debug (in /Users/mhall/perl5/perlbrew/perls/perl-5.18.2/bin/perl)

Summary of my perl5 (revision 5 version 21 subversion 8) configuration:

  Platform:
    osname=darwin, osvers=14.0.0, archname=darwin-2level
    uname='darwin lees-macbook-air.g3s.internal 14.0.0 darwin kernel version 14.0.0: fri sep 19 00:26:44 pdt 2014; root:xnu-2782.1.97~2release_x86_64 x86_64 '
    config_args='-de -Dprefix=/Users/leejohnson/perl5/perlbrew/perls/perl-5.21.8 -Dusedevel -Aeval:scriptdir=/Users/leejohnson/perl5/perlbrew/perls/perl-5.21.8/bin'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fno-common -DPERL_DARWIN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -I/opt/local/include',
    optimize='-O3',
    cppflags='-fno-common -DPERL_DARWIN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -I/opt/local/include'
    ccversion='', gccversion='4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags =' -fstack-protector -L/usr/local/lib -L/opt/local/lib'
    libpth=/usr/local/lib /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib /usr/lib /opt/local/lib
    libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
    perllibs=-lpthread -ldl -lm -lutil -lc
    libc=, so=dylib, useshrplib=false, libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup -L/usr/local/lib -L/opt/local/lib -fstack-protector'


Characteristics of this binary (from libperl):
  Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
                        PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
                        PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
                        PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
                        USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
                        USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME
                        USE_PERLIO USE_PERL_ATOF
  Built under darwin
  Compiled at Jan 21 2015 11:09:03
  %ENV:
    PERLBREW_BASHRC_VERSION="0.69"
    PERLBREW_HOME="/Users/leejohnson/.perlbrew"
    PERLBREW_MANPATH="/Users/leejohnson/perl5/perlbrew/perls/perl-5.21.8/man"
    PERLBREW_PATH="/Users/leejohnson/perl5/perlbrew/bin:/Users/leejohnson/perl5/perlbrew/perls/perl-5.21.8/bin"
    PERLBREW_PERL="perl-5.21.8"
    PERLBREW_ROOT="/Users/leejohnson/perl5/perlbrew"
    PERLBREW_VERSION="0.69"
  @INC:
    /Users/leejohnson/perl5/perlbrew/perls/perl-5.21.8/lib/site_perl/5.21.8/darwin-2level
    /Users/leejohnson/perl5/perlbrew/perls/perl-5.21.8/lib/site_perl/5.21.8
    /Users/leejohnson/perl5/perlbrew/perls/perl-5.21.8/lib/5.21.8/darwin-2level
    /Users/leejohnson/perl5/perlbrew/perls/perl-5.21.8/lib/5.21.8
    .

Devel::NYTProf (which I love, keep up the good work!) just sent me off
optimizing what turned out to be profiling overhead error :-/ -- I've
boiled down the behavior I was bitten by to a simple test case - it
appears that regular expressions perform very significantly slower with
Devel::NYTProf enabled. All versions were tested on a 64-bit ubuntu
10.04 (perl 5.10.1) install. 4.04 was also tested on a 64-bit debian
etch (perl 5.8.8) install.

ski@portege:~$ time perl -le 'my @x = (0..100_000); @x = grep { rand() <
1/2 } @x' 

real    0m0.076s
user    0m0.050s
sys 0m0.030s
ski@portege:~$ time perl -d:NYTProf -le 'my @x = (0..100_000); @x = grep
{ rand() < 1/2 } @x' 

real    0m0.108s
user    0m0.070s
sys 0m0.020s
ski@portege:~$ 
ski@portege:~$ time perl -le 'my @x = (0..100_000); @x = grep { /234/ } @x' 

real    0m0.197s
user    0m0.110s
sys 0m0.040s
ski@portege:~$ time NYTPROF=slowops=2 perl -d:NYTProf -le 'my @x =
(0..100_000); @x = grep { /234/ } @x' #default

real    0m1.744s
user    0m1.530s
sys 0m0.040s
ski@portege:~$ time NYTPROF=slowops=0 perl -d:NYTProf -le 'my @x =
(0..100_000); @x = grep { /234/ } @x' #default

real    0m0.801s
user    0m0.720s
sys 0m0.020s

Why is the slowdown large? I noticed that even using qr//, regular
expressions seem to get re-CORE::regcomp'd every run (although this may
be intended behavior, it's certainly counter-intuitive) - perhaps there
is a regular expression in the hooked-in code that deletes the
optimizations that perl does for when the last regular expression is
repeated?

Any ideas?

Thanks again!
Brian Szymanski

> Why is the slowdown large?

Because the profiler has to do a lot of work on each entry and return from a subroutine or 'slowop'.

That cost gets bigger, relatively, the fast the opcode is. A trivial regex on a short string is very fast 
indeed so the slow down appears large. It's still only about 8 micro seconds! (In the slowops=2 vs 
=0 case).

I'll add a note to the docs.

> I noticed that even using qr//, regular expressions seem to get re-CORE::regcomp'd every run

I don't see that in the examples you've given above.
perl -MO=Concise,-exec -e 'warn scalar grep { /12/ } (0..100);'
shows a 'match' opcode being generated.

If you mean code like
perl -MO=Concise -e '$q=qr/12/; warn scalar grep { /$q/ } (0..100);'
then yes, regcomp is called each time but has little to do (it just takes a temp copy of the regex).

I hope that helps. Sorry for any disappointment.

Tim.

On Mon Oct 11 16:39:12 2010, TIMB wrote:
> > Why is the slowdown large?
> 
> Because the profiler has to do a lot of work on each entry and return
> from a subroutine or 'slowop'.
> 
> That cost gets bigger, relatively, the fast the opcode is. A trivial
> regex on a short string is very fast
> indeed so the slow down appears large. It's still only about 8 micro
> seconds! (In the slowops=2 vs
> =0 case).
> 
> I'll add a note to the docs.

Well, NYTProf 1.x only slows the regular-expression-y code down by a
factor of 3, whereas in NYTProf 3/4.x the overhead is closer to a factor
of 5 even with slowops=0.

I understand why the overhead of tracking subroutines has to be high,
but (and perhaps this is my lack of knowledge of perl internals), I
don't understand why regexes would have to be so slow.

Perhaps there should be an option for "proportional overhead" ? IE do
all the (extraneous) work we would do for a slow opcode even for fast
opcodes that we would otherwise skip lots of unnecessary bookkeeping for?

On Tue, Oct 12, 2010 at 11:10:05PM -0400, Brian Szymanski via RT wrote:
> 
> Well, NYTProf 1.x only slows the regular-expression-y code down by a
> factor of 3, whereas in NYTProf 3/4.x the overhead is closer to a factor
> of 5 even with slowops=0.
> 
> I understand why the overhead of tracking subroutines has to be high,
> but (and perhaps this is my lack of knowledge of perl internals), I
> don't understand why regexes would have to be so slow.

With slowops=0 NYTProf is not involved with regexes at all.

Any difference in performance you're seeing between NYTProf versions
is due to other changes, such as additional logic when leaving a block
(see leave=0 in the docs) and more detail/accuracy in general.

Tim.

On Wed Oct 13 07:56:46 2010, [email protected] wrote:
> On Tue, Oct 12, 2010 at 11:10:05PM -0400, Brian Szymanski via RT wrote:
> > 
> > Well, NYTProf 1.x only slows the regular-expression-y code down by a
> > factor of 3, whereas in NYTProf 3/4.x the overhead is closer to a factor
> > of 5 even with slowops=0.
> > 
> > I understand why the overhead of tracking subroutines has to be high,
> > but (and perhaps this is my lack of knowledge of perl internals), I
> > don't understand why regexes would have to be so slow.
> 
> With slowops=0 NYTProf is not involved with regexes at all.

I'm not sure I understand what you are saying here... Something is
definitely affecting regex performance in ways many orders of magnitude
worse than DProf:

ski@editorial1:~$ time perl -le 'my @x = (0..1_000_000); @x = grep {
rand() < 1/2 } @x' #non-regexp

real    0m0.453s
user    0m0.404s
sys 0m0.052s
ski@editorial1:~$ time perl  -d:DProf -le 'my @x = (0..1_000_000); @x =
grep { rand() < 1/2 } @x' #non-regexp, dprof

real    0m0.489s
user    0m0.424s
sys 0m0.064s
ski@editorial1:~$ time perl  -d:NYTProf -le 'my @x = (0..1_000_000); @x
= grep { rand() < 1/2 } @x' #non-regexp, nytprof

real    0m0.462s
user    0m0.376s
sys 0m0.088s
ski@editorial1:~$ time perl -le 'my @x = (0..1_000_000); @x = grep {
/123/ } @x' #regexp

real    0m0.950s
user    0m0.764s
sys 0m0.192s
ski@editorial1:~$ time perl  -d:DProf -le 'my @x = (0..1_000_000); @x =
grep { /123/ } @x' #regexp, dprof

real    0m1.016s
user    0m0.828s
sys 0m0.176s
ski@editorial1:~$ time perl  -d:NYTProf -le 'my @x = (0..1_000_000); @x
= grep { /123/ } @x' #regexp, nytprof

real    0m8.251s
user    0m4.412s
sys 0m3.836s

On Wed, Oct 13, 2010 at 08:01:13PM -0400, Brian Szymanski via RT wrote:
> > 
> > With slowops=0 NYTProf is not involved with regexes at all.
> 
> I'm not sure I understand what you are saying here...

I'm simply saying that with slowops enabled (as it is by default)
NYTProf specifically intercepts the regex opcodes and so adds extra
overhead to every regex. With slowops=0 it doesn't do that:

$ NYTPROF=slowops=1 time perl  -d:NYTProf -le 'my @x = (0..1_000_000); @x = grep { /123/ } @x'
user        16.69
$ NYTPROF=slowops=0 time perl  -d:NYTProf -le 'my @x = (0..1_000_000); @x = grep { /123/ } @x'
user         7.76

Of course any profiling adds overheads. With slowops=0 the remaining
profiling overheads come mostly from the *statement* profiler.

> Something is definitely affecting regex performance in ways many
> orders of magnitude worse than DProf:

Naturally, because DProf is only a *subroutine* profiler but there are
no sub calls, so it's hardly involved in profiling that code at all.
(DProf is also so badly broken it's not worth using.)

You can achieve a similar effect with NYTProf by turning off the
statement profiler as well as slowops:

$ NYTPROF=slowops=0:stmts=0 time perl -d:NYTProf -le 'my @x = (0..1_000_000); @x = grep { /123/ } @x'
user         4.06

Then, for this example, NYTProf is almost as fast as not profiling at all:

$ time perl -le 'my @x = (0..1_000_000); @x = grep { /123/ } @x'
user         4.04

Tim.

p.s. I presume you've read
http://search.cpan.org/~timb/Devel-NYTProf/lib/Devel/NYTProf.pm#MAKING_NYTPROF_FASTER
If you have any suggestions for  clarifying the docs please let me know.

I wanted to profile an application suite running Apache 2 with mod_perl2
so I enabled Devel::NYTProf::Apache in my httpd.conf.

    PerlSetEnv NYTPROF trace=2:file=/tmp/mm-nytprof.out
    PerlModule Devel::NYTProf::Apache

I was seeing issues where "NYTProf is confused about CV" multiple times
for each thing that I did in my app. If I try to convert the out files
to html, it is very clear that something bad happened. (The top 15
subroutines doesn't get populated, the subroutine exclusive time treemap
is empty, etc.)

I bumped it up to trace=4 and captured one instance of it occurring
(attached as trace4.txt). I'm not sure what other information you need
from me
to assist you.

I have verified that when trace=0, the problem does not happen and the
resulting html output is as pretty as usual.

On Thu, Jan 27, 2011 at 05:57:46PM -0500, Matthew Musgrove via RT wrote:
> NYTProf is confused about CV 0x2ab2388a0568 called as /opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm at /opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm line 225 (please report as a bug)

Thanks for the report. Please try this change just to give me more infomation:

--- NYTProf.xs  (revision 1407)
+++ NYTProf.xs  (working copy)
@@ -2639,8 +2639,10 @@
                 logwarn("NYTProf is confused about CV %p called as %s at %s line %d (please report as a bug)\n",
                     (void*)called_cv, SvPV_nolen(sub_sv), OutCopFILE(prev_cop), (int)CopLINE(prev_cop));
                 /* looks like Class::MOP doesn't give the CV GV stash a name */
-                if (trace_level >= 2)
+                if (trace_level >= 2) {
                     sv_dump((SV*)called_cv); /* coredumps in Perl_do_gvgv_dump, looks line GvXPVGV is false, presumably on a Class::MOP wierdo sub */
+                    sv_dump((SV*)gv);
+                }
             }
         }

Tim.

Thanks Tim. Here's the latest dump from running at trace=2.

NYTProf is confused about CV 0x2b41eac6afb8 called as
/opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm
at
/opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm
line 225 (please report as a bug)
SV = PVCV(0x2b41eac6fa40) at 0x2b41eac6afb8
  REFCNT = 2
  FLAGS = (POK,pPOK,CONST)
  PROTOTYPE = ""
  COMP_STASH = 0x2b41eabb46f8   "ModPerl::RegistryCooker"
  XSUB = 0x2b41dc1665b0
  XSUBANY = 0x2b41eac41308 (CONST SV)
    SV = IV(0x2b41eac41300) at 0x2b41eac41308
      REFCNT = 2
      FLAGS = (PADMY,IOK,READONLY,pIOK)
      IV = 0
  GVGV::GV = 0x2b41eac41518     ""
  FILE =
"/opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm"
  DEPTH = 0
  FLAGS = 0xc00
  OUTSIDE_SEQ = 0
  PADLIST = 0x0
  OUTSIDE = 0x0 (null)
SV = IV(0x2b41eac41510) at 0x2b41eac41518
  REFCNT = 1
  FLAGS = (ROK)
  RV = 0x2b41eac41308
    SV = IV(0x2b41eac41300) at 0x2b41eac41308
      REFCNT = 2
      FLAGS = (PADMY,IOK,READONLY,pIOK)
      IV = 0

On Fri, Jan 28, 2011 at 11:08:31AM -0500, Matthew Musgrove via RT wrote:
>        Queue: devel-nytprof
>  Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=65200 >
> 
> Thanks Tim. Here's the latest dump from running at trace=2.
> 
> SV = IV(0x2b41eac41510) at 0x2b41eac41518
>   REFCNT = 1
>   FLAGS = (ROK)
>   RV = 0x2b41eac41308
>     SV = IV(0x2b41eac41300) at 0x2b41eac41308
>       REFCNT = 2
>       FLAGS = (PADMY,IOK,READONLY,pIOK)
>       IV = 0

Great. Now after the

    sv_dump((SV*)gv);   

add

    sv_dump((SV*)SvRV(gv));   

and re-run.

Thanks.

Tim.

NYTProf is confused about CV 0x2b397579fdd0 called as
/opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm
at
/opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm
line 225 (please report as a bug)
SV = PVCV(0x2b39757a4690) at 0x2b397579fdd0
  REFCNT = 2
  FLAGS = (POK,pPOK,CONST)
  PROTOTYPE = ""
  COMP_STASH = 0x2b39756e95a0   "ModPerl::RegistryCooker"
  XSUB = 0x2b39674715b0
  XSUBANY = 0x2b3975776280 (CONST SV)
    SV = IV(0x2b3975776278) at 0x2b3975776280
      REFCNT = 2
      FLAGS = (PADMY,IOK,READONLY,pIOK)
      IV = 0
  GVGV::GV = 0x2b3975776490     ""
  FILE =
"/opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm"
  DEPTH = 0
  FLAGS = 0xc00
  OUTSIDE_SEQ = 0
  PADLIST = 0x0
  OUTSIDE = 0x0 (null)
SV = IV(0x2b3975776488) at 0x2b3975776490
  REFCNT = 1
  FLAGS = (ROK)
  RV = 0x2b3975776280
    SV = IV(0x2b3975776278) at 0x2b3975776280
      REFCNT = 2
      FLAGS = (PADMY,IOK,READONLY,pIOK)
      IV = 0
SV = IV(0x2b3975776278) at 0x2b3975776280
  REFCNT = 2
  FLAGS = (PADMY,IOK,READONLY,pIOK)
  IV = 0

Hi Matthew.

> NYTProf is confused about CV 0x2ab2388a0568 called as /opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm at /opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm line 225 (please report as a bug)

Please hack RegistryCooker.pm to log the class of the object that
should_reset_inc_hash() is called on. Thanks.

Tim.

Howdy Tim,

I've never peered inside ModPerl::RegistryCooker before. I used
$self->log_error("should_reset_inc_hash $package"). I hope that was what
you were asking for. 

[Tue Feb 01 22:23:44 2011] [error] should_reset_inc_hash
ModPerl::ROOT::ModPerl::Registry::view_mm_2dnms_vobs_NMS_nms_www_cgi_2dbin_security_login_2ecgi
Marking 'Apache2::RequestRec::log_error' as xsub
Marking 'Apache2::RequestRec::notes' as xsub
Marking 'APR::Table::set' as xsub
NYTProf is confused about CV 0x2b7915c79368 called as
/opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm
at
/opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm
line 226 (please report as a bug)
SV = PVCV(0x2b7915c7ddf0) at 0x2b7915c79368
  REFCNT = 2
  FLAGS = (POK,pPOK,CONST)
  PROTOTYPE = ""
  COMP_STASH = 0x2b7915bc2850   "ModPerl::RegistryCooker"
  XSUB = 0x2b791bfc25b0
  XSUBANY = 0x2b7915c4f810 (CONST SV)
    SV = IV(0x2b7915c4f808) at 0x2b7915c4f810
      REFCNT = 2
      FLAGS = (PADMY,IOK,READONLY,pIOK)
      IV = 0
  GVGV::GV = 0x2b7915c4fa20     ""
  FILE =
"/opt/perl/lib/site_perl/5.12.3/x86_64-linux-thread-multi/ModPerl/RegistryCooker.pm"
  DEPTH = 0
  FLAGS = 0xc00
  OUTSIDE_SEQ = 0
  PADLIST = 0x0
  OUTSIDE = 0x0 (null)
SV = IV(0x2b7915c4fa18) at 0x2b7915c4fa20
  REFCNT = 1
  FLAGS = (ROK)
  RV = 0x2b7915c4f810
    SV = IV(0x2b7915c4f808) at 0x2b7915c4f810
      REFCNT = 2
      FLAGS = (PADMY,IOK,READONLY,pIOK)
      IV = 0


It looks like my login script is the package in question. I hope that I
haven't missed anything in my conversion from CGI to mod_perl to waste
your time.

I can reproduce in perl 5.14 but not 5.16 (or 5.17) so I'm inclined to consider it a perl bug.
I'll dig a little deeper anyway to try to identify the cause and see if a workaround is possible.

When I've captured data with stmts=1, I get a warning when examining it
with Devel::NYTProf::Data:

    $ perl -MDevel::NYTProf::Data -lwe 'Devel::NYTProf::Data->new({filename=>"/tmp/nytprof.t10.out.30294"})' 
    Reading /tmp/nytprof.t10.out.30294

    Eval '(eval 2)' (fid 9, flags:viastmt,savesrc) has unknown invoking fid 10
    Processing /tmp/nytprof.t10.out.30294 data

Don't get this warning when capturing similar activity with stmts=0:

    $ perl -MDevel::NYTProf::Data -lwe 'Devel::NYTProf::Data->new({filename=>"/tmp/nytprof.t6.out.26875"})' 
    Reading /tmp/nytprof.t6.out.26875

    Processing /tmp/nytprof.t6.out.26875 data

-zefram

This issue is documented in the TROUBLESHOOTING section of the docs.
I'l leave this case open as there's a possible fix I'd like to try at some point.

Greetings,

We have just tried to profile our application that uses Coro package and got an error:

NYTProf panic: stack is confused, giving up!
Segmentation fault

I understand that it's not so easy to add support of another low level implementation of threads, but still hope it's possible. Thank you in advance.

Cheers
Eugene.

On Wed, Sep 01, 2010 at 03:15:54PM -0400, Eugene Toropov via RT wrote:
> Transaction: Ticket created by [email protected]
>  Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=60950 >
> 
> Greetings,
> 
> We have just tried to profile our application that uses Coro package and got an error:
> NYTProf panic: stack is confused, giving up!
> Segmentation fault

I'm not surprised.

> I understand that it's not so easy to add support of another low level
> implementation of threads, but still hope it's possible. Thank you in
> advance.

Realistically it's not going to happen anytime soon, and not without
much help from the Coro developers. Your best bet is to submit a bug
report to Coro asking for them to work with NYTProf devs.

Tim.

Hi Tim,

I've had a discussion on this subject with Marc Lehmann and here is what he 
wrote to me:

<schmorp:#anyevent> well, all they need to do is use the current thread as 
extra index into their data structures
<schmorp:#anyevent> $NYTPROFDATA{$Coro::current+0} for example

Cheers
Eugene


> <URL: http://rt.cpan.org/Ticket/Display.html?id=60950 >
>
> On Wed, Sep 01, 2010 at 03:15:54PM -0400, Eugene Toropov via RT wrote:
>> Transaction: Ticket created by [email protected]
>>  Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=60950 >
>>
>> Greetings,
>>
>> We have just tried to profile our application that uses Coro package and 
>> got an error:
>> NYTProf panic: stack is confused, giving up!
>> Segmentation fault
>
> I'm not surprised.
>
>> I understand that it's not so easy to add support of another low level
>> implementation of threads, but still hope it's possible. Thank you in
>> advance.
>
> Realistically it's not going to happen anytime soon, and not without
> much help from the Coro developers. Your best bet is to submit a bug
> report to Coro asking for them to work with NYTProf devs.
>
> Tim.
> 

On Tue, Sep 07, 2010 at 04:19:43AM -0400, Eugene Toropov via RT wrote:
> 
> I've had a discussion on this subject with Marc Lehmann and here is what he 
> wrote to me:
> 
> <schmorp:#anyevent> well, all they need to do is use the current thread as 
> extra index into their data structures
> <schmorp:#anyevent> $NYTPROFDATA{$Coro::current+0} for example

So essentially we need to support threading first. That's unlikely to
get done by me anytime soon, but it would be reasonably straightforward
for someone else to do. Volunteers welcome!

Tim.

On Thu, Sep 23, 2010 at 05:12:06PM +0100, Roger Picton wrote:
>    Tim:

Hello Roger.

>    Excuse me emailing you directly but I haven't got a login id for the CPAN bug system (and I'm not sure
>    whether I'd be allowed to add a comment to this bug even if I had!).

Once logged in you can comment on any ticket, and add yourself as a CC
on any ticket.

>    I found the same problem when trying to profile a program which uses Coro.  Since the error message said
>    "stack is confused" and coroutines tinker with the stack I thought I'd try switching off subroutine
>    profiling (subs=0 in the NYTPROF environment variable).  When I did that the profiler worked fine.

That's useful to know. I'll add a note to the docs.

>    I
>    obviously only got line and block level results in that case, but that was certainly good enough to help
>    me identify the hot spots.  On the other hand, I presume this ploy wouldn't work for multiple threads.

Per the docs at
http://search.cpan.org/~timb/Devel-NYTProf-4.05/lib/Devel/NYTProf.pm#Threads_and_Multiplicity
    "Meanwhile, profiling is disabled when a thread is created, and NYTProf
    tries to ignore any activity from perl interpreters other than the first
    one that loaded it."
So, no. Sorry.

>    For reference I was using ActivePerl 5.12.2.1202 (which includes Devel::NYTProf 4.04) and Coro 5.23 on
>    Windows Vista Home Premium SP2.
> 
>    Thanks for taking on this excellent profiler!

You're welcome.

I'd be very supportive of anyone wanting to make NYTProf thread/coro safe.
It should be a relatively straightforward and typical process, i.e.,
create a struct, pass it around, start moving the globals into the struct.
That can be an incremental process, each in a separate commit, with the
test suite kept passing. Once that work is done, actual thread safety
shouldn't be hard to add as a final step.

I'm looking for a volunter!

Tim.

I've created an open quest on Play Perl for this: http://play-perl.org/quest/5164001d0d7965996100005a
(This is the most important feature missing from Devel::NYTProf for me and our team, but I don't have time to do it myself.)

Let's see if anyone picks it up.

nytprofhtml -f nytprof.out.15306 
Reading nytprof.out.15306
Processing nytprof.out.15306 data
Writing line reports to nytprof directory
 100% ... 
Extracting subroutine call data ...
Extracting subroutine links
Generating subroutine stack flame graph ...
Can't resolve fid of '0' at /bin/nytprofhtml line 1381

  FcgidCmdOptions /path/to/program.pl \
     InitialEnv PERL5OPT=-d:NYTProf \
     InitialEnv NYTPROF=file=/tmp/nytprof-main.out:addpid=1:start=no \
     MaxProcesses 1

The flame graphs (very useful btw.) already ignore the time spent in DB::finish_profile,
but the exclusive time treemap includes it, which isn't very useful. Ignoring it in there
would give more useful percentages.

$ prove t/test25-strevalb.t
t/test25-strevalb.t .. 1/145 Assertion failed: ((kid->op_type == OP_NULL && kid->op_targ == OP_NEXTSTATE) || kid->op_type == OP_STUB || kid->op_type == OP_ENTER), function Perl_rpeep, file op.c, line 12011.
Exit status 6 from /usr/local/bin/perl5.20.0 -d:NYTProf at t/test25-strevalb.t line 23.

$ NYTPROF='use_db_sub=1' perl -d:NYTProf -e 'use Benchmark'
Assertion failed: ((kid->op_type == OP_NULL && kid->op_targ == OP_NEXTSTATE) || kid->op_type == OP_STUB || kid->op_type == OP_ENTER), function Perl_rpeep, file op.c, line 12011.
Abort trap

$ NYTPROF='use_db_sub=1' perl5.21.4 -d:NYTProf -e 'use Benchmark'
Assertion failed: ((kid->op_type == OP_NULL && kid->op_targ == OP_NEXTSTATE) || kid->op_type == OP_STUB || kid->op_type == OP_ENTER), function Perl_rpeep, file op.c, line 11775.
Abort trap

When I tried to install 4.06 running cpan under 5.14.1 on Mac OS X.7.2, I got odd test failures. 
Building by hand I had no problem. This isn't a problem for me, but I'll just note that it 
happened in case someone else has the same problem.



$ cpan5.14.1 Devel::NYTProf
CPAN: File::HomeDir loaded ok (v0.98)
CPAN: Storable loaded ok (v2.27)
Going to read '/Users/brian/.cpan/Metadata'
  Database was generated on Sat, 19 Nov 2011 13:27:17 GMT
CPAN: LWP::UserAgent loaded ok (v6.02)
CPAN: URI::URL loaded ok (v5.03)
CPAN: YAML loaded ok (v0.77)
Going to read '/Volumes/Atlas/CPAN/authors/01mailrc.txt.gz'
CPAN: Compress::Zlib loaded ok (v2.033)
Going to read 127 yaml files from /Users/brian/.cpan/build/
...........................................................................DONE
Restored the state of none (in 1.9073 secs)
............................................................................DONE
Going to read '/Volumes/Atlas/CPAN/modules/02packages.details.txt.gz'
  Database was generated on Mon, 28 Nov 2011 13:29:31 GMT
...............
  New CPAN.pm version (v1.9800) available.
  [Currently running version is v1.9600]
  You might want to try
    install CPAN
    reload cpan
  to both upgrade CPAN.pm and run the new version without leaving
  the current session.


.............................................................DONE
Going to read '/Volumes/Atlas/CPAN/modules/03modlist.data.gz'
............................................................................DONE
Going to write /Users/brian/.cpan/Metadata
Running install for module 'Devel::NYTProf'
Running make for T/TI/TIMB/Devel-NYTProf-4.06.tar.gz
  Has already been unwrapped into directory /Users/brian/.cpan/build/Devel-NYTProf-4.06-
TKXkNF
CPAN: Parse::CPAN::Meta loaded ok (v1.4401)
CPAN: CPAN::Meta loaded ok (v2.110440)
CPAN: Module::CoreList loaded ok (v2.49_01)
  Has already been made
Running make test
cp blib/arch/auto/Devel/NYTProf/NYTProf.bundle 
blib/lib/Devel/auto/Devel/NYTProf/NYTProf.bundle
PERL_DL_NONLAZY=1 /usr/local/perls/perl-5.14.1/bin/perl "-MExtUtils::Command::MM" "-e" 
"test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-load.t ............ 1/2 
#   Failed test 'use Devel::NYTProf::Core;'
#   at t/00-load.t line 5.
#     Tried to use 'Devel::NYTProf::Core'.
#     Error:  Can't load '/Users/brian/.cpan/build/Devel-NYTProf-4.06-
TKXkNF/blib/lib/Devel/auto/Devel/NYTProf/NYTProf.bundle' for module Devel::NYTProf: 
dlopen(/Users/brian/.cpan/build/Devel-NYTProf-4.06-
TKXkNF/blib/lib/Devel/auto/Devel/NYTProf/NYTProf.bundle, 2): Symbol not found: _deflate
#   Referenced from: /Users/brian/.cpan/build/Devel-NYTProf-4.06-
TKXkNF/blib/lib/Devel/auto/Devel/NYTProf/NYTProf.bundle
#   Expected in: dynamic lookup
#  at /Users/brian/.cpan/build/Devel-NYTProf-4.06-TKXkNF/blib/lib/Devel/NYTProf/Core.pm 
line 19
# Compilation failed in require at (eval 4) line 2.
# BEGIN failed--compilation aborted at (eval 4) line 2.
# Testing Devel::NYTProf 4.06 started at Mon Nov 28 15:01:37 2011

#   Failed test 'use Devel::NYTProf::Constants;'
#   at t/00-load.t line 11.
#     Tried to use 'Devel::NYTProf::Constants'.
#     Error:  Attempt to reload Devel/NYTProf/Core.pm aborted.
# Compilation failed in require at /Users/brian/.cpan/build/Devel-NYTProf-4.06-
TKXkNF/blib/lib/Devel/NYTProf/Constants.pm line 5.
# BEGIN failed--compilation aborted at /Users/brian/.cpan/build/Devel-NYTProf-4.06-
TKXkNF/blib/lib/Devel/NYTProf/Constants.pm line 5.
# Compilation failed in require at (eval 5) line 2.
# BEGIN failed--compilation aborted at (eval 5) line 2.
Undefined subroutine &main::NYTP_DEFAULT_COMPRESSION called at t/00-load.t line 15.
# Looks like you failed 2 tests of 2.
# Looks like your test exited with 255 just after 2.
t/00-load.t ............ Dubious, test returned 255 (wstat 65280, 0xff00)

I'm sorry for the long delay in replying. Is this still a problem?

On Mon Jan 02 15:58:59 2012, TIMB wrote:
> I'm sorry for the long delay in replying. Is this still a problem?

With Perl 5.14.2 and Devel::NYTProf 4.06, it's still a problem. I think this is an odd interaction 
with CPAN.pm. I don't see the problem when I compile by hand or use cpanp. 

cschwenz@cochin:~/local/src/perl-modules/Devel-NYTProf-5.06$ ~/local/perl-5.20.1/bin/perl Makefile.PL
Looking for header files and functions...
Found clock_gettime in time.h
Found deflateInit2 in zlib.h
Your perl was compiled with gcc (version 4.4.3), okay.
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Devel::NYTProf
Writing MYMETA.yml and MYMETA.json
cschwenz@cochin:~/local/src/perl-modules/Devel-NYTProf-5.06$ make
cp lib/Devel/NYTProf/Constants.pm blib/lib/Devel/NYTProf/Constants.pm
cp lib/Devel/NYTProf/js/jit/gradient50.png blib/lib/Devel/NYTProf/js/jit/gradient50.png
cp lib/Devel/NYTProf/js/desc.png blib/lib/Devel/NYTProf/js/desc.png
cp lib/Devel/NYTProf/js/jquery-min.js blib/lib/Devel/NYTProf/js/jquery-min.js
cp lib/Devel/NYTProf/js/jquery-tablesorter-min.js blib/lib/Devel/NYTProf/js/jquery-tablesorter-min.js
cp lib/Devel/NYTProf/SubCallInfo.pm blib/lib/Devel/NYTProf/SubCallInfo.pm
cp lib/Devel/NYTProf/Test.pm blib/lib/Devel/NYTProf/Test.pm
cp lib/Devel/NYTProf/js/jit/gradient.png blib/lib/Devel/NYTProf/js/jit/gradient.png
cp lib/Devel/NYTProf/FileInfo.pm blib/lib/Devel/NYTProf/FileInfo.pm
cp lib/Devel/NYTProf/js/jit/gradient30.png blib/lib/Devel/NYTProf/js/jit/gradient30.png
cp lib/Devel/NYTProf/Apache.pm blib/lib/Devel/NYTProf/Apache.pm
cp lib/Devel/NYTProf/js/jit/Treemap.css blib/lib/Devel/NYTProf/js/jit/Treemap.css
cp lib/Devel/NYTProf/Util.pm blib/lib/Devel/NYTProf/Util.pm
cp lib/Devel/NYTProf/ReadStream.pm blib/lib/Devel/NYTProf/ReadStream.pm
cp lib/Devel/NYTProf/FileHandle.pm blib/lib/Devel/NYTProf/FileHandle.pm
cp lib/Devel/NYTProf/js/jit/jit.js blib/lib/Devel/NYTProf/js/jit/jit.js
cp lib/Devel/NYTProf/Run.pm blib/lib/Devel/NYTProf/Run.pm
cp lib/Devel/NYTProf/Core.pm blib/lib/Devel/NYTProf/Core.pm
cp lib/Devel/NYTProf/Data.pm blib/lib/Devel/NYTProf/Data.pm
cp lib/Devel/NYTProf/SubInfo.pm blib/lib/Devel/NYTProf/SubInfo.pm
cp lib/Devel/NYTProf/Reader.pm blib/lib/Devel/NYTProf/Reader.pm
cp lib/Devel/NYTProf/js/asc.png blib/lib/Devel/NYTProf/js/asc.png
cp lib/Devel/NYTProf/js/style-tablesorter.css blib/lib/Devel/NYTProf/js/style-tablesorter.css
cp lib/Devel/NYTProf/js/jit/gradient20.png blib/lib/Devel/NYTProf/js/jit/gradient20.png
cp lib/Devel/NYTProf/js/jit/gradient-cushion1.png blib/lib/Devel/NYTProf/js/jit/gradient-cushion1.png
cp lib/Devel/NYTProf.pm blib/lib/Devel/NYTProf.pm
cp lib/Devel/NYTProf/js/bg.png blib/lib/Devel/NYTProf/js/bg.png
cp lib/Devel/NYTProf/js/jit/jit-yc.js blib/lib/Devel/NYTProf/js/jit/jit-yc.js
cp MemoryProfiling.pod blib/lib/Devel/MemoryProfiling.pod
cp lib/Devel/NYTProf/js/jit/gradient40.png blib/lib/Devel/NYTProf/js/jit/gradient40.png
Running Mkbootstrap for Devel::NYTProf ()
chmod 644 "NYTProf.bs"
"/home/cschwenz/local/perl-5.20.1/bin/perl" "/home/cschwenz/local/perl-5.20.1/lib/5.20.1/ExtUtils/xsubpp"  -typemap "/home/cschwenz/local/perl-5.20.1/lib/5.20.1/ExtUtils/typemap" -typemap "typemap"  FileHandle.xs > FileHandle.xsc && mv FileHandle.xsc FileHandle.c
gcc -c   -D_REENTRANT -D_GNU_SOURCE -DPERL_REENTRANT_MAXSIZE=1048576 -DPERL_HASH_FUNC_SIPHASH -DPERL_PERTURB_KEYS_RANDOM -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/home/cschwenz/local/include -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g   -DVERSION=\"5.06\" -DXS_VERSION=\"5.06\" -fPIC "-I/home/cschwenz/local/perl-5.20.1/lib/5.20.1/x86_64-linux-thread-multi-ld/CORE"  -DHAS_CLOCK_GETTIME -DHAS_ZLIB -W -Wall -Wpointer-arith -Wbad-function-cast -Wno-comment -Wno-sign-compare -Wno-cast-qual -Wmissing-noreturn -Wno-unused-parameter FileHandle.c
FileHandle.xs: In function ‘NYTP_write_attribute_nv’:
FileHandle.xs:920: warning: format ‘%g’ expects type ‘double’, but argument 4 has type ‘NV’
FileHandle.xs:920: warning: format ‘%g’ expects type ‘double’, but argument 4 has type ‘NV’
"/home/cschwenz/local/perl-5.20.1/bin/perl" "/home/cschwenz/local/perl-5.20.1/lib/5.20.1/ExtUtils/xsubpp"  -typemap "/home/cschwenz/local/perl-5.20.1/lib/5.20.1/ExtUtils/typemap" -typemap "typemap"  NYTProf.xs > NYTProf.xsc && mv NYTProf.xsc NYTProf.c
gcc -c   -D_REENTRANT -D_GNU_SOURCE -DPERL_REENTRANT_MAXSIZE=1048576 -DPERL_HASH_FUNC_SIPHASH -DPERL_PERTURB_KEYS_RANDOM -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/home/cschwenz/local/include -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g   -DVERSION=\"5.06\" -DXS_VERSION=\"5.06\" -fPIC "-I/home/cschwenz/local/perl-5.20.1/lib/5.20.1/x86_64-linux-thread-multi-ld/CORE"  -DHAS_CLOCK_GETTIME -DHAS_ZLIB -W -Wall -Wpointer-arith -Wbad-function-cast -Wno-comment -Wno-sign-compare -Wno-cast-qual -Wmissing-noreturn -Wno-unused-parameter NYTProf.c
NYTProf.xs: In function ‘open_output_file’:
NYTProf.xs:1769: warning: format ‘%.6f’ expects type ‘double’, but argument 3 has type ‘NV’
NYTProf.xs: In function ‘close_output_file’:
NYTProf.xs:1799: warning: format ‘%.6f’ expects type ‘double’, but argument 2 has type ‘NV’
NYTProf.xs: In function ‘finish_profile’:
NYTProf.xs:2960: warning: format ‘%g’ expects type ‘double’, but argument 2 has type ‘NV’
NYTProf.c: At top level:
NYTProf.xs:1137: warning: ‘get_str_id’ defined but not used
rm -f blib/arch/auto/Devel/NYTProf/NYTProf.so
LD_RUN_PATH="/usr/lib:/home/cschwenz/local/lib" gcc  -shared -O2 -g -L/home/cschwenz/local/lib -L/usr/local/lib -fstack-protector FileHandle.o NYTProf.o  -o blib/arch/auto/Devel/NYTProf/NYTProf.so    \
       -lrt -lz     \

chmod 755 blib/arch/auto/Devel/NYTProf/NYTProf.so
"/home/cschwenz/local/perl-5.20.1/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- NYTProf.bs blib/arch/auto/Devel/NYTProf/NYTProf.bs 644
cp blib/arch/auto/Devel/NYTProf/NYTProf.so blib/lib/Devel/auto/Devel/NYTProf/NYTProf.so
cp bin/nytprofcsv blib/script/nytprofcsv
"/home/cschwenz/local/perl-5.20.1/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/nytprofcsv
cp bin/nytprofhtml blib/script/nytprofhtml
"/home/cschwenz/local/perl-5.20.1/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/nytprofhtml
cp bin/flamegraph.pl blib/script/flamegraph.pl
"/home/cschwenz/local/perl-5.20.1/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/flamegraph.pl
cp bin/nytprofcalls blib/script/nytprofcalls
"/home/cschwenz/local/perl-5.20.1/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/nytprofcalls
cp bin/nytprofmerge blib/script/nytprofmerge
"/home/cschwenz/local/perl-5.20.1/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/nytprofmerge
cp bin/nytprofcg blib/script/nytprofcg
"/home/cschwenz/local/perl-5.20.1/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/nytprofcg
Manifying 5 pod documents
Manifying 9 pod documents
cschwenz@cochin:~/local/src/perl-modules/Devel-NYTProf-5.06$ make test
Running Mkbootstrap for Devel::NYTProf ()
chmod 644 "NYTProf.bs"
cp blib/arch/auto/Devel/NYTProf/NYTProf.so blib/lib/Devel/auto/Devel/NYTProf/NYTProf.so
PERL_DL_NONLAZY=1 "/home/cschwenz/local/perl-5.20.1/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-load.t ............ 1/2 # Testing Devel::NYTProf 5.06 started at Mon Feb  9 12:29:57 2015
# Compression: default level is 6, zlib version 1.2.8
# --- Perl 5.020001 Config on x86_64-linux-thread-multi-ld:
#   d_gettimeod: define
#   d_sysconf: define
t/00-load.t ............ ok   
t/10-run.t ............. ok       
t/22-readstream.t ...... ok    
t/30-util.t ............ ok     
t/31-env.t ............. ok   
t/40-savesrc.t ......... ok       
t/42-global.t .......... ok       
t/44-model.t ........... ok     
t/50-errno.t ........... ok   
t/60-forkdepth.t ....... ok   
t/68-hashline.t ........ skipped: Currently a developer-only test
t/70-subname.t ......... ok    
t/71-moose.t ........... skipped: Test is incomplete (has no results defined yet)
t/72-autodie.t ......... skipped: Currently a developer-only test
t/80-version.t ......... ok   
t/90-pod.t ............. ok     
t/91-pod_coverage.t .... skipped: Currently a developer-only test
t/92-file_port.t ....... skipped: Test::Portability::Files required for testing filename portability. Can't locate Test/Portability/Files.pm
t/test01.t ............. ok       
t/test02.t ............. ok       
t/test03.t ............. ok       
t/test05.t ............. ok       
t/test06.t ............. ok       
t/test07.t ............. ok       
t/test08.t ............. ok       
t/test09.t ............. ok       
t/test10.t ............. ok       
t/test11.t ............. ok       
t/test12.t ............. ok       
t/test13.t ............. ok       
t/test14.t ............. ok     
t/test16.t ............. 1/113 given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
t/test16.t ............. 3/113 given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
t/test16.t ............. 45/113 given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
t/test16.t ............. 81/113 given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
given is experimental at test16.p line 10.
when is experimental at test16.p line 11.
when is experimental at test16.p line 14.
t/test16.t ............. ok       
t/test17-goto.t ........ ok     
t/test18-goto2.t ....... ok     
t/test20-streval.t ..... ok       
t/test21-streval3.t .... ok       
t/test22-strevala.t .... ok       
t/test23-strevall.t .... ok     
t/test24-strevalc.t .... ok     
t/test25-strevalb.t .... 1/145 perl: op.c:12011: Perl_rpeep: Assertion `(kid->op_type == OP_NULL && kid->op_targ == OP_NEXTSTATE) || kid->op_type == OP_STUB || kid->op_type == OP_ENTER' failed.
Exit status 6 from /home/cschwenz/local/perl-5.20.1/bin/perl -d:NYTProf at t/test25-strevalb.t line 23.
Profile data incomplete, no terminator for 1 pids (refer to TROUBLESHOOTING in the documentation)
Use of uninitialized value in division (/) at /home/cschwenz/local/src/perl-modules/Devel-NYTProf-5.06/blib/lib/Devel/NYTProf/Data.pm line 113, <DATA> line 6.
Use of uninitialized value in subtraction (-) at /home/cschwenz/local/src/perl-modules/Devel-NYTProf-5.06/blib/lib/Devel/NYTProf/Data.pm line 114, <DATA> line 6.

#   Failed test 'undef isa 'Devel::NYTProf::SubInfo''
#   at t/test25-strevalb.t line 34.
#     undef isn't defined

#   Failed test at t/lib/NYTProfTest.pm line 172.
# Test group aborted: Can't call method "subname" on an undefined value at t/test25-strevalb.t line 35, <DATA> line 6.
# Looks like you planned 145 tests but ran 4.
# Looks like you failed 2 tests of 4 run.
t/test25-strevalb.t .... Dubious, test returned 2 (wstat 512, 0x200)
Failed 143/145 subtests 
t/test30-fork-0.t ...... ok       
t/test40pmc.t .......... ok       
t/test50-disable.t ..... ok       
t/test51-enable.t ...... ok       
t/test60-subname.t ..... ok     
t/test61-submerge.t .... ok     
t/test62-subcaller1.t .. ok     
t/test70-subexcl.t ..... ok     
t/test80-recurs.t ...... ok     
t/test81-swash.t ....... ok     
t/test82-version.t ..... ok     
t/test90-strsubref.t ... ok     
t/zzz.t ................ 1/? # Tests ended at Mon Feb  9 12:32:13 2015
t/zzz.t ................ ok   

Test Summary Report
-------------------
t/test25-strevalb.t  (Wstat: 512 Tests: 4 Failed: 2)
  Failed tests:  3-4
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 145 tests but ran 4.
Files=53, Tests=4521, 136 wallclock secs ( 1.52 usr  0.14 sys + 77.08 cusr 13.07 csys = 91.81 CPU)
Result: FAIL
Failed 1/53 test programs. 2/4521 subtests failed.
make: *** [test_dynamic] Error 255
cschwenz@cochin:~/local/src/perl-modules/Devel-NYTProf-5.06$ ~/local/perl-5.20.1/bin/perl -V
Summary of my perl5 (revision 5 version 20 subversion 1) configuration:

  Platform:
    osname=linux, osvers=2.6.32-71-server, archname=x86_64-linux-thread-multi-ld
    uname='linux evl3300811 2.6.32-71-server #138-ubuntu smp thu dec 18 17:51:53 utc 2014 x86_64 gnulinux '
    config_args='-Dcc=gcc -Dprefix=/home/cschwenz/local/perl-5.20.1 -Dusethreads -Accflags=-DPERL_REENTRANT_MAXSIZE=1048576 -Duselargefiles -Duse64bitall -Duselongdouble -Accflags=-DPERL_HASH_FUNC_SIPHASH -Accflags=-DPERL_PERTURB_KEYS_RANDOM -Dinc_version_list=none -DEBUGGING=both -DNO_MATHOMS [email protected] [email protected]'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    use64bitint=define, use64bitall=define, uselongdouble=define
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DPERL_REENTRANT_MAXSIZE=1048576 -DPERL_HASH_FUNC_SIPHASH -DPERL_PERTURB_KEYS_RANDOM -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/home/cschwenz/local/include -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DPERL_REENTRANT_MAXSIZE=1048576 -DPERL_HASH_FUNC_SIPHASH -DPERL_PERTURB_KEYS_RANDOM -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/home/cschwenz/local/include -I/usr/local/include'
    ccversion='', gccversion='4.4.3', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
    alignbytes=16, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags ='-fstack-protector -L/home/cschwenz/local/lib -L/usr/local/lib'
    libpth=/home/cschwenz/local/lib /usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.4.3/include-fixed /usr/lib /lib/../lib /usr/lib/../lib /lib /lib64 /usr/lib64
    libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=libc-2.11.1.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.11.1'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/home/cschwenz/local/perl-5.20.1/lib/5.20.1/x86_64-linux-thread-multi-ld/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/home/cschwenz/local/lib -L/usr/local/lib -fstack-protector'


Characteristics of this binary (from libperl): 
  Compile-time options: DEBUGGING HAS_TIMES MULTIPLICITY PERLIO_LAYERS
                        PERL_DONT_CREATE_GVSV PERL_HASH_FUNC_SIPHASH
                        PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
                        PERL_NEW_COPY_ON_WRITE PERL_PERTURB_KEYS_RANDOM
                        PERL_PRESERVE_IVUV PERL_TRACK_MEMPOOL USE_64_BIT_ALL
                        USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
                        USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE
                        USE_LOCALE_NUMERIC USE_LONG_DOUBLE USE_PERLIO
                        USE_PERL_ATOF USE_REENTRANT_API
  Built under linux
  Compiled at Jan 16 2015 12:30:38
  %ENV:
    PERL_CPANM_HOME="/home/cschwenz/.local-cpanm"
    PERL_CPANM_OPT="--verbose --from=https://cpan.metacpan.org --auto-cleanup=1 --interactive --prompt"
  @INC:
    /home/cschwenz/local/perl-5.20.1/lib/site_perl/5.20.1/x86_64-linux-thread-multi-ld
    /home/cschwenz/local/perl-5.20.1/lib/site_perl/5.20.1
    /home/cschwenz/local/perl-5.20.1/lib/5.20.1/x86_64-linux-thread-multi-ld
    /home/cschwenz/local/perl-5.20.1/lib/5.20.1
    .
cschwenz@cochin:~/local/src/perl-modules/Devel-NYTProf-5.06$ 

Devel::NYTProf only records timings from one clock at a time.  (It's nice
that that's configurable, to use any of the clock_gettime() clocks.)
It ought to be possible to configure NYTProf to record multiple clocks
simultaneously.  A naive use of times(), in a very simple profiler,
gives both real time and CPU time, with CPU time split into user-mode
and system-mode.  NYTProf could equally well record both CLOCK_MONOTONIC
and CLOCK_PROCESS_CPUTIME_ID.  The comparison of the two sets of timing
data, on the same subroutine executions, provides useful information.

-zefram

NYTProf.xs:343:35: error: invalid suffix "ui64" on integer constant
 unsigned __int64 time_frequency = 0ui64;
                                   ^

NYTProf already 'hides' calls to the accept() builtin.

It would be nice to extend this to the commonly used FCGI::Accept sub (XS).

It would be even nicer to make this a generic mechanism for any sub.

Most of Devel::NYTProf::Data is undocumented.  For example, although it
says how to list the subroutines defined in a particular file identified
by numeric fid, it doesn't document enough to list all known subroutines.
A list of all known fids can be produced cleanly by "map { $_->fid }
$p->all_fileinfos", and that can be iterated to find subs from each file,
but the all_fileinfos method is undocumented.

-zefram

This is unlikely to be resolved soon. I'm reluctant to document the API as I know it is likely to 
change significantly in future (i.e. move to an SQLite based data model).

< 10 UNIVERSAL::can 58 (stack 9)
<  9 Moose::Object::DEMOLISHALL 280 (stack 10)
<  8 Moose::Object::__ANON__[/home/portfolio/perl5/lib/perl5/x86_64-linux-gnu-thread-multi/Moose/Object.pm:99] 291 (stack 9)
<  7 Try::Tiny::try 1234 (stack 8)
<  6 Moose::Object::DESTROY 588 (stack 7)
<  7 Scalar::Util::blessed 58 (stack 6)
<  6 Portfolio::STDIO::DESTROY 462 (stack 7)
<  7 Scalar::Util::blessed 58 (stack 6)
<  6 Portfolio::Variable::DESTROY 331 (stack 7)
<  7 Scalar::Util::blessed 55 (stack 6)
<  6 Portfolio::Request::thumbnail::DESTROY 323 (stack 7)
<  7 Scalar::Util::blessed 53 (stack 6)
<  6 Portfolio::Browser::DESTROY 327 (stack 7)
<  6 CGI::DESTROY 148 (stack 6)
<  7 Scalar::Util::blessed 55 (stack 6)
<  6 Portfolio::Language::DESTROY 321 (stack 7)
<  5 Portfolio::Context::clear_request 2671 (stack 6)
<  4 Portfolio::Context::run 5557 (stack 5)
<  3 Portfolio::FastCGI::handler 1252 (stack 4)
<  2 FCGI::Engine::handle_request 900 (stack 3)
<  4 FCGI::Engine::ProcManager::signals_received 170 (stack 2)
<  3 FCGI::Engine::ProcManager::received_signal 350 (stack 4)
<  4 FCGI::Engine::ProcManager::signals_received 139 (stack 3)
<  3 FCGI::Engine::ProcManager::received_signal 319 (stack 4)
<  3 FCGI::Engine::ProcManager::manager_pid 170 (stack 3)
<  3 FCGI::Engine::ProcManager::manager_pid 148 (stack 3)
<  3 FCGI::Engine::ProcManager::no_signals 167 (stack 3)
<  4 FCGI::Engine::ProcManager::sigaction_no_sa_restart 153 (stack 3)
<  4 POSIX::sigaction 167 (stack 4)
<  4 POSIX::sigaction 92 (stack 4)
<  3 FCGI::Engine::ProcManager::setup_signal_actions 605 (stack 4)
<  3 Moose::inner 254 (stack 3)
<  2 FCGI::Engine::ProcManager::post_dispatch 1120 (stack 3)
<  2 FCGI::Accept 92423400 (stack 2)
visit_node: 0 HASH(0xd112d0)
panic at /home/portfolio/perl5/bin/nytprofcalls line 191.

        (void)hv_fetch_ent((HV *)SvRV(sv), caller_subname_sv, 1, 0);

This is with NYTProf 5.00. I'm generating profiling data by calling DB::enable_profile and ::disable_profile inside a script, if that's relevant.

If you need more info please let me know.

On Wed Apr 17 15:26:52 2013, DROLSKY wrote:
> This is with NYTProf 5.00. I'm generating profiling data by calling
> DB::enable_profile and ::disable_profile inside a script, if that's
> relevant.
> 
> If you need more info please let me know.

This definitely seems to be related to starting the profiling inside the script. When I just profile it start to finish, there's no problem.

A small test case would be greatly appreciated.

On Thu, 18 Apr 2013, Tim_Bunce via RT wrote:

> <URL: https://rt.cpan.org/Ticket/Display.html?id=84704 >
>
> A small test case would be greatly appreciated.

Hmm, I don't know if I can come up with a small one. The thing I was 
profiling is one of our web services, and it's anything but small.

If this comes up again I'll look into more, but for now it's ok to mark 
this as stalled.


-dave

/*============================================================
http://VegGuide.org               http://blog.urth.org
Your guide to all that's veg      House Absolute(ly Pointless)
============================================================*/

After merging a small group of nytprof.out files using nytprofmerge,
reading the merged file using Devel::NYTProf::Data generates many warnings
that I didn't get from any of the individual files:

    $ perl -MDevel::NYTProf::Data -lwe '$|=1; Devel::NYTProf::Data->new({filename=>"nytprof-t6.out"})' 2>&1 | uniq -c
          1 Reading nytprof-t6.out
          1 
          1 Processing nytprof-t6.out data
          1 
       6300 Use of uninitialized value $srclines_array in join or string at /opt/perl-5.12.3/lib/site_perl/5.12.3/x86_64-linux-ld/Devel/NYTProf/FileInfo.pm line 491.

Looking at the code, I believe the warning is incorrectly identifying
the uninitialised value; it must be elements of @$srclines_array, not
$srclines_array itself, that are undef.

-zefram

I'd appreciate a test case for this.

A screencast about profiling perl code, including a detailed look at how to use NYTProf and how to optimize your code, is available at http://timbunce.blip.tv/file/3913278/

Separating this out from the Apache2::SizeLimit issue...

It ought to be possible to save profiling data without stopping collection
of profiling data, and it ought to be possible to save more than once in
one process.  Each save operation saves the data that has been collected
since the previous save.  Each save probably needs to go to a separate
file; you'll need nytprofmerge to combine them into a single file for
report generation.

The purpose of multiple saving is to get useful profiling data out of a
process that may die unpredictably in a way that D:NYTP doesn't catch.
This comes up a lot with Apache, where things like Apache2::SizeLimit
terminate a worker process after some request has been handled, with no
simple way to predict which request will do it.  With multiple saving,
you can set up a PerlCleanupHandler, running after each request, that
saves data without stopping profiling.  This means you can always pick up
data covering every request processed so far, regardless of the current
state of child processes.  You then lose very little if you fail to
trap child termination.  Processes can be killed uncleanly and you've
still got profiling data from everything they did except what they were
working on at the time of killing.

You pointed at the existing ability to start and stop data collection
repeatedly at runtime.  That doesn't do what I'm after, because it
doesn't save the data collected so far.  Currently, saving profile data
once means that no more data can be saved from that process.

-zefram

You can already call finish_profile(), which will write the accumulated sub profiler data and other end-of-run data.
I believe that calling enable_profile() after that will start a new profile.
So I think the foundations you need are already there.
Or am I missing something?

Tim_Bunce via RT wrote:
>I believe that calling enable_profile() after that will start a new profile.

Ah, looks like it does.  I got confused by the coupling between control
of data collection and control of writing, sorry.

-zefram

Great. I'll happily take a doc patch to clarify things.

p.s. I've just uploaded Devel-NYTProf-5.04.tar.gz

NYTProf provides average time per call in its reports, like so:

# spent 31.1ms (2.60+28.5) within Eval::Closure::eval_closure which was called 149 times, avg 
209µs/call:
#126 times (2.31ms+25.9ms) by Class::MOP::Method::Generated::_compile_code at line 63 of 
Class/MOP/Method/Generated.pm, avg 224µs/call
#  23 times (292µs+2.64ms) by Moose::Meta::TypeConstraint::_actually_compile_type_constraint 
at line 320 of Moose/Meta/TypeConstraint.pm, avg 128µs/call

However, averages hide a lot of data. One of the lessons learned from DTrace, eg in 
http://dtrace.org/blogs/brendan/2011/12/18/visualizing-device-utilization/ , is that averages 
hide many types of interesting data: the rare spike, bimodal distributions, etc. At the Moving to 
Moose Hackathon, we saw a practical example of how NYTProf's averages were not telling the 
whole story. regcomp was taking a significant chunk of a program's time, which NYTProf of 
course pointed out like a champ. But we scratched our heads thinking it was a bug in Perl until 
we generated a histogram using DTrace, which showed that 3/4ths of the regcomps were super 
fast, and 1/4th of them were many orders of magnitude slower. Once we attained that bit of 
information, we instantly saw that the problem was a particular regular expression, and tweaked 
it to not expose the problem. I'm sure there are many other examples of where a histogram 
would highlight performance bottlenecks.

So it would be fantastic if NYTProf's reports included a histogram of how long each call took. 
DTrace again is a good model for how to render this kind of data: list each power of 2 in the first 
column, representing milliseconds. Then, in the next column, list how many times the number of 
calls that took that chunk of time (so for example, there might be 500 calls between 16-31ms, 
and 4 calls between 1024 and 2048ms). Then you can also render that information visually. 
Here's an example of DTrace's output from quantize (note the first and last values will generally 
have no hits, they're just there to bookend the data).

           value  ------------- Distribution ------------- count
           32768 |                                         0
           65536 |@@                                       1
          131072 |@@@@@@@@@@@@@@@@@@@@                     9
          262144 |@@@@@@@@@@@@@@@@                         7
          524288 |                                         0
         1048576 |@@                                       1
         2097152 |                                         0

I'd be very happy to implement this if you're interested in the idea and you could guide me!

I agree that this is a good idea.

Currently the subroutine profiler stores its data in an array held in a hash that's keyed by 
caller_subname[fid:line].

See incr_sub_inclusive_time(). Especially subr_call_av and the NYTP_SCi_* macros.

There's a test using given/when that on recent perls generates warnings
about given/when being experimental.  The test should muffle the warning.
Attached patch fixes.

-zefram

FileHandle.xs:62: error: syntax error before "tTHX"
FileHandle.xs:62: warning: no semicolon at end of struct or union
FileHandle.xs:75: error: syntax error before '}' token
FileHandle.xs: In function `NYTP_tell':
FileHandle.xs:95: error: dereferencing pointer to incomplete type
FileHandle.xs:96: error: dereferencing pointer to incomplete type
FileHandle.xs:97: error: dereferencing pointer to incomplete type
FileHandle.xs:97: error: dereferencing pointer to incomplete type
FileHandle.xs:101: error: dereferencing pointer to incomplete type
FileHandle.xs:102: error: dereferencing pointer to incomplete type
FileHandle.xs: In function `NYTP_type_of_offset':
FileHandle.xs:109: error: dereferencing pointer to incomplete type
FileHandle.xs:120: error: dereferencing pointer to incomplete type
FileHandle.xs: In function `compressed_io_croak':
FileHandle.xs:144: error: dereferencing pointer to incomplete type
FileHandle.xs:156: error: dereferencing pointer to incomplete type
FileHandle.xs: In function `NYTP_start_deflate':
...

I'm not yet certain whose fault this is, but I'm submitting this anyway so
that someone else knows.

First: When I try to use Devel::NYTProf::Apache on my system, Apache
segfaults immediately on startup.

First, the numbers:
Debian wheezy (and all the debian patches that comes with)
Perl v5.14.2 (Debian version 5.14.2-20)
Apache2-mpm-prefork v2.2.22 (Debian version 2.2.22-13)
Devel::NYTProf       4.06


I have attempted to track down the problem, and narrowed it down to this
sequence of events:

1. Apache starts up and spins up mod_perl
2. mod_perl creates an interpreter
3. Devel::NYTProf is loaded.  XS_DB_init_profiler  patches the opcode table.
4. Apache decides to shut down mod_perl for some reason.
5. mod_perl unloads all the SOs loaded, including Devel::NYTProf

#5  0x00007ffff695800f in dlclose () from /lib/x86_64-linux-gnu/libdl.so.2
#6  0x00007ffff46be9f9 in modperl_sys_dlclose (handle=<optimized out>) at
modperl_sys.c:50
#7  0x00007ffff46b5b39 in modperl_xs_dl_handles_close
(handles=0x55555580a570) at modperl_util.c:319
#8  0x00007ffff46adea6 in modperl_interp_destroy (interp=0x555555806b10) at
modperl_interp.c:148
#9  0x00007ffff46adf19 in modperl_interp_pool_destroy (data=0x7ffff7e2e4c0)
at modperl_interp.c:202
#10 0x00007ffff7761957 in ?? () from /usr/lib/libapr-1.so.0
#11 0x00007ffff77607e9 in apr_pool_destroy () from /usr/lib/libapr-1.so.0
#12 0x00007ffff77606b7 in apr_pool_clear () from /usr/lib/libapr-1.so.0
#13 0x000055555558570f in main (argc=6, argv=0x7fffffffe618) at main.c:698

However, the Perl SO is not unloaded, and the opcode table remains patched.

6. Later (not much), a new interpreter is spun up
7. Perl goes to run some code and jumps to the address where NYTProf opcode
intercept functions were running
8. Those functions are no longer there, because the SO has been unloaded,
and segfault.


At this point I have not yet been able to determine if the problem is
mod_perl's (for not cleaning up the interpreter correctly) or NYTProf's
(for not cleaning up the opcode table before the interpreter is destroyed).


-- Stevie-O
Real programmers use COPY CON PROGRAM.EXE

skip "Sub::Name 0.06 required for subname line numbers", 2
            if $Sub::Name::VERSION <= 0.06;