run4flat / c-blocks Goto Github PK

I did a fresh install of Alien::TinyCCx and C::Blocks and figured this would work first try:

use C::Blocks;
cblock {
    printf("hello world\n");
}

However, it's failing for me with this error:

C::Blocks compiler error:
./c-blocks-test.pl:7: error: undeclared function 'printf'

I've no clue what the problem is, but I'll try to figure it out later if I
don't hear anything else.

I've been wondering: Blocks.xs is pretty big at this point. How would you feel about splitting some of it out into their own .c/.h files to decouple things a bit (where possible)?

Please accept my apologies for filing 200000 bug reports and not really contributing all the fixes. :(

Consider the following example which, while admittedly contrived, segfaults:

use strict; use warnings;

END { Foo->new }

use C::Blocks; use C::Blocks::PerlAPI;

package Foo;

sub new { my $c = shift; bless({@_} => $c) }
sub DESTROY {
  cblock { printf("Hello from DESTROY!\n"); }
}

This segfaults. The example is a bit more complicated than need be, but I'm using it in the hope that it's a bit more intuitively likely to happen than the following, simpler example:

use strict; use warnings;
END { foo() }

use C::Blocks; use C::Blocks::PerlAPI;

sub foo { cblock {} }

Perl is nice to us in that END{} blocks get executed in reverse order they're encountered, so under, let's say, "normal" usage, the C::Blocks END block will be executed AFTER all that are encountered by the (Perl) compiler AFTER C::Blocks is loaded. Since generally speaking, you don't have cblock{} and its siblings available before you're done loading C::Blocks, that means virtually all code is safe.

But this is once again where Perl's dynamism bites. It's perfectly legal to refer to a function before its defined. (Alternatively, we could throw in a "sub foo;" into the beginning of the second example to make that LOOK less fishy.)

I've encountered similar issues in Class::XSAccessor. There, I simply decided NOT to clean up the memory I'd allocated. In virtually any real-world Perl code, that's not a problem, but the trade-offs are that it makes other leak debugging harder, and that if somebody is seriously creating and destroying multiple perl's (not perl ithreads) in the same OS process, it's technically leaking. Not nice.

The easiest way to prevent segfaults in your case is likely to add a guard that, instead of segfaulting when a cblock is to be compiled or executed after _cleanup, croaks() with a user friendly message. All it takes to do that is to add a global (interpreter global, anyway) variable check to both the keyword plugin implementation and all custom ops (do you also generate any XSUBs? Haven't checked). Not a work of beauty, but I think still worth it for people's sanity.

If anybody every comes across this issue in a significantly sized code base, and that's really the only place you should ever accidentally run into this, he's in for a debugging session from hell...

Right now, C::Blocks will leak its internal state whenever there's an exception thrown in the keyword plugin.

The following commit (which might not cleanly apply to master since it's based off of the tsee/mutex branch) fixes one of several such leaks:

tsee@d144334

If I get some spare time I'll try to fix more of them, but in a nutshell, this shows the problem:

use strict;use warnings;use C::Blocks;
cblock {
  int i
}

Run that with valgrind after adding the missing semicolon. Note the leaks. Then run it as is. According to my machine, 11 more leaks (though they might end up pointing to the same one or two root causes).

The commit above fixes some of them by making sure cleanup_c_blocks_data gets called even on croak().

--Steffen

This is obviously just a wishlist item. :)

Looking at csub blocks, I can't but feel that it's the "wrong" API. Writing XSUBs by hand, dealing with the Perl stack and type conversions manually is something that is exceedingly difficult and OH MY GOD error prone. I dare say that even most XS module authors would struggle some. Having the ability to do it makes perfect sense, but it clearly is a measure of last resort.

I also realize that XS is ugh, kludgy and the typemap system is pretty awful. But it's pretty much what we have at our disposal to solve the problem I described above (unless one would reinvent that entirely).

As you, David, know, XS::TCC implements something like this. For bystander's context: XS::TCC exports a function that takes a code string as argument. That code will be scanned (by scary regexes wired together to a makeshift C parser, courtesy of Inline::C) for C function signatures. It then extracts the parameter and return types and tries to locate typemaps for them (either user supplied or the defaults that XS::TCC ships) and generates C code for the parameter and return value handling.

So essentially, it does the SANE part of the job of ExtUtils::ParseXS.

Now, a bunch of the code to do this in XS::TCC is a bit meh, but it works and seems reasonably straightforward to port to a use case in C::Blocks. But C::Blocks has the opportunity to have a much better interface! The scan-code-with-regexes thing should really be replaced.

Imagine you had instead:

use C::Blocks;

csub double my_exp(double x, double exponent) {
    return pow(x, exponent);
}

print my_exp(3, 4), "\n";

What this would require is building a C function signature parser using the Perl lexer API. That's not truly hard, but it's likely a fair amount of work (you have more recent experience working with it than I do, so maybe you feel it's not a huge deal).

I think if that bit were implemented and gave a structured parse of the C function signature, I could whip up the code gen reasonably easily...

The API bit that would be missing is that this does not allow for providing custom typemaps. Having a global registry isn't good enough for this (unless it offers local() alike semantics) because that would otherwise introduce action at a distance between unrelated libraries. I'm not sure what would be the cleanest way to do that. Maybe some kind of attribute syntax (ugh).

--Steffen

I just built C::Blocks on a rather vanilla Linux (4.4.0-53-generic #74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux) using the stock distribution perl 5.22.1.

Building C::Blocks takes "forever" in the Build stage because it falls into brute_force_screen in PerlAPI.xs.PL

tsee@l4tsee:~/perl/C-Blocks$ time ./Build 
Building C-Blocks
Creating lib/C/Blocks/PerlAPI.xs
tcc: error: undefined symbol 'PL_memory_wrap'
tcc: error: undefined symbol 'Perl_croak_nocontext'
^^^ Ignoring likely erroneous undefined symbol warning/error
Using executable screen to check 2755 identifiers

...

Using DynaLoader screen to check 2411 identifiers
Beginning brute-force screening
Using DynaLoader screen to check 2411 identifiers
ccache gcc -I/usr/local/share/perl/5.22.1/auto/share/dist/Alien-TinyCCx/include -I. -I/usr/lib/x86_64-linux-gnu/perl/5.22/CORE -DVERSION="0.42" -DXS_VERSION="0.42" -fPIC -c -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -o lib/C/Blocks.o lib/C/Blocks.c
ExtUtils::Mkbootstrap::Mkbootstrap('blib/arch/auto/C/Blocks/Blocks.bs')
x86_64-linux-gnu-gcc -shared -L/usr/local/lib -fstack-protector-strong -o blib/arch/auto/C/Blocks/Blocks.so lib/C/Blocks.o -L/usr/local/share/perl/5.22.1/auto/share/dist/Alien-TinyCCx/lib -ltcc
ccache gcc -I/usr/local/share/perl/5.22.1/auto/share/dist/Alien-TinyCCx/include -I. -I/usr/lib/x86_64-linux-gnu/perl/5.22/CORE -DXS_VERSION="0.42" -DVERSION="0.42" -fPIC -c -D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -o lib/C/Blocks/PerlAPI.o lib/C/Blocks/PerlAPI.c
ExtUtils::Mkbootstrap::Mkbootstrap('blib/arch/auto/C/Blocks/PerlAPI/PerlAPI.bs')
x86_64-linux-gnu-gcc -shared -L/usr/local/lib -fstack-protector-strong -o blib/arch/auto/C/Blocks/PerlAPI/PerlAPI.so lib/C/Blocks/PerlAPI.o -L/usr/local/share/perl/5.22.1/auto/share/dist/Alien-TinyCCx/lib -ltcc

real    6m59.862s
user    6m59.248s
sys     0m13.268s

(I think that the "starting brute force scan" showing up late in the output is a STDOUT/STDERR mixing or a PerlIO buffering effect.)

The bit that I found surprising was that this uses the system compiler rather than TCC, so there's clearly something about the purpose of this that I haven't intuitively understood (didn't dig).

HTH,
Steffen

Please see https://github.com/tsee/C-Blocks/tree/tsee/mutex for a partial fix to make C::Blocks threadsafe. The main commit in that branch outlines what I know is missing from it. Other than that, I think it's not a huge hassle.

HTH,
Steffen

This will make it easier to pull in the compiler flags by just using the name of the relevant package (e.g., PDL, Gimp, Leptonica, etc.). Plus, all Alien::Base packages have support for this right now.

CC: @Perl5-Alien

This ticket is just a PSA at this stage.

Your module is one of the very few things showing up on grep, filing this ticket to get your attention towards https://rt.cpan.org/Ticket/Display.html?id=113635. It may be possible your logic is similarly flawed (I have not tested your module on these perls, can do so at a later stage).

Cheers!

I had hoped to finish writing this and post it quickly, but that's taking a little longer than I had expected. The details are tricky! This system allows for single inheritance plus roles. It builds a class with fully equivalent representation in both C and Perl code. Derivative classes can override methods using both Perl and C. Anyway, the basic ideas follow:

Each object is a pointer to a struct which holds the object's attributes. The first element of every object is a pointer to the class method table (the vtable). Derived classes that have additional attributes always place those attributes after the attributes of the parent class. Thanks to C's structural equivalence, we can handle an object of a derived class as if it were an object of the parent class because the attributes are all in the same relative locations.

The details (metainformation) for the attributes, methods, and roles are maintained in Perl package globals within each class. The mechanics for handling derived classes amounts to copying the contents of these package globals and adding to them, then building the C and Perl code that reflects the class of the given structure. (Some day I'd like to figure out how to make this a Moose extension, but today isn't the day.)

The method tables hold all of the methods. Any new methods provided by derived classes are added on at the end o the method table. This means that an object of a derived class can be used as if it were an object of the parent class even including method invocation.

Roles were the tricky part of this design, particularly for C. Obviously in Perl if something is known to do a role, we just invoke the wanted role method. In C, we have the added issue of finding said role methods. I solve this by using role method tables. When a role is applied to a class, the role methods are added to the method table in the order specified by the role. The C implementation of the does method takes a role id (an integer) and finds the address of the first method in the method table. It then returns this address. Due to structural equivalence, this address can be safely cast as a role method table, because the order of the pointers starting from the first method are identical to the role method table's layout. Then, the C code can refer to the desired method from the role method table:

    My::Logger * role_table = obj->methods->does(My::Logger::Type);
    if (role_table) {
        role_table->log(obj);
    }

Note that there is no sensible way to directly access role attributes. Instead, attributes will have to be accessed via getters and setters.

Also, once a role is composed into the class, the class itself knows about the methods and attributes of the role and can use them directly.

So it isn't getting included in the tarball and isn't visible on metacpan.

The following variation on my thread safety test script will happily segfault (might need to tune the number iterations for your system or run a few times) even outside of any threaded environment:

use strict;
use warnings;

use C::Blocks;

for (1..500) {
  eval qq[
    clex {
      void foo$_() {}
    }
    1
  ];
}

This may be bizarre use, but it clearly shouldn't segfault.

The start of the backtrace looks like this:

#0  0x00007fdd7c931428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007fdd7c93302a in __GI_abort () at abort.c:89
#2  0x00007fdd7c9737ea in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fdd7ca8c2e0 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007fdd7c97be0a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7fdd7ca8c3f0 "double free or corruption (out)", action=3) at malloc.c:5004
#4  _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3865
#5  0x00007fdd7c97f98c in __GI___libc_free (mem=<optimized out>) at malloc.c:2966
#6  0x00007fdd7bb71f29 in execute_compiler (my_perl=0x7fdd7b956360, state=0x13c7e10, data=0x7fdd742b5c70, keyword_type=2073388360) at lib/C/Blocks.xs:1157
#7  0x000003cf000003cf in ?? ()
#8  0x00007fdd7b9563d0 in ?? ()
#9  0x00007fdd7bb730c9 in strcat_printf () from /home/tsee/perl/C-Blocks/blib/arch/auto/C/Blocks/Blocks.so
#10 0xbd83489001eb90ff in ?? ()
#11 0x48297400ffffff38 in ?? ()
#12 0x008bfffffef8858b in ?? ()
#13 0xe8c78900000000be in ?? ()
#14 0x48c68948ffff0c11 in ?? ()
#15 0x00b80002a9df3d8d in ?? ()
#16 0xffff003de8000000 in ?? ()
#17 0x8b002363fe058b48 in ?? ()
#18 0x481089ffffff2895 in ?? ()

This goes on for some 20000 call stacks before gdb gives an unhelpful error message. So I suppose some recursive thing in TCC is bombing?

--Steffen

Perl 5.35.10 converted newSV_type from a normal function (in sv.c) to an inline function (in sv_inline.h) for performance reasons. This caused test failures in C::Blocks, as discussed in Perl/perl5#19561.

Summarizing the points in that issue:

• Changing a function to an inline function is a pretty straightforward change, so there's not thought to be a bug in the perl interpreter.
• xenu suggested that it might be a bug in TCC.
• run4flat stated that C::Blocks is highly experimental and not currently under heavy development.

Since it seems to be a bug in TCC/C::Blocks, the perl issue will be closed.

See todo tests in

"For more on goals and milestones, see the distribution's README."

After browsing through the directory on MetaCPAN, I see that you have a README.pod, but it's not showing up on the distribution page. I presume this is a problem with MetaCPAN, and I'll be filing an issue, but in the mean time could you either change the above line to README.pod, or just re-upload the distribution with a plain README? Thanks.

See