Ox.parse stack overflow on 4-deep nested tags under Cygwin about ox

[ohler55]

Does seem odd it just happens on Cygwin. Have you tried running the same code in a file instead of with irb?

[mcarpenter]

Yes, I started out with a script and narrowed down the test case to the above in irb. I have just retested the minimal case in a script (in fact not even the initial <?xml... directive is required), the error persists.

However I cannot reproduce the issue on a new VirtualBox/Windows 7/Cygwin install, building the gem fresh from a clone of master.

So I went back to the original machine, uninstalled ox, cloned the repository, built the gem and installed it. The SystemStackError persists on four nested tags. But not on three.

I'll take a closer look again tomorrow.

[ohler55]

You have certainly done all the homework. Tried everything short of modifying the code. I'm afraid that is the next step. I don't have a Windows setup and even if I did there is a good chance I could not reproduce the error. If you are willing to work with me, can I have you put in a printf() statement in the parse.c file at the top of the read_element() function? Right after the line

ename = read_name_token(pi);
printf("*** read_element for %s\n", ename); // add this line

Run it and see if that gets carried away or if we need to look somewhere else.

[mcarpenter]

Here you go:

/tmp/ox$ ruby --version
ruby 1.9.3p374 (2013-01-15 revision 38858) [i386-cygwin]
/tmp/ox$
/tmp/ox$ git log -1 --format=oneline
27ca52570a391efcd0918fe4e785b513c687ed06 Merge pull request #47 from mcarpenter/cygwin
/tmp/ox$
/tmp/ox$ git diff
diff --git a/ext/ox/parse.c b/ext/ox/parse.c
index 3b0a880..b23a436 100644
--- a/ext/ox/parse.c
+++ b/ext/ox/parse.c
@@ -339,6 +339,7 @@ read_element(PInfo pi) {
     int                        done = 0;

     ename = read_name_token(pi);
+    printf("*** read_element for %s\n", ename); // add this line
     end = pi->s;
     elen = end - ename;
     next_non_white(pi);
/tmp/ox$ 
/tmp/ox$ rm ox-1.9.2.gem 
/tmp/ox$
/tmp/ox$ gem build ox.gemspec 
  Successfully built RubyGem
  Name: ox
  Version: 1.9.2
  File: ox-1.9.2.gem
/tmp/ox$ 
/tmp/ox$ gem uninstall ox 
Successfully uninstalled ox-1.9.2
/tmp/ox$ 
/tmp/ox$ gem install ox-1.9.2.gem 
Building native extensions.  This could take a while...
Successfully installed ox-1.9.2
1 gem installed
/tmp/ox$ 
/tmp/ox$ cat foo 
#!/usr/bin/ruby
require 'ox'
Ox.parse('<a><b><c><d></d></c></b></a>') 
/tmp/ox$ 
/tmp/ox$ ./foo 
*** read_element for a><b><c><d></d></c></b></a>
*** read_element for b><c><d></d></c></b></a>
*** read_element for c><d></d></c></b></a>
*** read_element for d></d></c></b></a>
./foo:3: stack level too deep (SystemStackError)
/tmp/ox$ 

[ohler55]

Odd that it only took 4 levels to run out of memory. How much memory do you have on the machine or VM?

[mcarpenter]

The failing machine is standalone OS install, i7 with 16G, rebooted before the above test.

I should look at ulimit(1) I guess.

Shame we can't see what's actually on the stack when it chokes. I'll try to add some more debug code over the weekend -- presumably one can't do anything really useful by catching SystemStackError?

[ohler55]

Sadly the stack error will probably only show the ruby stack. You could put more print statements in to see where the error occurs. That might help and is pretty easy to do. One thing that will help with that is changing the foo file to add the lib and ext directories to the path. That way you do not have to create a gem and go through those steps. Just compile and run the test. Take a look at the tests.rb file in the Ox test directory.

[mcarpenter]

$ ulimit -s
2025
$ ulimit -Hs
2048

(KB according to bash(1)). Same on both systems. Also cannot change the stack size in cygwin via ulimit(1): http://cygwin.com/faq/faq.programming.html#faq.programming.adjusting-heap

[ohler55]

Lets nail down exactly where it is failing first. Maybe there is something else going on.

[mcarpenter]

Stack blows when add_element() calls rb_obj_alloc() in gen_load.c.

/tmp/ox$ git diff 
diff --git a/ext/ox/gen_load.c b/ext/ox/gen_load.c
index d0fe99f..ad1c9e6 100644
--- a/ext/ox/gen_load.c
+++ b/ext/ox/gen_load.c
@@ -316,7 +316,9 @@ add_element(PInfo pi, const char *ename, Attr attrs, int hasChildren) {
        rb_funcall(s, ox_force_encoding_id, 1, pi->options->rb_enc);
     }
 #endif
+    printf("*** gen_load.c:add_element() 1\n");
     e = rb_obj_alloc(ox_element_clas);
+    printf("*** gen_load.c:add_element() 2\n");
     rb_ivar_set(e, ox_at_value_id, s);
     if (0 != attrs->name) {
         VALUE   ah = rb_hash_new();
diff --git a/ext/ox/parse.c b/ext/ox/parse.c
index 3b0a880..3b2f7b7 100644
--- a/ext/ox/parse.c
+++ b/ext/ox/parse.c
@@ -339,6 +339,7 @@ read_element(PInfo pi) {
     int                        done = 0;

     ename = read_name_token(pi);
+    printf("*** read_element for %s\n", ename); // add this line
     end = pi->s;
     elen = end - ename;
     next_non_white(pi);
@@ -385,7 +386,9 @@ read_element(PInfo pi) {
            hasChildren = 1;
            done = 1;
            ap->name = 0;
+    printf("*** parse.c:read_element() 1\n");
            pi->pcb->add_element(pi, ename, attrs, hasChildren);
+    printf("*** parse.c:read_element() 2\n");
            break;
        default:
            /* Attribute name so it's an element and the attribute will be */
/tmp/ox$
/tmp/ox$
/tmp/ox$ rake compile:clean
WARNING: rake-compiler found compiled files in 'ext/ox' directory. Please remove them.
rm -r tmp/i386-cygwin/ox/1.9.3
/tmp/ox$ 
/tmp/ox$ rake compile:clobber 
WARNING: rake-compiler found compiled files in 'ext/ox' directory. Please remove them.
rm -r tmp/i386-cygwin/ox/1.9.3
rm -r ext/ox/ox.so
rm -r tmp
/tmp/ox$
/tmp/ox$ rake compile:compile 
[elided]
/tmp/ox$ cat foo 
#!/usr/bin/ruby -I lib -I ext
require 'ox'
Ox.parse('<a><b><c><d></d></c></b></a>')
/tmp/ox$
/tmp/ox$ ./foo 
*** read_element for a><b><c><d></d></c></b></a>
*** parse.c:read_element() 1
*** gen_load.c:add_element() 1
*** gen_load.c:add_element() 2
*** parse.c:read_element() 2
*** read_element for b><c><d></d></c></b></a>
*** parse.c:read_element() 1
*** gen_load.c:add_element() 1
*** gen_load.c:add_element() 2
*** parse.c:read_element() 2
*** read_element for c><d></d></c></b></a>
*** parse.c:read_element() 1
*** gen_load.c:add_element() 1
*** gen_load.c:add_element() 2
*** parse.c:read_element() 2
*** read_element for d></d></c></b></a>
*** parse.c:read_element() 1
*** gen_load.c:add_element() 1
./foo:8: stack level too deep (SystemStackError)
/tmp/ox$

[ohler55]

That was very informative. The news is not good though. Basically you ran out of memory when allocating a simple object. There are no other functions called after that other than the ruby internals. Even the initialize method is not called. I'm afraid the only solution is out of my hands and with figuring out how to give ruby more memory.

[mcarpenter]

Oh how annoying. I tracked it down to Cygwin ruby 1.9.3p374 vs. p385. I wish I'd noticed that difference earlier but I suppose "it's an interpreter bug" wasn't my first reflex (p.374!!).

Thanks for your guidance in resolving this.

[ohler55]

Thanks for the help tracking it down. It would have been impossible for me to chase it on my own.