run k-script-build correctly about oil

[andychu]

OK I fixed the two bugs you mentioned.

Then I cloned the repo and did the self-hosting step. I was able to parse all ~2600 lines of code in the project with OSH.

Running it, OSH completes (in 15 seconds on my machine), but gives the wrong results. It looks like tabs are messed up, and the file is much shorter.

I will look into these issues more. I may have questions later. Thanks, this is a great test case!

[stephenreay]

Ah. The printf issue seems to be because of backslash escaping, see basic repro in https://gist.github.com/stephenreay/339d01d6fc6239c3089810cbdca2f4cf

[stephenreay]

Ok, and I've got a repro for the [ ... -a ... ] issue too. Same gist. https://gist.github.com/stephenreay/339d01d6fc6239c3089810cbdca2f4cf#file-test-and-or-sh

[andychu]

OK thanks, this is great! I just ran both gists and got the unexpected results. The first one is a simple oversight. I have to look deeper into the second one. Should have fixes soon though.

[stephenreay]

Great! I added you to the repo for (amongst other things) k-script-build.

Once you have it checked out ./configure && make k-script-build-static should produce (via self-hosting) a working script with everything inlined so you don't need to install the library files for the paths to work.

The makefile uses that same static script when building the distribution targets (ie the relatively linked source script builds a temporary static inlined script, which builds the absolutely linked installable script)

[stephenreay]

Nice work. Happy to help, so sure send me any questions you have, and/or I'm happy to trial against updated OSH builds too.

[andychu]

So after some inspection and debugging I think the rest of the errors are due to the fact that the read builtin doesn't currently respect IFS, and that's used in several places in the script.

I could implement that, but unfortunately I think it will make the script even slower than the 15 seconds it takes now! I already know I have to optimize the IFS splitting algorithm because it's a bottleneck elsewhere.

So I'm going to leave this open, but I probably won't get to it for a couple months, until after I optimize the parser.

But this is a really good benchmark because it spends significant time in shell! Most shell scripts spend most of their time in external utilities, but this does significant work in shell itself.

If you can think of any other shell scripts to run OSH on, please do and file bugs :) Thanks.

[stephenreay]

Well its great that you figured out the issue. No worries - I’ll be interested to see how it goes once you can optimise & implement IFS for read, but no rush from my side. Another one (which from memory also will rely on `IFS= read -r`) that does a lot in the shell is in that same repo - k-shell-doc. Basically *also* reads line by line and generates markdown api docs from inline comment docs.

…

On 16 Nov 2017, at 4:22 pm, andychu ***@***.***> wrote: So after some inspection and debugging I think the rest of the errors are due to the fact that the read builtin doesn't currently respect IFS, and that's used in several places in the script. I could implement that, but unfortunately I think it will make the script even slower than the 15 seconds it takes now! I already know I have to optimize the IFS splitting algorithm because it's a bottleneck elsewhere. So I'm going to leave this open, but I probably won't get to it for a couple months, until after I optimize the parser. This is a really good benchmark because it spends significant time in shell! Most shell scripts spend most of their time in external utilities, but this does significant work in shell itself. If you can think of any other shell scripts to run OSH on, please do and file bugs :) Thanks. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAyBz8Hzg8HJ2aQxeAXLOyqJFl_77Im1ks5s2_7rgaJpZM4Qd12b>.

[andychu]

Hi Stephen, I'm implementing IFS with read right now. (I ran into this in multiple places. I see other people using this for packaging shell scripts into a single file.)

After I do this, I think OSH should be able to pass your tests.

It looks like the repository still private (or at least I get a login screen visiting it)? If it's OK with you, I'd like to include your test as part of my test suite, since it is a self-contained program with a well-defined output. Is there something you are planning to do before making it public? Thanks.

[stephenreay]

Hey Andy,

It's great to hear you're making progress!

It's fine to include it, and I think I'm OK to make it public now too. If you want to include files/etc directly in the osh repo please feel free to proceed, if you want to link to my repo and have it work for others, I'll aim to get it done this week.

[andychu]

OK great, I finished implementing read with IFS. I ended up rewriting the hairy IFS algorithm with an explicit state machine, which fixed a lot of bugs and failing tests. I'm happy with how it turned out.

I will try it out on your scripts shortly. Thanks a lot!

[andychu]

OK I got it working! I checked the output script and both have the same checksum. I think the main issue was that I had to rewrite the IFS splitting algorithm and have read use it.

It takes 40 seconds with OSH vs 0.6 seconds for bash on my machine. Well, now that it works, it can be optimized :)

Thanks for the test case!

~/hg/wild/shell-script-library$ sha1sum k-script-build-static k-script-build-static-OSH 
7bf59775a7e3b175da284a1268ba437c94d77573  k-script-build-static
7bf59775a7e3b175da284a1268ba437c94d77573  k-script-build-static-OSH

[andychu]

Fixed in 0.4.0 http://www.oilshell.org/blog/2018/02/03.html

[stephenreay]

Hey @andychu, FYI, despite a huge delay the koalephant-shell-script-library project is now in an open repo.

The k-script-build tool has also had some changes, which may affect how/if it runs under OSH still - it's doing more of the key stuff directly in shell where it used to rely on sed. This mostly relates to pattern matching of strings.

The URL is the same (https://bitbucket.org/koalephant/shell-script-library) and I'm putting up release version tarballs in the 'downloads' page too, if thats easier for testing with.

Let me know if I can help at all.