Comments (25)
Possible solutions:
- Instead of finding the .zip file dynamically with
PYTHONPATH
/ open(), we can compile it into the binary. - We could use two files instead of a concatenated file. I don't like that distros put stuff in
/usr/local/share
though? But that means that arch-independent and arch-dependent files are separate, which is good.
from oil.
I don't follow what you mean by "The executable has to be set as PYTHONPATH in order to find the bytecode". Can you clarify?
from oil.
This blog post has some details:
http://www.oilshell.org/blog/2017/05/05.html
But to summarize, I wanted to treat the fact that Oil is written in Python as an implementation detail. So basically I created an app bundle with the compiled Python interpreter and a zip file of all the .pyc
files.
However it uses a trick about zip files.. you can simply do this:
cat python_interpreter app.zip > oil.ovm
Then make oil.ovm
executable. When it runs, it will run the Python interpreter -- the stuff at the end is ignored. The zip file header lives at the end of the file, so oil.ovm is both a valid ELF file and a valid zip file. And then the Python interpreter has to find ITSELF to open()
app.zip!!! This location is set to PYTHONPATH
dynamically (i.e. since PYTHONPATH=app.zip
is a conventional way of looking inside a .zip file.)
However this is actually a hard problem in Unix!!! There are Linux-specific solutions, but no generic Unix ones. argv[0] can be changed by the caller, so you can't necessarily find it by looking there. (See the stackoverflow link above.)
I need to fix this long term... it might even require ditching this app bundle concept altogether unfortunately.
That said, is there a workaround? Unfortunately I lost your Lobsters private messages. I think you could actually fix it by doing
./configure --prefix /usr # assume oil.ovm and osh go into `/usr/bin/`
make
# Then don't call install, instead move it into the arch location. The binary will NOT work
# in this location because --prefix is wrong.
# But then when the user installs it it might work???
Does Arch test the binary in the temporary location, or only in the final installed location?
from oil.
Since the executable is loaded into memory, would it be possible to read the zip contents from there, instead of looking back in the file system?
from oil.
Yes definitely, although it's not a trivial change in the build system. It couldn't be a .zip file anymore -- it would have to be some other in-memory format. Well maybe you could mmap()
it or something, not sure.
Also, compiling the .pyc into the executable makes building for multiple platforms slower in theory. That is, I wanted an architecture-independent and architecture-dependent part. If you put all the bytecode into a C program and compile it, everything becomes architecture-dependent.
Most software has that problem but I was trying to make Oil more "optimal"... but that created some different problems as you can see!
Also a bit of a tangent about Arch... I noticed here that you are only supposed to retrieve PKGBUILD one at a time, not get the whole thing:
https://wiki.archlinux.org/index.php/Arch_Build_System#Retrieve_PKGBUILD_source_using_Svn
I wonder if there is a way to get a copy of all PKGBUILD? I have parsed all Alpine APKBUILD here:
http://www.oilshell.org/git-branch/master/21552818/wild.wwz/distro/alpine-aports/index.html
That's 5,000+ shell scripts.
from oil.
That said, is there a workaround? Unfortunately I lost your Lobsters private messages.
I asked around the IRCs and the workaround I thought may be an option won't, understandably, be admissible in the AURs. Invoking sudo ./install
works in the PKGBUILD, but we can't use that because it goes against the convention.
Does Arch test the binary in the temporary location, or only in the final installed location?
There is a check() function available in PKGBUILD, but Arch doesn't do any testing unless that function is provided. In the PKGBUILD I'm currently making, I do not have that check() function, so there isn't any testing by Arch in the temporary location or the final location . The errors I'm getting are from trying to invoke osh
myself after installing using the PKGBUILD.
from oil.
OK then I think there should be some setting for --prefix
to make it work? Unless I am understanding something about Arch.
The assertion error happens when the binary can't find itself at runtime, i.e. PYTHONPATH is set to something invalid. It expects PYTHONPATH=/usr/bin/oil.ovm
essentially.
But if it's OK that it doesn't run in the temp location, then I think we should able to make it work in the final location only.
Do you have a link to the PKGBUILD file so I can see what's going on? (I guess Arch doesn't use GitHub because it's in SVN? ) And the output of the build process would be useful.
Also if you use IRC we could arrange a time to chat there, which could be easier.
from oil.
tl;dr I think ./configure --prefix /usr
should work, but if it doesn't send me the code :)
install
is a tiny file, so I think it could be duplicated in Arch (for now, although that's not ideal). Basically the build process outputs oil.ovm
, and then osh
is simply a symlink to it, in the style of Busybox.
from oil.
I haven't put the PKGBUILD file anywhere public yet, I'm just testing locally. I'll post a link to it later though so y'all can test it out as well.
OK then I think there should be some setting for --prefix to make it work? Unless I am understanding something about Arch.
I think I can clarify when we chat. There's a specific subdirectory (referred to as $pkgdir within the PKGBUILD) that we have to install binaries to. It's a fakeroot where $pkgdir represents the root of the filesystem. We install binaries relative to that and makepkg
theoretically does the actual placement on the real filesystem.
If I set the prefix to anything on the real filesystem, like /usr, it will actually install to that location and that violates the PKGBUILD convention. Now, if I set the prefix to /usr then manually change the actual installation location in the PKGBUILD to $pkdir/usr, I still get the same error as before. Here's the PKGBUILD file for that scenario: https://timetoplatypus.com/static/oil.PKGBUILD
from oil.
Any further thoughts on this? There's definitely something lost in translation when makepkg
places the binaries on the filesystem. I've tried a bunch of different permutations of options, but I still get that error. Let me know if there's anything else I should try. Otherwise, it seems like the solution will have to be more with how OSH is shipped
from oil.
Sorry I've been procrastinating on this. One reason is that I don't really have a good short term solution for the path issue.
I think you, along with some performance problems I noted in the latest blog post, have convinced me to get rid of the .zip file scheme in the long term. But in the short term I think the best I can do is improve the error message. It's possible to add a Linux-specific hack, but I'm not sure I want to go there yet.
One other possibility: does the Arch build system run "strip" on the binaries? See the dontStrip
option here in Nix:
https://github.com/NixOS/nixpkgs/blob/master/pkgs/shells/oil/default.nix
I didn't realize this, but apparently stripping a binary with a .zip file appended messes with the zip file. That is another reason that this error could occur.
I will definitely add an improved error message for the next release so we can see why the import is failing.
Thanks for publishing the code -- that helps!
from oil.
I've got some good news. Your guess about the binary stripping appears to have been accurate. makepkg
does strip binaries by default and after explicitly selecting an option to not strip the binary in the PKGBUILD, it's working for me.
Identifying that as the problem also greatly simplifies the PKGBUILD, as I can now simply invoke the install
script that ships with OSH.
from oil.
OK awesome! I will definitely put a better error message in the next release!
Thanks again for doing this and sorry for procrastinating :)
from oil.
No problem, I'm glad we got it figured out.
By the way, am I supposed to be able to invoke oil.ovm
directly? After installing OSH using the current corrected PKGBUILD, osh
works fine but trying to run oil.ovm
yields a "-bash: /usr/bin/oil.ovm: No such file or directory" error. It's worth noting that it only errors out when I try to run oil.ovm
, and I do not get an error when I run it using absolute path (/usr/local/bin/oil.ovm
)
from oil.
That's interesting, maybe try in a new bash session? I think bash cached the location wrong?
I know that bash has logic for caching entries in $PATH
, but I'm not sure how it's invalidated. If /usr/local/bin
is in your PATH, then clearly it's done something wrong.
I would debug it like this:
$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin: ...
$ which oil.ovm
/usr/local/bin/oil.ovm
$ oil.ovm
Usage: oil MAIN_NAME [ARG]...
MAIN_NAME [ARG]...
oil behaves like busybox. If it's invoked through a symlink, e.g. 'osh', then
it behaves like that binary. Otherwise the binary name can be passed as the
first argument, e.g.:
oil osh -c 'echo hi'
Missing name of main()
from oil.
In other words there's nothing special about oil.ovm -- if it's in your $PATH you should be able to run it, and if it isn't you won't. Since osh
is in your PATH, and osh and oil.ovm live in the same directory, there's something wrong. But I feel like this has to do with bash caching.
from oil.
Ohp, ya it looks like it's probably just a caching issue. I started a new bash
session and it worked.
from oil.
Yeah this is related to issue #44. I consider that a bash bug.
from oil.
Ok I fixed an issue like this in pyinstaller. I used objcopy to append the zip file to the executable in a valid elf section. This doesn't get stripped off the executable.
from oil.
@garthy Thanks, I didn't know about that! I added a better error message just now, but I will think about doing the objcopy instead.
Do you know if objcopy is GNU- or Linux- specific?
I might get rid of the app bundle thing altogether, so I'm not sure how much I want to fiddle with it.
from oil.
objcopy is a specifically a GNU development tool. It ships with the GNU binutils package.
from oil.
OK thanks, that's what I thought. I improved the error message for this release, so for now I'm closing this bug. But if people continue running into problems (mainly packagers I expect, not end users) then we can revisit it.
from oil.
I'm hitting this one when trying to use Oil as PID 1. I thought it could be a missing /bin/sh and /usr/bin/env but it didn't go away so i dont really know
from oil.
https://gist.github.com/MagicRB/7f9b9c5a7afd2bb97fdcb8169c8f19c1
this is the log i get if i max out verbosity
from oil.
@MagicRB Thanks for the info
I will reply on #1073 since I think there are multiple causes of this error message:
- If the .zip file is stripped
- If
oil.ovm
can't "find itself" to open the embedded zip file
from oil.
Related Issues (20)
- SIGINT and SIGTSTP breaks out of loops in bash/zsh, but not OSH HOT 6
- Implement shopt -s autocd HOT 3
- Error message shows wrong/confusing position for long commands HOT 3
- command -v "$emptyvar" returns zero HOT 3
- Oils 0.23.0 TODO
- Missing "Str=>lower()"
- Abort with += on missing dict key
- intermittent crash running amd-test script -- reproducible in dbg, opt HOT 4
- OpenBSD doesn't have RLIMIT_AS
- Add pp [x + 42] to print an expression and its value - like Rust dbg!() HOT 2
- github URL pastebin HOT 8
- bash allows pasting multi-line code snippets, but OSH/YSH don't HOT 3
- allow backspace from multiline expressions HOT 1
- allow any custom prompts HOT 2
- typeof / reflection / assign results of '=' HOT 3
- emoji/unicode identifiers
- Setting up GNU readline history HOT 1
- 'view latest' released version in documentation - because old links are often surfaced HOT 1
- crash in parsing return HOT 1
- BashArray could have a sparse representation (ble.sh)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from oil.