Build fails without readline about oil

[andychu]

There is a subtlety here -- it's the difference between the dev machine requirements and end user machine requirements again.

The dev machine must be Linux and have readline; the end user machine could be non-Linux and not have readline.

If the dev machine didn't have readline, then the tarball would NOT contain the Python readline libraries, and then the end user's machine couldn't use readline no matter what!

I added a note here about it:

https://github.com/oilshell/oil/wiki/Contributing

So basically the tarball always contains readline dependencies. That's what the app_deps step is for -- to figure out what goes in the tarball.

And then the end user runs ./configure and there is some logic to disable readline if it's not there. It's at the top of bin/oil.py with HAVE_READLINE. This mechanism might change but it works for now.

[andychu]

And yes I could have done something simpler than this whole app_deps step, which took awhile to get right... but I also want to bootstrap OPy with a second, distinct app bundle. So I thought it would be good to make it so that you can just write an arbitrary Python program, and then get a dynamically sliced "app bundle" from it.

It's a little weird but it's not exposed to end users. Only the tarball is exposed.

[lheckemann]

Even with readline present, the python build doesn't seem to be detecting it. I'm getting

Python build finished, but the necessary bits to build these modules were not found:
_bsddb            _curses           _curses_panel
_multiprocessing   _sqlite3          _ssl
_tkinter          bsddb185          bz2
dbm               dl                gdbm
imageop           nis               readline
sunaudiodev       zlib

although the configure script finds readline just fine:

$ cat _build/detected-config.sh
HAVE_READLINE=1
PREFIX=/usr/local

[andychu]

Hm did you re-run build/prepare.sh configure? That is the step that runs Python's autoconf, and detects readline with the CPython build.

Yeah it's a bit confusing but for developers, there are two places to detect readline -- in the CPython build (purely for detecting dependencies) and in the Oil/OSH build.

[lheckemann]

I'm running this stuff in nix so it's a pure environment and I'm not re-running anything — going from the start each time to make sure I've got the steps right :) Plus once I've got it working I'll have a nix package for it.

[andychu]

Try this:

$ grep -i readline _devbuild/cpython-full/pyconfig.h 
/* Define if you have the readline library (-lreadline). */
#define HAVE_LIBREADLINE 1
/* Define if you have readline 2.1 */

The pyconfig.h is the output of Python's autoconf, and it should detect readline if your Nix environment has it.

Although it might have to do with something like pkgconfig ... I know for "non-standard" distros like Nix they might have problems there. The configure script needs to know where readline is.

Are you running NixOS or the Nix package manager inside another distro?

[andychu]

Also, you can test a raw Python tarball inside your Nix environment and run ./configure. (Although that's pretty much what Oil has; I don't think there are any differences but I could be wrong)

[lheckemann]

Not only do I have readline there, but it's detected the features successfully by the looks of it as well.

$ grep -iA1 readline _devbuild/cpython-full/pyconfig.h
/* Define if you have the readline library (-lreadline). */
#define HAVE_LIBREADLINE 1

--
/* Define if you have readline 2.1 */
#define HAVE_RL_CALLBACK 1
--
/* Define if you can turn off readline's signal handling. */
#define HAVE_RL_CATCH_SIGNAL 1
--
/* Define if you have readline 2.2 */
#define HAVE_RL_COMPLETION_APPEND_CHARACTER 1
--
/* Define if you have readline 4.0 */
#define HAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK 1
--
/* Define if you have readline 4.2 */
#define HAVE_RL_COMPLETION_MATCHES 1
--
/* Define if you have readline 4.0 */
#define HAVE_RL_PRE_INPUT_HOOK 1
--
/* Define if you have readline 4.0 */
#define HAVE_RL_RESIZE_TERMINAL 1

I'm using NixOS, but that shouldn't make a difference as nix uses Linux namespaces to ensure that only the explicitly specified dependencies are available in the build environment.

[andychu]

OK so if that is found, then I would search in the logs above the build step to see what happened.

Here is what I just did:

rm -rf _devbuild
build/prepare.sh configure
build/prepare.sh build-python

Then I get:

gcc -shared build/temp.linux-x86_64-2.7/home/andy/git/oil/Python-2.7.13/Modules/_ctypes/_ctypes.o build/temp.linux-x86_64-2.7/home/andy/git/oil/Python-2.7.13/Modules/_ctypes/callbacks.o build/temp.linux-x86_64-2.7/home/andy/git/oil/Python-2.7.13/Modules/_ctypes/callproc.o build/temp.linux-x86_64-2.7/home/andy/git/oil/Python-2.7.13/Modules/_ctypes/stgdict.o build/temp.linux-x86_64-2.7/home/andy/git/oil/Python-2.7.13/Modules/_ctypes/cfield.o build/temp.linux-x86_64-2.7/home/andy/git/oil/Python-2.7.13/Modules/_ctypes/libffi/src/prep_cif.o build/temp.linux-x86_64-2.7/home/andy/git/oil/Python-2.7.13/Modules/_ctypes/libffi/src/closures.o build/temp.linux-x86_64-2.7/home/andy/git/oil/Python-2.7.13/Modules/_ctypes/libffi/src/x86/ffi64.o build/temp.linux-x86_64-2.7/home/andy/git/oil/Python-2.7.13/Modules/_ctypes/libffi/src/x86/unix64.o build/temp.linux-x86_64-2.7/home/andy/git/oil/Python-2.7.13/Modules/_ctypes/libffi/src/x86/ffi.o build/temp.linux-x86_64-2.7/home/andy/git/oil/Python-2.7.13/Modules/_ctypes/libffi/src/x86/sysv.o -L/usr/lib/x86_64-linux-gnu -L/usr/local/lib -o build/lib.linux-x86_64-2.7/_ctypes.so

Python build finished, but the necessary bits to build these modules were not found:
_bsddb            _curses           _curses_panel  
_multiprocessing   _sqlite3          _ssl           
_tkinter          bsddb185          bz2            
dbm               dl                gdbm           
imageop           sunaudiodev       zlib           
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

Whereas you are also getting readline missing for some reason.

Oops, look at this message:

To find the necessary bits, look in setup.py in detect_modules() for the module's name.

So Python is probably doing something special. It might not be using autoconf. It's using distutils.

I noticed this weirdness about the Python build system and wrote about it in my blog. Distutils duplicates the logic of autoconf. I don't see the exact problem now but let's look.

[lheckemann]

Yeah, I've been digging around in setup.py looking for the missing links just now. My guess would have been that it's also missing curses, but since yours didn't find the deps for that either I'm guessing that's not the issue here. I'll get back to you when I've found out more.

[andychu]

So if you look at Python-2.7.13/setup.py, it has the lines below.

So that's the OTHER method Python uses to detect readline -- don't ask me why!

There might some flags you can pass to change lib_dirs and detect it?

       # readline
        do_readline = self.compiler.find_library_file(lib_dirs, 'readline')
        readline_termcap_library = ""
        curses_library = ""
        # Determine if readline is already linked against curses or tinfo.
        if do_readline and find_executable('ldd'):
            fp = os.popen("ldd %s" % do_readline)
            ldd_output = fp.readlines()
            ret = fp.close()
            if ret is None or ret >> 8 == 0:
                for ln in ldd_output:
                    if 'curses' in ln:
                        readline_termcap_library = re.sub(
                            r'.*lib(n?cursesw?)\.so.*', r'\1', ln
                        ).rstrip()
                        break
                    if 'tinfo' in ln: # termcap interface split out from ncurses
                        readline_termcap_library = 'tinfo'
                        break

[andychu]

Oh maybe you can look in Nix's own Python package to get a clue? This problem should occur with Python, not just Oil.

You could also just build the tarball outside Nix, and build the binary inside Nix. Although I get that the point of Nix is to isolate dependencies like readline!

Long story but I tried to write a containerized package manager like 3 years ago, and I read the Nix paper a long time ago (but I haven't really used Nix). One of the things I didn't like was the non-standard build environment of Nix -- I recall that it did ld.so magic or something like that. Like intercepting system calls and rewriting them.

I think Nix was developed pre-Linux containers. But chroot() still existed. If they made a chroot for every package build rather than having /nix/store/<hash> I think that would go along way toward alleviating problems like this. The issue is that most packages assume various thing about the file system which as far as I understand Nix breaks.

But yeah I never finished my containerized package manager so I know it's a hard problem :)

[lheckemann]

Oh maybe you can look in Nix's own Python package to get a clue?

Of course, why didn't I think of that!

[andychu]

OK cool! Since this is for the dev build and not the actual Nix tarball build, I think you can just maintain a 2-line patch?

If there is more than one person developing Oil on Nix, we could add the patch, but for now I want to keep it more pristine for debuggability. Like I said, this whole app_deps thing is a little wonky and might go away in 6-12 months.

[lheckemann]

Yes, that's my plan. I'll just steal the relevant pieces from the python in nixpkgs.

More generally about nix, yes it breaks assumptions that a lot of software makes. IMHO that's that software's fault; people writing nix packages will not hesitate to patch the assumptions out and if appropriate try to get the patches accepted upstream. The benefits are worth it, I love my unprivileged package management, neatly mixed source/binary package model, how easy it is to package software, atomic upgrades that can easily be rolled back, multiple versions of the same software coexisting on the system, declarative configuration, etc etc. :)

[lheckemann]

It worked! Thanks for the help with this issue.

[andychu]

Yeah I like all of those properties of Nix. Definitely package management without root, easy access to source code, etc. (Although I wouldn't call it easy to package software if you have to debug upstream a lot.)

This is a tangent, but long term I want Oil to be a good language to write a package manager in. I haven't looked in awhile, but I remember I didn't really like Nix's C++. I don't really like the C++ in dpkg and apt-get too.

That might seem like a silly reason at first, but package managers are pretty fundamental for security and writing them in unsafe languages is a problem, e.g.:

https://www.twistlock.com/2017/06/25/alpine-linux-pt-1-2/

But yes Nix is probably the closest to what I want in principle, although along the way I learned way too much about apt-get and dpkg so I never really ran Nix. I think Nix and Oil share a philosophy -- it's Unix but also with some higher level principles, not just a bunch of hacks :)

---

Also I want Oil to be able to express a mix of declarative configuration (somewhat nginx-style) and imperative/functional shell code. From what I can tell, it's not too far from Nix's language. From what I remember it's a basically a functional language focused on computing tuples/records.

The declarative/imperative thing shows up a lot in:

• Makefiles: declarative dependencies vs build actions
• Nix config: there are embedded shell snippets that could be part of the language rather than strings
• Dockerfiles: also have embedded shell

There seems to be this pattern of shell in YAML lately too. So ideally you can do that in Oil and just use one language.

Anyway, that is far in the future, but I want to keep the goals for Oil in mind, as well as OSH.