jmmv / pkg_comp Goto Github PK
View Code? Open in Web Editor NEWAutomates the build of pkgsrc binary packages in a sandbox
License: BSD 3-Clause "New" or "Revised" License
Automates the build of pkgsrc binary packages in a sandbox
License: BSD 3-Clause "New" or "Revised" License
# unchanged from usual docs
pkg_add pkgin pkg_comp-cron
echo 'file:///var/pkg_comp/packages/All' >>/etc/pkgin/repositories.conf
pkgin update
# simple new way to install all the requested packages
pkgin import /var/pkg_comp/list.txt
(pkgin import
seems to obey the grep -v '^#'
rule, too.)
And I haven't had a chance to test it yet, but maybe if repositories.conf
had the local path as the only source, pkgin would even complain about missing packages (i.e. before trying to install them)!
I ran the pkg_comp auto
command on macOS Sierra 10.12.5:
# pkg_comp auto
and the pbulk build failed on the very first package (cwrappers):
Resolving...
Building...
Initialisation complete.
[1/417] Starting build of cwrappers-20170611
[1/417] Failed to build cwrappers-20170611
Building pkg_summary...
Building SHA512...
Build reports...
Sending report mail...
Uploading report...
Uploading packages...
pkg_comp: I: Destroying sandbox
The pbulk log shows that it built the package successfully, but the problem was in installing it; the following appears in bulklog/cwrappers-20170611/package.log
:
=> Creating binary package /pkg_comp/work/pkg/pkgtools/cwrappers/work/.packages/cwrappers-20170611.tgz
pkg_add: A different version of cwrappers-20170611 is already installed: cwrappers-20170112
pkg_add: 1 package addition failed
I'd find it very useful if pkg_comp
, still using sandboxctl
and requiring root to run, could be configured to create an unprivileged user inside the sandbox and do an --unprivileged
build as that user.
I'd use this in a VM configured to match the OS of a hosting provider where I have shell access but can't compile much of anything (thanks to process limits). I'd build binary packages in the VM, taking care that the sandbox $HOME
matches my hosting provider $HOME
, and then upload and install them.
Per my original comments:
It's still not clear to me how to see the output logs. It looks like only the cron job will log to /var/pkg_comp/log, but that's not clear from the blog post. ("...you can find the logs for all builds...") I also am still not clear on how to see the detailed logs from the individual packages, both during the build, and after. I found them once, but I don't remember where. (I know that's probably more of a pbulk issue, but it would help to explain it here.)
I think there's a 2x2 matrix of cases:
1a. During cron build
1b. After cron build
2a. During manual (auto -f
) build
2b. After manual build
Ideally it would always just be "here's where the active and/or residual logs are, from outside the sandbox" but maybe it's not that easy.
Per http://julio.meroh.net/2017/02/pkg_comp-2.0-tutorial-netbsd.html#comment-3187176460, it's not currently possible to bootstrap pkg_comp without pkgsrc or pkgconfig. It would be nice if it could be done.
My pkg_comp build system has been unable to build the erlang package since Erlang 24.
Erlang 24 introduced a JIT-compiler and uses that compiler during it's build process. The JIT-compiler implementation on NetBSD uses shm_open in virtmem.cpp. For erlang to build it needs a working shm_open setup, namely a correctly mounted /var/shm
mount.
By default the sandbox prepared by pkg_comp does not have /var/shm
mounted, hence the build fails.
My work around at the moment is the following hook inside sandbox.conf
:
pkg_comp_post_mount_hook() {
mount -t tmpfs -o rw -o -m1777 -o -sram%10 tmpfs /var/pkg_comp/sandbox/var/shm
true
}
Is that the best way to mount /var/shm
inside the sandbox? Could an option be added to pkg_comp as an alternative to the custom hook above?
NOTE: I have also submitted this as pkg/57158 becuase it may affect other packages & build systems.
Twice now I've uninstalled pkg_comp-cron and reinstalled it, and later realized that it has re-added the @daily cron task. I change it to @Weekly, so I end up with it configured twice.
It's not really surprising that it does that, but it would be nice if it would e.g. check if anything in the crontab already had pkg_comp4cron
in it, and just skipped adding it again.
Building individual packages via pkg_comp seems to end up overwriting the pkg_summary file(s) with just the packages that were built during that session, even if lots of other packages were already present in the package directory.
So if you e.g. just want to add a package to your system, and you build it via e.g. pkg_comp auto -f some/package
, and follow that with pkgin update
, you end up with a very short list of available packages.
I'm wanting to do my NetBSD 8.x package builds with clang. Newer NetBSD has clang in the base system, but 8.x doesn't, so I've built it from pkgsrc on the host.
What's a good way to get clang pkg_add'd in the sandbox and have it get used to build the bootstrap and all my packages?
On macOS Sierra 10.12.4, pkg_comp can't download the distfiles for packages via HTTPS. See sandboxctl issue #2 for why this is. This is problematic since a number of packages provide their distfiles via HTTPS only (e.g., devel/protobuf
and devel/googletest
). A workaround is to download the distfiles by hand outside of the sandbox and place them in /var/pkg_comp/distfiles
, but that's obviously a pain and far from automated.
I rebuild packages a lot sometimes and I don't want to wear down my disk more than I need to, so I always specify the work directory to point to a ram drive (also speeds up builds significantly). On my NetBSD systems I have a slightly modified pkg_comp which introduces a REAL_WORKDIR for this purpose.
It would be neat if pkg_comp 2.0 could do this out-of-the-box.
What I did: following the recent howto, updated my 1.x config to 2.0 and ran my usual weekly pkg_comp
build.
What I expected: roughly the same set of binary packages as last week's pkgtools/pkg_comp1
-powered build, including lang/php56
and a handful of extensions.
What happened instead: the same binary packages plus lang/php70
and extensions built against it and lang/php71
and extensions built against it.
Why it got in my way: my little script to pkg_add
everything failed due to conflicts between the various PHP versions.
What fixed it for me: adding NO_MULTI_PKG= #defined
to extra.mk.conf
so that pbulk
only builds the PHP I asked for.
What I'm suggesting for pkg_comp
: set this variable by default, and probably don't even worry about needing to allow it to be overridden by the user. As far as I can tell, the default behavior makes sense for typical pbulk
usage but is never desirable in pkg_comp
usage.
On macOS Sierra 10.12.4, executing the bootstrap
subcommand produces messages about mDNSResponder:
# pkg_comp bootstrap
pkg_comp: I: Bootstrapping pkg tools
pkg_comp: I: Setting up bootstrap in /opt/pkg using binary kit
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: service already loaded
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: service already loaded
pkg_comp: I: Setting up /pkg_comp/pkg.mk.conf
pkg_comp: I: Setting up /opt/pkg/etc/pkg_install.conf
pkg_comp: I: Bootstrapping pbulk tools
pkg_comp: I: Setting up bootstrap in /pkg_comp/pbulk using binary kit
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: service already loaded
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: service already loaded
pkg_comp: I: Setting up /pkg_comp/pbulk.mk.conf
pkg_comp: I: Setting up /pkg_comp/pbulk/etc/pkg_install.conf
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: service already loaded
===> Binary install for pbulk-0.65
=> Installing pbulk-0.65 from /pkg_comp/packages/pbulk/All
pkg_add: Warning: package `pbulk-0.65' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
pkg_add: Warning: package `rsync-3.1.2' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
mkdir: /etc/rc.d: Read-only file system
===========================================================================
The following files should be created for rsync-3.1.2:
/etc/rc.d/rsyncd (m=0755)
[/pkg_comp/pbulk/share/examples/rc.d/rsyncd]
===========================================================================
pkg_add: Warning: package `pbulk-base-0.53' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
pkg_add: Warning: package `digest-20160304' was built for a platform:
pkg_add: Darwin/x86_64 16.4.0 (pkg) vs. Darwin/x86_64 16.5.0 (this host)
pbulk-0.65: copying /pkg_comp/pbulk/share/examples/pbulk/pbulk.conf to /pkg_comp/pbulk/etc/pbulk.conf
pbulk-0.65 successfully installed.
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: service already loaded
Similarly for the build
subcommand:
# pkg_comp build devel/protobuf
pkg_comp: I: Adding packages to be built to (existent) pbulk.list
pkg_comp: I: Starting pbulk build in the sandbox
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: service already loaded
Scanning...
[...]
Building pkg_summary...
Building SHA512...
Build reports...
Sending report mail...
Uploading report...
Uploading packages...
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: service already loaded
Similarly for the sandbox-shell
subcommand:
# pkg_comp sandbox-shell
/System/Library/LaunchDaemons/com.apple.mDNSResponder.plist: Operation not permitted while System Integrity Protection is engaged
/private/var/pkg_comp/sandbox/private/tmp/com.apple.mDNSResponder.plist: service already loaded
sh-3.2#
https://github.com/truist/settings/blob/master/bin/util/pkg_comp_util.sh is a set of utility functions, including a "hygiene" function, to help catch errors/issues with the build builds. Maybe some of it will be useful to you.
https://github.com/truist/settings/blob/master/bin/server/pkg_comp_wrapper is my new interface to day-to-day usage of pkg_comp. It ensures I always start with a clean (but pre-created) sandbox, and then actually confirms that all the desired packages were built, and then leaves the sandbox behind (so I can inspect the errors) if they weren't. I'm also going to try running it (with no args) instead of pkg_comp4cron in my weekly cron job.
If that works, I think I'm going to fall back to bare pkg_comp instead of pkg_comp-cron. pkg_comp-cron was great for helping me get set up, but now that I've essentially merged the two packages (via shared configuration and a manual cron job), it seems extraneous.
All of this is just in the spirit of sharing ideas and "real-user" feedback; I don't have any specific critique or suggestion for pkg_comp. So feel free to close this issue; this was just the best way I knew to draw your attention to these things.
Per my comment on your blog post:
Then, because of that symlink (which I had to recreate, but I didn't realize that until after I'd gone a ways down the path without it), I ended up with an /etc/mk.comp (from bootstrap) that has an inside-the-sandbox path in it. That feels a little weird, but I haven't figured out a better option so I'm just living with it.
To rephrase: the mk.conf that is extracted from the bootstrap archive has .sinclude "/pkg_comp/pkg.mk.conf"
in it. That path only makes sense from inside the sandbox, but that file ends up in a place where it has an effect outside the sandbox. It's confusing, and worst-case someone could end up with a matching path and unintended consequences.
As per https://mail-index.netbsd.org/tech-repository/2017/06/10/msg000637.html, the old jsonn repos are going away.
On macOS Sierra 10.12.3, I followed the Easy pkgsrc on macOS with pkg_comp 2.0 tutorial (using pkg_comp-2.0-20170223-macos.pkg
) and then ran an auto build (as root):
# pkg_comp auto
(Just in case it matters, I actually ran two shorter auto builds before this where I listed a few packages on the command line after the auto
subcommand just to test that it works.)
It attempted to build 485 packages (some of which succeeded and some of which failed), and seemed to run OK and exited with an exit status of 0. However, when I went to review the build logs, I found that there were none:
% ls -alF /var/pkg_comp/log/
total 0
drwxr-xr-x 2 root wheel 68 Feb 23 16:05 ./
drwxr-xr-x 7 root wheel 238 Mar 22 04:59 ../
When you run
# pkg_comp -c <config> bootstrap
and you forgot to create the sandbox beforehand, it will give you a very misleading error message about a missing real path on mount (the error message is of course correct, but only a follow up of the actual root cause, the missing sandbox).
It would be great if there would be a sandbox-detection routine to either create the sandbox on the fly if it doesn't exist on calling bootstrap, or to at least give the user a proper error message like "please run pkg_comp -c sandbox-create, first".
The macOS installer comes with the program /usr/local/bin/bindfs, but not the shared library /usr/local/lib/libosxfuse.2.dylib that bindfs needs.
I'm guessing this is just an oversight in the packing list for the package.
The first version of pkg_comp 2.0 forced the use of a bootstrapped pkg_install for simplicity but this gets in the way of using binary packages on platforms where pkg_install is native (NetBSD). It also makes it impossible to mix pkg_comp-built packages with the official binary packages.
pkg_comp should be able to use the native pkg_install binaries for the final packages so that using those packages does not require having to use a different path to the package tools nor having to configure the host's pkg_path.conf
file.
To consider: are there any implications on the host's pkg_install.conf
file, for example, to properly configure e.g. PKG_DBDIR
to match whatever was done in the pkg_comp configuration?
After setting up pkg_comp-cron on a new host, I ran a full bulk build. It claimed success, and a visual scan of the output showed that all packages were built.
But when I went to install the packages, 5 of them (out of about 65) weren't built.
So I tried one of them (git-docs) with pkg_comp auto -f devel/git-docs
and it ran through the process, claimed that every package built successfully, and claimed overall success. But when I looked closely, the git-docs wasn't even listed as an attempted package.
Digging deeper, I couldn't find any build logs to explain what had happened. (Per issue #3.) I did try manually creating the sandbox (so it would be left behind) and running another build, and I still couldn't find any logs in the sandbox. I was able to find pbulk's reports, and report.txt
did show that there were errors, but it didn't show which packages actually had the errors. (This seems like a bug in pbulk, but I'm not sure.) I finally discovered that report.html
had more details than report.txt
had, and the underlying problem was that python wasn't building because it wanted X11 and I didn't have it installed. So apparently the python package was "masked" and that made dependent builds be automatically skipped.
I don't know if this is fixable via pbulk config; hopefully it is. But even if it is, I think pkg_comp still needs a way to verify if all the requested packages were actually built. It seems like pbulk can't be trusted to report success or failure reliably (at least not when building a specific list of packages), so some sort of post-build check is needed.
I have made earlier attempts at such a script; here is one and here is another that might be helpful.
As raised in my first comment to your blog post:
I was pretty confused about how the various mk.conf and extra.mk.conf files would come together. (And the bootstrap /usr/pkg/etc/mk.conf would have blown away my /etc/mk.conf if I hadn't already deleted /usr/pkg before extracting it [because of my symlink].) I don't think the post nor the docs really explain it well. I think it would help for you to just say "nearly everything you usually put in /etc/mk.conf should go in extra.mk.conf, except A, B, C".
devel/libidn2 2.0.3 is packaged from a .tar.lz
distfile.
As a one-off build directly from pkgsrc, it extracts fine. I've also been watching pkgsrc-bulk@
and not seeing any reported failures. That's why I suspect the following must be a bug in pkg_comp:
With my usual weekly build, I get the following in configure.log
:
=> Bootstrap dependency digest>=20010302: found digest-20160304
=> Checksum SHA1 OK for libidn2-2.0.3.tar.lz
=> Checksum RMD160 OK for libidn2-2.0.3.tar.lz
=> Checksum SHA512 OK for libidn2-2.0.3.tar.lz
===> Installing dependencies for libidn2-2.0.3
=> Tool dependency libtool-base>=2.4.2nb9: found libtool-base-2.4.6
=> Tool dependency lzip>=1.14: found lzip-1.19
=> Tool dependency pkgconf-[0-9]*: found pkgconf-1.3.5
=> Tool dependency checkperms>=1.1: found checkperms-1.11nb1
=> Build dependency cwrappers>=20150314: found cwrappers-20170611
=> Full dependency libunistring>=0.9.4: found libunistring-0.9.7
===> Overriding tools for libidn2-2.0.3
===> Extracting for libidn2-2.0.3
lzip: not found
tar: End of archive volume 1 reached
tar: Sorry, unable to determine archive format.
*** Error code 1
Stop.
bmake: stopped in /pkg_comp/pkgsrc/devel/libidn2
*** Error code 1
Stop.
bmake: stopped in /pkg_comp/pkgsrc/devel/libidn2
I can reproduce just as well with a simple pkg_comp auto libidn2
(though when I do it this way, the build logs disappear before I can save them off!).
Once a week or so, I run pkg_comp
targeting my NetBSD server. SANDBOX_TYPE=netbsd-release
, no X sets.
As of this pkgsrc/mk commit, my builds became incomplete, missing 100+ of the usual packages. (pkg_comp
reported no errors, so I only noticed the problem after attempting to update my system with the new package set.) The log showed pbulk
starting at [113/496]
instead of the expected [1/496]
.
Based on the above commit, I tried adding an explicit X11_TYPE=modular
in extra.mk.conf
. Indeed, this restored me to complete builds.
Possibly a second causal factor here is that freefonts
(one of the packages in my list) grew dependencies on encodings
and mkfontscale
since my last full build two weeks ago, perhaps forcing the X11 issue. Guessing this isn't particular to freefonts
but more generally to do with font handling in pkgsrc.
In the condition where pbulk
skips trying to build some packages, would it be possible for pkg_comp
to error out -- ideally right away?
Is there a helpful-but-not-too-clever way that pkg_comp
could auto-generate the needed X11_TYPE
setting? Or should users perhaps be required to explicitly set it?
I can't bootstrap on macOS High Sierra. I do the following:
$ sudo pkg_comp sandbox-create
$ sudo pkg_comp bootstrap
And the bootstrap fails like this:
===> Bootstrapping bmake
===> running: /bin/sh /pkg_comp/work/pkg/bootstrap/bin/install-sh -d -o root -g
wheel /pkg_comp/work/pkg/bootstrap/bmake
===> running: (cd /pkg_comp/work/pkg/bootstrap/bmake && /bin/sh configure --pre
fix=/pkg_comp/work/pkg/bootstrap --with-default-sys-path=/pkg_comp/work/pkg/boot
strap/share/mk --with-machine-arch=x86_64 )
checking for gcc... cc -isystem /usr/include
checking for C compiler default output file name...
configure: error: in `/pkg_comp/work/pkg/bootstrap/bmake':
configure: error: C compiler cannot create executables
See `config.log' for more details.
===> exited with status 77
aborted.
It seems that the C compiler does not work in the sandbox. I filed sandboxctl issue #7 for this.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.