gobolinux / compile Goto Github PK

Here is the output

/Data/Compile/LocalRecipes/Glamor-EGL  29s
❯ Compile Glamor-EGL
Compile: Locating a recipe for Glamor-EGL ...
Compile: Found recipe for Glamor-EGL 
GetRecipe: Trying to get /Data/Compile/LocalRecipes/Glamor-EGL/0.6.0
Compile: Recipe placed in /Data/Compile/LocalRecipes/Glamor-EGL/0.6.0
Compile: Runner is available but the sandbox can not be created.
Compile: Checking dependencies...
Compile: Compiling Glamor-EGL version 0.6.0.
Compile: Glamor-EGL will be installed as part of Xorg-Driver 7.7.
[...]
UnionSandbox: Cleaning up.
UnionSandbox: Moving entries to: /Programs/Xorg-Driver/7.7/.SandboxInstall_Root
SandboxInstall: Postprocessing Sandbox
Compile: Stripping executables...
SymlinkProgram: Directory /Programs/Glamor-EGL does not exist.

SymlinkProgram: Did you mean one of these? GParted
 Gegl
 Lame
tee: /Programs/Glamor-EGL/0.6.0/Resources/UseFlags: No such file or directory
Compile: Glamor-EGL 0.6.0 - Failed installing file.

If Runner is not found in the path hence not used it ends up in the correct place /Program/Xorg-Drive.

I will try to fix the logic after Runner finish compilation.

Probably is related with this: #5 but not sure

Hello lucas, hisham and everyone else,

How about giving Compile the ability to show which recipes are available at the gobo store?

Here is a syntax proposal BUT keep in mind, this is just a proposal, there can be other ways; I am more interested in the functionality and less in the syntax (though syntax of course also matters):

Compile htop --versions?

The above currently fails with a message such as:

Compile: Unknown option: --versions?

I propose that Compile may instead output this:

"The following recipes are available for htop (showing not more than the 5 latest)"

Htop 2.0.2-r1
Htop 2.0.1-r1
Htop 2.0.0-r1
Htop 1.0.3-r1
Htop 1.0.2-r1

I mention "5 latest" or "five latest" because it should not spam the user with like when there are 20 recipes.

Reasoning for this feature:

• Well, I think this is again mostly out of convenience. Perhaps you may want to or have to install an older version, such as python 2 versus 3, or different glib versions and so forth. This would also work offline :D - admittedly when one can browse the web, this feature is not so useful, but it may be mostly a small convenience feature I'd think.

The default behavior can't be to ignore the certificates when downloading files. It should be a flag that users enable by their own responsibility.

With the newest version of the Compile tool supporting using Runner to construct a sandbox during build (see this forum post [BACKUP]), focus now moves to verifying the various different Recipe types.

Test Recipes

Each category should be populated with 2-3 test recipes for verification.

See the Recipe Format Specification documentation for the supported types.

Each test should be conducted with updated versions of /Data/Compile/Recipes, /Programs/Compile/Current, /Programs/Scripts/Current on a system where all steps in the Known Issues and Fixes post-install guide have been completed. Note that the GrepReplace step in particular is important, otherwise build failures WILL occur!

Please use the command Compile --debug --pure <Recipe> and report any issues with a full build log.

Notes on Compile Dependencies

The new Dependencies required during Compile --pure runs should actually belong to
Compile/Resources/Dependencies.

We already have GCC and Python. Lua and CMake should be there too.

Also, the way that I implemented Compile --pure will allow any hand-written dependency file which states a
specific > version (or a range of versions) of a program to get prioritized over dependencies that simply
list the program name.

That is, if you add CMake to Compile/Resources/Dependencies and add CMake >= x.y.z to your Recipe,
the version requested by your Recipe will be honored.

First pass of the above was added in Compile PR #49.

Cabal

• (TBD)

Cmake

• Bullet (useful because of #34, which is fixed with Recipes PR 104)
• (TBD)

Configure

• Lazy (randomly chosen as a simple example)
• GCC (9.3.0 now builds on my end, but I need some guidance re. what belongs GCC deps and what belongs in Compile deps)
• Glibc (2.31 builds with the addition of Bison and Flex to BuildDependencies, see Recipes PR 103)
• Dracut
• Kmod

Makefile

• Linux (5.9.16 builds with some additions to BuildDependencies and some Dracut changes)
• (TBD)

Manifest

• (TBD)

Meson

• (TBD)

Meta

• (TBD)

Python

• (TBD)

Scons

• (TBD)

WAF

• Samba

xmkmf

• (TBD)

It seems that Compile, in the case of Pahole 1.18, does not download the second archive specified in the urls section.

To reproduce try Compile Pahole 1.18. It will fail with the message:

ls: cannot access '/Data/Compile/Archives/libbpf-0.2.tar.gz': No such file or directory

(If you are unable to reproduce, make sure libbpf-0.2.tar.gz is not already present under /Data/Compile/Archives/)

Looking at the output, It appears that Compile does not even attempt to download the archive.

When making a command like this:

NewVersion wifi-radar 2.0.s10 http://wifi-radar.tuxfamily.org/pub/wifi-radar-2.0.s10.tar.bz2

• the link is altered to 'http://wifi-radar.tuxfamily.org/pub/wifi-radar-2.0.s10.s10.tar.bz2' and obviously the recipe fails, because the source is not found.

The problem is when there are letters in the program version numbering ( s10 , i586 , x64 ).

It will get more complicated now with Gobo-016, since there will be the need for 32bit and 64bit recipe versions, with 'i686' or 'x86_64' or 'x64' in the version numbering.

Another example:

NewVersion jre 8u112_i586 http://ftp.osuosl.org/pub/funtoo/distfiles/oracle-java/jre-8u112-linux-i586.tar.gz

• the link will be changed to 'http://ftp.osuosl.org/pub/funtoo/distfiles/oracle-java/jre-8u112_i586-linux-i586.tar.gz'

I have found that some recipes contain multibyte characters and NewVersion is complaining and failing to generate a new version of the Recipe.

❯ NewVersion Wireshark 2.2.3 https://1.na.dl.wireshark.org/src/wireshark-2.2.3.tar.bz2
NewVersion: Locating a recipe for Wireshark...
NewVersion: Found recipe for Wireshark at /Data/Compile/LocalRecipes/Wireshark/2.2.0
GetRecipe: Trying to get /Data/Compile/LocalRecipes/Wireshark/2.2.0
NewVersion: Recipe placed in /Data/Compile/LocalRecipes/Wireshark/2.2.0
NewVersion: Creating recipe for Wireshark 2.2.3 based on 2.2.0.
Recipe: línea 2: Coincidencia: no se encontró la orden
Recipe: línea 2: Coincidencia: no se encontró la orden
NewVersion: Downloading source code...
/Data/Compile/LocalRecipes/Wireshark/2.2.3/Recipe: línea 2: Coincidencia: no se encontró la orden

When one builds a recipe featuring a Resources/Defaults/Settings directory, Compile will place those files under /Programs/Name/Version/Resources, and SymlinkProgram will take care of copying them over to /Programs/Name/Settings.

When a recipe does not include Resources/Defaults/Settings, however, "make install" will overwrite settings files from previous versions of the same package at /Programs/Name/Settings. It looks like the most sensible thing to do is to:

1. Let 'make install' copy settings files to /Programs/Name/Version/Resources/Defaults/Settings
2. Overwrite files at /P/N/V/Resources with a recursive copy of the recipe's default settings, if that exists
3. Invoke UpdateSettings through SymlinkProgram (we already do this)

This approach, at the very least, prevents problems with lost or missing settings files after the compilation of a frequently updated package.

When using overlayfs as unionfs implementation Compile may misbehave depending on the order of the dependencies listed in the Dependencies file. The problem goes down to the way in which overlayfs merges two given directories. For instance:

$ mount -t overlay overlay -o lowerdir=/Programs/LibXFCE4Util/4.14.0/include:/Programs/XFConf/4.14.3/include,upperdir=/tmp/upper_layer/include,workdir=/tmp/write_layer/include /Mount
$ ls /Mount/
xfce4
$ ls /Mount/xfce4 
libxfce4util

When the arguments are swapped then the contents of both packages are shown under the mountpoint:

$ mount -t overlay overlay -o lowerdir=/Programs/XFConf/4.14.3/include:/Programs/LibXFCE4Util/4.14.0/include,upperdir=/tmp/upper_layer/include,workdir=/tmp/write_layer/include /Mount
$ ls /Mount/
xfce4
$ ls /Mount/xfce4
libxfce4util  xfconf-0

This is with kernel 5.6.10-Gobo. We need to test with a recent version before contacting upstream or coming up with changes to our tools.

As reported by Stéphane Aulery on gobo-devel:

This code [1] accepts a lot of archive formats but this code is narrower [2]. Expected ?
[1] https://github.com/gobolinux/Compile/blob/master/Functions/Compile#L88
[2] https://github.com/gobolinux/Scripts/blob/master/Functions/Archive#L7

When both meson and ninja are installed with Alien --install PIP3:<name>, Compile fails to import them when building something with recipe_type=meson.

This could be due to PYTHONPATH being reset when we launch Compile with Runner, but it needs more investigation.

The docs docs=() directive does not appear to do anything at all, unless I am mistaken.
It seems to be broken.

See package GLM.

More and more projects seems to be moving to releases that are basically a version tag on services like github and gitlab, so it would be nice if Gobolinux would learn download them as a versioned recipe rather than a generic git recipe.

Compile Ruby
Compile: Locating a recipe for Ruby ...
Compile: Found recipe for Ruby
GetRecipe: Trying to get http://gobolinux.org/recipe-store/Ruby--2.4.2-r1--recipe.tar.bz2
GetRecipe: Downloading recipe from http://gobolinux.org/recipe-store/Ruby--2.4.2-r1--recipe.tar.bz2
Compile: Recipe placed in /Data/Compile/Recipes/Ruby/2.4.2-r1
Compile: Checking dependencies...
Compile: Compiling Ruby version 2.4.2, revision r1.
--2018-01-11 21:34:42-- https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.2.tar.gz
Resolving cache.ruby-lang.org... 2a04:4e42:1b::434, 151.101.113.178
Connecting to cache.ruby-lang.org|2a04:4e42:1b::434|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14187859 (14M) [application/octet-stream]
Saving to: 'ruby-2.4.2.tar.gz'

ruby-2.4.2.tar.gz 100%[=======================================================================================>] 13.53M 7.38MB/s in 1.8s

2018-01-11 21:34:52 (7.38 MB/s) - 'ruby-2.4.2.tar.gz' saved [14187859/14187859]

Compile: Unpacking file /Data/Compile/Archives/ruby-2.4.2.tar.gz...

PrepareProgram: Preparing...
PrepareProgram: Autoconf configure script detected.

checking for ruby... /System/Index/bin/ruby
config.guess already exists
config.sub already exists
checking build system type... mkdir: cannot create directory '/Depot/Temp/cg9251-32697': No such file or directory
mkdir: cannot create directory '/Depot/Temp/cg-9251': No such file or directory
config.guess: cannot create a temporary directory in /Depot/Temp
configure: error: cannot guess build type; you must specify one
PrepareProgram: configure failed.
Compile: Ruby 2.4.2 - Configuration failed.

^^^ the above is on a fresh install on a laptop; sorry for the typos, typing on
this thing is AWFUL ...

Anyway, I usually use /Depot/Temp for temp files

in the case above, I forgot to create it

My suggestion to fix this:

• Have Compile check whether a target directory exists, before
  continuing. Right now it fails with "cannot guess build type;
  you must specify one" but I believe this message is bogus
  since the real error is that the directory does not exist yet.

Consider updating Resources/Environment, as that file may hold a string with the installation path.

It would be nice to have the option to tell Compile to continue the installation a package after it has been built successfully. Too many times a package like Glibc has too many file exists errors due to symlink conflicts from its previous installation. It like playing a game of Cat and Mouse where each time a silly failure like that occurs, I have to manually remove the symlink and rebuild again, hoping it will succeed the next time until it actually does. I have to rebuild Glibc over 10 times now because of this.

I am on a freshly-installed GoboLinux system, and I have never used GoboLinux before.

When I run, sudo Compile LXDM, it responds with this:

Password: 
Compile: Initializing recipes repository
Cloning into '/Data/Compile/Recipes'...
[...hangs for several minutes...]
fatal: unable to access 'https://github.com/gobolinux/Recipes.git/': getaddrinfo() thread failed to start

fatal: not a git repository (or any of the parent directories): .git
Compile: Could not find a recipe for LXDM

When I sudo git clone the Recipes repository by hand, it simply says Cloning into 'Recipes'... and hangs.
Even when I git clone as a regular user into my home directory, it hangs.
I have also tried git clone with a smaller repository, and that hangs as well.
I have found that when I curl anything, that hangs too.
Perhaps this is an incompatibility with the WiFi service?
Thank you in advance!

Please see this gobolinux/Scripts#2

how to make it all work and make use of this can some body give me an simple example of use

Motivation

Compile has a long history -- and if we're being honest, it tends to show in the mishmash of styles in the scripts.

Since the plan is to stay with Bash for the most part and only port the more complex scripts to Python3 (per @lucasvr ), it seems like it would make sense to go through all scripts and update them to Bash-5.0 compatible syntax to the benefit of readability and uniformity.

If, during the cleanup process, it turns out that small changes can affect speedups or simplification, that'll just be a nice side benefit.

Style Guide references:

• https://wiki.gobolinux.org/Overview/GoboLinux-Scripts/Guidelines-for-script-authors/index.html
• https://www.daveeddy.com/bash/ (also at https://github.com/bahamas10/bash-style-guide)

List of Compile scripts

• ApplyVariables - nothing major, just `foo` to $(foo)
• AutoPatch - mostly test/conditions (([ "$x" = "y" ] to [[ "$x" == "y" ]]) and -a/-o to &&/||) and a smattering of `foo` to $(foo)
• ColorMake - single test/condition change
• Compile
• ContributeRecipe
• EditRecipe
• FetchArchive
• GetRecipe
• GoboPath2Ruby
• MakeRecipe
• NewVersion
• NoRecipe
• PrepareProgram
• RecipeLint
• SandboxInstall
• UnionSandbox
• UpdateRecipes

Notes

There doesn't appear to be a rigidly enforced approach to variables -- I've seen all of $var vs. "$var" vs. ${var} vs. "${var}" so far. "$var" and "${var}" are apparently equivalent and I happen to like the latter more as it tends to stand out in code as a variable and works everywhere.

The same can be said about var=`foo` vs. var="`foo`" vs. var=$(foo) vs. var="$(foo)". It seems that var=$(foo) is safe, so that's what I'm going with until something breaks (at which point I'll do "$(foo)" instead).

Certain packages fail to build if they are already installed under /Programs/*.

It can be reproduced by installing LibBPF 0.2 and trying to reinstall it using Compile LibBPF 0.2. In the presented scenario, the installation will fail with the following error:
real_install: cannot change permissions of ‘/usr/include/bpf’: No such file or directory.

Doing a RemoveProgram LibBPF 0.2 prior to calling Compile LibBPF 0.2 will make it work.

Unless a user already knows this, this can be a very frustrating experience.

Hello lucas and hisham and everyone else,

I just had run Compile but python 3.x linked in. Compile complained but it also showed
the following:

Compile: Your Scripts package is too old. Please update it by running 'InstallPackage Scripts'.

My suggestion is:

(1) Perhaps Compile itself could also have a way to update (itself and/or also the Scripts collection)

I know from the ruby issue tracker that the ruby core team often wants explanations or even more importantly, a use case as to why, so here goes.

• I believe that this is just a convenience feature, nothing that important either way. Out of convenience, I think it may be useful if people could ALSO use e. g. "Compile --update-scripts" or "Compile --update-itself" or something like that. While it is admittedly not a real problem, since people can just use "InstallPackage Scripts" instead anyway, I thought it may be convenient. Of course you can also say that InstallPackage has to be used instead, which is perfectly fine too - making a distribution is about making choices and decisions. Thank you for reading, please feel free to close this issue at any moment in time.

Hi,

I write this down on behalf of ThePub from the IRC channel.

He commented that the scripts were moved to git + github, which is fine, but there was no "scripts upgrade path" to get git repository support in the current system available. (I think he referred to 015 default install).

The git-move happened after the 015 installation .iso, so it is understandable that it offers no git-support, but perhaps that could be added? Some way that people can make use of git-update of components (in the event that bugs were fixed etc...)

In the past I think this happened when the Compile-Recipe was updated, perhaps this could now happen via git (no idea if anything else changed too).

The idea would be to have the Gobolinux Scripts work reliably.

Perhaps something like Compile --sync or something, or compile --git - no idea about the specific command.

I've been trying to compile the Ranger recipe on GL 016.01 in a VirtualBox VM (Win 7 host), but it fails with these messages:

"/usr/bin/env: 'python': No such file or directory
Compile: Your Scripts package is too old. Please update it by running 'InstallPackage Scripts'."

I've confirmed that I have Python 2.7 installed using the 'which' command, and I've also confirmed that I have Scripts 016.01 installed by looking in the /Programs directory. So I don't know why these errors are happening, but they are preventing the package from compiling.

Output from SystemInfo:
Kernel: Linux 4.9.16-Gobo x86_64
Compilation: #0 SMP PREEMPT Mon Mar 20 08:53:55 BRT 2017
Processor: Intel(R) Core(TM) i7-3940XM CPU @ 3.00 GHz
Memory: 16423312 kB

I'm not sure if this is the right place to bring this up, or if I should go to the recipe maker, or the Ranger devs themselves. If you need more information, please let me know. This is preventing me from installing the recipe, so any help you can give will be much appreciated. Thanks!

Would be good to know. Hopefully a permissive one for those of us who'd like to incorporate the GoboLinux approach to the filesystem under *BSD.

I've been experimenting with building my own updated ISO of GoboLinux using GoboALFS and this bug has been inconvenient for when building meta recipes like Xorg-Lib, where certain packages within the meta recipe depend on one and another and it requires that each individual package to be symlinked each time so that dependent programs/libs can detect the dependencies.

To temporarily get around this problem, I had to manually use Compile for each recipe in the meta recipe and have it detect and build automatically for the meta recipe sub-directory in /Programs.

I just fresh install gobolinux and completed all instuctions in GoboLinux-017-Known-Issues-and-Fixes.
but i see Missing import: UnionFS if use Compile util...

I'm unsure if this should be posted under Recipes, Compile, or Scripts. if this is the wrong place for it feel free to move it or lmk.

Feature Request

Having an internal versioning system for recipes, as a way to check if the recipe you used has had any changes since you compiled the program.

Potentially through logging the recipe's source and commit to a Resources file.

Issue

Say I install Bash 5.1 from Recipe, and then a few days later an issue is found+fixed in the recipe. Currently I'd have no way of knowing (w/o manual checking of dates) if I used the version before or after the Recipe's change.

This issue is found first hand with the installed version of Bzip2 1.0.8 provided in Gobo017, having used an outdated Recipe file, leading to some programs failing to find it's shared library.

Proposed Method

Having Compile (or an adjacent script) log some information about the Recipe, for exanple using git log [FOLDER_PATH] |head -1 > $target/Resources/RecipeCommit to get the commit.

Then be able to check it against the current version in Recipes as follows:

local name="$1"     # program name
local version="$2"  # program version

local recipesfolder="/Data/Compile/Recipes/${name}/${version}"
local installedfolder="/Programs/${name}/${version}"

local recipescommit=$(git log "${recipesfolder}" |head -1)
local installedcommit=$(cat "${installedfolder}/Resources/RecipeCommit")
if [[ "${recipescommit}" == "${installedcommit}" ]]
then
   echo "up to date"
else
   echo "out of date"
fi

Ideally, a bit more than just the commit would be logged, potentially the git repository/fork, branch, and commit; just something concrete, fast to generate, and easy to compare.

Hello,
I've encountered this issue updating the bullet recipe.
The saved log says:

Compile: Unpacking file /Data/Compile/Archives/bullet-2.87.tar.gz...
Compile: Directory /Data/Compile/Sources/bullet-2.87 not found.

The recipe is

# Recipe (MakeRecipe) for Bullet by Filipe Vieira, on Fri Jan 18 20:42:37 WET 2008
compile_version=1.10.0
url="https://github.com/bulletphysics/bullet3/archive/2.87.tar.gz"
file_size=56691047
file_md5=7566fc00d925a742e6f7ec7ba2d352de
file="bullet-2.87.tar.gz"
recipe_type=configure
autogen_before_configure=yes

pre_link()
{
   # Enlightenment 18 will ask for the missing headers below
   for i in ./src/BulletSoftBody/btSoftBodySolvers.h \
            ./src/BulletSoftBody/btDefaultSoftBodySolver.h \
            ./src/BulletSoftBody/btSoftBodySolverVertexBuffer.h \
   do
      cp -v $i $target/include/bullet/BulletSoftBody
   done
              
   cp -v ./src/BulletDynamics/ConstraintSolver/btFixedConstraint.h \
      $target/include/bullet/BulletDynamics/ConstraintSolver
}

$ Compile XProto 7.0.31
...
Compile: Compiling XProto version 7.0.31.
Compile: XProto will be installed as part of Xorg-Proto 7.7.
...
SandboxInstall: Postprocessing Sandbox
Compile: Stripping executables...
SymlinkProgram: Directory /Programs/XProto does not exist.