archhaskell / cblrepo Goto Github PK

This is necessary for maintaining repositories that depend on [haskell-core]:

cblrepo add -d <pkg>,<ver>_<xrev>,<rel>

Consequently the x-revision should also appear in the dependency list of generated PKGBUILDs.

I am working in archhaskell/habs, trying to add a few packages.

Rebuilding every package in the docker container, I repeatedly runs into the following sort of issue:

:: Proceed with installation? [Y/n]
:: Retrieving packages...
 haskell-extensible-excepti...    31.2 KiB  0.00B/s 00:00 [###############################] 100%
[...]
:: Processing package changes...
(1/2) installing haskell-extensible-exceptions            [###############################] 100%
[...]
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Downloading alsa-core-0.5.0.1.tar.gz...
[...]
  -> Found original.cabal
==> Validating source files with sha256sums...
    alsa-core-0.5.0.1.tar.gz ... Passed
    original.cabal ... Passed
==> Extracting sources...
  -> Extracting alsa-core-0.5.0.1.tar.gz with bsdtar
bsdtar: Failed to set default locale
==> Starting prepare()...
==> Starting build()...
Configuring alsa-core-0.5.0.1...
Setup: Encountered missing dependencies:
extensible-exceptions >=0.1.1 && <0.2
==> ERROR: A failure occurred in build().
    Aborting...
*** ERROR: Trapped ERR, something went very wrong.

In this example, despite extensible-exceptions having been installed from a freshly built package, the next package build can't find it. I can work around the issue by restarting the container and manually running the register.sh script for that package, after which the build for that package succeeds, only to fail on the next similar issue.

$ docker start blissful_mcnulty
$ docker exec -it blissful_mcnulty sudo bash -x /usr/share/haskell/haskell-extensible-exceptions/register.sh
$ docker attach blissful_mcnulty
==> Finished making: haskell-alsa-core 0.5.0.1.x0-252 (Sat Feb 17 04:44:01 UTC 2018)                                                                                                             
==> Extracting database to a temporary location...                                                                                                                                               
bsdtar: Failed to set default locale                                                                                                                                                             
==> Extracting database to a temporary location...                                                                                                                                               
bsdtar: Failed to set default locale                                                                                                                                                             
==> Adding package 'haskell-alsa-core-0.5.0.1.x0-252-x86_64.pkg.tar.xz'     

Digging down, I can see that haskell-extensible-exceptions (created with cblrepo pkgbuild extensible-exceptions) has a haskell-extensible-exceptions.install which should call the register.sh script properly in the post_install hook. However, the PKGBUILD for this module never mentions the install script, which is therefore ignored.

This seems to be an issue with cblrepo, which fails to add this install section to the PKGBUILD.

The value ghcPkgs (https://github.com/magthe/cblrepo/blob/master/src/Util/Translation.hs#L331) ought to be completely useless now that the GhcPkg type exists in the database.

-> haskell-alsa-core
-> haskell-alsa-mixer
-> haskell-hinotify
-> haskell-libmpd
-> haskell-stm
-> haskell-timezone-olson
-> haskell-timezone-series

The haskell-libmpd would be nice to have for xmonad-extras (control your music player from XMonad), as well.

Cblrepo generates pkgbuild based on local, not the cabal file, so it cause some dependency errors.

The following example command from the README doesn't work anymore, because it requires xrev to be specified. Shouldn't xrev really be optional when there might not necessarily be one, or should the README be changed?

$ cblrepo add -d xmonad-contrib,0.10,1 -d zlib,0.5.3.1,2.1

Currently pkgbuild requires that all packages are on Hackage. Adding a flag (.e.g. --alternative-source), where cblrepo can look for sources itself would allow working with non-Hackage packages.

Only pkgbuild should need to be modified: the alternate source should be consulted before generating the PKGBUILD.

From xyne:

• srcdir and pkgdir variables should be properly quoted in PKGBUILD output

It was proposed at tensor5/haskell-happstack#11 to run gitit with systemd without superuser privileges. For this purpose it would be necessary to create a gitit user and group, and add the CAP_NET_BIND_SERVICE capability to the executable /usr/bin/gitit. This would be done in gitit.install, thus it requires the ability to handle <package>.install patches, to a way similar to how it's done with <package>.pkgbuild patches.

Thanks

The current scheme

<pkgver>_<xrev>-<pkgrel>

is not good. Real-world examples of it are

warning: haskell-vector-algorithms: local (0.7_1-2) is newer than haskell-core (0.7.0.1_0-1)
warning: haskell-monadrandom: local (0.4_2-1) is newer than haskell-core (0.4.1_0-1)

A better scheme was suggested on the Arch Linux mailing list:

<pkgver>.x<xrev>-<pkgrel>

Currently we ghc-pkg unregister then register. ghc-pkg has update option that does 2 actions in one step, possibly saving installaton time.

As there's no runhaskell Setup update option, update.sh can be generated from the generated register.sh just by replacing register with update.

If you make a new patch, it would be handy if you could increase the pkgrel. I think something like storing the pkgrels in cblrepo's database, and a command like 'cblrepo increl pkg-name'. pkgrel could be reset with cblrepo add.

Some packages require preprocessing of the source with an external tool. For example clckwrks.cabal contains the line

Build-tools: hsx2hs

cblrepo doesn't consider build dependencies, and it can happen that cblrepo build base outputs clckwrks before hsx2hs.
Also cblrepo pkgbuild should include a makedepends=("haskell-hsx2hs") line in the resulting PKGBUILD.

Thanks

cblrepo pkgbuild puts dependencies it detects as is, which is wrong. There is pkgfile tool that finds which package if any provides a given file.

Calls of pkgfile /usr/lib/lib$1.so will avoid many .pkbuild patches in habs:

• GLURaw
• OpenGLRaw
• Unixutils
• X11 (3 of 4)
• digest
• persistent-sqlite
• ...

cblrepo rm hkjhkhk should display that the package isn't there

Hi,

I wrote a pkgbuild patch for gitit that adds two files to the package, gitit.conf and gitit.service:

Index: haskell-happstack/haskell-gitit/PKGBUILD
===================================================================
--- haskell-happstack.orig/haskell-gitit/PKGBUILD
+++ haskell-happstack/haskell-gitit/PKGBUILD
@@ -52,13 +52,22 @@ depends=("ghc=7.10.1-1"
          "haskell-uuid=1.3.10_0-7"
          "haskell-xml=1.3.14_0-80"
          "haskell-xss-sanitize=0.3.5.5_0-7"
-         "haskell-zlib=0.5.4.2_0-76")
+         "haskell-zlib=0.5.4.2_0-76"
+         "mime-types")
+optdepends=('git: Git filestore support'
+            'darcs: Darcs filestore support'
+            'mercurial: Mercurial filestore support'
+            'postfix: notification emails support')
 options=('strip' 'staticlibs')
 source=("http://hackage.haskell.org/packages/archive/${_hkgname}/${_ver}/${_hkgname}-${_ver}.tar.gz"
-        "original.cabal")
+        "original.cabal"
+        "gitit.conf"
+        "gitit.service")
 install="${pkgname}.install"
 sha256sums=("c94271744ec5fe846e1b3994fc0349f4a38120a8d60618bedc1909367acf63a0"
-            "e1f3c4b311d9e03575fd15ad2b00fa6b0eecffd634a976337846ba9048db8094")
+            "e1f3c4b311d9e03575fd15ad2b00fa6b0eecffd634a976337846ba9048db8094"
+            "d39af21ac72bbba302d76d3f0798665a4cedfc3fd8341987903682afbfe36286"
+            "1b163ed91af72cc15eed9cbbf9309631417c5fe576cae582e4d268e3bbd42ee7")

 # PKGBUILD functions

@@ -93,4 +102,10 @@ package() {
     runhaskell Setup copy --destdir="${pkgdir}"
     install -D -m644 "${_licensefile}" "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE"
     rm -f "${pkgdir}/usr/share/doc/${pkgname}/${_licensefile}"
+
+    install -d -m755 "${pkgdir}/etc"
+    install    -m644 "${srcdir}/gitit.conf" "${pkgdir}/etc"
+
+    install -d -m755 "${pkgdir}/usr/lib/systemd/system"
+    install    -m644 "${srcdir}/gitit.service" "${pkgdir}/usr/lib/systemd/system"
 }

However cblrepo pkgbuild fails to generate the PKGBUILD with this patch:

Failed patching /tmp/cblrepo.xs2Rim/PKGBUILD with /home/nicola/haskell-happstack/patches/gitit.pkgbuild

I think this has to do with sha256sums lines.
Is there a fix for this, or I should write a gitit.source patch, adding the two extra files to the source instead?

Currently cblrepo upgrades always shows stack-9.9.9 which was explicitly filtered out. It would be nice to comply with the given preferred-versions.
stack: 1.3.2:x0 (9.9.9:x4)

The file stack/preferred-versions in the index contains:
stack <9.9.9 || >9.9.9

cblrepo pkgbuild generates a PKGBUILD with "ghc=7.4.1-1" in makedepends, but the current official version of GHC in Arch is 7.4.1-2, so the PKGBUILD must be edited/patched for each package that cblrepo generates.

Can you either bump the hard-coded version number to match the current version, or better yet, have cblrepo detect the current version of GHC instead of hard-coding it?

Hi, is there a way to sync cblrepo.db with the currently installed --ghc-pkg and --distro-pkg? The use case is creating PKGBUILDs for local or AUR use, assuming [haskell-core] and [haskell-web] origin for everything else. But more generally, some clever combination of ghc-pkg --list and pacman (...) should be able to do this? Or am I using a completely misunderstanding the intended workflow?

Currently full upgrade time (e.g. when a new ghc is released) is quadratic in the number of packages as haddock must scan N packages N times to update the documentation index.

It takes a netbook more than 3 hours to pacman -Suy with 324 Haskell packages to upgrade. It is an extreme case of course, but it is not pacman but haddock that takes unreasonable amount of time.

The right solution is to wait for pacman post-transaction hooks proposal to be implemented.

Until that we can implement a trick with starting backround jobs or a similar approach to avoid running haddock many times. Or we can just disable the /usr/share/doc/ghc/html/libraries/arch-gen-contents-index line in the generated .install files.

You can do a benchmark on a decent machine to see if performance improvements are substantial, and if they are (e.g. 10 minutes vs 1) then I think having to run arch-gen-contents-index by hand is a good trade off. After all, not everybody needs the index or local documentation in the Internet age. It is good to have the documentation shipped as Hackage web documentation is often broken, but I think lack of auto-indexing can be tolerated in favor of lightning fast installation/upgrade performance.

It seems that cblrepo only patches during additions. Once a package is in cblrepo.db its constraints are set in stone and cannot be altered without losing pkgrel.

Here are minimal files to reproduce the issue:

supervisor-0.1.cabal:

name: supervisor
version: 0.1

library
    build-depends: extra ==0.1.*

extra-0.1.cabal:

name: extra
version: 0.1

extra-0.2.cabal:

name: extra
version: 0.2

First we install extra-0.1 and supervisor-0.1 and bump them so pkgrels are not 1:

$  cblrepo add -f extra-0.1.cabal -f supervisor-0.1.cabal
$ cblrepo bump --inclusive supervisor extra
$ cblrepo list
extra  0.1-2
supervisor  0.1-2

Now extra-0.2 is out, and it turns out that the constraint in supervisor-0.1 cabal is superficial as it is still compatible with extra-0.2. So I prepare a patch and put it to patches/supervisor.cabal:

Index: cblrepo-bug/supervisor-0.1.cabal
===================================================================
--- cblrepo-bug.orig/supervisor-0.1.cabal
+++ cblrepo-bug/supervisor-0.1.cabal
@@ -2,4 +2,4 @@ name: supervisor
 version: 0.1

 library
-    build-depends: extra ==0.1.*
+    build-depends: extra

Now I try adding extra-0.2.cabal and it fails:

$ cblrepo add -f extra-0.2.cabal 
Adding extra 0.2 would break:
  supervisor : extra ==0.1.*

If I add old unpatched supervisor-0.1.cabal along with extra-0.2 it works because the patch is applied:

$ cblrepo add -f extra-0.2.cabal -f supervisor-0.1.cabal

But pkgrel of supervisor is reset:

$ cblrepo list 
extra  0.2-1
supervisor  0.1-1

Is there a way to update to extra-0.2 without losing pkgrel of supervisor?

If it's not possible then one way to fix is to repatch all dependencies of the package being added during addition.

It would be nice if .install files could be patched. I also think cblrepo should generate the install= line automatically in the PKGBUILD if a patch is found.

If I need to delete multiple dependent packages, it doesn't work unless I pass them in reverse build order:

$ cblrepo rm distributed-process-async distributed-process-client-server \ 
distributed-process-extras distributed-process-supervisor distributed-process-task
Can't delete package distributed-process-async, dependants:
  distributed-process-client-server
  distributed-process-supervisor
  distributed-process-task

$ cblrepo build distributed-process-async distributed-process-client-server \
distributed-process-extras distributed-process-supervisor distributed-process-task \
| tac | xargs cblrepo rm
$

The situation has a bigger impact with additions. In this case async, client-server,
extras and supervisor packages have to be updated at once. Updating one by one is not possible because of incompatibility. And there is 5th package task that doesn't upgrade because it's compatible with both versions so it has to be just bumped.

$ cblrepo updates
distributed-process-async: 0.2.0 (0.2.1)
distributed-process-client-server: 0.1.1 (0.1.2)
distributed-process-extras: 0.1.1 (0.2.0)
distributed-process-supervisor: 0.1.1 (0.1.2)
$ cblrepo add distributed-process-async,0.2.1 distributed-process-client-server,0.1.2 \
distributed-process-extras,0.2.0 distributed-process-supervisor,0.1.2
Failed to satisfy the following dependencies for distributed-process-async:
  distributed-process-extras >=0.2.0 && <0.3
Failed to satisfy the following dependencies for distributed-process-client-server:
  distributed-process-async >=0.2.1 && <0.3
  distributed-process-extras >=0.2.0 && <0.3
Failed to satisfy the following dependencies for distributed-process-supervisor:
  distributed-process-client-server >=0.1.2 && <0.2
  distributed-process-extras >=0.2.0 && <0.3
Adding distributed-process-extras 0.2.0 would break:
  distributed-process-task : distributed-process-extras >=0.1.1 && <0.2

As it can be seen from above, I can't add then bump, but I have to perform a more complex procedure of removal, addition (in build order) and manual correction of task revision (because it was 2 in the beginning and after addition it's again 1 instead of correct 3):

$ cblrepo build distributed-process-async distributed-process-client-server \
distributed-process-extras distributed-process-supervisor distributed-process-task \
| tac | xargs cblrepo rm
$ cblrepo add distributed-process-extras,0.2.0 distributed-process-async,0.2.1 \
distributed-process-client-server,0.1.2 distributed-process-supervisor,0.1.2 \
distributed-process-task,0.1.0
$ cblrepo bump distributed-process-async distributed-process-client-server \
distributed-process-extras distributed-process-supervisor 
$ cblrepo list distributed-process-task
distributed-process-task  0.1.0-1 (-perf)
$ vim cblrepo.db
$ cblrepo list distributed-process-task
distributed-process-task  0.1.0-3 (-perf)
$

The atomic addition should proceed like this:

1. add the packages in the list, ignoring constraints
2. check if constraints are satisfied for each added package in new version context
3. revert all if not satisfied

Same for removal, but for removal there is an alternative approach of topological sorting, I'm not sure which is easier.

Currently there is nothing that warns about converting a base package to a non-base package. This causes issues like issue 53 in archhaskell/habs.

The easiest possible fix is to use lastNote from safe package to improve error message. Currently it is:

$ cblrepo versions -l kdjk
kdjk,cblrepo: Prelude.last: empty list

See http://www.well-typed.com/blog/2015/07/hackage-security-alpha/.

From Xyne:

• there is a typo in the help message for the convertdb command: "convert and old" -> "convert an old"

Hi,

I'm trying to generate PKGBUILDs for my repo, but I get a sequence of:

makepkg: error while calculating the source hashes for <mypkg>

Does this have something to do with pacman 5?

Thanks

The reason seems to be that cblrepo add (and cabal unpack btw) convert line endings in .cabal file but bsdtar doesn't.

Anyway here is a minimal example. Run it in an empty folder.

cblrepo add -g array,0.5.0.0 -g base,4.7.0.2 -g containers,0.5.5.1 \
     -g ghc-prim,0.3.1.0 -g integer-gmp,0.5.1.0 -d mtl,2.1.3.1,5 -d random,1.1,1
cblrepo add arithmoi,0.4.1.1 # fails because Hackage version requires random 1.0
mkdir patches
quilt new arithmoi.cabal
cabal unpack arithmoi
quilt add arithmoi-0.4.1.1/arithmoi.cabal
sed -i 's/< 1.1/< 1.2/g' arithmoi-0.4.1.1/arithmoi.cabal
quilt refresh
cblrepo add arithmoi,0.4.1.1 # succeeds
cblrepo pkgbuild --ghc-version=7.8.4 arithmoi
cd haskell-arithmoi
makepkg # fails with Hunk #1 FAILED at 38 (different line endings).

It is possible to prepare a patch that satisfies makepkg: just replace cabal unpack with

wget http://hackage.haskell.org/package/arithmoi-0.4.1.1/arithmoi-0.4.1.1.tar.gz
bsdtar -xzvf arithmoi-0.4.1.1.tar.gz

but now cblrepo add fails with Failed patching /tmp/cblrepo.0dE6Cj/arithmoi.cabal with patches/arithmoi.cabal

Currently I have added Cabal==1.24.0.0 as GHC pkg in cblrepo.db, but cabal-install 1.24.0.2 appears in cblrepo upgrades and I think it shouldn't.

The definition in cblrepo.db:
["Cabal",{"GhcPkg":{"gpVersion":"1.24.0.0"}}]

The constraint in cabal-install/1.24.0.2/cabal-install.cabal:
Cabal >= 1.24.2 && < 1.25,

And the output of cblrepo upgrades includes:
cabal-install: 1.24.0.0:x2 (1.24.0.2:x0)

Few packages are just umbrellas and don't have any .hs files. Cabal builds them just fine but cblrepo chokes on haddock.

A notable example is diagrams meta-package.

This is the current error message:

[magnus@vs0 habs]$ cblrepo pkgbuild dataenc hashed-storage hS3
cblrepo: /home/magnus/.cblrepo/00-index.tar.gz: openFile: does not exist (No such file or directory)

Surely it can be improved!

See http://www.well-typed.com/blog/2015/07/cabal-setup-deps/.

Am I right in thinking that if we include a PKGBUILD patch file in a tarball on Hackage, it will automatically be applied to the one generated from the .cabal file? This is specifically for packages to be included in [haskell-core] on Arch.

I propose to have configuration files to substantially reduce the need for patches, as patches are brittle. As can be seen from recent GHC update, version updates require recommitting of patches as they contain version numbers.

Patches are a good idea because they are universal and allow any type of modification. However, it turns out that most often the only changes needed are changes to the list of dependencies detected by cblrepo. So I propose someting like patches/<pkgname>.override with

depends=+foo -bar +baz

So this information is applied internally by cblrepo to its detected list of dependencies before generation of PKGBUILD and it doesn't contain version numbers so there is no need to maintain it on every update. Then patching proceeds as usual.

The .override files are technically not patches, but putting them in patches/ lets a user to uniformly inspect/remove all overrides by ls patches/<pkgname>.*, rm, vim -p etc. And the config can be extended to support overrides other from depends if we ever decide so.

cblrepo convertdb without further arguments operates silently and doesn't indicate a failure to find and convert the database.

I could only make it actually converting by supplying an -o argument. But cblrepo convertdb -h says that -o argument is optional.

In gitit.cabal there is

Flag plugins
  description:       Compile in support for plugins.  This will increase the size of
                     the executable and the memory it uses, so those who will not need
                     plugins should disable this flag.
  default:           True

[...]

  if flag(plugins)
    exposed-modules: Network.Gitit.Interface
    build-depends:   ghc, ghc-paths
    cpp-options:     -D_PLUGINS

cblrepo ignored the additional dependency on ghc-paths when I added gitit to the database, and didn't include "haskell-ghc-paths" in the depends list of the PKGBUILD. However since I happened to have ghc-path installed on my system, the Arch package that I built was silently depending on it. I had to fix it manually with a gitit.pkgbuild patch.

Thanks

Few packages don't have Setup.*. Cabal builds them but cblrepo doesn't. An example is definitive-base.

A heredoc cat in PKGBUILD seems to be the easiest option.

In the cabal file, flag testing defaults to False:

flag testing
    description: Testing mode, only build minimal components
    default: False

But when I use cblrepo add xmonad,0.12, it was set to True by default. Is there something I miss that made cblrepo change the flag?

Currently there is support for Cabal and PKGBUILD patches, but support for a build-time patch is missing.

It'd fix stuff like haddock not being able to generate docs for HaXml 1.22.2.

The .cabal file can contain lines like

Build-tools:         c2hs

Currently cblrepo can't handle this, but it really ought to.

I would rely like to see a --debug er --verbose option. This would make it allot easier, well, to debug in general (pkgbuilds, patches, cblrepo itself).
I think this would make it a lot easier for people with low skill level to join the ranks.
And in general be useful and a good addition, to a great program.

EDIT: Don't know if this is the right place for Future requests

Some packages (e.g. hsmagick) specify dependencies using pkgconfig-depends: Cabal option.

With pkgfile the dependencies can be resoved using pkgfile /usr/lib/pkgconfig/$1.pc

pkgconfig-depends is a more robust way to specify dependencies as it specifies all compiler and linker options, e.g. an include path as well. So it's a good idea to encourage upstream .cabal developers to switch to pkgconfig-depends when applicable.