ushis / local-repo

A local repository manager for Arch Linux with AUR support

local-repo repo [options]

For full docu see http://ushi.wurstcase.net/local-repo/ or have a look at doc/local-repo.html

usage: local-repo path [options]

This program helps to manage local repositories. Specify the path to the
repository with the first argument. If no option is specified, some repo
information will be printed.

positional arguments:
  path                  path to the repo or the repo name if set up in your
                        config file '~/.config/local-repo'

optional arguments:
  -h, --help            show this help message and exit
  -a path [path ...], --add path [path ...]
                        add a package to the repo - path can point to a local
                        or remote package file, pkgbuild file or pkgbuild
                        tarball - supported protocols are HTTP(S) and FTP
  -A name [name ...], --aur-add name [name ...]
                        download, build and add a package from the AUR to the
                        repo
  -c, --check           run an integrity check
  -C, --clear-cache     clear the cache
  -e, --elephant        the elephant never forgets
  -f, --force           force an operation - use this with -a or --add to
                        up-/downgrade a package
  -F path, --config-file path
                        use an alternative config file (instead of
                        '~/.config/local-repo')
  -i name [name ...], --info name [name ...]
                        display info for specified packages
  -l, --list            list all packages from the repo
  -r name [name ...], --remove name [name ...]
                        remove packages from the repo
  -R, --restore         restore repo database
  -s term, --search term
                        find packages
  -U, --aur-upgrade     upgrade all packages in the repo, which are available
                        in the AUR
  -V, --vcs-upgrade     upgrade all packages in the repo, which are based on a
                        VCS and available in the AUR

Please report bugs at: <https://github.com/ushis/local-repo/issues>

Examples

Creating a new repo

One way of creating a repo is to create an empty diretory and add some packages, eg from the AUR using -A.

$ mkdir /tmp/repo
$ local-repo /tmp/repo -A package1 package2 package3

If you already have some packages in a directory, you can use the -R option

$ cd /path/to/packages
$ ls
package1.pkg.tar.xz
package2.pkg.tar.xz
package3.pkg.tar.xz
$ local-repo ./ -R

Config

The default config file is stored in ~/.config/local-repo. See /usr/share/local-repo/config.example if you want to create one or run

$ cp /usr/share/local-repo/config.example ~/.config/local-repo

and edit the file. The coolest feature of the config file is the path option. Write something like this

[myrepo]
path = /srv/http/repo

Now you can type $ local-repo myrepo -l instead of $ local-repo /srv/http/repo -l

Translators

I am very happy about any contribution. The easiest way to contribute is to add a translation.

How?

Go to https://www.transifex.net/projects/p/local-repo/ and translate into your preferred language. Its that easy.

Dont like Transifex?

This is a little bit more complicated, but no problem for experienced git users.

1. Fork local-repo and clone it

2. Check out the devel branch

   $ cd /path/to/local-repo
   $ git checkout devel
   

3. Check if your language already exists. mylang is something like 'en' or 'de'

   $ cd share
   $ ls translations/ | grep mylang
   

4. Copy the template into the translations folder. Replace mylang by your language. If your language already exists, you should skip this.

   $ cp messages.pot translations/mylang.po
   

5. Edit the language file.

6. If you want to test your translation (This would be very nice!), compile the language files and launch the programm. You need to have installed the gettext package for doin this.

   $ ./po.sh compile
   $ ../local-repo path [options]
   

7. Add, commit and push your changes and send me a pull request. Choose 'devel' as integration branch.

Happy translating!

LICENSE

Copyright (c) 2012 ushi

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

local-repo's Issues

repo db's filename is the same name as folder in path

Instead of providing to repodb file the same name as repo, the repodb filename is created with the same name as the folder's. For example, reponame=myrepo and path=/var/cache/anothername : if the database is rebuilt, the repodb filename will be "anothername.db.tar.gz", inside /var/cache/anothername.

This is partically weird if I add my local repository to '/etc/pacman.conf'. I would expect to have [myrepo] name, but pacman would find only [anothername] as repo name.

The repodb filename shouldn't follow the name of the repo's name?

check gettext strings

Check all gettext strings before pushing the new messages template. Something like

$ egrep "_\('[^']*'\)" --color=auto local-repo localrepo/*.py

should help doing the job.

Create repo

Add a create repo option
This should create an empty repo (and imports all packages in the directory, if available)

LocalRepo.aur_upgrade() has bad version comparison

The lexical comparison of the version strings in LocalRepo.aur_upgrade() is crap. The result is that something like '100' > '59' is False.

Parse variables in PKGBUILD when upgrading from AUR

If a PKGBUILD has "depends=( ${_pkgbase} )" or in 'makedepends', local-repo should translate this variable to the correct value, instead of looking for package '${_pkgbase}' which is the current behavior when trying to upgrade from AUR.

The presence of this variable in 'depends' is very common in "lib32" packages in AUR and [multilib] repo.

locale

make it multilingual...

Recreate database

Add a recreate database option wich is simply running

repo-add self._db *.pkg.tar.xz

test if packages are available...

extend vcs upgrade

--vcs-upgrade should also upgrade the vcs packages found in the 'pkgbuild' dir, if set up in the config

send an error, if no option is specified

at the moment local-repo loads the package list a exits if no operation option is given

ftp uploader?

not sure...

Dont use wget

write the the download stuff in python, use less external tools as possible...

something like:

def progress(blocks, blocksize, total):
    print(math.floor(blocks * blocksize / total * 100) + '%')

urllib.request.urlretrieve('url', 'dest', progress)

Repo.remove() & multiple packages

it should possible to remove multiple packages a time. what about Repo.add()?

Won't update, but it is out-of-date

I have some packages in this situations, but I haven't figure out what is the pattern. As a workaround I remove and add again to the repo db.

See an example:

$ LC_ALL=C lr -s lib32-glib-networking

Loading repo: /var/cache/rafaelrepo
lib32-glib-networking 2.30.2-1

$ LC_ALL=C yaourt -Ss lib32-glib-networking
rafaelrepo/lib32-glib-networking 2.30.2-1
Network-related giomodules for glib (32 bit)
aur/lib32-glib-networking 2.32.1-1 (5)
Network-related giomodules for glib (32 bit)

$ LC_ALL=C lr -U

Loading repo: /var/cache/rafaelrepo
254 packages found
Retrieving package info from the AUR
82 packages found
Checking for updates
All packages are up to date

You're using sudo but it's not in your PKGBUILD as a dependency

Handle KeyboardInterrupt

Handle Ctrl+C Exception

rename some methods in the Repo class

• Repo.find_packages() -> Repo.find()
• Repo.has_package() -> Repo.has()

short and nice and more consistent with Repo.add(), Repo.remove() and stuff.

Poor elephant forgotten :/

Prior today (march 24 2012), " --elephant" didn't need any parameter, as it stated in the help output. Right now, it will not work. Can you please fix this funny feature?

$ LC_ALL=C local-repo --help | grep elephant
-e, --elephant the elephant never forgets

$ LC_ALL=C local-repo --elephant
usage: local-repo path [options]
local-repo: error: too few arguments

Failed to load $CARCH from /etc/makepkg.conf

Trying to build gngeo from AUR but it seems local-repo didn't load $CARCH variable. This variable is set in /etc/makepkg.conf (in my case, should return 'x86_64') and gngeo's PKGBUILD is loading it outside and inside build() function. This PKGBUILD normally would work fine.

See output :

$ LC_ALL=C lr -a http://aur.archlinux.org/packages/gn/gngeo/gngeo.tar.gz

Loading repo: /var/cache/rafaelrepo
Forging a new package: http://aur.archlinux.org/packages/gn/gngeo/gngeo.tar.gz
[=========================] 100%
/tmp/local-repo-1rckl4/gngeo/PKGBUILD: line 12: [: ==: unary operator expected
/tmp/local-repo-1rckl4/gngeo/PKGBUILD: line 15: [: ==: unary operator expected
==> Making package: gngeo 0.8-4 (Sat Apr 21 23:20:56 BRT 2012)
==> WARNING: Skipping dependency checks.
==> Retrieving Sources...
-> Found gngeo-0.8.tar.gz
-> Found 4dec1ccfb85d.patch
==> Validating source files with md5sums...
gngeo-0.8.tar.gz ... Passed
4dec1ccfb85d.patch ... Passed
==> Extracting Sources...
-> Extracting gngeo-0.8.tar.gz with bsdtar
==> Starting build()...
patching file src/star_interf.c
patching file src/raze_interf.c
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gcc... gcc -m32
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc -m32 accepts -g... yes
checking for gcc -m32 option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc -m32... gcc3
checking for ranlib... ranlib
checking dependency style of gcc -m32... gcc3
checking for sdl-config... /usr/bin/sdl-config-32
checking for SDL - version >= 1.2.0... ./configure: line 4251: /usr/bin/sdl-config-32: No such file or directory
./configure: line 4252: /usr/bin/sdl-config-32: No such file or directory
./configure: line 4255: /usr/bin/sdl-config-32: No such file or directory
./configure: line 4257: /usr/bin/sdl-config-32: No such file or directory
./configure: line 4259: /usr/bin/sdl-config-32: No such file or directory
no
*** Could not run SDL test program, checking why...
*** The test program failed to compile or link. See the file config.log for the
*** exact error that occured. This usually means SDL was incorrectly installed
*** or that you have moved SDL since it was installed. In the latter case, you
*** may want to edit the sdl-config script: /usr/bin/sdl-config-32
configure: error: *** SDL version 1.2.0 not found!
==> ERROR: A failure occurred in build().
Aborting...
An error occurred while running: /usr/bin/makepkg -d -f -L -m --nosign

Only load what you need

Dont load the whole database on startup. That could be quite painful, when managing big repos. Only load needed info and cache them...

add .install file that prints a hint about config.example

I think you could add a install file to your PKGBUILD that prints
a hint to users, mentioning this config.example as base config. Some
might miss this file, which would be a waste of time for them.

Rafael

Think thats great!

Forward --force to makepkg [-f]

Trying --aur-upgrade, but it seems the package is already built. It suggests to use --force/-f to overwrite, but it doesn't work.

$ LC_ALL=C local-repo rafaelrepo --force -U

Loading repo: /var/lib/local-repo
258 packages found
Retrieving package info from the AUR
84 packages found
Checking for updates
lib32-icu (4.8.1.1-1 -> 4.8.1.1-2)
Upgrade? [y|N] y
Forging a new package: https://aur.archlinux.org/packages/li/lib32-icu/lib32-icu.tar.gz
[=========================] 100%
==> ERROR: A package has already been built. (use -f to overwrite)
An error occurred while running: /usr/bin/makepkg -d -L -m --nosign

better error messages

messages like invalid databse entry are studpid!
say something like: Invalid db entry for package: foo: Missing field: sha256sum

makes life easier

Missing string to translate

The following strings are part of 'localrepo/localrepo.py' and are not available in the po file:

• "Installed following packages as dependencies"
• "Uninstall?"

improve bash_completion

complete multiple packages after --remove --info and --search

it should be possible to add packages from the web

lets allow something like this:

$ local-repo myrepo -a http://ushi.wurstcase.net/arch/local-repo-1.5.1-1-any.pkg.tar.xz

Don't build a package if won't add it

If local-repo is told to add a package from its source tarball, it will build then add. However, the package is already in the database, the addition will only be denied after the compilation. Can't it be told to the user before the compilation is done?

Example:

$ LC_ALL=C lr -s gngeo

Loading repo: /var/cache/rafaelrepo
gngeo 0.8-3

$ LC_ALL=C lr -a https://aur.archlinux.org/packages/gn/gngeo/gngeo.tar.gz

Loading repo: /var/cache/rafaelrepo
Forging a new package: https://aur.archlinux.org/packages/gn/gngeo/gngeo.tar.gz

[=========================] 100%
==> Making package: gngeo 0.8-4 (Sun Apr 22 00:36:09 BRT 2012)
(... hiden build stuff ...)
==> Tidying install...
-> Purging unwanted files...
-> Compressing man and info pages...
-> Stripping unneeded symbols from binaries and libraries...
==> Creating package...
-> Generating .PKGINFO file...
-> Adding install file...
-> Compressing package...
==> Leaving fakeroot environment.
==> Finished making: gngeo 0.8-4 (Sun Apr 22 00:37:25 BRT 2012)

Adding package to the repo: gngeo
Package is already in the repo: gngeo

Some english speaking people should review the docs

You can find them here: doc/local-repo.html and here: http://ushi.wurstcase.net/local-repo/

no dont ask the AUR for upgrades, if repo is empty

check repo size in LocalRepo.aur_upgrade() before sending request.

more output

local-repo should give mire feedback

Dont use tar

Remove the workaround in localrepo/package.py Package.from_file(), when the tarfile module supports lzma. This should come with next python release, its already implemented: http://hg.python.org/cpython/file/default/Lib/tarfile.py

add/remove/update multiple packages

it should be possible to run operations on multiple packages

Add exception to '--aur-upgrade' option

Sometimes, the package's PKGBUILD in AUR is messed up and it won't build until the maintainer get it fixed. It would be nice to have implemented an exception option to this command. Something like '--ignore ' (quoting an option of "pacman -S").

print some repo info, if no option is specified

num of packages
path to repo
size (of all packages)
last update
...

Can't remove bogus package from repo

An already added package doesn't have URL, which is interpreted as a "invalid db entry" by local-repo. Due to this fact, I can not do some (all?) operations with this repo. e.g. can't remove this package.

$ LC_ALL=C local-repo rafaelrepo -f -r gtksourceview2-pkgbuild

Loading repo: /var/lib/local-repo
Invalid db entry: gtksourceview2-pkgbuild-0.1.0-3/desc: Missing fields: url

p.s.: I used an ugly workaround to solved this issue (manually removed pacakge and rebuilt the repo rb).

replace useless regex

there are some useless reg expressions in local-repo, which can be replaced with str.endswith() or str.startswith() this is faster and easier to read.

Using 'su' to get root privileges does not work

$ local-repo /path/to/localrepo -A oilrush
> Loading repo database: /home/localrepo/repo 
> Retrieving package info from the AUR 
> Making a new package 
Need following packages as dependencies: qt, qtwebkit
Install? [y|N] y
su: invalid option -- 'S'
Try `su --help' for more information.
An error occurred while running: su -c ' pacman -S qt qtwebkit --asdeps ' 

Using sudo works.

use pythons list comprehensions/generator expressions

makes your code more readable and compact

bash completion

write a bash completion. see: http://bash-completion.alioth.debian.org/

add bug report url + transifex url to help

use epilog='...' in the argparser constructor for this

Allow multiple root entries in Package.from_tarball()

Extract tarball to a subdir, if it has multiple root entries.

Logging!

What about logging? It should be optional. The log file should be specified in the config - if not, no logging...

failed to automatically remove [pkgname>=pkgver], installed as dependency

When building a lib32-nss [1] , had to install lib32-nspr [2] as dependency. However 'lib32-nspr>=4.9' how it is represented in the source array. So, after compilation is done, local-repo asks if it should remove the dependency, but fail to get the correct package name.

Procedure to reproduce:

• compile lib32-nspr and add to repo db
• update pacman's db
• start compilation of lib32-nss with local-repo
• install needed package, lib32-nspr
• after compilation, it asks if it should remove 'lib32-nspr>=4.9'
• After 'y', it fails.

Output:

==> Finished making: lib32-nss 3.13.3-1 (Sat May 26 02:08:36 BRT 2012)
Installed following packages as dependencies:
[lib32-nspr>=4.9]
Uninstall? [y|N] y
error: target not found: lib32-nspr>=4.9
An error occurred while running: /usr/bin/sudo /usr/bin/pacman -R lib32-nspr>=4.9

introduce -f --force options & get rid of -u --upgrade

use force instead of upgrade and get rid of Repo.upgrade() which only has a version check... we can use Repo.add(pkg, force=True) and also allow downgrades.

nicer info ouput for --info

make the --info ouput nicer, e.g. human readable package size,...

Install depencies from the AUR, if needed

It is very great that you added --aur-upgrade. Just be aware the if a
package in the repo needs to install a dependency from AUR, it
(makepkg, in matter of fact) will fail. I get this problem every time
compiling my packages locally.
e.g.: --aur-upgrade> lib32-gtk3> depends on lib32-colord> which was
not found installed> installing from AUR ...> build fail

will think about that

Dont abuse Exceptions for error messages

Just write your own Exceptions.

Unittests

Hate it... But when its done, it will make things easier...

Won't install/update package with plus sign in pkgname

This package is in AUR, but it seems that the '++' is messing things up.

$ LC_ALL=C lr -A lib32-libsigc++

Loading repo: /var/cache/rafaelrepo
Retrieving package info from the AUR
AUR responded with error: No results found

Clean up the main script file

--> it looks like shit

config file in ~

add support for config files in ~/.config/ where people can add shortcuts and stuff... something like ~/.ssh/config

Repo fancystuff
    Path /srv/ftp/fancystuff
    Foo Bar

People can call local-repo fancystuff -l