osbuild / weldr-client Goto Github PK
View Code? Open in Web Editor NEWGo client library and cmdline tool for WELDR API servers like lorax-composer and osbuild-composer
License: Apache License 2.0
Go client library and cmdline tool for WELDR API servers like lorax-composer and osbuild-composer
License: Apache License 2.0
I wanted to test a version of weldr-client
with the support for cross-distro building and while trying to build RPMs from the git checkout, I got an error due to failed signing:
$ make rpm
sed -e "s/%%VERSION%%/35.1/g" -e "s/%%GPGKEY%%/908AF740C0F898A1079968CDC5887AD51D9F3C2D/g" < weldr-client.spec.in > weldr-client.spec
make -s changelog >> weldr-client.spec
mkdir -p /home/thozza/devel/weldr-client/rpmbuild/SPECS
cp weldr-client.spec /home/thozza/devel/weldr-client/rpmbuild/SPECS
git archive --prefix=weldr-client-35.1/ --format=tar.gz HEAD > weldr-client-35.1.tar.gz
gpg --armor --detach-sign -u 908AF740C0F898A1079968CDC5887AD51D9F3C2D weldr-client-35.1.tar.gz
gpg: skipped "908AF740C0F898A1079968CDC5887AD51D9F3C2D": No secret key
gpg: signing failed: No secret key
make: *** [Makefile:53: sign] Error 2
Please consider not signing the archive as part of rpm
target.
According to the command help, compose log is meant to get information of the running build, however, it complains the compose is still running, which sounds like a contradiction.
[root@fedoravm image-builder]# composer-cli compose list
ID Status Blueprint Version Type
a4abfd36-0b79-4ccb-a6ac-3edea46ec575 RUNNING fedora-nx 0.0.1 image-installer
17bda711-098a-413d-bbd8-c4f6f7ac6c34 FAILED fedora-nx 0.0.2 image-installer
b2f165f0-cb6d-4803-880d-1d62b365f6dd FAILED fedora-nx 0.0.0 image-installer
[root@fedoravm image-builder]# composer-cli compose log a4abfd36-0b79-4ccb-a6ac-3edea46ec575 100
Build a4abfd36-0b79-4ccb-a6ac-3edea46ec575 is still running.
weldr-client
currently speaks to the weldr-api
as provided by osbuild-composer. In the longer term I'd like to move this over to the cloud-api
, also provided by osbuild-composer.
Most important is how we deal with blueprints. These are currently being parsed on the weldr-api
side of things and with this change they would need to be parsed and converted into a cloud-api
type request on the client side.
We would likely split out the parsing and validation of blueprints to a separate library as the service has also indicated interest in copy/pasting in a blueprint.
Describe the bug
composer-cli blueprints save
creates a blueprint .toml where the user's UID and GID are floating point numbers, e.g. a UID of 1200 will be 1200.0 in the resulting .toml
Environment
/etc/os-release
and /etc/redhat-release
): Fedora 35
rpm -qi osbuild-composer)
Name : osbuild-composer
Version : 42
Release : 1.fc35
Architecture: x86_64
Install Date: Tue 01 Feb 2022 02:08:27 PM CET
Group : Unspecified
Size : 17577
License : ASL 2.0
Signature : RSA/SHA256, Wed 12 Jan 2022 02:59:40 PM CET, Key ID db4639719867c58f
Source RPM : osbuild-composer-42-1.fc35.src.rpm
Build Date : Wed 12 Jan 2022 02:53:03 PM CET
Build Host : buildvm-x86-17.iad2.fedoraproject.org
Packager : Fedora Project
Vendor : Fedora Project
URL : https://github.com/osbuild/osbuild-composer
Bug URL : https://bugz.fedoraproject.org/osbuild-composer
Summary : An image building service based on osbuild
Description :
A service for building customized OS artifacts, such as VM images and OSTree
commits, that uses osbuild under the hood. Besides building images for local
usage, it can also upload images directly to cloud.
It is compatible with composer-cli and cockpit-composer clients.
To Reproduce
Steps to reproduce the behavior:
Create an original_blueprint.toml
file with these contents:
name = "BPSaveBug"
description = ""
version = "0.0.1"
[[customizations.user]]
name = "admin"
password = "$6$qL0yRFzeHbv5xvh9$DLgYdwRL5dYj0S9R941AWbjA6lN/ecJr/htnyPvct93tke2sp81qy9F7EHAZmIRMQtlCUT82Q2DYaoAqRJ3CG."
groups = ["wheel"]
uid = 1200
gid = 1200
Push and save the blueprint:
composer-cli blueprints push originalblueprint.toml
composer-cli blueprints save originalblueprint
The resulting BPSaveBug.toml has incorrect floating point values for gid and uid:
description = ""
distro = ""
groups = []
modules = []
name = "BPSaveBug"
packages = []
version = "0.0.1"
[customizations]
[[customizations.user]]
gid = 1200.0
groups = ["wheel"]
name = "admin"
password = "$6$qL0yRFzeHbv5xvh9$DLgYdwRL5dYj0S9R941AWbjA6lN/ecJr/htnyPvct93tke2sp81qy9F7EHAZmIRMQtlCUT82Q2DYaoAqRJ3CG."
uid = 1200.0
Expected behavior
description = ""
distro = ""
groups = []
modules = []
name = "BPSaveBug"
packages = []
version = "0.0.1"
[customizations]
[[customizations.user]]
gid = 1200
groups = ["wheel"]
name = "admin"
password = "$6$qL0yRFzeHbv5xvh9$DLgYdwRL5dYj0S9R941AWbjA6lN/ecJr/htnyPvct93tke2sp81qy9F7EHAZmIRMQtlCUT82Q2DYaoAqRJ3CG."
uid = 1200
Additional context
Add any other context about the problem here.
Hello,
Is there any plans on structuring this so that other Go applications can
use the weldr client that this has?
I don't see a good way to import it right now so I think we'll have to copy the code.
Best regards
The JSON output for commands like list
that send more than one request is not a JSON list, it is a series of distinct objects. It would be nice if this could be JSON compliant so it can be passed directly into a parser.
$ rpm -q composer-cli
composer-cli-34.9-5.fc34.x86_64
$ sudo dnf install --enablerepo=updates-testing weldr-client
Last metadata expiration check: 0:23:11 ago on Čt 22. dubna 2021, 15:05:27.
Dependencies resolved.
===============================================================================================================================================================
Package Architecture Version Repository Size
===============================================================================================================================================================
Installing:
weldr-client x86_64 35.1-1.fc34 updates-testing 2.6 M
Transaction Summary
===============================================================================================================================================================
Install 1 Package
Total size: 2.6 M
Installed size: 8.6 M
Is this ok [y/N]: y
Downloading Packages:
[SKIPPED] weldr-client-35.1-1.fc34.x86_64.rpm: Already downloaded
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction test error:
file /usr/bin/composer-cli from install of weldr-client-35.1-1.fc34.x86_64 conflicts with file from package composer-cli-34.9-5.fc34.x86_64
file /usr/share/man/man1/composer-cli.1.gz from install of weldr-client-35.1-1.fc34.x86_64 conflicts with file from package composer-cli-34.9-5.fc34.x86_64
I think that maybe Obsoletes: composer-cli < 35.0
might solve it, I didn't test it though. We might also want Conflicts
? Not sure...
Document how to make a new upstream release.
Currently, composer-cli compose image
firstly downloads the image into /tmp
. This causes the osbuild-composer's test suite to fail because on AWS, /tmp
is on tmpfs
and we don't have enough RAM to store the whole image. My proposal is to drop the temporary file and save the image directly in the current working directory.
BurntSushi/toml is unmaintained: https://github.com/BurntSushi/toml
We should probably switch to using something else sooner rather than later.
As the subject says, currently composer-cli compose image xxx
downloads to a pre-defined image name (based on the UUID) but this isn't optimal for automation, as it forces one to parse the output and rename the image in a 2nd step and, because Ansible doesn't support moving/renaming of files, it requires a copy and/or using a command to move it. Anyway, it's less than nice and being able to give directly the name of the downloaded image, would make it easier.
When calling sudo composer-cli compose results
I get not feedback on stdout (as opposed to the docs which suggest the name and size of the result should be printed).
I know this is a minor / cosmetic issue but at first I thought something had gone wrong until I figured it out.
(Using F34 and osbuild from the repos.)
The compose info response from the server can also contain the status of uploads, these should also be printed.
The trick is to figure out how to include them in the current output without making it messy.
As pointed out in pr #7 the flags don't get reset between tests because the Cobra struct persists for all the tests.
Need to figure out a nice way to reset things so that you don't need to clear them all at the start of every test.
the old composer-cli
:
$ sudo composer-cli --json blueprints changes empty
{
"blueprints": [
{
"changes": [
{
"commit": "14cf307a846a66b4ae184aa79b00ffb5d7de2cb1",
"message": "Recipe empty, version saved.",
"revision": null,
"timestamp": "2021-04-22T15:59:36Z"
},
...
],
"name": "empty",
"total": 15
}
],
"errors": [],
"limit": 15,
"offset": 0
}
the new weldr-client
:
$ sudo composer-cli --json blueprints changes empty
{
"method": "GET",
"path": "/blueprints/changes/empty?limit=0",
"status": 200,
"body": {
"blueprints": [
{
"changes": [
{
"commit": "14cf307a846a66b4ae184aa79b00ffb5d7de2cb1",
"message": "Recipe empty, version saved.",
"revision": null,
"timestamp": "2021-04-22T15:59:36Z"
},
...
],
"errors": [],
"limit": 0,
"offset": 0
}
}
{
"method": "GET",
"path": "/blueprints/changes/empty?limit=15",
"status": 200,
"body": {
"blueprints": [
{
"changes": [
{
"commit": "14cf307a846a66b4ae184aa79b00ffb5d7de2cb1",
"message": "Recipe empty, version saved.",
"revision": null,
"timestamp": "2021-04-22T15:59:36Z"
},
...
],
"name": "empty",
"total": 15
}
],
"errors": [],
"limit": 15,
"offset": 0
}
}
I'm not sure what's the expected behavior here. I would say that a user expects a valid JSON when passing the --json
flag. On the hand, the output is actually a valid JSON stream. Not sure, what do you think @bcl?
The README.md file contains a user guide which could live in the guides repository. We could either synchronize it there or move it completely and only reference it.
It would be nice to be able to sort the output of composer-cli compose status
by the different fields that are shown. i.e. composer-cli compose status --sort time
or composer-cli compose status --sort status
I'm most interested in sorting by the time
field, but I guess it could be useful to sort by other fields too.
This easy reproducer says it all:
[root@kvm-08-guest31 osbuild-composer]# composer-cli compose start whatever qcow2
ERROR: UnknownBlueprint: Unknown blueprint name: whatever
[root@kvm-08-guest31 osbuild-composer]# echo $?
1
[root@kvm-08-guest31 osbuild-composer]# composer-cli --json compose start whatever qcow2
{
"method": "POST",
"path": "/api/v1/compose",
"status": 400,
"body": {
"errors": [
{
"id": "UnknownBlueprint",
"msg": "Unknown blueprint name: whatever"
}
],
"status": false
}
}
[root@kvm-08-guest31 osbuild-composer]# echo $?
0
Ran using weldr-client-35.1-4.el9.x86_64
user@desktopa weldr-client € make scratch-srpm
sed -e "s/%%VERSION%%/35.6/g" -e "s/%%GPGKEY%%//g" < weldr-client.spec.in > weldr-client.spec
make -s changelog >> weldr-client.spec
mkdir -p /home/supakeen/dev/src/work/weldr-client/rpmbuild/SPECS
cp weldr-client.spec /home/supakeen/dev/src/work/weldr-client/rpmbuild/SPECS
git archive --prefix=weldr-client-35.6/ --format=tar.gz HEAD > weldr-client-35.6.tar.gz
mkdir -p /home/supakeen/dev/src/work/weldr-client/rpmbuild/SOURCES
cp weldr-client-35.6.tar.gz* rpmbuild/SOURCES/
rpmbuild -bs \
--define "_topdir /home/supakeen/dev/src/work/weldr-client/rpmbuild" \
--define "commit 35.6" \
--with tests \
--without signed \
rpmbuild/SPECS/weldr-client.spec
setting SOURCE_DATE_EPOCH=1661472000
Wrote: /home/supakeen/dev/src/work/weldr-client/rpmbuild/SRPMS/weldr-client-35.6-1.fc36.src.rpm
user@desktopa weldr-client € rpmlint rpmbuild/SRPMS/weldr-client-35.6-1.fc36.src.rpm
================================================================================================================================== rpmlint session starts ==================================================================================================================================
rpmlint: 2.2.0
configuration:
/usr/lib/python3.10/site-packages/rpmlint/configdefaults.toml
/etc/xdg/rpmlint/fedora.toml
/etc/xdg/rpmlint/licenses.toml
/etc/xdg/rpmlint/scoring.toml
/etc/xdg/rpmlint/users-groups.toml
/etc/xdg/rpmlint/warn-on-functions.toml
checks: 32, packages: 1
weldr-client.src: W: unexpanded-macro URL %{gourl}
weldr-client.spec:30: W: macro-in-comment %check
weldr-client.spec:168: W: macro-in-%changelog %gobuild
weldr-client.src: W: invalid-url URL %{gourl}
Working on this in #125
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.