fedora-modularity / libmodulemd Goto Github PK

View Code? Open in Web Editor NEW

30.0 30.0 52.0 4.08 MB

C Library for manipulating module metadata files

License: MIT License

C 89.23% Meson 1.46% Shell 1.62% Python 7.47% Roff 0.14% Makefile 0.08%

libmodulemd's People

Contributors

Stargazers

Watchers

libmodulemd's Issues

RPM components aren't in the output of `dumps`

When running dumps on a Modulemd.Module object, the string output doesn't include the RPM components.

This is blocking the Module Build Service from migrating to libmodulemd, which inherently blocks work on Module Stream Expansion.

Here is an example:

>>> import gi
>>> gi.require_version('Modulemd', '1.0')
>>> from gi.repository import Modulemd
>>> mmd = Modulemd.Module().new_from_string('document: modulemd\nversion: 1\ndata:\n    summary: A test module in all its beautiful beauty\n    description: >-\n        This module demonstrates how to write simple modulemd files And\n        can be used for testing the build and release pipeline.\n    license:\n        module: [ MIT ]\n    dependencies:\n        buildrequires:\n            platform: f28\n        requires:\n            platform: f28\n    references:\n        community: https://docs.pagure.org/modularity/\n        documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules\n    profiles:\n        default:\n            rpms:\n            - tangerine\n    api:\n        rpms:\n        - perl-Tangerine\n        - tangerine\n    components:\n        rpms:\n            perl-List-Compare:\n                rationale: A dependency of tangerine.\n                ref: master\n            perl-Tangerine:\n                rationale: Provides API for this module and is a dependency of tangerine.\n                ref: master\n            tangerine:\n                rationale: Provides API for this module.\n                buildorder: 10\n                ref: master\n')
>>> mmd.get_rpm_components()
{'perl-Tangerine': <Modulemd.ComponentRpm object at 0x7fc8ec39eeb0 (ModulemdComponentRpm at 0x558af5756820)>, 'tangerine': <Modulemd.ComponentRpm object at 0x7fc8ee5746e0 (ModulemdComponentRpm at 0x558af5756e00)>, 'perl-List-Compare': <Modulemd.ComponentRpm object at 0x7fc8ee574690 (ModulemdComponentRpm at 0x558af56222c0)>}
>>> mmd2 = Modulemd.Module().new_from_string(mmd.dumps())
>>> mmd2.get_rpm_components()
{}

libmodulemd shouldn't allow mixing positive and negative requires

Right now, libmodulemd allows the following

requires:
platform: [-f26, -f27, f28, -f29]
even though that doesn't make much sense. It should only allow all positive or all negative requires.

Add _dup() and _peek() functions for libmodulemd

After conversations with GNOME folks, it was made clear that a "proper" GObject API should have both object_peek() and object_dup() methods. The object_peek() methods would be the same functionality as currently provided by object_get() in libmodulemd and should be implemented as deprecating the latter.

object_dup() methods must return a deep copy of the property.

Support boolean variant type for XMD

When setting boolean variant in xmd, libmodulemd fails with:

** Message: Unhandled variant type
** Message: Error writing arbitrary mapping
** Message: Error emitting variant hashtable
** Message: Failed to emit xmd
** Message: Failed to emit data
** Message: Failed to process root
** Message: Could not emit YAML
** Message: Error emitting YAML string: Unhandled variant type

We would like to use this for "mse" flag in XMD section in MBS.

RFE: Add a way to return the raw YAML documents that failed validation for logging

Right now, the parse_yaml() function only returns the subset of results that validated the parser and reports errors using g_message(). We should instead have some mechanism for returning an error code and the failed YAML document string to the calling function to handle its own logging.

implement "buildafter" or something similar for components/rpms

Only one of buildorder or buildafter should be used. Then MBS could take advantage of this submitting all builds at once and let koji handle build queue and priorities.

I picked first name which came to my mind, you can choose different one.

Intents need to be handled while merging defaults

When adding support for Intents, they weren't figured into default-merging. (Presently, they are ignored during the merge, so only the Intents set by the first defaults subdocument for a particular module will be used.

The proposed logic is to treat it like the Profiles: if the key does not exist, it is added with the new document's value. If the key does exist, override with the higher-valued subdocument when in override mode, otherwise simply check that they are identical and throw an error if they are not.

Why does `modulemd_module_get_dependencies` return an array?

I'm working on porting MBS to libmodulemd and noticed that when running mmd.get_dependencies() on a v2 modulemd (became v2 via upgrade), I seem to always get a list with one element, no matter how many requires or buildrequires the modulemd specifies. To actually get those requires and buildrequires, I have to run mmd.get_dependencies()[0].get_requires() and mmd.get_dependencies()[0].get_buildrequires(). These functions return a dictionary with all the dependencies in the modulemd.

So with that said, what circumstance causes mmd.get_dependencies() to return a list that is larger than one element? I would have expected that mmd.get_dependencies() would return a ModulemdDependencies directly instead of this one element list.

Make libmodulemd buildable on CentOS 7 / RHEL 7

We'll need libmodulemd built for YUM4 CentOS SIG:
https://wiki.centos.org/SpecialInterestGroup/ConfigManagementSIG/YUM4

The build is also needed for several infrastructure tools as they run on RHEL 7.

We can't build libmodulemd as it is, because RHEL doesn't have python3 and meson.

Prioritizer: add() should throw an error on priority that is out of range

When priority is a too big number it either overflows or add() fails:
module_defaults_prioritizer.add(defaults, 99999999999)
The result is that the added defaults are not present in the prioritizer output.

Add functions for reading from FILE *

Using path to file is fine, but not when you are dealing with something complex.

Dumping Modulemd.TranslationEntry to YAML crash if entry is missing summary or description

This should not crash, but should throw an error.

libmodulemd 1.4.0 can't read file 1.3.0 had no problems with

#!/usr/bin/env python

import gi
gi.require_version('Modulemd', '1.0') # noqa
from gi.repository import Modulemd

s = '''
---                       
document: modulemd        
version: 2                
data:                     
  name: testmodule        
  stream: master          
  version: 1              
  context: 2017           
  summary: foo            
  description: >-         
    foo                   
  license:                
    module:               
    - GPL                 
  components:             
    rpms:                 
      pkg:                
        rationale: Needed for test                   
        multilib: [x86_64]                           
  artifacts:              
    rpms:                 
    - pkg-1.0.0-1.i686    
    - pkg-1.0.0-1.x86_64  
...
'''

mmd = Modulemd.Module.new_from_string(s)

$ python repl.py
Traceback (most recent call last):
  File "repl.py", line 34, in <module>
    mmd = Modulemd.Module.new_from_string(s)
TypeError: constructor returned NULL

Fix Travis CI build

When Ubuntu 17.04 disappeared from the Ubuntu mirrors, the automated builds (including the one that submits the code to Coverity for static analysis) broke. Instead of relying on Ubuntu, we should update the Travis environment to use a Fedora Rawhide container.

SIGSEGV in Prioritizer when no defaults were added

libmodulemd crashes when no defaults to Prioritizer were added.

reproducer:

import gi
gi.require_version('Modulemd', '1.0')

import gi.repository.Modulemd
prioritizer = gi.repository.Modulemd.Prioritizer()
prioritizer.resolve()

Prioritizer: throw an error when merging different data with identical priority

I'm not sure if it's expected that provided example ends with 'master' default stream.
I was assuming that this behavior is allowed only when 2nd record has higher priority (lower number).
Shouldn't the Prioritizer raise an error instead?

document: modulemd-defaults
version: 1
data:
  module: testmodule
  stream: 2.0
  profiles:
    master: [default]
---
document: modulemd-defaults
version: 1
data:
  module: testmodule
  stream: master
  profiles:
    master: [default]

Provide convenience functions for modulemd-defaults

It'd be nice to have modulemd-style convenience functions for the defaults, too. I would like to load and dump multiple defaults (and only defaults) from/to strings and files.

Add name property to profiles.

As discussed on IRC: Currently, components have a name property and profiles don't. They might have more reason than profiles to carry that information (it identifies them "more") but it's surprising for a user of the API to remember that one thing has it and the other doesn't.

buildorder: -1 is broken

from gi import Repository
r = Repository.get_default() # noqa
r.require('Modulemd') # noqa
from gi.repository import Modulemd

data = """
document: modulemd
version: 1
data:
    summary: A test module in all its beautiful beauty.
    description: This module demonstrates how to write simple modulemd files And can be used for testing the build and release pipeline.
    license:
        module: [ MIT ]
    dependencies:
        buildrequires:
            platform: el8
        requires:
            platform: el8
    references:
        community: https://fedoraproject.org/wiki/Modularity
        documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules
        tracker: https://taiga.fedorainfracloud.org/project/modularity
    profiles:
        default:
            rpms:
                - tangerine
    api:
        rpms:
            - perl-Tangerine
            - tangerine
    components:
        rpms:
            attr:
                rationale: A build dependency of acl
                ref: rhel-8.0
                buildorder: -1
"""

mmd = Modulemd.Module.new_from_string(data)
mmd.upgrade()
print mmd.get_rpm_components().values()[0].get_buildorder()

$ python mmd.py 
18446744073709551615

Do not warn about missing stream from profile defaults.

Example data:

document: modulemd-defaults
version: 1
data:
  module: testmodule
  stream: 2.0
  profiles:
    master: [default]

This triggers following output:

** Message: 09:15:48.608: Default stream missing from profile defaults
** Message: 09:15:48.608: Invalid [ModulemdDefaults] document [Default stream missing from profile defaults].
** Message: 09:15:48.608: Invalid document [Default stream missing from profile defaults]. Skipping it.

I think this behavior is incorrect. Consider following use cases:

specify default stream only; use the 'default' profile in modulemd or the empty 'default' fallback
override installation profiles without touching the default stream (or eventually don't have a default at all)

[RFE] Remove warning on v1->v2 upgrade on modulemds that don't have `eol` set

When upgrading a v1 modulemd to v2 that doesn't have the eol property set, you get the warning __main__:1: Warning: g_date_valid: assertion 'd != NULL' failed. This warning isn't useful since a v1 modulemd is valid without having the eol property set. It'd be friendlier to not have this warning, especially since it can pollute application logs if the upgrade function is often run.

Here is how to produce the warning message:

>>> import gi
>>> gi.require_version('Modulemd', '1.0')
>>> from gi.repository import Modulemd
>>> mmd = Modulemd.Module().new_from_string('document: modulemd\nversion: 1\ndata:\n    summary: A test module in all its beautiful beauty\n    description: >-\n        This module demonstrates how to write simple modulemd files And\n        can be used for testing the build and release pipeline.\n    license:\n        module: [ MIT ]\n    dependencies:\n        buildrequires:\n            platform: f28\n        requires:\n            platform: f28\n    references:\n        community: https://docs.pagure.org/modularity/\n        documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules\n    profiles:\n        default:\n            rpms:\n            - tangerine\n    api:\n        rpms:\n        - perl-Tangerine\n        - tangerine\n    components:\n        rpms:\n            perl-List-Compare:\n                rationale: A dependency of tangerine.\n                ref: master\n            perl-Tangerine:\n                rationale: Provides API for this module and is a dependency of tangerine.\n                ref: master\n            tangerine:\n                rationale: Provides API for this module.\n                buildorder: 10\n                ref: master\n')
>>> mmd.upgrade()
__main__:1: Warning: g_date_valid: assertion 'd != NULL' failed
True

Add a SimpleSet and hash table sorting function

The get_as_strv() routine currently returns in the order provided by the GHashTable iterator. We should allow passing a sorting comparison function to get the results in an expected order.

Add a SimpleSet comparison function

This will make it possible to avoid always setting SimpleSets when saving, even if they have the same effective contents.

See https://github.com/sgallagher/c-modulemd/blob/a4e1617/modulemd/modulemd-profile.c#L100 for an example use-case.

modulemd_module_dumps () Adds fields which where not initially present on the yaml document.

In [1]: import gi

In [2]: gi.require_version('Modulemd', '1.0')

In [3]: from gi.repository import Modulemd

In [4]: x = '/home/ipanova/Downloads/fcb39b9a0b045c35442f1ab05dfd230148b4662fe1ab6ad7cd3e9722ffce8b84-modules.yaml'

In [5]: modules = Modulemd.objects_from_file(x)

In [6]: modules
Out[6]: 
[<Modulemd.Module object at 0x7fc91c111370 (ModulemdModule at 0x55f1aeadf040)>,
 <Modulemd.Module object at 0x7fc91c08d640 (ModulemdModule at 0x55f1aeadf130)>,
 <Modulemd.Module object at 0x7fc91c08d5f0 (ModulemdModule at 0x55f1aeadf220)>,
 <Modulemd.Module object at 0x7fc91c08d410 (ModulemdModule at 0x55f1aeadf310)>,
 <Modulemd.Module object at 0x7fc91c08d2d0 (ModulemdModule at 0x55f1aeadf400)>,
 <Modulemd.Defaults object at 0x7fc91c08d3c0 (ModulemdDefaults at 0x55f1aea3c280)>,
 <Modulemd.Defaults object at 0x7fc91c08d5a0 (ModulemdDefaults at 0x55f1aea3c2c0)>,
 <Modulemd.Defaults object at 0x7fc91c08d550 (ModulemdDefaults at 0x55f1aea3c300)>]

In [7]: module = modules[0]
In [8]: module.dump('/tmp/lal')

$ diff temp-django.yaml /tmp/lal
54a55,56
>   buildopts:
>     rpms: {}

Same behaviour is observed with dump() method.

Version: libmodulemd-1.5.2-1.fc28.x86_64

Cannot use libmodulemd with libdnf (c++)

Hello,

I am not able to compile libdnf, when including any header from libmodulemd.

Here is minimal reproducer:
main.cpp.txt

Compiling using:

$ g++ main.cpp `pkg-config --cflags --libs modulemd`

implement "buildonly" for components/rpms

For instance, in rust ecosystem we just want to pull components for the buildroot. Yes, you can use filter, but since that doesn't support globs/regex, it would be to complex to list 2-30 rpms per component.

Python segfaults when running `add_rpm_component`

The following causes Python to segfault:

>>> import gi
>>> gi.require_version('Modulemd', '1.0')
>>> from gi.repository import Modulemd
>>> mmd = Modulemd.Module().new_from_file('tests/staged_data/formatted_testmodule.yaml')
>>> component = Modulemd.ComponentRpm()
>>> component.set_name('test')
>>> mmd.add_rpm_component(component)

This is the modulemd I'm using:
https://pagure.io/fm-orchestrator/blob/master/f/tests/staged_data/formatted_testmodule.yaml

dup_nsvc() prints version in hexadecimal

I can replicate this with 1.6.1 and 1.6.2 through the gobject introspection bindings:

>>> mmd.dup_nsvc()
'mod:master:125ab64228a8:9edba152'
>>> mmd.peek_version()
20180814145704L

How should the multilib work?

I'm going to implement proper multilib in Pungi, spec.v2.yaml says:

# A list of architectures with multilib
# installs, i.e. both i686 and x86_64
# versions will be installed on x86_64.
# TODO: This needs to be reworked or dropped as it's not at all useful in the current state.
# Optional, defaults to no multilib.
multilib: [ x86_64 ]

I talked with @contyk about this and he says the TODO basically means it is not clear what the arches should be in multilib field and if the current way how multilib field is defined is enough. We should have the final way how to define multilib so I can implement that in Pungi.

From the Pungi perspective, the current way with multilib: [x86_64] would be enough. It would work like this:

When component has x86_64 multilib set in modulemd, the binary RPMs built from this component for x86_64 and also for other multilib arches would be considered for x86_64 compose.
Depending on compose variant the Pungi's multilib strategy (see "multilib" here) would really be the option defining whether the particular RPMs end up in compose or not. So it might be possible that component with "multilib" set in modulemd won't appear in resulting variant, because the multilib Pungi option will disallow that for particular variant.

set_rpm_artifacts() doesn't enforce NEVRA format

Recently the parsing code was made to enforce that the RPM artifacts are in name-epoch:version-release form, but the setters weren't changed to match, so it's possible to use libmodulemd to generate a file it can't read back in.

I would suggest that set_rpm_artifacts() should warn and not set the artifacts at all if the contained strings are invalid. Stripping only non-validating elements would be possible too, but I don't think it's better to have a "corrupted" set of artifacts then no set at all.

The alternative of a replacement function that returns a GError doens't strike me as much better - it's not like we need to propagate the result back to a dialog, and there's typically nothing reasonable that a programmer could do to "recover" from this. Most likely the programmer just needs to fix the code that generates the artifacts list. The only pro is that in Python/Javascript it would turn the error into a backtrace rather than warning - which is nice - unless the caller uses props.artifacts_rpm = ...

Also, where to draw the line is unclear - GError for this is somewhat reasonable but GError for modulemd_module_set_name(module, NULL) is not.

Module with version "0" cannot be loaded

Or if I generate it manually, "dup_nsvc()" of such module returns "None". Reproducer:


from gi import Repository
r = Repository.get_default() # noqa
r.require('Modulemd') # noqa
from gi.repository import Modulemd
print "Modulemd.get_version() ==", Modulemd.get_version()
data = """
document: modulemd
version: 1
data:
    name: foo
    stream: 0
    version: 0
    context: 0
    summary: A test module in all its beautiful beauty.
    description: This module demonstrates how to write simple modulemd files And can be used for testing the build and release pipeline.
    license:
        module: [ MIT ]
    dependencies:
        buildrequires:
            platform: f28
        requires:
            platform: f28
    references:
        community: https://fedoraproject.org/wiki/Modularity
        documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules
        tracker: https://taiga.fedorainfracloud.org/project/modularity
"""

mmd = Modulemd.Module.new_from_string(data)
mmd.upgrade()
print mmd.dup_nsvc()

Segfault using Modulemd.Module.new_all_from_file_ext()

The following script will cause a segmentation fault:

from gi.repository import Modulemd
mmdd = Modulemd.Module.new_all_from_file_ext("defaults.yaml")

print (mmdd)

The reason is incorrect memory management of the returned array.

Intents are not copied by modulemd_defaults_copy()

Intents support was added for 1.5.0, but copying existing intents when copying a defaults object was missed. This needs to be fixed before 1.5.0 is released.

Refactor YAML parser and Emitter

Right now, the YAML parser and emitter logic is all in monolithic files with all the YAML parsing.

It would be better and more maintainable in the future if the parser and emitter sections were handled by the C files containing the objects they represent.

modulemd_objects_from_file fails with `error` set to NULL

  g_autoptr(GPtrArray) objects = modulemd_objects_from_file (pool_tmpjoin (pool, TEST_DIR, fname, NULL), NULL);

  for (guint i = 0; i < objects->len; i++)
    {
      if (MODULEMD_IS_MODULE (g_ptr_array_index (objects, i)))
        {
          ModulemdModule *module = g_ptr_array_index (objects, i);
          add_module_solvables (repo, module);
        }
    }

code like this fails with

** Message: 11:54:53.974: GError is initialized.
fish: “and ./build/main” terminated by signal SIGSEGV (Address boundary error)

Moreover, it segfaults:

#0  0x00007ffff6e33cdf in free () at /lib64/libc.so.6
#1  0x00007ffff58a2c12 in yaml_parser_delete (parser=parser@entry=0x7fffffffdea0) at api.c:218
        __PRETTY_FUNCTION__ = "yaml_parser_delete"
#2  0x00007ffff73e2495 in parse_yaml_file (path=0x62e670 "/home/brain/tmp/m/data/repodata/8e7a1438e8668c6a8a74eb6d9ff3362f8039dd5f4ab25a4bb2db30e685e639d8-modules.yaml.gz", data=data@entry=0x7fffffffe0c0, error=<optimized out>) at ../modulemd/modulemd-yaml-parser.c:113
        result = 0
        yaml_file = 0x0
        parser = 
          {error = 544497952, problem = 0x74737961682f2f3a <error: Cannot access memory at address 0x74737961682f2f3a>, problem_offset = 0, problem_value = 0, problem_mark = {index = 0, line = 0, column = 0}, context = 0x0, context_mark = {index = 0, line = 0, column = 0}, read_handler = 0x0, read_handler_data = 0x0, input = {string = {start = 0x0, end = 0x0, current = 0x0}, file = 0x0}, eof = 4202117, buffer = {start = 0x629250 "\360\222b", end = 0x8000000100000007 <error: Cannot access memory at address 0x8000000100000007>, pointer = 0x610730 "", last = 0x629250 "\360\222b"}, unread = 6478576, raw_buffer = {start = 0x62af0d "\201C\216z\024\070\350f\214j\212t\353m\237\363\066/\200\071\335_J\262ZK\262\333\060\346\205\346\071\330]\r=\263\006\\3\203mvi<\373('\214\036\032\235P\030W\347\b\a\244\177\222\341\260U\252repodata/8e7a1438e8668c6a8a74eb6d9ff3362f8039dd5f4ab25a4bb2db30e685e639d8-modules.yaml.gz", end = 0x62af0f "\216z\024\070\350f\214j\212t\353m\237\363\066/\200\071\335_J\262ZK\262\333\060\346\205\346\071\330]\r=\263\006\\3\203mvi<\373('\214\036\032\235P\030W\347\b\a\244\177\222\341\260U\252repodata/8e7a1438e8668c6a8a74eb6d9ff3362f8039dd5f4ab25a4bb2db30e685e639d8-modules.yaml.gz", pointer = 0x0, last = 0x62d534 "\002"}, encoding = 6521616, offset = 195, mark = {index = 2, line = 0, column = 0}, stream_start_produced = 0, stream_end_produced = 0, flow_level = 0, tokens = {start = 0x0, end = 0x0, head = 0x0, tail = 0x0}, tokens_parsed = 0, token_available = 0, indents = {start = 0x0, end = 0x0, top = 0x0}, indent = 0, simple_key_allowed = 0, simple_keys = {start = 0x0, end = 0x0, top = 0x0}, states = {start = 0x0, end = 0x0, top = 0x0}, state = YAML_PARSE_STREAM_START_STATE, marks = {start = 0x0, end = 0x0, top = 0x0}, tag_directives = {start = 0x0, end = 0x0, top = 0x0}, aliases = {start = 0x0, end = 0x0, top = 0x0}, document = 0x0}
        __func__ = "parse_yaml_file"
#3  0x00007ffff73ca168 in modulemd_objects_from_file (yaml_file=<optimized out>, error=<optimized out>) at ../modulemd/modulemd-common.c:53
        data = 0x0
        __func__ = "modulemd_objects_from_file"
#4  0x0000000000401cd6 in main () at ../main.c:205
        fp = 0x62e510
        pool = 0x610730
        repo = 0x629250
        chksumtype = 52
        chksum = 0x62af0f "\216z\024\070\350f\214j\212t\353m\237\363\066/\200\071\335_J\262ZK\262\333\060\346\205\346\071\330]\r=\263\006\\3\203mvi<\373('\214\036\032\235P\030W\347\b\a\244\177\222\341\260U\252repodata/8e7a1438e8668c6a8a74eb6d9ff3362f8039dd5f4ab25a4bb2db30e685e639d8-modules.yaml.gz"
        fname = 0x62af4f "repodata/8e7a1438e8668c6a8a74eb6d9ff3362f8039dd5f4ab25a4bb2db30e685e639d8-modules.yaml.gz"
        objects = 0x7ffff7192369 <_g_signal_init+153>

libmodulemd should treat multiple defaults for the same module as a failure

When reading in a YAML stream, libmodulemd must keep track of the set of modules for which a defaults object exists and throw a validation error if defaults for the same module occur twice in one stream.

Warning: g_object_unref: assertion 'G_IS_OBJECT (object)' failed

For this particular mmd file, the new_from_string method shows following warning, but things are working, so not a critical issue :).

from gi import Repository
r = Repository.get_default() # noqa
r.require('Modulemd') # noqa
from gi.repository import Modulemd

data = """
data:
  api:
    rpms: [tangerine, perl-Tangerine]
  components:
    rpms:
      perl-List-Compare: {cache: 'http://pkgs.fedoraproject.org/repo/pkgs/perl-List-Compare',
        rationale: A dependency of tangerine., ref: 76f9d8c8e87eed0aab91034b01d3d5ff6bd5b4cb,
        repository: 'git://pkgs.fedoraproject.org/rpms/perl-List-Compare'}
      perl-Tangerine: {cache: 'http://pkgs.fedoraproject.org/repo/pkgs/perl-Tangerine',
        rationale: Provides API for this module and is a dependency of tangerine.,
        ref: 4ceea43add2366d8b8c5a622a2fb563b625b9abf, repository: 'git://pkgs.fedoraproject.org/rpms/perl-Tangerine'}
      tangerine: {buildorder: 10, cache: 'http://pkgs.fedoraproject.org/repo/pkgs/tangerine',
        rationale: Provides API for this module., ref: fbed359411a1baa08d4a88e0d12d426fbf8f602c,
        repository: 'git://pkgs.fedoraproject.org/rpms/tangerine'}
  dependencies:
    buildrequires:
      zebra: stripes
      base-runtime: master
      fluffy: slippers
    requires:
      fidget: spinner
      base-runtime: master
      silly: puddy
      glass: water
  description: This module demonstrates how to write simple modulemd files And can
    be used for testing the build and release pipeline.
  filter: {}
  license:
    module: [MIT]
  name: testmodule
  profiles:
    default:
      rpms: [tangerine]
  references: {community: 'https://fedoraproject.org/wiki/Modularity', documentation: 'https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules',
    tracker: 'https://taiga.fedorainfracloud.org/project/modularity'}
  stream: master
  summary: A test module in all its beautiful beauty
  version: 20170109091357
  xmd:
    mbs:
      buildrequires:
        zebra:
          ref: 08f8671f5925ecbd7c55d83e8368dd0be81ef8ed
        base-runtime:
          ref: ae993ba84f4bce554471382ccba917ef16265f11
        fluffy:
          ref: ea83325b231f64c575ea104bb698e9d43b80469a
      requires:
        fidget:
          ref: 5aa32bd61aadaec8295ef90f79daaf190e387509
        base-runtime:
          ref: ae993ba84f4bce554471382ccba917ef16265f11
        silly:
          ref: 0e2a1f9246118180d61aad731923ebd85154bc6e
        glass:
          ref: ea8753e832974dd5b0d95cd6bb7d26727d2ba742
      commit: 7fea453bc362cc8e5aa41e129e689baea853653d
      scmurl: git://pkgs.stg.fedoraproject.org/modules/testmodule.git?#7fea453
document: modulemd
version: 1

"""

mmd = Modulemd.Module.new_from_string(data)

rpm_buildopts lost after dumps and new_from_string with 1.1.0

It also prints "Incompatible modulemd version" while running these methods. Reproducer:

from gi import Repository
r = Repository.get_default() # noqa
r.require('Modulemd') # noqa
from gi.repository import Modulemd

data = """
document: modulemd
version: 1
data:
  name: testmodule
  stream: master
  version: 20180205135154
  context: c2c572ec
  summary: A test module in all its beautiful beauty
  description: This module demonstrates how to write simple modulemd files And can
    be used for testing the build and release pipeline.
  license:
    module:
    - MIT
  dependencies:
    buildrequires:
      platform: f28
    requires:
      platform: f28
  references:
    community: https://docs.pagure.org/modularity/
    documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules
  xmd:
    mbs:
      buildrequires:
        platform:
          filtered_rpms: []
          ref: virtual
          stream: f28
          version: '3'
          context: '00000000'
      commit: 65a7721ee4eff44d2a63fb8f3a8da6e944ab7f4d
      requires:
        platform:
          filtered_rpms: []
          ref: virtual
          stream: f28
          version: '3'
      rpms:
        perl-List-Compare:
          ref: ac0f3bccca9dcb8465c434ac4c38974bf6205c28
        perl-Tangerine:
          ref: 7e96446223f1ad84a26c7cf23d6591cd9f6326c6
        tangerine:
          ref: c0f9a7dbd8cf823a2bdc19eeeed20d22b0aa52bf
      mse: true
      scmurl: https://src.fedoraproject.org/modules/testmodule.git?#65a7721ee4eff44d2a63fb8f3a8da6e944ab7f4d
  profiles:
    default:
      rpms:
      - tangerine
  api:
    rpms:
    - perl-Tangerine
    - tangerine
  components:
    rpms:
      perl-List-Compare:
        rationale: A dependency of tangerine.
        repository: git://pkgs.fedoraproject.org/rpms/perl-List-Compare
        ref: master
        cache: http://pkgs.fedoraproject.org/repo/pkgs/perl-List-Compare
      tangerine:
        rationale: Provides API for this module.
        buildorder: 10
        repository: git://pkgs.fedoraproject.org/rpms/tangerine
        ref: master
        cache: http://pkgs.fedoraproject.org/repo/pkgs/tangerine
      perl-Tangerine:
        rationale: Provides API for this module and is a dependency of tangerine.
        repository: git://pkgs.fedoraproject.org/rpms/perl-Tangerine
        ref: master
        cache: http://pkgs.fedoraproject.org/repo/pkgs/perl-Tangerine
"""

mmd = Modulemd.Module.new_from_string(data)
mmd.upgrade()
mmd.set_rpm_buildopts({'macros': '%my_macro 1'})
mmd = Modulemd.Module.new_from_string(mmd.dumps())
assert mmd.get_rpm_buildopts() != {}

modulemd: include checksum information into artifacts.rpm

Lookup by NEVRA is extremely slow... See filelists.xml for example how checksum is used.

I was pretty sure I have filled this RFE before, but can't find it.

Make the stream and profiles optional in modulemd-defaults

This is a backwards compatible change.

Use cases for optional default stream: modules providing alternatives to non-modularized software and documents defining default streams under intents

User cases for optional default profiles: perhaps highly specialized modules that do not provide any profiles whatsoever (profiles are optional) and those defining their profiles under intents

build: add `modulemd_dep`

By using declare_dependency() for ease of bundling libmodulemd..

In theory, should be something like

modulemd_dep = declare_dependency(
    include_directories : include_directories('.'),
    link_with : modulemd_lib,
)

modulemd_defaults_dumps () Adds fields which where not initially present on the yaml document.

In [1]: import gi

In [2]: gi.require_version('Modulemd', '1.0')

In [3]: from gi.repository import Modulemd

In [4]: x = '/home/ipanova/Downloads/fcb39b9a0b045c35442f1ab05dfd230148b4662fe1ab6ad7cd3e9722ffce8b84-modules.yaml'

In [5]: modules = Modulemd.objects_from_file(x)

In [6]: modules
Out[6]: 
[<Modulemd.Module object at 0x7fb68a5dd2d0 (ModulemdModule at 0x5645864fb000)>,
 <Modulemd.Module object at 0x7fb689cc8d70 (ModulemdModule at 0x5645864fb0f0)>,
 <Modulemd.Module object at 0x7fb689cc8d20 (ModulemdModule at 0x5645864fb1e0)>,
 <Modulemd.Module object at 0x7fb689cc8cd0 (ModulemdModule at 0x5645864fb2d0)>,
 <Modulemd.Module object at 0x7fb689cc8c80 (ModulemdModule at 0x5645864fb3c0)>,
 <Modulemd.Defaults object at 0x7fb689cc8be0 (ModulemdDefaults at 0x564586493280)>,
 <Modulemd.Defaults object at 0x7fb689cc8aa0 (ModulemdDefaults at 0x5645864932c0)>,
 <Modulemd.Defaults object at 0x7fb689cc8a50 (ModulemdDefaults at 0x564586493300)>]

In [7]: defaults = modules[-1]

In [8]: defaults
Out[8]: <Modulemd.Defaults object at 0x7fb689cc8a50 (ModulemdDefaults at 0x564586493300)>

In [9]: defaults.dumps()
Out[9]: '---\ndocument: modulemd-defaults\nversion: 1\ndata:\n  module: django\n  profiles:\n    1.6: [default]\n  intents: {}\n...\n'

Meanwhile in the downloaded modules.yaml file the defaults for django look like:

---
document: modulemd-defaults
version: 1
data:
  module: django
  profiles:
    1.6: [default]
...

Version : libmodulemd-1.5.2-1.fc28.x86_64
NOTE: Intents where NOT present in the original yaml document

Same behaviours is noticed with modulemd_defaults_dump ()

priorities resolver doesn't de-duplicate?

I tried this today:

import gi
gi.require_version('Modulemd', '1.0')
from gi.repository import Modulemd  # noqa

# Here, load the same file twice.
# All entries in these two lists should be duplicates of each other.
objects_from_repo_a = Modulemd.objects_from_file('modules.yaml')
objects_from_repo_b = Modulemd.objects_from_file('modules.yaml')

prioritizer = Modulemd.Prioritizer()
prioritizer.add(objects_from_repo_a, 0)
prioritizer.add(objects_from_repo_b, 1)
supposedly_merged_objects = prioritizer.resolve()

# I would expect all three of these to be the same length... but they are not.
print(len(objects_from_repo_a))
print(len(objects_from_repo_b))
print(len(supposedly_merged_objects))

Am I doing something wrong?

Modulemd: enforce N-E:V-R.A format in artifacts/rpms

The DNF team proposes a change in how libmodulemd handles RPM NEVRAs.
We'd like to always get records in N-E:V-R.A format.
libmodulemd SHOULD fail on reading data in invalid format.
libmodulemd MUST fail on writing data in invalid format.

As part of this, we'd like to enforce '0:' epoch when the epoch is zero or empty in RPM.
In some cases missing epoch can represented as "any epoch", which may result in bad UX.

Module component refs are lost when dumping the object to a string

If I have a module component that has a ref specified, the ref is lost when I dump the module to a string.

Here is the reproducer in Python:

>>> import gi
>>> gi.require_version('Modulemd', '1.0')
>>> from gi.repository import Modulemd
>>> mmd = Modulemd.Module().new_from_string("""
... document: modulemd
... version: 1
... data:
...     summary: A test module in all its beautiful beauty.
...     description: This module demonstrates how to write simple modulemd files And can be used for testing the build and release pipeline.
...     license:
...         module: [ MIT ]
...     dependencies:
...         buildrequires:
...             platform: el8
...         requires:
...             platform: el8
...     references:
...         community: https://fedoraproject.org/wiki/Modularity
...         documentation: https://fedoraproject.org/wiki/Fedora_Packaging_Guidelines_for_Modules
...         tracker: https://taiga.fedorainfracloud.org/project/modularity
...     profiles:
...         default:
...             rpms:
...                 - acl
...     api:
...         rpms:
...             - acl
...     components:
...         rpms:
...             acl:
...                 rationale: needed
...                 ref: rhel-8.0
...         modules:
...             testmodule:
...                 ref: private-x
...                 rationale: Testing module inclusion.
...                 buildorder: 10
... """)
>>> assert mmd.get_module_components()['testmodule'].get_ref() == 'private-x'
>>> mmd2 = Modulemd.Module().new_from_string(mmd.dumps())
>>> assert mmd2.get_module_components()['testmodule'].get_ref() == 'private-x'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError

`version` must be set before `data` in the modulemd when dependencies are specified

When loading a modulemd from a file, the following fails with "Invalid document [Unknown modulemd version]. Skipping it":

document: modulemd
data:
    dependencies:
        buildrequires:
            platform: f28
        requires:
            platform: f28
version: 1

When specifying the version before dependencies is processed, it works as expected.

document: modulemd
version: 1
data:
    dependencies:
        buildrequires:
            platform: f28
        requires:
            platform: f28

I'm not sure how to workaround this in MBS as both should be valid modulemds, or at least they were in the Python implementation of modulemd.

Methods on mmd.props.buildopts aren't available unless a buildopt is set

You can test this with the following Python code:

import gi
from gi.repository import Modulemd
import requests
mmd_str = requests.get('https://src.fedoraproject.org/modules/testmodule/raw/master/f/testmodule.yaml').text
mmd = Modulemd.Module.new_from_string(mmd_str)
mmd.props.buildopts is None

I'm currently using libmodulemd v1.5.1.

`new_from_file` and `new_from_string` hang indefinitely on bad YAML

When trying to create a Moduemd.Module object with bad YAML, it will hang as long as the bad YAML starts with document: modulemd\n.

For example:

from gi.repository import Modulemd
Modulemd.Module().new_from_string('document: modulemd\nBad YAML')

We need libmodulemd to be able fail on bad modulemd files that users submit to MBS.

libmodulemd should provide a prioritized merging routine for modules and defaults

When dealing with multiple repositories, we need to handle how to merge content. For example, if we have a release repository (fedora-modular) and an updates repository (fedora-modular-updates) and both provides differing profiles in a modulemd-defaults file, we need to prefer the fedora-modular-updates version.

Getting a validation error when context is not set

When parsing a modules.yaml file and one of the modulemd entries doesn't contain a context element, I get:

ValidationError (Modulemd:3cc94ec5-27ea-4d19-b942-a80ec0317d84) (Field is required: ['context'])

However, this seems to disagree with what is present in the spec.

fedora-modularity / libmodulemd Goto Github PK

libmodulemd's People

Contributors

Stargazers

Watchers

Forkers

libmodulemd's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs