qiboteam / qibodb

qibodb

qibodb's Issues

Predefined queries

• platform qpu:iqm5q latest
• full collection
  • print a summary table with ids and most relevant information
• full database
  • i.e. the index, implementing the "runcard" mode
  • another summary table

Tests

It should be possible to write meaningful unit tests with pytest-mongodb (and some even stand-alone).

Moreover, thanks to the container server, it should be possible to run full integration tests on any platform which provides podman and a web connection.

• unit tests
• with pytest-mongodb
• integration tests

Update document/runcard

• dynamically derive an update object from the insert objects
• enforce the behavior of dataclasses.replace at DB level
  • updates are inserted as new records

For some collections, like package, better to allow mutation.

Stub for generated Models

Unfortunately, there is a bunch of duplication involved in inserting and retrieving information from the DB.
You could get a grasp for this from a simple (official) example:
https://www.mongodb.com/developer/languages/python/python-quickstart-fastapi/

This led to #8 and the ReadModel and UpdateModel creation.

Unfortunately, being dynamically generated, they are pretty useless for the static analyzers and the intellisense.
For this reason, it would be useful to generate stub files (I won't generate code in Python files, especially in existing ones, to keep generated code separate from manually written one).
https://mypy.readthedocs.io/en/stable/stubs.html

This is something that should be pretty simple to do automatically, starting with the stubgen CLI.

The only thing I'm not sure about is whether I want to commit to the repo or not.
Contra: I don't like adding in Git and reviewing in GitHub generated files (even though they will look very close to the manually written ones)
Pro: they will be available in development as well.

Validation

@scarrazza @stavros11 I'm now facing the problem of whom delegating the validation to.

I have essentially two ingredients, that generate two possible options:

• MongoDB: by default it has no validation, but I can set it with a command passing a JSON schema
• Pydantic, that is made on purpose, and it will turn dictionaries in objects

The database has of course has to be there, but it only works with dictionaries (and at most TypedDict for type hints). However, Pydantic gives me the objects, it should be able to turn the model specified by them into a JSON schema (which I would not need any longer), and it has a better integration with type checks.

On the one side, I wonder if I should keep insisting with Pydantic adoption, since I could do everything with MongoDB (giving up on type hints, and writing manually the JSON schema). On the other, I really want the objects, rather than dictionaries around, and Pydantic is a dependency actively developed, with >13k stars on GitHub... all in all it seems quite reliable.

So, the two tools are so complete that they span part of the things done by the other, and in particular validation is in both.
But they are also complementary, so I would just decide to whom attributing the intersection, and keep using both to have the full set of features.

Get variables from environment

Like the QIBOPORT or the CONTAINER name.

Read API

As in #6 an update variant of insert classes is described, a further read variant should be generated.

In particular, the only additional information should be:

• the id
• the ctime

It would be rather useful to return objects, rather than dicts, to the packages directly consuming Qibodb in Python (of course for CLI dicts are perfectly fine).

Consider TerminusDB

It has native Git-like features (versioning, diff, ...) and graph queries.

https://terminusdb.com/products/terminusdb/

Import/export database or collection

• export DB or collection to JSON or other format
• import DB or collection from ...

Actually possible out of the box with Compass.

Impose uniqueness

There are collections in which it makes sense not to insert records, if a certain value is already contained.
E.g. the package collection, for which it does not make sense to have multiple entries for the same version.

In SQL the UNIQUE attribute is doing exactly this. There should be a way to do it even in MongoDB, let's just check if it's convenient, or better to manually search.

https://www.mongodb.com/docs/manual/core/index-unique/
https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.create_index

Versions management

Add subcommand to update the relation between package versions (essentially add new versions, or modify the supersedes key).

Bundle should be written in transactions

If an element insertion fails, nothing should be written at all.
https://www.mongodb.com/docs/manual/core/transactions/

Info command

• retrieve available dbs
• retrieve available collections per db
• retrieve all available collections (db.collection)
• retrieve settings