hochfrequenz / mig_ahb_utility_stack Goto Github PK

https://github.com/Hochfrequenz/edifact-templates/commit/8f4e6c524758dc28c61e186ec232e7bfe98c345a

Hier scheint die 11042er Maus noch nicht ganz korrekt zu sein:

https://github.com/Hochfrequenz/edifact-templates/blob/606d81dffe3621fbcf84a59be9e34a5cea0e3b8a/maus/FV2210/UTILMD/11042_maus.json#L702

Dort zeigt der EdiseedPfad auf ein "Kündigungstermin des Vertrags", welches es nicht gibt in der 11042 statt auf das "Datum zum geplanten Leistungsbeginn"

src\maus\models\anwendungshandbuch.py:156: error: Argument "validator" to "field" has incompatible type "Callable[[Any, Attribute[Optional[str]], Optional[str]], Any]"; expected "Union[Callable[[Any, Attribute[str], str], Any], Sequence[Callable[[Any, Attribute[str], str], Any]], None]" [arg-type]

Problem: Der Predecessor Qualifier Z01 kommt aus der SG8 und die SG9 wird separat behandelt, unabhängig von ihrer Mutter SG8.

relates to Hochfrequenz/ahbicht#119

https://github.com/Hochfrequenz/mig_ahb_utility_stack/actions/runs/4566827926/jobs/8059895848

Warning: Unexpected input(s) 'token', valid inputs are ['config-name', 'name', 'tag', 'version', 'publish', 'prerelease', 'commitish', 'header', 'footer', 'disable-releaser', 'disable-autolabeler']
Run release-drafter/release-drafter@v5
"pull_request_target.opened" is not a known webhook name (https://developer.github.com/v3/activity/events/types/)
"pull_request_target.reopened" is not a known webhook name (https://developer.github.com/v3/activity/events/types/)
"pull_request_target.synchronize" is not a known webhook name (https://developer.github.com/v3/activity/events/types/)
"pull_request_target.edited" is not a known webhook name (https://developer.github.com/v3/activity/events/types/)
Warning: Hochfrequenz/mig_ahb_utility_stack: Invalid config file
{
name: 'event',
id: '4566827926',
stack: 'Error: Configuration file .github/release-drafter.yml is not found. The configuration file must reside in your default branch.\n' +
' at getConfig (/home/runner/work/_actions/release-drafter/release-drafter/v5/dist/index.js:142719:13)\n' +
' at async drafter (/home/runner/work/_actions/release-drafter/release-drafter/v5/dist/index.js:142347:20)\n' +
' at async Promise.all (index 0)',
type: 'Error'
}
Error: Invalid config file

https://github.com/Hochfrequenz/edifact-templates/blob/606d81dffe3621fbcf84a59be9e34a5cea0e3b8a/maus/FV2210/UTILMD/11042_maus.json#L709

In our public repos, we use CodeQL to scan the code:
https://github.com/search?q=org%3AHochfrequenz+%22codeql-action%22&type=code

CodeQL Action v1 won't be supported anymore by December 2022:

CodeQL Action v1 will be deprecated on December 7th, 2022. Please upgrade to v2. For more information, see https://github.blog/changelog/2022-04-27-code-scanning-deprecation-of-codeql-action-v1/

Please upgrade all our repositories that use CodeQL v1 (see search query above) to CodeQL v2.
It's a rather dumb task that is not urgent.

• ValuePool
• FreeText
• DateTime

In the docx file MSCONSAHB-informatorischeLesefassung3.1aKonsolidierteLesefassungmitFehlerkorrekturenStand27.09.2022_20230331_20221001.docx on page 51 we have the following situation:

So the FlatAHB instance can not get created.

The following Prüfidentifikatoren have this issue:

# MSCONS
13002
13003
13005
13006
13007
13008
13009
13010
13011
13012
13013
13014
13015
13016
13017
13018
13019
13020
13021
13022
13023
13025
13026
# INVOICE
31001
31002
31003
31004
31005
31006
31007
31008
31009
31010
31011

Make Package PEP621 Compatible

Das schmerzt mich hier etwas, da es den tag-driven approach etwas kaputt macht.
Soll ich hier ein Issue aufnehmen damit wir es nicht vergessen?

Originally posted by @hf-krechan in #189 (comment)

Probably also relevant for other pruefis, but here in the 11042.maus the MP-ID for Absender and Empfänger should be a Freefield Data Element and not a Value Pool: https://github.com/Hochfrequenz/edifact-templates/blob/03559bccf5f29d3c457667aec393e57d5338cbe8/maus/FV2110/UTILMD/11042_maus.json#L108
Also they are missing their ahb_expression:
Current state:

{
"data_element_id": "3039",
"discriminator": "$[\"Dokument\"][0][\"Nachricht\"][0][\"MP-ID Absender\"][0][\"MP-ID\"]",
"value_pool": [],
"value_type": "VALUE_POOL"
}

Expected:

{
"ahb_expression": "X",
"data_element_id": "3039",
"discriminator": "$[\"Dokument\"][0][\"Nachricht\"][0][\"MP-ID Absender\"][0][\"MP-ID\"]",
"entered_input": null,
"value_type": "TEXT"
}

• AHB-Expression Absender
• value_type Absender
• AHB-Expression Empfänger
• value_type Empfänger

Can be hardcoded if it is urgent, but this always has the danger to be overwritten again when the maus is newly generated.

z.B. Geplante Turnusablesung des MSB (Strom)) schauen? Die hängt irgendwie unter der "Referenz auf die ID der Marktlokation f\u00fcr Termine der Marktlokation" bzw. hat den Diskriminator

Even though segment groups in segment groups are optional we still needs this line otherwise you'll get an error:

    SegmentGroup(
E   TypeError: __init__() missing 1 required keyword-only argument: 'segment_groups'

At this moment (2022-02-22) we think that if in an ahb there is not expression it does not appear in this Prüfi.

relates to #150 . see marshmallow-code/marshmallow#2044

super. wenn es da größere erkenntnisse gab, kannst du sie ja noch in der readme oder im template-repo dokumentieren

Originally posted by @hf-kklein in #84 (review)

Also in https://github.com/Hochfrequenz/edifact-templates/pull/151 there is a hardcoded 11042.maus fix from Kevin, where there are ahb_expressions missing and false. It would be good to fix this in the generated maus and not hardcode it in the maus file directly.
(But you can fix it in the AHB csv if that would be the problem)

Das Feld MP-ID Absender sollte ein Freifeld-Text sein.

{
    "data_element_id": "3039",
    "discriminator": "$[\"Dokument\"][0][\"Nachricht\"][0][\"MP-ID Absender\"][0][\"MP-ID\"]",
    "value_pool": [],
    "value_type": "VALUE_POOL"
},

    ist hier der falsche scope, hätte `get_format_of_pruefidentifikator` schöner gefunden

Originally posted by @hf-krechan in Hochfrequenz/kohlrahbi#48 (comment)

Tracking issue for:

• https://github.com/Hochfrequenz/mig_ahb_utility_stack/security/code-scanning/3

This part of the regular expression may cause exponential backtracking on strings starting with 'A:9:9[9:9=' and containing many repetitions of '009:9='.

type_check run-test: commands[2] | mypy --show-error-codes tests/integration_tests
tests/integration_tests/edifact-templates/extract_all_conditions_from_ahbs.py:10: error: Cannot find implementation or library stub for module named "ahbicht.mapping_results" [import]
tests/integration_tests/edifact-templates/extract_all_conditions_from_ahbs.py:10: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 16 source files)
ERROR: InvocationError for command /home/runner/work/mig_ahb_utility_stack/mig_ahb_utility_stack/.tox/type_check/bin/mypy --show-error-codes tests/integration_tests (exited with code 1)
___________________________________ summary ____________________________________
unit_tests: commands succeeded
integration_tests: commands succeeded
linting: commands succeeded
coverage: commands succeeded
ERROR: type_check: commands failed

I don't get it. Why does the usual CI just pass but the upload behaves differently?
There are in fact unresolved (ahbicht) dependencies in the script mentioned above but they shouldn't let the package build fail.

https://github.com/Hochfrequenz/mig_ahb_utility_stack/runs/6796916740?check_suite_focus=true#step:5:175

I just ignored the warnings for now. Sorry for that.

• too-many-branches
• too-many-nested-blocks

(May also be relevant for other Prüfis and other fields.)
If you have a look at the addresses, f.ex. in the 11042.maus you can see for the "Korrespondenzanschrift des Kunden des Messstellenbetreibers" that the Name, Ort, Postleitzahl, etc. only have their name as Discriminator/Path, but we need an EDISeedPath. This only works for "Struktur" which has $[\"Dokument\"][0][\"Nachricht\"][0][\"Vorgang\"][0][\"Korrespondenzanschrift des Kunden des Messstellenbetreibers\"][0][\"Struktur\"] as Discriminator/Path. We would expect and need this also for the other address fields.
Example Ort:

• current state: "discriminator": "Ort"
• expected value: "discriminator": "$[\"Dokument\"][0][\"Nachricht\"][0][\"Vorgang\"][0][\"Korrespondenzanschrift des Kunden des Messstellenbetreibers\"][0][\"Ort\"]"

At the moment this was hardcoded in https://github.com/Hochfrequenz/edifact-templates/pull/151/files (here you can also see reality vs. expections).
⚠You need to look out to not overwrite this state for the wimbee-backend.

• Transaktionsgrund im MIG + Transaktionsgrund im AHB ==> JsonPath für EdifactSeed
• Mehrere mögliche Wert im AHB ==> Wertevorrat pro Datelement/pro Jsonpath + angehägnte Bedingungen

{
    "$['Nachricht'][0]['Dokument'][0]['Vorgang'][0]['Transaktionsgrund']": {
        "ahb_expression": "Muss[2061]",
        "dataelements": {
            "9015": {
                "mig_name": "Statuskategorie",
                "possible_values": {
                    "7": {
                        "meaning": "Transaktionsgrund",
                        "ahb_expression": "X"
                    }
                }
            }
            "9013": {
                "mig_name": "Statusanlass",
                "possible_values ": {
                    " E01 ": {
                        " meaning ": " Einzug / Auszug(Umzug)",
                        " ahb_expression ": "X"
                    },
                    "E02 ": {
                        " meaning ": " Einzug / Neuanlage ",
                        " ahb_expression ": "X",
                    },
                   ...E03, ZJ4...
                }
            }
        }
    }
}

• Add a Cache to the file based maus provider (not all maus providers) such that, if the files are stable, the result is returned faster
• Allow the user to manually disable the cache
• See here: Hochfrequenz/ahbicht@fd2135d for a similar case

Bei der Korrespondenzanschrift des Messstellenbetreibers haben zwei Datenelemente den Discriminator null, vermutlich Name und Straße&Hausnummer, da die fehlen.

Related Azure Devops Ticket:
https://dev.azure.com/christianwessel0427/WiM.BEE/_workitems/edit/109

Actual Behaviour:

> PS  .\maus_cli.exe
Traceback (most recent call last):
  File "maus\__init__.py", line 83, in <module>
  File "click\core.py", line 1130, in __call__
  File "click\core.py", line 1055, in main
  File "click\core.py", line 1404, in invoke
  File "click\core.py", line 760, in invoke
  File "maus\__init__.py", line 46, in main
TypeError: expected str, bytes or os.PathLike object, not NoneType
[17792] Failed to execute script '__init__' due to unhandled exception!

Expected Behaviour: Meaningful error message

"und Summenteil" is alone in this line, we can't handle this yet but maybe could merge the line with its predecessor.

• The type name is DataElementDataType
• The name of the attribute having this type is (DataElement)ValueType

Downstream services (namely ahbicht) now need to decide whether to use the name DataType or ValueType. This confusion/ambiguity is unnecessary.

Ideas

• From the DataElements perspective the name "value type" seems more appropriate imo, because it characterizes the data elements value.
• For (also non python) downstream dependents it is easier if we don't change the attribute name but the type name (because this doesn't affect the json serialization)

    anmerkung von @hf-fvesely wir wollen den edi-qualifier nicht komplett verlieren, sondern ihn irgendwie im zugriff behalten. z.b. im meaning einfachen appenden?

Originally posted by @hf-kklein in #149 (comment)

The pipeline: "Build and publish Python 🐍 distributions 📦 to PyPI and TestPyPI" does not work.

What happened:

https://github.com/Hochfrequenz/mig_ahb_utility_stack/runs/4581156534?check_suite_focus=true

Run pypa/gh-action-pypi-publish@master
/usr/bin/docker run --name a6825ce3aad4b5f5d402d96be993ed9e93a9d_1f59ca --label 6a6825 --workdir /github/workspace --rm -e INPUT_PASSWORD -e INPUT_USER -e INPUT_REPOSITORY_URL -e INPUT_PACKAGES_DIR -e INPUT_VERIFY_METADATA -e INPUT_SKIP_EXISTING -e INPUT_VERBOSE -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_REF_NAME -e GITHUB_REF_PROTECTED -e GITHUB_REF_TYPE -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e RUNNER_OS -e RUNNER_ARCH -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/mig_ahb_utility_stack/mig_ahb_utility_stack":"/github/workspace" 6a6825:ce3aad4b5f5d402d96be993ed9e93a9d "token" "***" "" "dist" "true" "false" "false"
Checking dist/maus-0.0.2-py3-none-any.whl: PASSED
Checking dist/maus-0.0.2.tar.gz: PASSED
Uploading distributions to https://upload.pypi.org/legacy/
Uploading maus-0.0.2-py3-none-any.whl

0%| | 0.00/23.1k [00:00<?, ?B/s]
35%|███▍ | 8.00k/23.1k [00:00<00:00, 48.4kB/s]
100%|██████████| 23.1k/23.1k [00:00<00:00, 49.8kB/s]
Error during upload. Retry with the --verbose option for more details.
HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
File already exists. See https://pypi.org/help/#file-name-reuse for more information.

What I expected:

The package to be updated on pypi: https://pypi.org/project/maus/ (still 0.0.2, released on 2021-12-13)

Ideally you could check in O(1) if a line with specific properties is contained in an anwendungshandbuch:

https://github.com/Hochfrequenz/kohlrahbi/blob/c9143fdc134977548199e3374aaa98bd226e1eba/src/kohlrahbi/unfoldedahb/unfoldedahbtable.py#L51

idea: maybe build something like a lookup structure or make the ahbline hashable?

It's broken since a while. Don't know why.
Someone should investigae

anwendungshandbuch.py:193: RemovedInMarshmallow4Warning: The 'default' argument to fields is deprecated. Use 'dump_default' instead.
maus_version = fields.String(required=False, allow_none=True, default=_VERSION)

anwendungshandbuch.py:194: RemovedInMarshmallow4Warning: The 'default' argument to fields is deprecated. Use 'dump_default' instead.
description = fields.String(required=False, allow_none=True, default=None)

anwendungshandbuch.py:195: RemovedInMarshmallow4Warning: The 'default' argument to fields is deprecated. Use 'dump_default' instead.
direction = fields.String(required=False, allow_none=True, default=None)

Implement the new Format Version 2210 into the Maus

This part of the regular expression may cause exponential backtracking on strings starting with 'A:9:9[9:9=' and containing many repetitions of '009:9='.