cimt-ag / data_vault_pipelinedescription Goto Github PK

a not historized raference table must not declare a diff hash columns

Check an adapt to provide Plans for tables, without connected satellites.

"This should only load a hub.
Main concern, if a plan will be generated whithout any ""leaf table"""

• Writing complete
• MWG QA
• Joscha QA
• Change DVDP Structure diagram to new method
• Final tweaks

"A source may have a complete or partitioned set of a business objects. On this knowledge an internal deletion detection can be implemented. "

• define syntax
• load and check syntax with compiler
• provide derivations by compiler
• provide test cases for different scenarios

• Case aligned with current variation catalog
• test case(s) created (3110-3130,3170-3210)

When a table is targeted by multiple relations, the resulting column structure must be the same for all relations
same number of columns
same column names and types

if this is violated, the compiler must log a proper error message and stop

There is already a stub function in the code:
check_multifield_mapping_consistency_of_column

This test should determine, if the declaration of the field order for a key hash is transported correctly to the DVPI and
has an impact on the DVPI summary

• Test created

• specification (incl default of data vault profile)
• parsing
• presentation in table properties

• test created

"Explicit for bk or dck
Implicit for links"

First approach of document created. Further writing according to the list below, will be added later

use case analysis

Information representation needed, annotatef with sourceendpoints and fields

Source specification

Breakdown of Endpoint data structure to single table representation

Fields, types, business key parts, parsing rules, increment pattern, tracking/deletion detection method

Tools: metadata discovery, content analysis

Result: Pipelines with field list

data vault modelling

Table structure and mapping of fields

Tool/Resource: Modelling tool, already established model

Vault model completion and verification

Full/Conform naming of tables and columns
Essential Naming of key and diff hash columns
Integration into established model (no conflicts)

Tools: Generators, check routines

implementation

Deployable and ""executable"" Artifact for

• Deployment of DB tables
• Processing and loading incoming data
  Bandwidth of methods
• Can be just the dvpd = full generic engine.
• Dvpd + copy of current engine
• Generated process (dvpd only provided as documentation)
• Generated template, with final manual work

(Discussion about pro/cons of full coded artifacts against generic solutions)

Generation of Fetch

Test of pipeline

• All increment scenarios
• All historization scenarios

Tools: generated vault to source views, generated testdata (variety, change over time)

Deployment

• Schedule

operations

usage of data

Tools: Vault model, columns, types, comments, linage

Currently there can be only one additional HK of a Hub in a link. (Hard coded as H1 and H0) For the very rare case of a multi back reference in the same relation, this would be not sufficient.

"Depends on target database and source system or fetch processing
Translation and normalisation is respobility of processing engine.
Recommendenstion: upper case, remove spaces, check syntax
Possible support: separate configuration JSON, mapping of source to target types for every product (needs a optional product specification in the DVPD for source and targets)"

The final physical table name must be unique to all others

• check implemented
• test created

Declaration will be used to check and force the field mappings to meet these names and types and provide consistent declarations about hashing concatenation

• Case checked and aligned with new catalog (see 3490)

Stettings of property must be unique and numeric

• Test cases implemented
• DVPDC Check implemented
• Test cases integrated in automatic tests

• Challenge: How to declare explicitly the recursive content? Allow usage of field names, indicated by prefixing with "!" (maybe force the use of field names in case of recursive parents)

• add declaration of key name to recursion declaration of link

This test should determine, if the declaration of the field order for a diff hash is transported correctly to the DVPI and
has an impact on the DVPI summary

• Test created

"When the same satellite content is loaded for all field groups, currently the process mapping and staging provide a row hash for every field group. Maybe it is possible zu reduce this to one row hash.
The probability of occurence for this constellation is very low, so dont hurry."

• Written
• QA Matthias
• QA ####

• change syntax reference
• add propertiy to loader
• use propterties in column derivation logic
• change / review syntax examples

• Case aligned with current variation catalog
• test case(s) created (6xx)

Currently the position of a filed in the incoming resultset must be declare explicitly. This is due to the fact, postgresSQL array expandation hat no way to provide the index of the array. If somehow this can be fixed, we could remove the explicit index.