absaoss / fixed-width Goto Github PK

Background

Improve or add code coverage support to be able measure code quality.

Feature

New ability to measure current code coverage as one of QA metrics.
Add GH action to check changed file coverage.

Feature

Add support for spark 3.2.1

Background

If a special column, which name is specified by Spark spark.sql.columnNameOfCorruptRecord setting, exists in the schema, it is used to log parsing errors into it.

Feature

If the column of the name specified in spark.sql.columnNameOfCorruptRecord exists in the schema, use it for logging parsing errors there.

Related to and dependent on #1

Describe the bug

If the compulsory width metadata value type is not string, it's not recognized. Integer type value should be accepted too.

To Reproduce

Define a schema, where the metadata value of a field is an integer value, not string. (e.g. 5 instead of 5). Processing will fail with an exception: "Unable to parse metadata: width of column:..."

Expected behaviour

Integer definition of width is accepted.

Background

Options are passed to Spark data sources as key-value pairs. It is very easy to make a typo in an option name. If the data source has a default behaviour for this option (e.g. the option is not mandatory), the passed option will be unrecognized and therefore ignored.

It would be very useful if the data source would detect unrecognized options.

Feature

1. Detect unrecognized options and log them as warnings.
2. Add .option("pedantic", true), which, when enabled, will fail the Spark Application if there is at least one unrecognized option passed to the source.

Additional context

As an inspiration for possible implementation, Cobrix uses a wrapper that takes incoming options. All configuration is read from that wrapper instead of the original configuration. The wrapper records which options are queried at least once. At any given point it allows getting the list of options that haven't been used.
An advantage of such implementation is that if there are dependencies between options, e.g. a particular option is only used if some other option is enabled, it is tracked automatically.

https://github.com/AbsaOSS/cobrix/blob/master/spark-cobol/src/main/scala/za/co/absa/cobrix/spark/cobol/parameters/Parameters.scala

Description
build.sbt was not ready for the cross-compilation signed release. Figure out how to do it best. Steal it from Cobrix.

Background

When dateFormat is specified it either represents a date or a date + time. That effectively disables to be able both timestamp and date within the same file.

Feature

Add timestampFormat parameter used similarly as dateFormat and make each of them exclusive to use with timestamp and date respectively.

Background

Spark has the mode setting to describe readers behavior in case of parsing error.

Feature

Reflect PERMISSIVE and FAILFAST settings in case of parsing errors.

• FAILFAST - in case of parsing error, the process immediately fails with exception
• PERMISSIVE - the parsing continues with best effort results

Background

Add code coverage support to be able measure code quality.

Feature

New ability to measure current code coverage as one of QA metrics.

Background

usually the values in fixed width forma columns might have some trailing spaces. That can disable parsing of non-string types.

Feature

Create a code accepting a metadata setting driving the trimming of the column values, overriding the global setting.

Proposed Solution

Suggested metadata field name: trim.
Accepted type boolean or string convertible to boolean.