moov-io / metro2 Goto Github PK

Metro2 Version: 0.3.3

What were you trying to do?
I am trying to generate a json and convert it into metro.
This is the json
{ "header": { "recordDescriptorWord": 426, "recordIdentifier": "HEADER", "equifaxProgramIdentifier": "5555555555", "transUnionProgramIdentifier": "6666666666", "activityDate": "2022-01-20T18:35:42.648+03:00", "dateCreated": "2022-01-20T18:35:42.648+03:00", "programDate": "2022-01-20T18:35:42.648+03:00", "programRevisionDate": "2022-01-20T18:35:42.648+03:00", "reporterName": "name", "reporterAddress": "Reporter address", "reporterTelephoneNumber": 19999999999 }, "data": [ { "base": { "recordDescriptorWord": 730, "processingIndicator": 1, "timeStamp": "2022-01-20T18:35:42.648+03:00", "identificationNumber": "unknown", "consumerAccountNumber": "L21350000010", "portfolioType": "I", "accountType": "01", "dateOpened": "2021-12-16T11:35:24.854+03:00", "highestCredit": 100, "termsDuration": "3", "termsFrequency": "M", "scheduledMonthlyPaymentAmount": 34, "actualPaymentAmount": 0, "accountStatus": "11", "paymentRating": "0", "paymentHistoryProfile": "0", "currentBalance": 104, "amountPastDue": 0, "dateAccountInformation": "2022-01-20T15:35:39.051+03:00", "surname": "surname", "firstName": "name", "socialSecurityNumber": 1111, "dateBirth": "1963-03-28T00:00:00+03:00", "ecoaCode": "1", "countryCode": "US", "firstLineAddress": "add", "secondLineAddress": "", "city": "cc", "state": "TX", "zipCode": "78249", "addressIndicator": "Y" }, "j1": [], "j2": [] } ], "trailer": { "recordDescriptorWord": 426, "recordIdentifier": "TRAILER", "totalBaseRecords": 0, "totalStatusCodeDF": 0, "totalConsumerSegmentsJ1": 0, "totalConsumerSegmentsJ2": 0, "blockCount": 0, "totalStatusCodeDA": 0, "totalStatusCode05": 0, "totalStatusCode11": 0, "totalStatusCode13": 0, "totalStatusCode61": 0, "totalStatusCode62": 0, "totalStatusCode63": 0, "totalStatusCode64": 0, "totalStatusCode65": 0, "totalStatusCode71": 0, "totalStatusCode78": 0, "totalStatusCode80": 0, "totalStatusCode82": 0, "totalStatusCode83": 0, "totalStatusCode84": 0, "totalStatusCode88": 0, "totalStatusCode89": 0, "totalStatusCode93": 0, "totalStatusCode94": 0, "totalStatusCode95": 0, "totalStatusCode96": 0, "totalStatusCode97": 0, "totalECOACodeZ": 0, "totalEmploymentSegments": 0, "totalOriginalCreditorSegments": 0, "totalPurchasedToSegments": 0, "totalMortgageInformationSegments": 0, "totalPaymentInformationSegments": 0, "totalChangeSegments": 0, "totalSocialNumbersAllSegments": 0, "totalSocialNumbersBaseSegments": 0, "totalSocialNumbersJ1Segments": 0, "totalSocialNumbersJ2Segments": 0, "totalDatesBirthAllSegments": 0, "totalDatesBirthBaseSegments": 0, "totalDatesBirthJ1Segments": 0, "totalDatesBirthJ2Segments": 0, "totalTelephoneNumbersAllSegments": 0 } }

What did you expect to see?
I expect to get a valid metro2 file

What did you see?

1. I see errors like that (may vary depending on the json I pass there, but still errors)
   ./metro2 convert --format metro -g --input ./b.json b.metro2
   Error: total_base_records in trailer record has a required field
2. However, the provided json converts into metro file. But the resulting metro file doesn't convert back into json.

How can we reproduce the problem?

1. Convert the json into metro, Now convert the result back to json.

Readme should document using the CLI for taking an input file and converting to an output type. Should also show validation of the file.

Placeholder for larger discussion

The highest credit field in the base_segment.go struct has the required json flag, which doesn't allow for 0 int values:

https://pkg.go.dev/gopkg.in/go-playground/validator.v8#hdr-Required

This validates that the value is not the data types default zero value. For numbers ensures value is not zero. For strings ensures value is not "". For slices, maps, pointers, interfaces, channels and functions ensures the value is not nil.

Metro2 Version: moov/metro2:latest image id 1fd1f2e9b67e

What were you trying to do?
Generate any super-basic metro2 file with more than one user and furnish it to Experian

What did you expect to see?
Since all the validations passed I was expecting Experian to accept it for processing.

What did you see?
The feedback from Experian was:
There was an issue with the file you sent to us and it couldn’t be processed. There is no Line Feed Characters (CRLF) to break the file into “clean” records. We cannot process files like this. We need you to fix and resend this file.

A similar issue was also reported twice by other Moov metro2 users in the Slack channel in the last few months:
https://moov-io.slack.com/archives/C014L9CAW77/p1637747898029500
https://moov-io.slack.com/archives/C014L9CAW77/p1648746025803759

How can we reproduce the problem?
Generate any valid metro2 file with more than one base record and observe the lack of Line Feed Characters

It seems that moov-io/metro2 does not depend on github.com/gogo/protobuf any more, both directly and indirectly.
So, replace directive left in go.mod makes no sense. Should it be dropped?

$ go mod why github.com/gogo/protobuf
# github.com/gogo/protobuf
(main module does not need package github.com/gogo/protobuf)

https://github.com/moov-io/metro2/blob/master/go.mod#L5

replace github.com/gogo/protobuf => github.com/gogo/protobuf v1.3.2

Metro2 Version: 0.3.1

What were you trying to do?

I was trying to create a metro2 file with a phone number of digits greater than 10.

Are we supposed to ignore the US +1 code in phone numbers when reporting to the bureaus? Does the metro2 standard only accept 10-digit phone numbers?

Our API system includes the US +1 code with phone numbers. Example: 1-XXX-XXX-XXXX. The metro2 lib uses an int32 and assumes that all phone numbers are at most 10 digits. int32 doesn't support a number greater than 10 digits. The easiest solution is to truncate our phone numbers down to just 10 digits.

What did you expect to see?

I guess I expected that we support phone numbers with the country extension, so an int64 is required.

What did you see?

A runtime error.

How can we reproduce the problem?

Try 15103334444 in place of any of the required phone number fields like the reporter or primary customer phone.

We should generate an OpenAPI specification for this project's HTTP server. This allows us better documentation and to generate clients in other languages.

Useful Editor: https://editor.swagger.io/
Example: https://github.com/moov-io/ach/blob/master/openapi.yaml

Metro2 Version: 0.3.1

What were you trying to do?

I am trying to create a metro2 file via file.CreateFile(json)

What did you expect to see?

I expect an error to be returned if I pass an invalid date-string for data[0].base.dateBirth.

func badMetro2JSON() string {
	return `{
        "header": {
          ....
        },
        "data": [
          {
            "base": {
              // .. truncated data for easy viewing  
              "dateLastPayment": "2002-08-02T00:00:00Z", 
           
              "dateBirth": "1972-03-18", // <--- Note the lack of timezone
              ....
            }
          }
        ]
    }
    `
}

What did you see?

I see a partially complete metro2 file that has all the header fields but no customer data in the output file.

How can we reproduce the problem?

Use the test case from https://github.com/moov-io/metro2/blob/master/test/testdata/unpacked_fixed_file.json and remove the timezone from data[0].base.dateBirth

Metro2 Version: ``

What were you trying to do?

Used http service with large file

What did you expect to see?

want to use large metro file

What did you see?

How can we reproduce the problem?

http server returns error response when trying to validate a large ( 63MB / ~140k lines) txt based metro2 file.

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Metro2 Version: moov/metro2:latest

What were you trying to do?
Convert Json to Metro2 using API call

What did you expect to see?
Converted response text in metro 2 format

What did you see?
Got the converted response but the encoding seems to be wrong. please find the attached screenshot for reference.

How can we reproduce the problem?
Run docker container using docker for windows.
Execute /convert API passing sample data.
Read the response and save in a file.
check file encoding.

Zero balances for both current balances and highest credit responds with a validation error that reads, "current_balance in base segment has a required field". However things do work appropriately when using v0.2.7

Metro2 Version: v0.3.1

What were you trying to do?

Validate a json with current balance set at 0.

What did you expect to see?

Values of 0 should be accepted

What did you see?

Validation error stating "current_balance in base segment has a required field"

How can we reproduce the problem?

Using version 0.3.1, try to validate a json using 0 for current balance or highest credit.

Metro2 Version: v0.2.6

What were you trying to do?
go run ./cmd/metro2/ print an unpacked file.

What did you expect to see?
A printed out version of an unpacked file.

What did you see?
Error, without verbose explanation of the issue.

How can we reproduce the problem?
Attempt to Go Run Metro2 Print an unpacked file using:

go run ./cmd/metro2/ print on the unpacked_fixed_file.dat in testdata folder with format output as JSON.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Other Branches

These updates are pending. To force PRs open, click the checkbox below.

• chore(deps): update all (major) (actions/cache, actions/checkout, actions/setup-go, bulma-clean-theme, catchpoint/workflow-telemetry-action, github/codeql-action)

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update golang.org/x/exp digest to 4c93da0

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Metro2 Version: 0.7.1

What were you trying to do?
I am trying to deploy your docker image to AWS ECR but I don't know docker very well. Is there anyone that has an example Docker file that can be used?

What did you expect to see?
moov metro 2 running via AWS API gateway

What did you see?
When I deploy your docker image directly to ECR I get a Runtime unknown error which I assume means that the Go runtime environment is missing.

How can we reproduce the problem?
follow this tutorial to deploy docker repo to ECR and see the errors
https://medium.com/@tud7/aws-deploy-lambda-function-with-docker-image-and-invoke-over-https-ae27b137fffc

Metro2 Version: v0.2.6

What were you trying to do?
Print more verbose errors on what field / location in the input file is causing the error.

What did you expect to see?
A specific part of the file which caused the program to error so I could resolve it.

What did you see?
Generic error

How can we reproduce the problem?

Run the print command on unpacked_fixed_file.dat for instance.

Metro2 Version: v0.3.3

What were you trying to do?
trying to play with API and understand metro2 format

What did you expect to see?
https://github.com/moov-io/metro2/blob/master/test/testdata/unpacked_variable_file.json
I tested validator with above test data from your repo, noticed totalConsumerSegmentsJ1 is 2
and validator returns error if I change it to 1
I'm a bit confused as I also don't find much explanation on this value in the guide provided by CDIA
I was expecting it to be 1 since there is only one j1 item found, or is the base record also considered?

What did you see?
see above

How can we reproduce the problem?
N/A

Thank you

Placeholder issue for coordinating updates.

Metro2 Version: v0.4.1

What were you trying to do?

Validate test file test/testdata/unpacked_fixed_file.json.

What did you expect to see?

In the trailer record, totalSocialNumbersAllSegments should have value 3, and totalSocialNumbersBaseSegments should have value 1.

What did you see?

The current values are incorrect because they don't include the social security number of the base record.

Similar to how dates of birth are counted, this method should increase the counter for social security numbers:

I believe the following lines are missing right above line 412:

trailer.TotalSocialNumbersAllSegments++
trailer.TotalSocialNumbersBaseSegments++

How can we reproduce the problem?

Validator returns "the file is valid", but test/testdata/unpacked_fixed_file.json is not a valid file. Set the following two values in the test file:

"totalSocialNumbersAllSegments": 3,
"totalSocialNumbersBaseSegments": 1,

and then validate:

go run cmd/metro2/main.go validator --input test/testdata/unpacked_fixed_file.json

Metro2 Version: v0.5.1

What were you trying to do?

Generate a metro2 using the metro2 tool. I inputted a 0 value for the CurrentBalance on one of the base segments.

What did you expect to see?

A metro2 with a base segment and 0 current balance.

What did you see?

An error stating "current_balance in base segment is a required field"

How can we reproduce the problem?
Input a 0 current balance on one of the base segments and try to generate a metro2.

Note I created a similar ticket a couple of weeks ago here. I believe the issue this time is coming from this: https://github.com/moov-io/metro2/blob/master/pkg/client/model_base_segment.go#L40, when it converts the lib model into the client model and expects a non-zero value.

Metro2 Version: latest

What were you trying to do?
We were trying to validate and convert large JSON files to the Metro2 format using the HTTP Docker server.

What did you expect to see?
We ran tests with JSON files consisting of 10k, 25k, 50k, and ~250k records. We expected to see each file validate and, if valid, have the server return a Metro2 file.

What did you see?
The JSON payload holding 10k records validated and was converted with no issues.

The 25k record payload validated, but when hitting the /convert endpoint we received a 502 Bad Gateway response. Our cloud architects don't think this is an issue with server resources or how we've set up the server. At times we have also seen a 504 Gateway Time-out response and an SSL "end of file" error.

On the two larger files, we received a 400 status code on validation with the message "is not a metro file". We are unsure if this is from the server not receiving the whole file or otherwise having trouble parsing it, or if we're just sending bad payloads.

Here's an example of how we've used curl to test these payloads:

curl --form "file=@./25k_record.json" <URL>/validator
curl --form "file=@./25k_record.json" --form "format=metro" <URL>/convert

How can we reproduce the problem?
I've attached the payloads we've been testing with. The problem can be reproduced by sending these payloads to the /validator and /convert endpoints of the HTTP server. If you'd like to use our instance, feel free to test with this url:

https://workato-container-service.ou23u2ki0eaiq.us-west-2.cs.amazonlightsail.com

Payloads:
10k_20k_50k.zip

Fuzzing is a technique for sending arbitrary input into functions to see what happens. Typically this is done to weed out crashes/panics from software, or to detect bugs. In some cases all possible inputs are ran into the program (i.e. if a function accepts 16-bit integers it's trivial to try them all).

We should support fuzzing and ship a docker image that can be ran which will help automate fuzzing. There are examples we can use from moov-io/ach:

• Fuzzing setup from moov-io/ach
• Self-Contained Dockerfile for Fuzzing

Metro2 Version: moov/metro2:latest docker image as of this writing (apparently v0.4.1)

What were you trying to do?

Validate the example file using the same commands in the README

What did you expect to see?

valid file

What did you see?

total_social_numbers_all_segments in trailer record has an invalid value

How can we reproduce the problem?

By following the steps in the README:

docker pull moov/metro2:latest
docker run -p 8080:8080 moov/metro2:latest

...then, from the root of this repo (on commit 53a5e86, the tip of master as of this writing):

curl -X POST --form "file=@./test/testdata/packed_file.json" http://localhost:8080/validator

Metro2 Version: 0.3.1

What were you trying to do?

I wanted to use the library.

What did you expect to see?

I expected some reference explaining that the main library usage starts at "github.com/moov-io/metro2/pkg/file"
with file.CreateFile.

What did you see?

Instead, I had to dig into the CLI and API packages to learn how the metro2 lib was actually used.

How can we reproduce the problem?