travishorn / csval Goto Github PK

I'm running csval against a 2.5GB csv and I get the error
Error: Cannot create a string longer than 0x1fffffe8 characters
after about 5 seconds of execution.

Your readme shows the following JSON Schema fragment

{
  "type": "object",
  "properties": {
    "salary": {
      "type": "number"
    }
  }
}

However, this is not valid JSON Schema, right?
I mean, the $schema and definitions nodes are missing as shown in this example.

Does csval really support complete JSON Schema as the specification documents?

Thanks for your help,
Mario

{
    "$schema": "http://json-schema.org/draft-06/schema#",
    "definitions": {
        "Artist": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "name": {
                    "type": "string"
                },
                "founded": {
                    "type": "integer"
                },
                "members": {
                    "type": "array",
                    "items": {
                        "type": "string"
                    }
                }
            },
            "required": [
                "founded",
                "members",
                "name"
            ],
            "title": "Artist"
        },

...

Hi Travis,

I've started playing around with this csval and so far I like it but I've got a few questions

1. Are delimiters other than comma supported?
2. I'm running into an issue specifying a second field for validation. I believe I've got a valid CSV (csval told me that when I tried it without a schema) but when I add a second field for validation, it fails for all rows. Can I get some help?

Thanks.

Is there a flag (or something that I need to specify in json schema) to get strict validation of the csv columns? What I'm looking for is

1. The columns specified in the schema (and only those) are valid. Changes to those column headers (order or type) cause failures.

Is there a way to do that in current functionality? Would you like me to provide an example to better illustrate it?

Hey - I wanted to use your tool but I couldn't see an option to pass in a custom delimiter

Have you ever considered streaming the csv in on a chunk basis? I think that could help this really work for very large CSVs. I'd be very interested in this.

I just built a docker image with the latest version from this repo and it's not working as expected. I had built one docker image locally (npm link) with my changes and it does respect additionalProperties. The difference I'm seeing is that for some reason the latest from npm is flagging my test csv as having missing values when the regex for the column allows for empty values. Also the column is not marked as a required field so I'm not sure of the difference.

I've verified inside the docker image that my changes are there but now it's not behaving as it should. Is something possibly messed up with the release?

Hi,
I experiment some trouble by implementing the required option. It doesn't seem to work. In your example, the usage of required it's pretty confusing. You seem to look for the entire colone, not for a field in this colone.

You can require certain fields, as well. Consider this rules file.

{
"properties": {
"age": {
"type": "number"
}
},
"required": ["age"]
}
This CSV file would pass.

name,age,salary
John,30,100000
Jane,50,150000
This one would fail.

name,salary
John,100000
Jane,150000

Beside I try to implement a conditionnal required, I have tried all the combination on this post :
https://stackoverflow.com/questions/38717933/jsonschema-attribute-conditionally-required

Without any success.

Best regards

hoek subject to prototype pollution via the clone function.
Dependabot cannot update @hapi/hoek to a non-vulnerable version
The latest possible version that can be installed is 6.2.4 because of the following conflicting dependencies:

[email protected] requires @hapi/[email protected] via @hapi/[email protected]
[email protected] requires @hapi/hoek@^6.0.0 via [email protected]
[email protected] requires @hapi/hoek@^8.3.0 via a transitive dependency on @hapi/[email protected]
No patched version available for @hapi/hoek
The earliest fixed version is 9.0.4.

hoek versions prior to 8.5.1, and 9.x prior to 9.0.3 are vulnerable to prototype pollution in the clone function. If an object with the proto key is passed to clone() the key is converted to a prototype. This issue has been patched in version 9.0.3, and backported to 8.5.1.

We use txt file and csv format. Is there a way to pass in .txt files and received the same validation.

Thank you for this great project!

I am trying to come up with a schema i.e.:

{
  "properties" : {
    "aaa": {
      "type": "string",
      "minLength": 0
    },
    "bbb": {
      "type": "string",
      "minLength": 0
    }
  }
}

such that bbb can not be "" when aaa is not ""; or bbb must be "" when aaa is ""; so the validate() returns the error when either aaa or bbb is "", returns true when not both "" or both having values.

I am very new to programming; and I have tried to look through the JSON Schema documentations and your examples but I have not found a right direction. Would you please give me some tips? Thank you so much.