Automatically validate in fromJSON about jsonlite

[johnlaing]

I neglected to include:

> sessionInfo()
R version 3.1.0 (2014-04-10)
Platform: x86_64-pc-linux-gnu (64-bit)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] jsonlite_0.9.7

[jeroen]

The libjson library implements parsing and validation in separate functions for performance reasons. The fromJSON parser assumes the json is valid. Indeed, this can lead to unexpected parser behavior when parsing invalid json.

I recommend that if you expect your application/service might receive invalid json, you manually run validate(myjson) before calling fromJSON(myjson).

I'll look if there is anything I can do to make the parser more strict.

[johnlaing]

Thanks for the quick response. I assume my situation to be a common one, in that you never expect to receive invalid JSON until after it has already happened. Calling validate each time may be a bit inelegant, but it will certainly solve the problem.

[jeroen]

Can try to modify some options in JSONOptions.h to make the parser more strict, but not sure if that is a good idea.

[johnlaing]

I took a look at the options for flags that might be useful. The only idea I had was to undefine JSON_SAFE, which leads to a segfault in the event of invalid JSON. So that makes it worse.

Would you consider adding a validate argument to fromJSON? That function already includes some convenience-oriented code like automatic handling of file or URL arguments instead of actual JSON text. Adding validation seems to me in the same spirit. I'd be happy to submit a patch if you're interested.

[jeroen]

Mmm not sure. The toJSON and fromJSON functions already have a lot of arguments. It's not much shorter to type

fromJSON(mydata, validate=TRUE)

rather than

validate(mydata)
fromJSON(mydata)

The only reason to add it would be if we set it to TRUE by default. I guess I need to do some benchmarking to test the relative performance overhead of validating everything.

[jeroen]

I'm adding this in jsonlite 0.9.10: 678135f. However it is set to FALSE by default, because I know that many people currently using fromJSON won't appreciate the unasked performance overhead when upgrading jsonlite.

[jeroen]

On cran now.

[jeroen]

FYI starting jsonlite version 0.9.12 we are using another parser which always validates the JSON while parsing. The validate argument has been deprecated.