iarna / iarna-toml

Better TOML parsing and stringifying all in that familiar JSON interface.

@iarna/toml

Better TOML parsing and stringifying all in that familiar JSON interface.

** TOML 1.0.0-rc.1 **

TOML Spec Support

The most recent version as of 2019-04-21: 1.0.0-rc.1

Other Versions

1.0.0-rc.1 parsers can load almost any TOML 0.4 and TOML 0.5 document, but TOML 1.0.0-rc.1 docs are not always compatible with TOML 0.4 and TOML 0.5 parsers. If you're using this to generate TOML documents and you want an older parser to be able to read them you may want to use the latest TOML 0.5 version of this module.

Example

const TOML = require('@iarna/toml')
const obj = TOML.parse(`[abc]
foo = 123
bar = [1,2,3]`)
/* obj =
{abc: {foo: 123, bar: [1,2,3]}}
*/
const str = TOML.stringify(obj)
/* str =
[abc]
foo = 123
bar = [ 1, 2, 3 ]
*/

Visit the project github for more examples!

Why @iarna/toml

• Support for TOML 1.0.0-rc.1!
• Highly correct! Careful adherence to spec.
• See TOML-SPEC-SUPPORT for a comparison of which TOML features are supported by the various Node.js TOML parsers.
• Speedy! See benchmarks at end.
• BigInt support on Node 10!
• 100% test coverage.
• Small parser bundle (if you use @iarna/toml/parse-string).
• No deps.
• Detailed and easy to read error messages‼

> TOML.parse(src)
Error: Unexpected character, expecting string, number, datetime, boolean, inline array or inline table at row 6, col 5, pos 87:
5: "abc\"" = { abc=123,def="abc" }
6> foo=sdkfj
       ^
7:

TOML.parse(str) → Object (example)

Also available with: require('@iarna/toml/parse-string')

Synchronously parse a TOML string and return an object.

TOML.stringify(obj) → String (example)

Also available with: require('@iarna/toml/stringify')

Serialize an object as TOML.

[your-object].toJSON

If an object TOML.stringify is serializing has a toJSON method then it will call it to transform the object before serializing it. This matches the behavior of JSON.stringify.

The one exception to this is that toJSON is not called for Date objects because JSON represents dates as strings and TOML can represent them natively.

moment objects are treated the same as native Date objects, in this respect.

TOML.stringify.value(obj) -> String

Also available with: require('@iarna/toml/stringify').value

Serialize a value as TOML would. This is a fragment and not a complete valid TOML document.

Promises and Streaming

The parser provides alternative async and streaming interfaces, for times that you're working with really absurdly big TOML files and don't want to tie-up the event loop while it parses.

TOML.parse.async(str[, opts]) → Promise(Object) (example)

Also available with: require('@iarna/toml/parse-async')

opts.blocksize is the amount text to parser per pass through the event loop. Defaults to 40kb.

Asynchronously parse a TOML string and return a promise of the resulting object.

TOML.parse.stream(readable) → Promise(Object) (example)

Also available with: require('@iarna/toml/parse-stream')

Given a readable stream, parse it as it feeds us data. Return a promise of the resulting object.

readable.pipe(TOML.parse.stream()) → Transform (example)

Also available with: require('@iarna/toml/parse-stream')

Returns a transform stream in object mode. When it completes, emit the resulting object. Only one object will ever be emitted.

Lowlevel Interface (example) (example w/ parser debugging)

You construct a parser object, per TOML file you want to process:

const TOMLParser = require('@iarna/toml/lib/toml-parser.js')
const parser = new TOMLParser()

Then you call the parse method for each chunk as you read them, or in a single call:

parser.parse(`hello = 'world'`)

And finally, you call the finish method to complete parsing and retrieve the resulting object.

const data = parser.finish()

Both the parse method and finish method will throw if they find a problem with the string they were given. Error objects thrown from the parser have pos, line and col attributes. TOML.parse adds a visual summary of where in the source string there were issues using parse-pretty-error and you can too:

const prettyError = require('./parse-pretty-error.js')
const newErr = prettyError(err, sourceString)

What's Different

Version 3 of this module supports TOML 1.0.0-rc.1. Please see the CHANGELOG for details on exactly whats changed.

TOML we can't do

• -nan is a valid TOML value and is converted into NaN. There is no way to produce -nan when stringifying. Stringification will produce positive nan.
• Detecting and erroring on invalid utf8 documents: This is because Node's UTF8 processing converts invalid sequences into the placeholder character and does not have facilities for reporting these as errors instead. We can detect the placeholder character, but it's valid to intentionally include them in documents, so erroring on them is not great.
• On versions of Node < 10, very large Integer values will lose precision. On Node >=10, bigints are used.
• Floating/local dates and times are still represented by JavaScript Date objects, which don't actually support these concepts. The objects returned have been modified so that you can determine what kind of thing they are (with isFloating, isDate, isTime properties) and that their ISO representation (via toISOString) are representative of their TOML value. They will correctly round trip if you pass them to TOML.stringify.
• Binary, hexadecimal and octal values are converted to ordinary integers and will be decimal if you stringify them.

Changes

I write a by hand, honest-to-god, CHANGELOG for this project. It's a description of what went into a release that you the consumer of the module could care about, not a list of git commits, so please check it out!

Benchmarks

You can run them yourself with:

$ npm run benchmark

The results below are from my desktop using Node 13.13.0. The library versions tested were @iarna/[email protected], [email protected], [email protected], @sgarciac/[email protected], @ltd/[email protected], and [email protected]. The speed value is megabytes-per-second that the parser can process of that document type. Bigger is better. The percentage after average results is the margin of error.

New here is fast-toml. fast-toml is very fast, for some datatypes, but it also is missing most error checking demanded by the spec. For 0.4, it is complete except for detail of multiline strings caught by the compliance tests. Its support for 0.5 is incomplete. Check out the spec compliance doc for details.

As this table is getting a little wide, with how npm and github display it, you can also view it seperately in the BENCHMARK document.

	@iarna/toml	toml-j0.4	toml	@sgarciac/bombadil	@ltd/j-toml	fast-toml
Overall	28MB/sec 0.55%	-	-	-	-	-
01-small-doc-mixed-type-inline-array	5.3MB/sec 0.48%	-	-	-	-	12MB/sec 0.13%
Spec Example: v0.4.0	25MB/sec 0.40%	9.9MB/sec 0.15%	0.9MB/sec 0.37%	1.3MB/sec 1.02%	28MB/sec 0.33%	-
Spec Example: Hard Unicode	63MB/sec 0.47%	17MB/sec 0.21%	2MB/sec 0.25%	0.6MB/sec 0.47%	65MB/sec 0.27%	79MB/sec 0.09%
Types: Array, Inline	7.2MB/sec 0.53%	4.1MB/sec 0.09%	0.1MB/sec 0.69%	1.4MB/sec 0.86%	10MB/sec 0.33%	9MB/sec 0.16%
Types: Array	6.8MB/sec 0.09%	6.8MB/sec 0.20%	0.2MB/sec 0.81%	1.3MB/sec 0.82%	8.9MB/sec 0.36%	29MB/sec 0.16%
Types: Boolean,	20MB/sec 0.22%	9.3MB/sec 0.29%	0.2MB/sec 0.91%	1.9MB/sec 0.85%	16MB/sec 0.29%	8.6MB/sec 0.22%
Types: Datetime	17MB/sec 0.09%	11MB/sec 0.17%	0.3MB/sec 0.75%	1.6MB/sec 0.42%	9.8MB/sec 0.40%	6.5MB/sec 0.11%
Types: Float	8.5MB/sec 0.29%	5.8MB/sec 0.33%	0.2MB/sec 0.91%	2.2MB/sec 0.91%	14MB/sec 0.25%	7.9MB/sec 0.33%
Types: Int	5.8MB/sec 0.13%	4.5MB/sec 0.14%	0.1MB/sec 0.63%	1.5MB/sec 0.73%	9.8MB/sec 0.14%	8.1MB/sec 0.16%
Types: Literal String, 7 char	25MB/sec 0.15%	8.3MB/sec 0.38%	0.2MB/sec 0.71%	2.3MB/sec 1.04%	23MB/sec 0.28%	14MB/sec 0.21%
Types: Literal String, 92 char	44MB/sec 0.23%	12MB/sec 0.14%	0.3MB/sec 0.63%	13MB/sec 1.12%	100MB/sec 0.14%	77MB/sec 0.15%
Types: Literal String, Multiline, 1079 char	23MB/sec 0.35%	7.2MB/sec 0.34%	0.9MB/sec 0.86%	47MB/sec 1.07%	380MB/sec 0.13%	641MB/sec 0.14%
Types: Basic String, 7 char	25MB/sec 0.09%	7MB/sec 0.08%	0.2MB/sec 0.82%	2.3MB/sec 1.02%	15MB/sec 0.12%	13MB/sec 0.14%
Types: Basic String, 92 char	44MB/sec 0.15%	8MB/sec 0.39%	0.1MB/sec 1.52%	12MB/sec 1.53%	70MB/sec 0.17%	71MB/sec 0.16%
Types: Basic String, 1079 char	24MB/sec 0.36%	5.7MB/sec 0.12%	0.1MB/sec 3.65%	42MB/sec 1.67%	93MB/sec 0.13%	617MB/sec 0.14%
Types: Table, Inline	9.4MB/sec 0.21%	5.2MB/sec 0.23%	0.1MB/sec 1.18%	1.4MB/sec 1.20%	8.5MB/sec 0.68%	8.7MB/sec 0.30%
Types: Table	6.8MB/sec 0.13%	5.5MB/sec 0.22%	0.1MB/sec 1.10%	1.5MB/sec 1.05%	7.3MB/sec 0.54%	19MB/sec 0.21%
Scaling: Array, Inline, 1000 elements	40MB/sec 0.27%	2.4MB/sec 0.20%	0.1MB/sec 1.90%	1.6MB/sec 1.14%	18MB/sec 0.16%	32MB/sec 0.12%
Scaling: Array, Nested, 1000 deep	2MB/sec 0.17%	1.6MB/sec 0.09%	0.3MB/sec 0.62%	-	1.8MB/sec 0.80%	13MB/sec 0.19%
Scaling: Literal String, 40kb	59MB/sec 0.26%	10MB/sec 0.14%	3MB/sec 0.91%	13MB/sec 0.40%	479MB/sec 0.25%	19kMB/sec 0.20%
Scaling: Literal String, Multiline, 40kb	61MB/sec 0.23%	5.3MB/sec 0.30%	0.2MB/sec 1.78%	12MB/sec 0.55%	276MB/sec 0.16%	21kMB/sec 0.10%
Scaling: Basic String, Multiline, 40kb	61MB/sec 0.21%	6MB/sec 0.40%	2.8MB/sec 0.75%	12MB/sec 0.60%	1kMB/sec 0.13%	27kMB/sec 0.14%
Scaling: Basic String, 40kb	60MB/sec 0.13%	6.6MB/sec 0.13%	0.2MB/sec 1.67%	13MB/sec 0.30%	504MB/sec 0.26%	19kMB/sec 0.22%
Scaling: Table, Inline, 1000 elements	26MB/sec 0.17%	7.3MB/sec 0.83%	0.3MB/sec 0.95%	2.5MB/sec 1.24%	5.4MB/sec 0.22%	13MB/sec 0.22%
Scaling: Table, Inline, Nested, 1000 deep	8MB/sec 0.10%	5.2MB/sec 0.25%	0.1MB/sec 0.45%	-	3.1MB/sec 0.58%	10MB/sec 0.19%

Tests

The test suite is maintained at 100% coverage:

The spec was carefully hand converted into a series of test framework independent (and mostly language independent) assertions, as pairs of TOML and YAML files. You can find those files here: spec-tests.

Further tests were written to increase coverage to 100%, these may be more implementation specific, but they can be found in coverage and coverage-error.

I've also written some quality assurance style tests, which don't contribute to coverage but do cover scenarios that could easily be problematic for some implementations can be found in: test/qa.js and test/qa-error.js.

All of the official example files from the TOML spec are run through this parser and compared to the official YAML files when available. These files are from the TOML spec as of: 357a4ba6 and specifically are:

• github.com/toml-lang/toml/tree/357a4ba6/examples
• github.com/toml-lang/toml/tree/357a4ba6/tests

The stringifier is tested by round-tripping these same files, asserting that TOML.parse(sourcefile) deepEqual TOML.parse(TOML.stringify(TOML.parse(sourcefile)). This is done in test/roundtrip-examples.js There are also some tests written to complete coverage from stringification in: test/stringify.js

Tests for the async and streaming interfaces are in test/async.js and test/stream.js respectively.

Tests for the parser's debugging mode live in test/devel.js.

And finally, many more stringification tests were borrowed from @othiym23's toml-stream module. They were fetched as of b6f1e26b572d49742d49fa6a6d11524d003441fa and live in test/toml-stream.

Improvements to make

• In stringify:
  • Any way to produce comments. As a JSON stand-in I'm not too worried about this. That said, a document orientated fork is something I'd like to look at eventually…
  • Stringification could use some work on its error reporting. It reports what's wrong, but not where in your data structure it was.
• Further optimize the parser:
  • There are some debugging assertions left in the main parser, these should be moved to a subclass.
  • Make the whole debugging parser thing work as a mixin instead of as a superclass.

iarna-toml's Issues

NPM does not seem up-to-date

Description

The version on npm seem to be slightly off compared to the latest branch

It's still the 2.2.5 and not the 3.0

How to 'merge' toml files

I have an application with a default set of options and another set of options to override the default (either the subset or its entirety ) . How do pass 2 .toml in a specific order so I get a final object that represents the merged / overridden values.

Eg:

Eg: default.toml

[db]
readURI = "mysql://user:[email protected]/mydb?charset=utf8"
writeURI = "mysql://user:[email protected]/mydb?charset=utf8"

[db.options]
ConnMaxLifetime = 1
MaxOpenConns = 10

Eg: prod.toml

[db]
readURI = "mysql://secureuser:[email protected]/mydb?charset=utf8"
writeURI = "mysql://secureuser:[email protected]/mydb?charset=utf8"

So hypothetically an api like -

parser.parse(['default.toml',' prod.toml']

would yield a final toml object as below:

[db]
readURI = "mysql://secureuser:[email protected]/mydb?charset=utf8"
writeURI = "mysql://secureuser:[email protected]/mydb?charset=utf8"

[db.options]
ConnMaxLifetime = 1
MaxOpenConns = 10

Eg: the go implementation takes in an object and 'decodes' a given toml on that object. This helps us to have a simpler override functionality. More details at https://godoc.org/github.com/BurntSushi/toml#Decode .

Is there a similar api / hack available to repeatedly parse and process a bunch of .toml files so we can override default options ?

TOML.parse throws on some hex literals containing the character E/e

Simple test cases:

TOML.parse("hex = 0xe");   // Throws
TOML.parse("hex = 0x0e");  // Works

TOML.parse("hex = 0x0_e"); // Throws
TOML.parse("hex = 0x0_f"); // Works

The issue seems to be caused by .parseNoUnder() being re-used for hex literals and not being hex character aware.

The benchmarks is too wide

Maybe this will look better? ;)

	name @0.0.0	name @0.0.0
item	1MB/s 1%	1MB/s 1%

Use `.cjs` file extension for CommonJS modules

In consideration of ESM support (pending #36), I think it makes sense for this package to be explicit about CommonJS modules by utilizing the .cjs extension.

• JavaScript has standardized on ESM, and Node doesn't care if CommonJS module file extensions are .js or .cjs when package.type is commonjs or undefined — that's the case here. When package.type is module (not the case here) CommonJS files must end in .cjs. (See https://nodejs.org/api/esm.html)

• It provides immediate clarity to anyone casually browsing the repository: they can see the .cjs extension and instantly know it's a CommonJS module. Similarly, .mjs serves as an indicator that a file is an ES module for Node (vs. for browser, etc.). This cognitive benefit is the biggest win, IMO.

Known Bugs

As seen in: https://github.com/iarna/iarna-toml/blob/latest/KNOWN-BUGS.md

This list was dug up by running @iarna/toml against the toml and toml-j0.4 test suites.

Known Bugs:

• Current string literals are supported as key names. This is not valid per TOML 0.4:

  Quoted keys follow the exact same rules as basic strings and allow you to use a much broader set of key names.

• Accessing a higher level table after accessing a deeper attribute is valid. The following should not crash:

  [a.b]
  c = 1
  
  [a]
  d = 2

• Invalid unicode should crash. Currently we catch String.fromCodePoint errors and rethrow them, but other sequences are invalid like \uD800 which becomes the Unicode Replacement Char, but probably should crash instead.

• Empty bare keys are errors, eg [.abc] or [abc.] or [abc..def] or [] should crash. Versus ["".abc] or [abc.""] or [abc."".def], or [""] which are all valid. That is, value should not return an empty string if it didn't match anything. Possibly should stop initializing this.state.buf to ''.

• Multiline \ trimming nees to support CRs

• Multline post opener trimming needs to support CRs

• Strings MAY NOT contain: the control characters (U+0000 to U+001F

• toml-j0.4 and toml both think that keys in inline tables may not be quoted. The spec doesn't say much:

  Key/value pairs take the same form as key/value pairs in standard tables.

  But keys in standard tables can be double quoted, and the ABNF definitely thinks they can be quoted. I think the others are wrong here.

TOML v1.0.0-rc.1 Support

As of last week, there's now a RC for 1.0.

Changes:

• Clarify in ABNF how quotes in multi-line basic and multi-line literal strings
• are allowed to be used.
• Leading zeroes in exponent parts of floats are permitted.
• Clarify that control characters are not permitted in comments.
• Clarify behavior of tables defined implicitly by dotted keys.
• Clarify that inline tables are immutable.
• Clarify that trailing commas are not allowed in inline tables.
• Clarify in ABNF that UTF-16 surrogate code points (U+D800 - U+DFFF) are not
• allowed in strings or comments.
• Allow raw tab characters in basic strings and multi-line basic strings.
• Allow heterogenous values in arrays.

I don't want to commit to doing this, but I might be able to find time... eventually. :)

Can i do a PR to represent dates with timezone offset?

Hi totally new to this project... love it btw!

Question: Are you interested in a PR to add timezone relative dates to the library?

ie

const thingy = {
date: new Date(),
};
TOML.stringify(thingy);

// date = 2020-07-03T22:21:32.426Z

I'd like to make it more human editable for my people using the product, so is it possible / plausible to change the stringified date into the user's local timezone?

date = 2020-07-03T21:21:32.426-01:00

Looks like I'd do this in stringify.js

How many bytes would you allow me to add? Should I do this in some sort of external way so you can turn it on and off? Should it be the default behavior?

How can i stringify toml comment?

hey iarna~ this parser is so convenient to convert between toml and JSOject.
but i have one question about that how can i insert the toml comment when i use TOML.stringify method to convert one js object to toml text?

i will appreciate it if u can give me some advice. thx a lot.

Benchmarks Feedback / Suggestions / Discussion

Hello.

I've been working on some Toml related tooling lately and came across your benchmarks
Which helped me so thanks for that! 👍

Asserting as part of the benchmark.

    fixtures.forEach(_ => {
      assertIsDeeply(parseIarnaToml(_.data), _.answer)
})

This means the time it takes to perform the deep compare is taken into account in the benchmark.
In your own package's case that is ~50% of each iteration's run time.

Would it be more precise to perform these assertions outside the benchmark, or at least not every single time?

Asserting equality of output when comparing different parsers

A more general question for discussion is if the assertion should even be done.
Assume someone creates a Toml Parser that outputs a data structure that also represents
the comments, if I understand the "assertIsDeeply" code correctly that would fail
due to additional keys (for the comments properties). But would that be a true failure?

Focusing on micro benchmarks.

It looks like there is a great deal of focus on specific parts of the parser.
e.g one benchmark for literals of type X, another for nested arrays.
But won't end users be more interested in real world performance?
Particularly for large toml files?

See reference on V8 performance testing methodology.
https://v8.dev/blog/real-world-performance

I've tried to create such a real world benchmark by collecting large TOML samples
from github.com.

• https://github.com/bd82/toml-tools/tree/master/packages/parser/benchmark

I am sure more samples are needed to make it more representative,
maybe the best case would be to scrap 1,000 toml files from popular OSS projects for the sample
instead of large files from only a few repos...

But would not the results of a more real world benchmark be more relevant, interesting and easy to understand by potential end users?

Cheers.
Shahar.

Package is not in npm registry

I've got error that '@iarna/toml@^2.2.3' is not in the npm registry

Date throws an incomplete datetime exception

Hi, depending on what comes after a date, it parses correctly or it raises and exception TomlError: Incomplete datetime.

Reproduction steps

const toml = require('@iarna/toml')

const case1 = 'foo = 2019-07-14'
const case2 = 'foo = 2019-07-14\nbar = "lorem"'
const case3 = 'foo = 2019-07-14\n\nbar = "lorem"'

toml.parse(case1) // works!
toml.parse(case2) // don't work :(
toml.parse(case3) // works!

The exception for case2:

> toml.parse(case2)
Thrown:
TomlError: Unexpected character, expected only whitespace or comments till end of line at row 2, col 1, pos 18:
1: foo = 2019-07-14
2> bar = "lorem"
   ^


    at TOMLParser.parseWhitespaceToEOL (/Users/arnau/kitchen/blog/node_modules/@iarna/toml/lib/toml-parser.js:311:26)
    at TOMLParser.runOne (/Users/arnau/kitchen/blog/node_modules/@iarna/toml/lib/parser.js:64:30)
    at TOMLParser.goto (/Users/arnau/kitchen/blog/node_modules/@iarna/toml/lib/parser.js:87:17)
    at TOMLParser.recordAssignStatement (/Users/arnau/kitchen/blog/node_modules/@iarna/toml/lib/toml-parser.js:337:19)
    at TOMLParser.runOne (/Users/arnau/kitchen/blog/node_modules/@iarna/toml/lib/parser.js:64:30)
    at TOMLParser.returnNow (/Users/arnau/kitchen/blog/node_modules/@iarna/toml/lib/parser.js:107:17)
    at TOMLParser.recordAssignValue (/Users/arnau/kitchen/blog/node_modules/@iarna/toml/lib/toml-parser.js:380:19)
    at TOMLParser.runOne (/Users/arnau/kitchen/blog/node_modules/@iarna/toml/lib/parser.js:64:30)
    at TOMLParser.parse (/Users/arnau/kitchen/blog/node_modules/@iarna/toml/lib/parser.js:45:22)
    at Object.parseString [as parse] (/Users/arnau/kitchen/blog/node_modules/@iarna/toml/parse-string.js:13:12) {
  name: 'TomlError',
  fromTOML: true,
  wrapped: null,
  line: 1,
  col: 0,
  pos: 18
}

Environment

OS: macos 10.14.5
nodejs: v12.6.0
npm: 6.9.0
@iarna/toml: 2.2.3

Add positional informations to result

HI,
It would be very helpfull to be able to have positional informations (line, column) to each element of the resulting object.
I guess this would be optional.
I would be willing to propose a PR for this and would be interested to discuss the best way to achieve this.
Is this something to could be of interest ?
Fred

Add `Buffer` to `parse` input type

require("@iarna/toml").parse(fs.readFileSync("abc"))

This works but doesn't type check with TypeScript.

stringify adds underscore as integer decimal separator

Hello 👋,
I am a bit intrigued with the fact that integers are added an underscore as a decimal separatorssuch as:

const a = {number:10000};
toml.stringify(a) // number=10_000

repro: https://codesandbox.io/s/iarnatoml-stringify-45080?file=/src/index.js

This is being used for both integers and floats

iarna-toml/stringify.js

Lines 188 to 191 in 1374bf2

	function stringifyInteger (value) {
	/* eslint-disable security/detect-unsafe-regex */
	return String(value).replace(/\B(?=(\d{3})+(?!\d))/g, '_')
	}

But it would great allowing users to disable it in case they prefer it without separators in order to improve editability?

0.5.0 support

The current state of TOML was released as 0.5.0, as such, the previously planned 1.0.0 branch will be 0.5.0.

Parse error on local date (`d = 2020-01-02`) followed by any field

toml fails to parse the following block and errors with " Unexpected character, expected only whitespace or comments till end of line"

date = 2020-01-02
postType = "post"

Env

node: v13.3.0
toml: "@iarna/toml": "2.2.3"

Repro:

describe('toml', () => {
  it('date works', () => {
    const value = `date = 2020-01-02`;
    toml.parse(value)
  });
  
  it('date + newline + text works', () => {
    const value = `date = 2020-01-02
    
postType = "post"`;
    toml.parse(value)
  });

  it('date + text fails', () => {
    const value = `date = 2020-01-02
postType = "post"`;
    toml.parse(value)
  });

  it('date + number fails', () => {
    const value = `date = 2020-01-02
postType = 2`;
    toml.parse(value)
  });
});

Stack trace:

TomlError: Unexpected character, expected only whitespace 
                  or comments till end of line at row 2, col 1, pos 19:
1: date = 2020-01-02
2> postType = "post"
   ^

    at TOMLParser.parseWhitespaceToEOL (@iarna/toml/lib/toml-parser.js:311:26)
    at TOMLParser.runOne (@iarna/toml/lib/parser.js:64:30)
    at TOMLParser.goto (@iarna/toml/lib/parser.js:87:17)
    at TOMLParser.recordAssignStatement (@iarna/toml/lib/toml-parser.js:337:19)
    at TOMLParser.runOne (@iarna/toml/lib/parser.js:64:30)
    at TOMLParser.returnNow (@iarna/toml/lib/parser.js:107:17)
    at TOMLParser.recordAssignValue (@iarna/toml/lib/toml-parser.js:380:19)
    at TOMLParser.runOne (@iarna/toml/lib/parser.js:64:30)
    at TOMLParser.parse (@iarna/toml/lib/parser.js:45:22)
    at Object.parseString [as parse] (@iarna/toml/parse-string.js:13:12)

Question about the tests

Hi Iarna! Author of sgarciac/bombadil here. I'm wondering how you ran the tests against bombadil, since I see many tests in your comparison stating that bombadil fails, but that it actually passes, mostly of them for invalid toml files. I'm guessing you are expecting bombadil to throw an error in those cases? i.e., the test for float leading zero:

var bombadil = require('@sgarciac/bombadil')
var input = 'leading-zero = 03.14'
var reader = new bombadil.TomlReader
reader.readToml(input)
reader.result // -> undefined
reader.errors.length // -> 1

The comparison page states bombadil fails to signal an error, but it actually works as expected, putting an error in the reader.errors array and keeping reader.result undefined.

Outdated Node.js version support

This is based on travis coverage, as I couldn't find a better indicator of supported versions (such as "engines" in package.json).
Node 8, 11, and 13 are no longer maintained and shouldn't be supported, and Node 14 is missing support. ESM is also unflagged in Node 14, and backwards-compatible support is relatively easy if you're willing to add some ease-of-use.

Using with webpack

I'd like to use this in a webpacked project, i.e. create-react-app

Trying to import it:

import TOML from '@iarna/toml'

results in this error:

Uncaught ReferenceError: require is not defined

It's coming from here:

Do you have a suggestion for using this with webpack?

TOML.stringify adds "_" each three numbers when it has to convert large numbers

this is the input

what i expect is just the number

output:

Request: Deno port

Hi! I am very interested in porting this library to Deno (new runtime for JS/TS). Would you mind if I make a PR where I add a folder with Deno compliant code?

The code itself (as far as I can see) is not depending on any other packages or Node (except for util.inspect) so porting it should be a relatively simple task. The steps would be as follows:

1. Add rollup (or any other bundler which transforms code to ESM syntax from CommonJs) as a dev dependency.
2. Add to the scripts the command which transforms the code to ESM in a deno folder.
3. Change line 5 and 3 so that they work with Deno.

After that, it should be possible using this module with Deno by simply importing it from e.g. https://raw.githubusercontent.com/iarna/iarna-toml/latest/toml.js

Let me know what you think! 😄

Build Error in Webpack UglifyJs

Hi, I want use this package in browser side, but it seems have some problems with uglify plugin:

webpack -p --progress --hide-modules --env.prod

Hash: 85f427aa06d3a4f60d6e                                                                 
Version: webpack 2.4.1
Time: 69750ms
                                 Asset       Size  Chunks                    Chunk Names
         index.js?7b9782135fcefaae71be    1.13 MB       1  [emitted]  [big]  index
  d961fdfabbe512d2675de09af09f598b.svg      31 kB          [emitted]         
  674f50d287a8c48dc19ba404d20fe713.eot     166 kB          [emitted]         
  912ec66d7572ff821749319396470bde.svg     444 kB          [emitted]  [big]  
  b06871f281fee6b241d60582ae9369b9.ttf     166 kB          [emitted]         
af7ae505a9eed503f8b8e6982036873e.woff2    77.2 kB          [emitted]         
 fee66e712a8a08eef5805a46892932ad.woff      98 kB          [emitted]         
        vendor.js?df031a4f6465e884b124    4.39 MB       0  [emitted]  [big]  vendor
  6f0a76321d30f3c8120915e57f7bd77e.ttf      11 kB          [emitted]         
      manifest.js?086828a58800f469472c    1.49 kB       2  [emitted]         manifest
    vendor.js.map?df031a4f6465e884b124      23 MB       0  [emitted]  [big]  vendor
     index.js.map?7b9782135fcefaae71be    1.29 MB       1  [emitted]  [big]  index
  manifest.js.map?086828a58800f469472c    14.3 kB       2  [emitted]         manifest
                           favicon.ico    15.4 kB          [emitted]         
                            index.html  431 bytes          [emitted]   

ERROR in index.js?7b9782135fcefaae71be from UglifyJs
Invalid assignment [./~/@iarna/toml/lib/format-num.js:2,0][index.js?7b9782135fcefaae71be:5150,26]
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! Exit status 2
npm ERR! 
npm ERR! Failed at the build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Currently I have to disable the uglify.

Stringifying dependencies as one-line per depenency

Is there a way to stringify to a format where dependencies are not listed as [dependencies.<dependency>] blocks but in one line per dependency?

Please support toml spec 1.0

Calling toml.stringify on an object with literal key "toJSON" throws.

For drop-in compatibility with JSON.stringify toml.stringify should only attempt to call toJSON if it's a function. Example:

const a = { toJSON: { table: true } };
JSON.stringify(a); // {"toJSON":{"table":true}}
toml.stringify(a); // TypeError: obj.toJSON is not a function

const b = { toJSON() { return { table: true }; } };
JSON.stringify(b); // {"table":true}
toml.stringify(b); // table = true

Add test for TOMLParser low-level API stability

👋 opening this issue to remind myself of this tweet which asks for a test to ensure that the API used in https://github.com/googleapis/release-please/blob/e0c7b00034837d728770fb27ddf6d746c3dd181e/src/updaters/rust/toml-edit.ts remains available throughout iarna-toml releases.

I'm honestly not too sure how I'd go about writing a test like this - maybe extending TOMLParser and making sure the expected "events" are fired? I'm not sure what would even be asserted in the test - should it test the output of the extended parser? Should it test the internal state used in the parser?

I'm sure we can figure something out 😊

import toml using import statement in ts but failed

Hello, everyone. I tried to import toml using import statement in ts but failed, can someone help me

import toml from '@iarna/toml';
...
configdata = toml.parse(configstr)

hmr.ts:204  ReferenceError: global is not defined
    at node_modules/.pnpm/@[email protected]/node_modules/@iarna/toml/lib/create-date.js (create-date.js:3:18)
    at __require (chunk-Y2F7D3TJ.js?v=4804ad59:3:50)
    at node_modules/.pnpm/@[email protected]/node_modules/@iarna/toml/lib/toml-parser.js (toml-parser.js:26:20)
    at __require (chunk-Y2F7D3TJ.js?v=4804ad59:3:50)
    at node_modules/.pnpm/@[email protected]/node_modules/@iarna/toml/parse-string.js (parse-string.js:4:20)
    at __require (chunk-Y2F7D3TJ.js?v=4804ad59:3:50)
    at node_modules/.pnpm/@[email protected]/node_modules/@iarna/toml/parse.js (parse.js:2:18)
    at __require (chunk-Y2F7D3TJ.js?v=4804ad59:3:50)
    at node_modules/.pnpm/@[email protected]/node_modules/@iarna/toml/toml.js (toml.js:2:17)
    at __require (chunk-Y2F7D3TJ.js?v=4804ad59:3:50)
warnFailedUpdate @ hmr.ts:204
fetchUpdate @ hmr.ts:237
await in fetchUpdate(异步)
(anonymous) @ client.ts:182
handleMessage @ client.ts:180
(anonymous) @ client.ts:92
显示 5 更多框架
显示简略信息
hmr.ts:206  [hmr] Failed to reload /src/components/Index.vue. This could be due to syntax errors or importing non-existent modules. (see errors above)

All comments are syntax errors

New lines at the end of any comment cause an immediate error due to false flagging as control characters. This makes use of any comments impossible. This problem doesn't occur in v2.2.5, only in v3.0.0.

TomlError: Control characters (codes < 0x1f and 0x7f) are not allowed in comment
s, use \u000d instead at row 3, col 5, pos 22:
2: yea = 2
3> # ye
       ^

Package doesn't work with Content Security Policy unsafe-eval

NetlifyCMS uses this package as a dependency. This package includes an eval function that is not permitted with CSP 'unsafe-eval' which prevents NetlifyCMS from working in secure environments.

decaporg/decap-cms#4367 (comment)

Please, let me know if you're open to removing the eval - I can submit a PR.

setImmediate is not defined

Unhandled Rejection (ReferenceError): setImmediate is not defined
when invoke await TOML.parse.async(something) in Chrome.

Would you help to compare my parser?

Hello! Would you mind to add my parser to the compare table, after @iarna/[email protected], [email protected], [email protected], @sgarciac/[email protected]? My parser is https://GitHub.com/LongTengDao/j-toml/, it was published at https://www.npmjs.com/package/@ltd/j-toml.

[Feature] Comment Preservation Proposal

In addition to ideas from #22 I wanted to make some specific suggestions about comments support.

const result = toml.parse('...', {
  preserveComments [boolean] = true,  // default: preserve comments
  enumerableComments: [boolean] = false // default: comments property is not enumerable
});

• preserveComments when not truthy will exclude the ___COMMENTS___ property altogether
• enumerableComments (See Object.defineProperty.) would allow for the ___COMMENTS___ property to be enumerable. This would allow for preservation via JSON.stringify, default would be false. toml.stringify would explicitly check for values and include them if non-whitespace comments are present.

The property for comments would literally be result.___COMMENTS___; (three underscores before and after)

The format for comments would be:

• result,___COMMENTS___.${entry}.BEFORE - entries and document
• result,___COMMENTS___.${entry}.INLINE - only for entries
• result,___COMMENTS___.${entry}.AFTER

The entry key for the document is result.__COMMENTS__.___DOCUMENT___

What would be returned, should be the same object structure as before with an optionally enumerable (false by default) ___COMMENTS___ property, three underscores, all caps.

As mentioned in @scriptcoded 's comment Multiple comments together should be merged with \n.

# Top of file
# Multiline?
      << must have an empty line to break top of file and before names, otherwise will be before names
# before names
[names]
# before foo
"foo" = "Mr. Foo" # inline foo
# after foo
   << blank line, otherwise comment above would be attached to names.bar
# before bar
"bar" = "The big Bar"
  << blank line, anything below attached to next section
# before baz
"bop" = "bopper"
# before boom
"boom" = "boomer"
# after boom
  << blank line...
# bottom or file

Edge cases... For multiple interspersed comments/lines, the first block with a blank line after, and no blank after the node will belong to the previous node's AFTER, all additional nodes will be part of the next node's BEFORE, The document nodes will be before/after as such.

# comment1
# comment 2
   << blank
# comment 3
  << blank
# comment 4
[first]

Becomes:

{
  ...
  ___COMMENTS___: {
     ...
    ___DOCUMENT___: {
      BEFORE: " comment1\n comment2\n\n comment3"
    }
  }
}

NOTE: spaces after # preserved, no whitespace lines preserved except for the case of \n \n in between comment blocks, but any whitespace on the original, otherwise blank lines are removed.

With two \n\n together the output blank line stays blank... if there was a block between, even if no whitespace, should parse with \n \n space between the \n and will output # with no space on the blank line, even if the original had one. This may not align with the original input, but is probably the best option for dealing with this edge case.

Comment sections beginning with \n mean there are blank lines at the top of said block, similarly for the end, any dangling \n indicate blank lines.

# block
     << has whitespace, no other comments

becomes block\n whitespace on blank line removed, but dangling \n is blank line.

\n\n comment

becomes:

# comment

empty line(s) at start.

Carriage returns (\r) are dropped.

# start
#    
# more

The middle line (# ) with whitespace, will result in start\n \n more where the extra whitespace is dropped to a single space, similar for otherwise blank lines being replaced with nothing, but having extra \n

Standalone `\r` is accepted by iarna-toml but forbidden by the spec

A multi-line literal string like:

foo = '''\r'''

Is successfully parsed by this module, but per the spec \r is only allowed in multi-line literal strings when followed by \n.

I found this when I came across a string with \r\r that threw an error in the Rust toml crate, but toml-lint.com (powered by iarna-toml) said it was valid.

Table should be object extends null

toString = 1

This will cause an error:

Can't redefine existing key

There is two way to fix this,

1. use hasOwnProperty, not in
2. use Object.create(null) to create a table, not {}

But the first way is slow, and can't deal with __proto__ property, so the way 2 is better.

1.0.0-rc.1 stringifier can produce results incompatible with 0.5 and 0.4

In and of itself, this is unsurprising: Use new features, it won't work. The place where this is maybe surprising is with the support for mixed-type arrays. Consider the following examples:

#	json	1.0.0-rc.1	0.5
1	{a: [1, 2]}	a = [ 1, 2 ]	a = [ 1, 2 ]
2	{a: [1.0, 2]}	a = [ 1, 2 ]	a = [ 1, 2 ]
3	{a: [1.1, 2]}	a = [ 1.1, 2 ]	a = [ 1.1, 2.0 ]
4	{a: [1.1, 2n]}	a = [1.1, 2 ]	a = [ 1.1, 2.0 ]
5	{a: [1.1, 9999999999999999n]}	a = [ 1.1, 9_999_999_999_999_999 ]	a = [ 1.1, 9_999_999_999_999_999.0 ]

So my thinking:

1. this is all good
2. numbers are numbers in js, we can't tell 1.0 apart from 1, so this is fine
3. the 1.0.0-rc.1 version is perfectly valid, but nonetheless, let's update this to be compat with 0.5. no reason to introduce gratuitous backward incompatibility -- a compat doc under 0.5 should still produce a compat doc on 1.0.0-rc.1
4. no change to 1.0.0-rc.1 version, 0.5 should throw as a bigint is defintiely not a float
5. same as 4, as the fact that bigints are not floats is even more important here as if you load the 0.5 doc you'll get a value of 10000000000000000 due to lack of available precision.

Stringify produces invalid floating point number

This example is for iarna-toml 2.2.5 and node v16.13.1.

const TOML = require('@iarna/toml')
undefined
> config = TOML.parse('[config]\nvalue=1e-11')
{ config: { value: 1e-11 } }
> TOML.stringify(config)
'[config]\nvalue = 1e-11.0\n'
> TOML.parse(TOML.stringify(config))
Uncaught:
TomlError: Unexpected character, expected only whitespace or comments till end of line at row 2, col 14, pos 23:
1: [config]
2> value = 1e-11.0
                ^
3: 


    at TOMLParser.parseWhitespaceToEOL (/root/mnt/fds-core-workflow/node_modules/@iarna/toml/lib/toml-parser.js:311:26)
    at TOMLParser.runOne (/root/mnt/fds-core-workflow/node_modules/@iarna/toml/lib/parser.js:64:30)
    at TOMLParser.goto (/root/mnt/fds-core-workflow/node_modules/@iarna/toml/lib/parser.js:87:17)
    at TOMLParser.recordAssignStatement (/root/mnt/fds-core-workflow/node_modules/@iarna/toml/lib/toml-parser.js:337:19)
    at TOMLParser.runOne (/root/mnt/fds-core-workflow/node_modules/@iarna/toml/lib/parser.js:64:30)
    at TOMLParser.returnNow (/root/mnt/fds-core-workflow/node_modules/@iarna/toml/lib/parser.js:107:17)
    at TOMLParser.recordAssignValue (/root/mnt/fds-core-workflow/node_modules/@iarna/toml/lib/toml-parser.js:380:19)
    at TOMLParser.runOne (/root/mnt/fds-core-workflow/node_modules/@iarna/toml/lib/parser.js:64:30)
    at TOMLParser.returnNow (/root/mnt/fds-core-workflow/node_modules/@iarna/toml/lib/parser.js:107:17)
    at TOMLParser.parseNumberExponent (/root/mnt/fds-core-workflow/node_modules/@iarna/toml/lib/toml-parser.js:903:21) {
  fromTOML: true,
  wrapped: null,
  line: 1,
  col: 13,
  pos: 23
}
> quit