kantord / emuto

manipulate JSON files

Website • Tutorial • Live demo • CLI version • Compile-to-JS version

---

Emuto is a small language for manipulating and restructuring JSON and other data files. Emuto is inspired by jq and GraphQL

Features

• Transform and query data structures
• Integrate with unix commands in the command line
• Conversions between different file formats
• Supported input formats: JSON, text, csv, tsv, dsv
• Supported output formats: JSON, text
• Available as a Webpack loader

Getting started

Emuto as a CLI tool

npm install -g emuto emuto-cli

Read more in the tutorial

For Arch Linux users, also available as an AUR package

Webpack loader for emuto

yarn add --dev emuto emuto-loader

Read more in the Webpack guide

What is emuto good for? Examples

Number of items in JSON file

curl my_file.json | emuto 'length'

Your karma on HackerNews

curl https://hacker-news.firebaseio.com/v0/user/kantord.json -s | emuto '$.karma'

Convert another command's output to JSON

ls | emuto -i=raw '$[0:-1]'

See number of NPM dependencies

cat package.json | emuto -c '$.dependencies | keys | length'

List available scripts in package.json

cat package.json | emuto -c '$.scripts | keys | join " · "'

Get only the relevant data from a huge JSON file

curl https://api.github.com/repos/stedolan/jq/commits |\
emuto -c 'map ($ => $ { commit { message } committer { login } } )'

Automate the restructuring of data by creating scripts with emuto

restructure.emu

#! emuto -s

$
  | map ($ => $ { commit { message } committer { login } } )
  | map ($ => {
      "committer": $.committer.login,
      "message":   $.commit.message,
    })

Calling your script

curl https://api.github.com/repos/stedolan/jq/commits | ./restructure.emu

Contributing Guide

Read our contributing guide to learn about our development process, how to create bugfixes and improvements, and how to build and test your changes to emuto.

emuto's Issues

Allow extracting object projection fragments to functions

$ {
  from { ...Coordinate }
  to { ...Coordinate }
}
where
  $Coordinate = $ => $ { x, y }

Implement ternary

The syntax should by Python-like:

"Reject" if $salary < 1000 else "Accept"

Precedence should probably be the same as ternary in JavaScript

Add animated gif to README.md

Idea: Disallow non-zero-element pipes in operations

If there's only one item in a pipe, that should be handled as a value

Otherwise a type error should be thrown (there's no way that a generator could be multiplied by 4 for example)

Valid:

(4) * 2

Not valid:

(4 | 4) * 2

Implement .*. projection

For this input:

{
  "users": [
    {
      "posts": [
        {
          "title": "foo"
        },
        {
          "title": "bar"
        }
      ],
      "articles": [
        {
          "title": "baz"
        }
      ]
    },
    {
      "posts": [],
      "articles": []
    }
  ]
}

The following filter:

.users.*.*.title

Should produce the following output:

["foo", "bar", "baz"]

Add documentation for emuto-repl

Implement built in function 'has(key)'

Implement a function that returns whether the input object has the given key, or the input array has an element at the given index.

A code example that should work:

{"foo": "bar"} | has "foo" // results true
{"foo": "bar"} | has "bar" // results false

Allow nesting array projections under object projections

$ {
  users[1, 2] {
    name
  }
}

Add tutorial

Docs: add GitHub link to menu bar

Implement lambda functions using fn keyword

map (fn(): $ * 2)

should be equivalent to

map $ => $ * 2

Inside the parentheses it should be possible to use projections in this fashion:

(fn({ user }): $user.name)

Simplify primitive parsers into a single regexp

Make spread operator actually work for objects

Don't show regular expressions in syntax error messages

When a syntax error is reported, a bunch of regular expressions are shown:

SyntaxError: Expected '!' or '$' or '(' or '+' or '-' or '//' or '[' or '' or 'false' or 'null' or 'true' or '{' or /("(((?=\)\(["\/bfnrt]|u[0-9a-fA-F]{4}))|[^"\\\0-\x1F\x7F]+)")/ or /((?!null|false|true)[a-zA-Z][0-9a-zA-Z_$]|(null|false|true)[0-9a-zA-Z_$])/ or /-?(?:0|[1-9]\d*)(?:.\d+)?(?:[eE][+-]?\d+)?/ or /.[$A-Z_][0-9A-Z_$]*/i on line 1 column 82, found '}' instead

This is completely useless and annoying for the user. Instead of regular expressions it'd be much more useful to show names, such as 'Identifier' instead of /((?!null|false|true)[a-zA-Z][0-9a-zA-Z_$]*|(null|false|true)[0-9a-zA-Z_$])/

Luckily, Parsimmon has a really nice feature to solve this: the desc method. This code example shows how to use it: https://github.com/jneen/parsimmon/blob/master/API.md#parserparseinput

Therefore it's sufficient to add a desc call to each parser. To test this, I would just create a separate test file which parses the same invalid code (say ¡) with every parser and examines the result.

Incorrect return value of ["hello"] | map $ => [...$, 4]

["hello"] | map $ => [...$, 4] returns ["hello4"]

This is nonsense, it should return ["hello", 4]

Idea: datasources

emuto could be used as a GraphQL-like query language to retrieve data from the backend.

To determine which file/database table/API should be queried, one possible way could be creating a datasource function which could work like this:

datasource "userPhotos" : {"userId": $.userId}

this would simply return the data from datasource, passing {"userId": $.userId} as variables. The result could be transformed by using projections or a pipeline:

datasource "userPhotos" : {"userId": $.userId} {
    title
    photoURL
    user {
        name
        avatar
    }
}

In this case the server would pass the variables received along with the query as the input to the transformation

Idea: add static type analysis

This feature would consist in 3 different parts:

• Add a syntax for type casting (it should actually be a special way to use functions that take a value and either return a casted value or throw an exception). It should handle type arguments, for example, array of string vs array of integer
• Add casting functions for basic types (integer, array, etc.)
• A static type checker as a separate package that can be used to find errors and generate flow/typescript types for the entire application (or modules)

Improve tuple performance

Use the same tactic as used here:
cc324d1

Design pattern matching syntax + roadmap

The basic idea is to have a syntax similar to this:

(
  ["foo", $bar] -> $bar.value * 2,
  [$first, {"baz": $value}] -> [$value, $first],
  $value, $value >= 3.14 -> "Bigger than 3.14",
  otherwise -> error "Nonsense"
)

Refactor `:` as an operator

Ensure that only functions can be called

Implement object projection

{"foo": 1, "bar": 2, "baz": 3} | ${foo, bar}  // {"foo": 1, "bar": 2}
{"foo": 1, "bar": 2, "baz": 3}{"foo", "bar"}  // {"foo": 1, "bar": 2}

Document spread operator

Allow infix function calls

Example:

({"foo": "bar"} has "foo") // returns true
({"foo": "bar"} has "bar") // returns false

({"foo": "bar"} has "foo") should be equivalent to {"foo": "bar"} | has "foo"

Allow passing variable to interpreter

The variables should

• be available in the filter itself
• should be cleaned to ensure they only contain valid JSON values and not potentially dangerous stuff (https://github.com/debitoor/safe-json-stringify)

Is this an infinite loop?

{"columns": [(sort_by(-(.calories | tonumber)) | .[] | [.name, .calories])]}

Document optional chaining/projection

This is weird

Implement `::` operator

a :: b should be the same as [a, ...b]

Add git hook

yarn test and yarn checks should be always executed before commit, as well as commit message format enforcement

Implement nested projection

Projection rules should be nestable (GraphQL-style), for example:

{"foo": 1, "bar": {"x": [1, 0, 0]}, "baz": 3} | ${foo, bar{x[2,0]}}  // {"foo": 1, "bar": {"x": [0, 1]}}

Implement cross product

Fork JSON-Splora to support emuto

https://github.com/wellsjo/JSON-Splora

Set up docusaurus for documentation

Allow for a generator of inputs

Allow input to be a generator.

When the input is a generator, the filter should be executed on each element separately.

Two special filters should be added:

slurp - this filter should take every element in the input and convert it to an array

nibble - this should convert an array into a generator
nibble | $ * 2 should have the same result as map $ => $ * 2

Allow projecting each element of array in nested object projections

$ {
  foo {
    baz[] {
      bar
    }
  }
}

Add code examples to documentation site home page

Add contributing guidelines to website

The solution has the meet these criteria:

• There should be a development section in the sidebar on the website, with the contributing guide with it
• Contributing guide should still be in the root of the repo so that people can find it in its 'default' place without looking at the website

Allow for aliases in object projection

$ {
  fooBar: foo_bar,
  baz
}

Use transducers

https://github.com/cognitect-labs/transducers-js

Create emuto-cli

Add regexp support

Add contribution guidelines

Add error function

error "Hello World"

should throw an exception

((function() { throw new Error('Hello World') })())

Implement comprehension syntax

[.title forEach $ if (.author.posts | length) >= 3 sortBy .author.age]

The sortBy .author.age part should actually just be a function call. :-) So this should also be valid

[.title forEach $ if (.author.posts | length) >= 3 reverse]

Or of course without the extra function call:

[.title forEach $ if (.author.posts | length) >= 3]

Set up slackin

https://github.com/rauchg/slackin

Fix syntax error in this code example

https://kantord.github.io/emuto/docs/variables

Idea: @ syntax for applying function to pipeline

The following example:

map ($ => $ * 2) | map ($ => $ - 1) | @slurp | length

Should be roughly equivalent to

slurp (map ($ => $ * 2) | map ($ => $ - 1)) | length

Enable search in documentation

https://docusaurus.io/docs/en/search

Add some form of grammar check to CI for documentation

It would be great to use the CI to run some grammar checks on the docs folder