loganbarnett / flow-degen

A Flow based refiner/validator generator for Javascript. Generate repetitious type safe refiners that Flow can type check!

flow-degen README

flow-degen

This is a deserialization generator for JavaScript objects that are under Flow. Some deserializer/validator projects use $ObjMap and some clever casting to achieve a type safe means of deserialization/validation. flow-degen aims to leverage Flow itself in ensuring type safety by generating all of that ugly deserialization/validation code you would have written by hand.

Pros:

1. flow-degen introduces no runtime dependencies to consumers, other than itself.
2. flow-degen emits generators that use plain Flow type checking. There is no any casting internally, and there are no magic types.

Cons:

1. flow-degen does not currently provide a list of deserializer errors, but instead bails on the first error.
2. There is potentially a memory/storage footprint concern for non-trivial sizes and amounts of deserializers. A minifier may significantly mitigate this con. Using degenRefiner to reference other refiners can also reduce the footprint by calling other refiners instead of duplicating refinement logic in multiple places.

installation

yarn add -E -D flow-degen

config structure

You’ll need a config file to work as inputs to flow-degen. It has the structure below. Any paths or file names that start with ./ are intended to show a relative directory.

{
  "baseDir": "",
  "generatedPreamble": "",
  "generators": [
    {
      "exports": {
        "fooGenerator": "fooRefiner"
      },
      "inputFile": "./dir/generator-input-file.js",
      "outputFile": "deserializer-output-file.js"
    }
  ],
  "importLocations": {
    "importName": "./dir/runtime.js"
  },
  "typeLocations": {
    "TypeName": "./dir/type.js"
  }
}

baseDir

baseDir is the directory that the generated file will be assumed to be living out of relative to the imports.

generatedPreamble

generatedPreamble is the code or text you want to appear at the top of the file. You can use this to insert a copyright comment block, include linter rules (such as disabling ESLint’s no-used-expressions rule which can be an issue for disjoint unions).

generators

This is a list of generators and how they produce refiners. Generators here have the following structure:

{
  "exports": {
    "fooGenerator": "fooRefiner",
    "barGenerator": "barRefiner",
    "bazGenerator": "bazRefiner"
  },
  "inputFile": "foo-generator.js",
  "outputFile": "foo-refiner.js"
}

exports

exports is a mapping of identifiers exported from the generator file that flow-degen can find, and it maps to identifiers it will generate as refiners for that associated generator. In the sample configuration, fooGenerator is found in foo-generator.js, and it will emit a fooRefiner to foo-refiner.js that you can then import or require to use.

exports is also implicitly added to importLocations such that your refiners can refer to each other, and even achieve recursive calls if your structure requires recursion.

importLocations

This is a mapping of import names (which must be valid JavaScript identifiers) to files. The identifiers must map to export entities inside of your module. These will be hoisted to the top of the generated deserializer files if they are used. Including entries in here does not mean it will be used in your files, it is simply a lookup for flow-degen to use.

typeLocations

Just like importLocations, typeLocations is a reference for export type identifiers so the generated deserializer can find them if any of the combined deserializers use it.

usage

internal deserializer generators

import from flow-degen to get deserialization generators.

degenField

degenField is meant to be used in conjunction with degenObject.

degenFilePath

This is just an alias for degenString currently, but could one day encompass a Flow opaque type that, while represented by a string, is ensured to be a valid file path.

degenList

Requires a deserializer to be used for the element type, which is provided as its only argument. This will produce an Array<T>.

Suppose we have a foos-generator.js:

import { degenList, degenNumber} from 'flow-degen'

const numberType = { name: 'number', typeParams: [] }
export const foosGenerator = () => degenList(numberType, degenNumber())

Upon importing the emitted file, you can now refine into an Array of number:

import { deFoos } from './foos-refiner.js'

deFoos([1, 2, 3]) // Produces [1, 2, 3].
deFoos('farsnaggle') // Produces Error object.

declare var someInput: mixed

const eitherResult = deFoos(someInput)

if(eitherResult instanceof Error) {
  // Here the result did not refine correctly.
  console.error('How did this happen?', eitherResult)
} else {
  // Now you have an Array of number.
  console.log(eitherResult.map(x => x + 1))
}

degenMapping

A “mapping” is of the type {[A]: B} although usually it will be {[string]: mixed}. It takes the key meta type, the value meta type, a key deserializer, and a value deserializer for A and B respectively.

degenMaybe

The degenMaybe generator is for creating refiners for maybe types (e.g. type Foo = ?string). The maybe type will still require additional refinement after passing through the refiner. For example, given the type:

export type Foo = {
  bar: ?string,
}

And generator:

import { degenObject, degenField, degenMaybe, degenString } from 'flow-degen'

const fooType = { name: 'Foo' }
const stringType = { name: 'string' }
export const fooGenerator = () => degenObject(fooType, [
  degenField('bar', degenMaybe(stringType, degenString())),
])

The refiner would be used like so:

import { deFoo } from './foo-refiner.js'

declare var someInput: mixed

const eitherResult = deFoo(someInput)

if(eitherResult instanceof Error) {
  // Here the result did not refine correctly.
  console.error('How did this happen?', eitherResult)
} else {
  // We have a foo, but bar may not have been present
  if (eitherResult.bar != null) {
    console.log(eitherResult.bar + ' was refined')
  } else {
    console.log('result had a null bar')
  }
}

degenNumber

The degenNumber deserializer simply deserializes a value as a number.

degenObject

An “Object” can be thought of as a collection of “fields”. See degenField as these go together except for empty objects. degenObject takes the type of the object and a list of required fields that degenField can emit, and a second list of degenField results that represent the optional fields.

Assume the object Cat.

export type Cat = {
  // Cats always have demands.
  demands: number,
  // Cats can have no love sometimes.
  love?: number,
}

const catType = { name: 'Cat' }
const catGenerator = () => degenObject(catType, [
  degenField('demands', degenNumber()),
], [
  degenField('love', degenNumber()),
])

It is well known that cats always have demands but only sometimes have love. It is fallacious to assume love will always be present.

import { catRefiner } from './cat-refiner.js'

// It's pretty easy to get an unsanitized cat from anywhere, really.
handleUnsanitizedCat((input) => {
  const catOrError: string | Error = catRefiner(input)
  if(catOrError instanceof Error) {
    goGetADog()
  } else {
    // We have a cat! But we can't expect love.
    // Flow will also settle for a null check for love.
    if(catOrError.hasOwnProperty('love')) {
      console.log(`My cat loves me ${catOrError.love} love units!`)
    } else {
      console.log('My cat does not have any love for me at all...')
    }
  }
})

degenString

The degenString deserializer simply deserializes a value as a string.

Say we have a name-generator.js:

import { degenString } from 'flow-degen'
export const nameGenerator = () => degenString()

And this is configured to produce a name-refiner.js, this is how it would be used:

import { nameRefiner } from './name-refiner.js'

// This could be an HTTP POST handler on a server, or a form handler on a UI
handleUnsanitizedInput((input) => {
  const nameOrError: string | Error = nameRefiner(input)
  if(nameOrError instanceof Error) {
    console.error(nameOrError)
  } else {
    // Here can we use the name.
    storeName(nameOrError)
  }
})

degenSentinelValue

This deserializer is to be used in conjunction with degenSum to produce deserializers for a sum type. This represents one member of the union. It needs a key, which is a string value for the sentinel value, and the object deserializer itself, which will likely be degenObject.

degenSum

The degenSum deserializer handles sum type objects. It takes the type of the union, the sentinel field name, the sentinel field type, and a list of sentinel object deserializers (which can just come from degenObject) from degenSentinelValue.

degenValue

The degenValue deserializer takes a type (as a string) and a value (which could be anything). It checks for the literal equivalence of that value. This can be helpful when using Flow’s sentinel properties for sum types of objects.

degenRefiner

The degenRefiner refiner simply imports a symbol for use. This allows recursion to work when the refined data structure is recursive. Also it allows for reuse of other refiners of any kind. This reduces the size of generated refiners significantly. Otherwise the refiners are inlined.

Suppose we have a foo-generator.js whose generator builds the deFoo refiner:

import { degenObject, degenField, degenString } from 'flow-degen'

const fooType = { name: 'Foo' }
export const fooGenerator = () => degenObject(fooType, [
  degenField('first', degenString()),
  degenField('last', degenString()),
])

And we have a bar-generator.js:

import {
  degenObject,
  degenField,
  degenString,
  degenRefiner,
} from 'flow-degen'

// This is the same fooType in foo-generator.js, and could be imported.
const fooType = { name: 'Foo' }
const barType = { name: 'Bar' }
export const fooGenerator = () => degenObject(barType, [
  degenField('foo', degenRefiner(fooType, 'deFoo')),
])

Here the generator will simply invoke deFoo to refine the foo field. Any import, type, and hoist information will be made available in this refiner.

Note that this symbol must be one that is managed by flow-degen in your configuration file, or your configuration file must specify in the imports how to find this symbol.

creating meta types

Objects of type MetaType are passed into many generator functions and contain information flow-degen uses to build imports and type signatures in the generated code. The MetaType type can be found in src/generator.js but at a minimum contains the type name:

type Foo = {
  bar: string,
}

const fooType = { name: 'Foo' }

In the case of generic types, the optional typeParams field in MetaType can be used to list the meta types to be specified in the type signature:

type Foo<K: string, V: string> = {
  [K]: V,
}

const stringType = { name: 'string' }
const fooType = { name: 'Foo', typeParams: [stringType, stringType]}

Some types (for example flow utility types like $PropertyType) take literal strings or numbers instead of a type. The MetaType has an optional literal boolean to indicate these usages:

type Foo = {
  bar: {
    baz: string,
  },
}

const fooType = { name: 'Foo' }
const bazPropertyType = { name: "'baz'", literal: true }
const barType = { name: '$PropertyType', typeParams: [fooType, bazPropertyType] }

Note that string literals (and other literals with delimiters) need to include the delimiters in the name (e.g. “‘baz’” instead of “baz” or ‘baz’).

building custom deserializer generators

All deserializers must satisfy the following contract:

• They must be a function.
• The function returns a DeserializerGenerator<CustomType: string, CustomImport: string>, which is a tuple of a function that returns a string (the code) and a CodeGenDep<CustomType: string, CustomImport: string>. The exacts of these types can be found in ./src/generator.js.
• The code returned by the function must accept a mixed as a parameter. This is your input provided from your mystery variable. It is assumed to be “deserialized” already in the sense that it is not a string of JSON but perhaps the result of JSON.parse.
• If any imports are used, they must be enumerated in the imports list of the CodeGenDep. Any imports used by the generated function will also need to be part of the CustomImport type parameter of the generator as well as included in importLocations in your flow-degen configuration file (adding an import to importLocations is not necessary if the import is an export from a refiner defined in your flow-degen config).
• If any type imports are used, they must be enumerated in the types list of the CodeGenDep. Any types used by the generated function will also need to be part of the CustomType type parameter of the generator as well as included in typeLocations in your flow-degen configuration file.
• Consider that your generated code could likely be embedded deep within a function chain. If you need some “root” access to the module to declare things such as throw-away types, use the hoists list to place code.
• If your generator delegates to other generators (such as degenList delegating to a deserializer for the elements), you must honor the results of its CodeGenDep when you call the generator. This could mean merging the CodeGenDep with your own. The mergeDeps function in ./src/generator.js does this for you. It is found by flow-degen consumers as a top-level export (=import { mergeDeps } from ‘flow-degen’=).
• Try testing your refiner with an opaque type. This seems to be a good way to ensure Flow cannot run into issues with type inferencing. We suspect this is a good test because opaque types can never be inferred, and therefore will always need explicit types at the call site of a refiner.

Let’s create an custom generator example where we have an uppercase string.

import {
  degenString,
  mergeDeps,
  type DeserializerGenerator,
} from 'flow-degen'
import {
  type UppercaseString,
  uppercase,
} from './my-string-utils.js'

type UppercaseGeneratorType =
  | 'UppercaseString'

type UppercaseGeneratorImport =
  | 'uppercase'

export const degenUppercaseString = (
): DeserializerGenerator<UppercaseGeneratorType, UppercaseGeneratorImport> => {
  const [ stringGenerator, stringDeps ] = degenString()
  return [
    () => {
      return `(x: mixed): UppercaseString => {
         return uppercase(${stringGenerator()})
      }`
    },
    mergeDeps(
      stringDeps,
      {
        hoists: [],
        imports: [ 'uppercase' ],
        types: [ { name: 'UppercaseString' } ],
      },
    ),
  ]
}

Custom generators are no different from the built-in generators.

import {
  degenUppercaseString,
} from './custom-degens.js'

export const generateUppercaseStringRefiner = () => degenUppercaseString()

The built-in generators in src/generators.js can be used as more complex examples for building your own generators.

command line

Once installed, you can use the flow-degen script to generate your deserializers:

yarn flow-degen degen-config.json

consuming generated deserializers

The output files you indicate will export refiner functions defined in the exports config for the generator. The refiner functions take the form of (mixed) => T | Error.

import fs from 'fs'
import { fooDeserializer } from './foo.deserializer.js'

const unvalidatedFoo = JSON.parse(fs.readFileSync('foo.json', 'utf8'))
const fooOrError = fooDeserializer(unvalidatedFoo)

// Refine the result.
if(fooOrError instanceof Error) {
  console.error('Error deserializing foo:', fooOrError)
} else {
  doStuffWithFoo(fooOrError)
}

editing generated deserializers

Do not edit these files directly except for debugging purposes. The files will be overwritten on subsequent runs of the generator. Also, the code written there is not designed with human maintainability as its chief concern.

source control

Tooling could be built to make the generation process opaque to a consumer, but at the time that method is not known to flow-degen maintainers. It is fine and even recommended to check your generated deserializers into source control.

known issues

no-unused-expressions

When using degenSum, ESLint has a no-unused-expressions rule that fails during a cast in the default case. This expression doesn’t do anything in the runtime, but Flow needs it to tie the “everything else” match to the default case. This makes Flow flag an error when a member of the union isn’t enumerated in the switch. To work around this issue, you can add // eslint-disable no-unused-expressions to your configuration’s generatedPreamble.

bragging rights

The config object above is generated from config-generator.js which in turn must deserialize itself in order to build the generator. mind-blown.gif

flow-degen's Issues

Handling Nested Anonymous Types

This isn't an issue per se, more just meant for kicking off some discussion. Say I've got a library of types that I'm creating refiners for, and that the library has types like this:

type SomeData = {
  childData: {
    foo: string,
  }
}

The type of childData doesn't have a name, which makes it hard to create a MetaType for it (let's also assume that I can't go in to the library and extract { foo: string } into a named type). I could easily add a type alias, like so:

type SomeDataChildData = $PropertyType<SomeData, 'childData'>

I can then use SomeDataChildData in my MetaType, pass it into degenObject and all will work fine. This is the approach I'm using currently.

But should flow-degen support utility types more directly? I.e. to allow users to do something like this:

const someDataMetaType = { name: 'SomeData' }
const childDataPropertyNameMetaType = { name: 'literal', value: 'childData' }
const someDataChildDataMetaType = { name: '$PropertyType', typeParams: [ someDataMetaType, childDataPropertyNameMetaType ] }

const generator = () => degenObject(someDataMetaType, [
  degenField('childData', degenObject(someDataChildDataMetaType, [
    degenField('foo', degenString()),
  ], [])),
], [])

I've done up two possible avenues to supporting utility types: option one and option two. The above example is based on option two, which I have a slight preference for over option one. I'll also note that option two makes the slightly unrelated but desired change to make typeParams in MetaType optional.

I could see this being a beneficial addition, but it also seems like a fairly niche problem. In most of the cases I've used flow-degen, I'm in full control of the types and can introduce named types for nested types quite easily.

Add Github Actions

It's been a while since there's been much activity on flow-degen. We have accumulated a number of changes and the Rambda upgrade never got published. What would make this easier is configuring Github Actions so we have an automatic (or nearly automatic) build and release process. It also doubles as documentation to describe how the package gets released.

It would be nice if we could capture the dependabot changes in the changelog but for now let's just cover a simple package publish.

Intersection of degenField, degenRefiner and opaque type causes error

This could be a case of PEBCAK, but I was using opaque types and ran into an issue with flow finding an error in the resultant generated file. The types look (approximately) like this:

opaque type MyCustomId = string

export type MyObject = {
  id: MyCustomId,
}

There's a custom refiner in the same file as the opaque type that looks like this:

export const refineMyCustomId = (x: mixed) => MyCustomId | Error {
  return deString(x)
}

and then in a separate file the following generator:

export const genRefineMyObject = () => degenObject(
  { name: 'MyObject', typeParams: [] },
  [degenField('id', degenRefiner('refineMyCustomId'))],
  [],
)

The generated refiner has the following error:

Error ┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈ my-object-refiner.js:52:13

Cannot call deField with refineMyCustomId bound to deserializer because string [1] is incompatible with
Error [2] in the return value.

     my-object-refiner.js
     49│         } else {
     50│           const id = deField(
     51│             'id',
     52│             refineMyCustomId,
     53│             json.id,
     54│           )
     55│           if (refineMyCustomId instanceof Error) {

     node_modules/flow-degen/src/deserializer.js
 [2] 34│   deserializer: (a: mixed) => (T | Error),

     my-object.js
 [1] 25│ export const refineMyCustomId = (x: mixed): MyCustomId | Error => {

The only workaround I've found so far is to manually edit the generated file like so:

const id = deField<MyCustomId>(
  'id',
  refineMyCustomId,
  json.id,
)

Which is not ideal, as the change will be reverted the next time the refiner is regenerated.

Flow error when a custom generator's dependencies contains a non-flow-degen import

I've created a branch in my fork that exercises this issue, but in summary, if I have something like this:

const customDegen = <CustomType: string, CustomImport: string>(
  refinerType: MetaType<CustomType, CustomImport>,
  refinerGenerator: DeserializerGenerator<CustomType, CustomImport>,
) => {
  const [refinerCode, deps] = refinerGenerator
  const header = typeHeader(refinerType)

  return [
    () => {
      return `
        (x: mixed): ${header} | Error => {
          const refiner = ${refinerCode()}
          return refiner(importedFn())
        }
      `
    },
    mergeDeps<CustomType, CustomImport>(
      deps,
      { types: [], imports: ['importedFn'], hoists: [] },
    ),
  ]
}

The following error results:

$ yarn flow
yarn run v1.19.0
$ /Users/justin.shepard/Development/third-party-repos/flow-degen/node_modules/.bin/flow
Error ┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈ test/custom-import.js:44:30

Cannot call mergeDeps with object literal bound to b because string [1] is incompatible with enum [2] in array element
of property imports.

     test/custom-import.js
     41│     },
     42│     mergeDeps<CustomType, CustomImport>(
     43│       deps,
 [1] 44│       { types: [], imports: ['importedFn'], hoists: [] },
     45│     ),
     46│   ]
     47│ }

     src/generator.js
 [2] 40│   imports: Array<DeImport | CustomImport>,

Running the generator doesn't have any errors, and flow is happy with the resultant refiner, it's just the generator itself that flow doesn't like. It's slightly happier if the 'importedFn' string is changed to something flow-degen provides, like 'stringify'. It seems to be related to mergeDeps as if you comment out lines 42, 43 and 45 flow is happy (though the test fails at that point).

Certain Refiners Can't Be Used Alone

Some refiners (deList and deMapping that I've found so far) can't be used by themselves due to missing type parameters. For example:

const degenMyStringList = () => degenList(degenString())

will create the refiner:

export const refineMyStringList = deList.bind(
  null,
  deString,
)

which leads to the following error:

Missing type annotation for E. E is a type parameter declared in function [1] and was implicitly instantiated at call of
method bind

The error for deMapping is a bit different:

Missing type annotation for `K`. `K` is a type parameter declared in function [1] and was implicitly instantiated at
call of method `bind` [2].
...
Missing type annotation for `V`. `V` is a type parameter declared in function [1] and was implicitly instantiated at
call of method `bind` [2].

But has the same root cause. There are a couple of workarounds, one is manually editing the generated file after the fact and add typing so flow is happy (not ideal, since the manual edits will disappear when the file is regenerated). The second workaround is to wrap the list/map in an object like so:

type Container = {
   mapping: { [string]: string },
}

export const degenContainer = () => degenObject(
  { name: 'Container', typeParams: [] },
  [degenField('mapping', degenMapping(degenString(), degenString()))],
  [],
)

Which may not be possible, depending on the source of the data the type is modeling.

Union UnreachableFix and ExhaustiveUnionFix types are unused

When using degenSum (I think this is the culprit) two types are added to the generated file UnreachableFix / ExhaustiveUnionFix. These types are referenced in comments, but not in the code. On projects that use a linter to ensure there are no unused declarations, these types will cause an error.

I'm assuming the bug is really that these types /should/ be used somewhere but aren't.

Flow annotation could be strict

The flow annotation added to the generated file could be strict, to benefit projects that aim for the tightest flow coverage possible.