nvie / decoders

Elegant validation library for type-safe input data (for TypeScript and Flow)

Elegant and battle-tested validation library for type-safe input data for TypeScript.

Introduction

Data entering your application from the outside world should not be trusted without validation and often is of the any type, effectively disabling your type checker around input values. It's an industry good practice to validate your expectations right at your program's boundaries. This has two benefits: (1) your inputs are getting validated, and (2) you can now statically know for sure the shape of the incoming data. Decoders help solve both of these problems at once.

Basic example

import { array, iso8601, number, object, optional, string } from 'decoders';

// Incoming data at runtime
const externalData = {
  id: 123,
  name: 'Alison Roberts',
  createdAt: '1994-01-11T12:26:37.024Z',
  tags: ['foo', 'bar'],
};

// Write the decoder (= what you expect the data to look like)
const userDecoder = object({
  id: number,
  name: string,
  createdAt: optional(iso8601),
  tags: array(string),
});

// Call .verify() on the incoming data
const user = userDecoder.verify(externalData);
//    ^^^^
//    TypeScript automatically infers this type as:
//    {
//      id: number;
//      name: string;
//      createdAt?: Date;
//      tags: string[];
//    }

Installation

npm install decoders

Requirements

You must set strict: true in your tsconfig.json in order for type inference to work correctly!

// tsconfig.json
{
  "compilerOptions": {
    "strict": true
  }
}

Documentation

Documentation can be found on https://decoders.cc.

Building your own decoders

There is a dedicated page in the docs that explains how to build your own decoders — it’s fun!

decoders's Issues

URL decoder does not accept URLs with / in the query part

Description of the bug
As subject says, D.url will reject URLs that have forward slashes in the query part.

Expected behavior
Those URLs are valid and should be accepted. See https://datatracker.ietf.org/doc/html/rfc3986#section-3.4 where / is explicitly called out as a valid character in query.

To Reproduce

const D = require('decoders')
D.url.verify("https://example.com/foo?q=foo/bar")

Additional context
Decoders version: 2.0.3
TypeScript version: 5.1.3
Node version: 18.14.2

Migration help needed: .map to .transform

Please confirm:

• I have read the migration instructions

What is the problem?

• I found a bug in the migration instructions
• I ran into an issue that the migration instructions don't give clear guidance on
• Something else

Description
I had a map which was converting a string to a luxon DateTime.

import { Decoder, map, string } from 'decoders'
import { DateTime } from 'luxon'
export const transformDateTime: Decoder<DateTime> = map(string, (isoDate) => {
    const dt = DateTime.fromISO(isoDate)
    if (dt.invalidExplanation) {
      throw `Unable to parse date ${isoDate}: ${dt.invalidExplanation}`
    }
    return dt
  }
)

I use this whenever I use DateTime - e.g.:

const decodeFoo = object({
  id: string,
  expires: transformDateTime,
})

I migrated it like this (simple!):

import { Decoder, string } from 'decoders'
import { DateTime } from 'luxon'
export const transformDateTime: Decoder<DateTime> = string.transform(
  (isoDate) => {
    const dt = DateTime.fromISO(isoDate)
    if (dt.invalidExplanation) {
      throw `Unable to parse date ${isoDate}: ${dt.invalidExplanation}`
    }
    return dt
  }
)

Now my page fails to load with an error:

Uncaught TypeError: string.transform is not a function

I've looked at the generated code of the private package and it still looks correct to me. I get this (the .d.ts):

import { Decoder } from 'decoders';
import { DateTime } from 'luxon';
export declare const transformDateTime: Decoder<DateTime>;

and this (the .js):

import { string } from 'decoders';
import { DateTime } from 'luxon';
export const transformDateTime = string.transform((isoDate) => {
    const dt = DateTime.fromISO(isoDate);
    if (dt.invalidExplanation) {
        throw `Unable to parse date ${isoDate}: ${dt.invalidExplanation}`;
    }
    return dt;
});

Possibly useful extra info:

• this is in a private package which I include in my app
• transformDateTime is only used internally in the private package (none of the decoders are exported from the package)
• it's quite possible I'm misunderstanding something - I'm a C++ guy, not a JS/TS expert :-)

Any suggestions? Thank you!

How to decode discriminated union types with nested json?

Hi again 😅,

in my front-end TypeScript code I have the following discriminated union (DU) type:

type FooA = {
    duKind: "FooA",
    myPropA: string
}

type FooB = {
    duKind: "FooB",
    myPropB: number
}

type Foo = FooA | FooB

My back-end REST API returns either

{
    "foo" : {
        "fooA": {
            "myPropA": "Hello World"
        }
    }
}

or

{
    "foo" : {
        "fooB": {
            "myPropB": 42
        }
    }
}

depending on the DU case.

How can I map this to my DU types using decoders?

const fooADecoder: Decoder<FooA> = ???;
const fooBDecoder: Decoder<FooB> = ???;
const fooDecoder: Decoder<Foo> = either2<Foo>(fooADecoder, fooBDecoder);

I guess for the duKind I can use the constant decoder, but the main problem is that I have to reduce the "object nesting" from { "foo": { "fooA": { "myPropA": ...} } } in JSON to { myPropA: ... } in TypeScript.

This problem probably is not limited to DUs, but a general question of how to reduce/compress/squeeze JSON object nesting levels within a decoder.

Any help would be much appreciated, since this is a show stopper for me right now.

Avoiding passwords in error messages

Imagine a view where you verify input, like a user setting a new password:

const bodyGuard = guard(object({
  oldPassword: string,
  newPassword: string,
}));
const body = bodyGuard(req.body);

And then a user does not send in a new password, so the guard throws an error which contains the following error message:

{
  oldPassword: 'ohNo3s-thisSldntBeXposed!',
  newPassword: undefined,
               ^^^^^^^^^ Must be string
}

Normally, these values provide great debugging context, but in some cases you may want to omit sensitive data from those places. Some ideas:

1. Solve this in debrief by simply never outputting values for keys recognized as "typically sensitive", like password, passwd, secretKey, etc. This is pretty rigid.
2. Allow people to annotate which places should be considered sensitive. For example, by making a special decoder for it, like password instead of string. Or sensitive(string) for a slightly more composable touch.
3. Solve it at the guard level, i.e. use guard(decoder, { echo: false }) or something like that.

Design Approach

Currently, debrief only collects values and then annotates a string message to it. As of right now, it does not have a way to transform the values it serializes, but adding support for that in should be possible.

Improve error message serialization

This case throws a horrendously looking error (all useful information is invisible):

const g = guard(
    either(
        object({ foo: string }),
        object({ bar: number }),
    )
);
g({
    // Number + string deliberately flipped, so both branches throw a field error
    foo: 123,
    bar: 'not a number',
})

This outputs:

Decoding error:
{
  "foo": 123,
  "bar": "not a number",
}
^
Either:
- 
- 

Note the missing error info in the Either: section.

See also https://github.com/nvie/decoders/tree/improve-either-errs for a demonstration.

Composing `lazy` and `either` can become slow.

Description of the bug

Using a combination of lazy and either, decoders can become very slow.

Expected behavior

The ~13 second runtime mentioned below would be a few milliseconds.

This may not be an issue. While writing the first iteration of this code I imagined it would be slow--but this does feel needlessly slow.

In the below example each AST type has a constant type field. I decreased the runtime of the below code by using the commented out "fast ast decoder." The fast version of this decoder first checks for the constant type field, then will choose the appropriate decoder to use (formerly returned in lazy calls) within a .transform. Given these type fields are constant I think decoders could implement an optimization to handle this specific case.

To Reproduce

// test.js
const value = {
  "type": "label",
  "tok": "eq",
  "body": {
    "type": "lambda",
    "params": {
      "type": "ident",
      "tok": "x",
      "sibling": {
        "type": "ident",
        "tok": "y"
      }
    },
    "body": {
      "type": "eq",
      "left": {
        "type": "ident",
        "tok": "x"
      },
      "right": {
        "type": "ident",
        "tok": "y"
      }
    }
  },
  "sibling": {
    "type": "label",
    "tok": "not",
    "body": {
      "type": "lambda",
      "params": {
        "type": "ident",
        "tok": "status"
      },
      "body": {
        "type": "cond",
        "body": {
          "type": "condset",
          "left": {
            "type": "ident",
            "tok": "status"
          },
          "right": {
            "type": "false"
          },
          "sibling": {
            "type": "condset",
            "left": {
              "type": "true"
            },
            "right": {
              "type": "true"
            }
          }
        }
      }
    },
    "sibling": {
      "type": "list",
      "body": {
        "type": "ident",
        "tok": "not",
        "sibling": {
          "type": "list",
          "body": {
            "type": "ident",
            "tok": "eq",
            "sibling": {
              "type": "strlit",
              "tok": "this was a",
              "sibling": {
                "type": "strlit",
                "tok": "triumph"
              }
            }
          }
        }
      }
    }
  }
}

const decoders = require('decoders');

const typeDecoder = decoders.either(
    decoders.constant("cdr"),
    decoders.constant("cons"),
    decoders.constant("cond"),
    decoders.constant("condset"),
    decoders.constant("lambda"),
    decoders.constant("ident"),
    decoders.constant("label"),
    decoders.constant("list"),
    decoders.constant("atom"),
    decoders.constant("quote"),
    decoders.constant("eq"),
    decoders.constant("strlit"),
    decoders.constant("numlit"),
    decoders.constant("true"),
    decoders.constant("false"),
    decoders.constant("car"),
);

// fast ast decoder
// const astDecoder = decoders.inexact({type: typeDecoder}).transform(value => {
//     switch (value.type) {
//         case "cdr": return cdrDecoder.verify(value);
//         case "cons": return consDecoder.verify(value);
//         case "cond": return condDecoder.verify(value);
//         case "condset": return condSetDecoder.verify(value);
//         case "lambda": return lambdaDecoder.verify(value);
//         case "ident": return identDecoder.verify(value);
//         case "label": return labelDecoder.verify(value);
//         case "list": return listDecoder.verify(value);
//         case "atom": return atomDecoder.verify(value);
//         case "quote": return quoteDecoder.verify(value);
//         case "eq": return eqDecoder.verify(value);
//         case "strlit": return StrlitDecoder.verify(value);
//         case "numlit": return NumlitDecoder.verify(value);
//         case "true": return TDecoder.verify(value);
//         case "false": return NilDecoder.verify(value);
//         case "car": return carDecoder.verify(value);
//     }
// })

// slow ast decoder
const astDecoder = decoders.either(
    decoders.lazy(() => TDecoder),
    decoders.lazy(() => NilDecoder),
    decoders.lazy(() => listDecoder),
    decoders.lazy(() => identDecoder),
    decoders.lazy(() => atomDecoder),
    decoders.lazy(() => quoteDecoder),
    decoders.lazy(() => eqDecoder),
    decoders.lazy(() => StrlitDecoder),
    decoders.lazy(() => NumlitDecoder),
    decoders.lazy(() => carDecoder),
    decoders.lazy(() => cdrDecoder),
    decoders.lazy(() => consDecoder),
    decoders.lazy(() => condDecoder),
    decoders.lazy(() => lambdaDecoder),
    decoders.lazy(() => labelDecoder),
)

const listDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('list'),
    body: astDecoder,
});

const identDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('ident'),
    tok: decoders.string,
})

const atomDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('atom'),
    body: astDecoder,
})

const quoteDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('quote'),
    body: astDecoder,
})

const eqDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('eq'),
    left: astDecoder,
    right: astDecoder,
})

const StrlitDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('strlit'),
    tok: decoders.string,
})

const NumlitDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('numlit'),
    tok: decoders.string,
})

const TDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('true'),
})

const NilDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('false'),
})

const carDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('car'),
    body: astDecoder,
})

const cdrDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('cdr'),
    body: astDecoder,
})

const consDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('cons'),
    left: astDecoder,
    right: astDecoder,
})

const condSetDecoder = decoders.object({
    sibling: decoders.optional(decoders.lazy(() => condSetDecoder)),
    type: decoders.constant('condset'),
    left: astDecoder,
    right: astDecoder,
})

const condDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('cond'),
    body: condSetDecoder,
})

const lambdaArgDecoder = decoders.object({
    sibling: decoders.optional(decoders.lazy(() => lambdaArgDecoder)),
    type: decoders.constant('ident'),
    tok: decoders.string,
})

const lambdaDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('lambda'),
    params: decoders.optional(lambdaArgDecoder),
    body: astDecoder,
})

const labelDecoder = decoders.object({
    sibling: decoders.optional(astDecoder),
    type: decoders.constant('label'),
    tok: decoders.string,
    body: lambdaDecoder,
})

const start = new Date().getTime();
astDecoder.verify(value);
const end = new Date().getTime();
console.log(`run time: ${(end - start) / 1000} seconds`)

$ node test.js

The fast version of this decoder commonly runs in 0.002 seconds while the slow verson runs in ~13 seconds on my machine.

Additional context
Decoders version: 2.0.0
TypeScript version: N/A
Flow version: N/A
Node version: v16.14.0

Type is not inferred correctly from a dynamically created decoder with generics

A bit cryptic description, I know, but maybe the following example clarifies the issue:

import { array, Decoder, object } from "decoders"

type GenericArrayContainer<F> = {
    value: F[]
}

// inferred type matches declared type
const genericArrayContainer = <F>(decoder: Decoder<F>): Decoder<GenericArrayContainer<F>> => object({
    value: array(decoder),
})

type GenericValueContainer<F> = {
    value: F
}

// inferred type does not match declared type
const genericValueContainer = <F>(decoder: Decoder<F>): Decoder<GenericValueContainer<F>> => object({
    value: decoder,
})

as commented, in the first case the type is inferred correctly when the generic value is wrapped in another decoder, but if the field is declared with the dynamically passed decoder directly, type inference no longer works.

This was tried with:

    "decoders": "^2.0.1",
    "typescript": "^4.9.4"

Let me know if there is another approach to achieving what I'm trying to do here or if any further info is required.

Export Annotation and Formatting function from package

Description of the bug

When I use the .decode method, I end up with errors of type Annotation, but none of the utility functions to do anything with that type are exported. My error handling library only accepts instances of type Error. The non-exported format function handles converting and formatting an Annotation into an Error for verify, but it is not exported/accessible.

I must use .decode for other behaviors, but would still like to be able to use the error as an Error, not an Annotation in places.

Expected behavior

Be able to import formatInline and other custom formatting functions/utils to process the annotation before handling/logging it.

To Reproduce

Try to import formatInline from package.

Additional context

Decoders version: 2.0

TypeScript optional keys for DecoderType inference

If we create a Decoder called dFoo:

import * as d from 'decoders';

const dFoo = d.object({
  id: d.string,
  name: d.optional(d.string)
});

Then infer the type from it via:

type Foo = d.$DecoderType<typeof dFoo>;

The inferred type is:

type Foo = {
   id: string;
   name: string | undefined;
}

That means when we use Foo we need to set a name key equal to undefined, instead of just omitting it:

const someFoo: Foo = {
  id: '42',
  name: undefined
};

Is it possible to have the inferred type use ? so we can just omit the optional key:

type Foo = {
    id: string;
    name?: string;
}

Thanks

Decoder error with either(number, generalDecoder)

Dispatch does not work when the field differentiating which decoder to use has a non-string value

I have a set of 30 possible objects, all of which have different type properties. Their additional properties also differ based off of which type they have. For example,

const certificateSignerExample = { type: 1, signer: 'BillyGates' } 
// ...
const antimalwarePathExample = { type: 30, path: 'lol/not/a/path' } 

To decode these objects, I needed to use dispatch, since (quickly does some math) 30 is more than 9, which means I cannot use one of the friendly either composite decoders.

However, in my case, I could not use the type field, since the object of decoders that we supply to dispatch is... well, a javascript object :), i.e., all of the keys are inevitably strings.

const certSignerMatcher = object({ id: constant(1 as 1), signer: string })
// ...
const antimalwarePathMatcher = object({ id: constant(30 as 30), path: string })

const matcherDecoder: Decoder<PolicyMatcherDetails> =  dispatch(
    'type',
    {
      1: certSignerMatcher,
     // ...
      30: antimalwarePathMatcher
    }
  )

I was able to work around this, of course, thanks to the very friendly map and compose higher-order decoders (thanks for those, by the way).

My solution was to:

1. Use map on the input objects to add an additional __type field (which is just the stringified version of the object's type)
2. compose said map decoder with a Dispatch decoder that makes use of the __type field instead

const addStringifiedTypeDecoder = map(
  jsonObject,
  o => ({ ...o, __type: o?.type?.toString() })
)

const mapHackyStringTypeToDecoder = dispatch(
    '__type',
    {
      1: certSignerMatcher,
      // ...
      30: antimalwarePathMatcher
    }
  )

const matcherDecoder: Decoder<PolicyMatcherDetails> = compose(
  addStringifiedTypeDecoder,
  mapHackyStringTypeToDecoder
)

This solution works fine for me, but it made me think that the dispatch composite decoder could possibly be rewritten to be a tiiiiiny bit more flexible.

One thought I had is that there could be an optional transformation of the value located at field (in my case, type), which would convert it to a stringified key:

const matcherDecoder: Decoder<PolicyMatcherDetails> =  dispatch(
    'type',
    {
      1: certSignerMatcher,
     // ...
      30: antimalwarePathMatcher
    },
    (valueOfTypeField: number): string => `${valueOfTypeField}`
  )

At any rate, I'm curious to hear what others think!

Thoughts on adding an equivalent of Flow's $PropertyType to give access to nested properties of a decoder?

In Flow, if you have a type User:

   type User = {|
      name: string, 
      age: number, 
      address: {
         street: string, 
         city: string, 
         state: string, 
         zip: number, 
      }
   |}

You can grab the address type by using $PropertyType utility type:

   type Address = $PropertyType<User, 'address'>

I have found this to be extremely useful in the case of using flow types that are generated from graphql queries, because it allows for better consistency in types throughout the code. My proposal is to add some similar functionality for decoders. If you had a similar decoder for that same user, looking like this:

   import { object, number, string } from 'decoders';  

   const userDecoder = object({
      name: string, 
      age: number, 
      address: object({
         street: string, 
         city: string, 
         state: string, 
         zip: number, 
      })
   }); 

That you would be able to access the address in this way:

   import { propertyType } from 'decoders';  
   ...
   const addressDecoder = propertyType(userDecoder, 'address'); 

It would also be ideal to be able to access these even when nested within an array, similar to $ElementType.

I would find this extremely useful with generated decoders, for the same reason that it is useful with flow types. I would love to get your thoughts on this (or maybe it is already possible!)

Support for Enums

Let's say there is an enum like this:

enum Alphabet {
  a = "a",
  b = "b"
}

A possible decoder would be:

const decoder = either2(constant("a"), constant("b"))

This doesn't work for enums with more than 9 cases atm. What do you think of introducing another method that checks input agains an array of values with the same type?

Something like oneOf(values: [T])

Union type is not inferred correctly from either decoder

Either decoders signature does not seem to infer types correctly from the input decoders:

import { object, string, either } from "decoders";

const UnionDecoder = either(
    object({
        /* dependent fields */
        dependency: string,
        dependant: string,
        /* common fields */
        common: string
    }),
    object({
        common: string
    })
)

type UnionDecoderType = DecoderType<typeof UnionDecoder>
/**
 * { common: string; }
 */

 /* this is not assignable */
 const decoderTypeValue: UnionDecoderType = {
     dependency: "this does not compile",
     dependant: "value",
     common: "hello"
}

the use case here is that some of the fields have to be defined together, so they cannot be simply optional

I'm not sure what exactly is wrong with the type signature, but I've had a similar pattern in a personal project and found that the following signature yields correct type:

import { Decoder, DecoderType, object, string } from "decoders";

const either = <A extends Array<Decoder<DecoderType<A[number]>>>>(
    ...arr: A
): Decoder<DecoderType<A[number]>> => {
    throw Error("implementation")
}

const UnionDecoder = either(
    object({
        /* dependent fields */
        dependency: string,
        dependant: string,
        /* common fields */
        common: string
    }),
    object({
        common: string
    })
)

type UnionDecoderType = DecoderType<typeof UnionDecoder>
/*
    {
        dependency: string;
        dependant: string;
        common: string;
    } | {
        common: string;
    }
 */

/* now this is assignable */
const decoderTypeValue: UnionDecoderType = {
    dependency: "this does compile",
    dependant: "value",
    common: "hello"
}

/* and this is not assignable */
const decoderTypeValue2: UnionDecoderType = {
    dependency: "this does not compile",
    dependant: "value"
}

the signature can of course be cleaned up and extracted to utility types for readability, but that's the gist of it.

Tested with versions:

    "decoders": "^2.0.1",
    "typescript": "^4.9.4"

Decoder for object with specified key

Can I create a Decoder such as this one

const withKey = <K extends string, T>(k: K, d: Decoder<T>) => object({[k]: d})

so that the resulting type is exact? For example Decoder<{a: number}> for withKey("a", number)?

Currently this results in something like Decoder<{[x: string]: number}> (simplified).

---

Perhaps related, is it possible to take a Decoder<{...}> and "merge" a property into it / "extend" it?

zod equivalent: https://github.com/colinhacks/zod#extending-objects

Ability to filter out incorrect items during array decoding

I am currently using your library for type-safe decoding of API responses, which sometimes contain arrays of items. In its current implementation, the D.array decoder will throw an error if any of the items in the array fail to decode against the provided schema, like so:

export const item: D.Decoder<Item> = D.object({
  id: D.uuidv4,
  title: D.string,
});

export const paginatedItems: D.Decoder<PaginatedResponse<Item>> = D.object({
  results: D.array(item),
  next_page: D.optional(D.nullable(D.string)),
});

I'd like to propose a feature where the array decoder has a "resilient" or "filter" mode that continues decoding despite individual errors, collects these errors separately, and omits the failing items from the resulting array. This could provide a more user-friendly, error-resilient approach while still being within the bounds of the library's design philosophy.

Self-referential objects break decoders

If an object is passed in that refers to itself (like the express req object), an infinite loop occurs.

Smallest reproducible example:

const { string } = require('decoders');

// Take any decoder and pass in some self-referential object
const req = {};
req.myself = req;

console.log(string(req)); // <--- RangeError: Maximum call stack size exceeded

Extracting types from a Decoder<T>

You likely already know this, but I felt pretty chuffed when I got it working, and figured others might benefit from it!

https://gist.github.com/girvo/b4207d4fc92f6b336813d1404309baab

This demonstrates how one can extract the T from a given Decoder<T> into a real type definition, for use elsewhere in a code-base. Prior to this, I was duplicating it: writing a type Example = ... then writing a const example: Guard<Example> = guard(object(... with the run-time versions duplicated.

This lets one define them once for run-time, and re-use them at type-time!

About the only "gripe" I have, is that I'd love to be able to make the type exact.

// @flow
type Thing = {|
  name: string
|}

But I don't know whether this is possible through inference yet?

Perhaps by using the $Exact<T> utility type this can be achieved: https://flow.org/en/docs/types/utilities/#toc-exact

Unwrap Error Message from Guard

I don't want to use a guard because I don't want to deal with a bunch of try catch blocks but I do want to know when it has failed so I can log the issue and respond with an HTTP message. This write-up got me down the path of calling the decoder directly, rather than always wrapping a decoder in a guard.

I'm trying to do decoder(VAL).map(() => {}).mapError((e) => {}) and execute logic when I know VAL has been decoded and typed or do something with e in mapError. I've examined the code and the docs in lemons.js and it's not clear to me how to get the nice err.message that using a guard gives you. I see the inbound argument to mapError is an Annotation; how can I extract the message from an Annotation?

I've tried e.message and e.errValue() and neither of them worked. I am using version 1.25 of decoders.

Required types

Firstly, apologies if this isn't an actual issue and is instead related to my naivety as to how to properly use the library.

Secondly, this is a great library.

My issue is relating to determining that a type is required, and not ever undefined, as far as the Typescript compiler is concerned.

If I have some code:

const testDecoder = object({ success: boolean });

TS determines the type to be:

Decoder<{
    success?: boolean;
}, unknown>

My question is, how can I have typescript infer that the key success is not optional?

upgrading to TS4

The following code works fine on TS 3.4.5:

type Example = {
    eg: string;
};

const exampleDecoder: Decoder<Example> = object({
    eg: string,
});

In TS 4.0.3 however, I got this message:

Type 'Decoder<{ eg?: string; }, unknown>' is not assignable to type 'Decoder<Example, unknown>'.
  Type '{ eg?: string; }' is not assignable to type 'Example'.
    Property 'eg' is optional in type '{ eg?: string; }' but required in type 'Example'.ts(2322)

Any ideas?

Is the documentation for dispatch incorrect?

Hello 👋

I'd like to check if the documentation for dispatch is correct because I'm running into a problem with it at the moment!

import {object, string, constant, dispatch, guard } from 'decoders'

type SandboxEnviroment = {
  status: "sandbox";
  name: string;
}

type LiveEnvironment = {
  status: "live";
  name: string;
}

type Environment = SandboxEnviroment | LiveEnvironment

const decodeSandbox = object({
  name: string,
  status: constant('sandbox')
})

const decodeLive = object({
  name: string,
  status: constant('live')
})

const decoder = dispatch('status', {
  live: decodeLive,
  sandbox: decodeSandbox
})

const run = (): Environment =>
  guard(decoder)({
    status: 'live',
    name: 'my-app'
  })

The error I get (in run) is the following:

TS2322: Type '$Values<{ live: { status: string; name: string; }; sandbox: { status: string; name: string; }; }>' is not assignable to type 'Environment'.
Type '{ status: string; name: string; }' is not assignable to type 'Environment'.
Type '{ status: string; name: string; }' is not assignable to type 'LiveEnvironment'.
Types of property 'status' are incompatible.
Type 'string' is not assignable to type '"live"'

However the documentation for dispatch indicates that the inner type of the decoder will be a union! Is this a bug or an error in the documentation?

Of course, I can leave off the annotations in my run function to get this compiling so it's not a blocker. Thought I'd give you a heads up about it though! 🙂

Mixed parameter for the decoders

Hi,

We wanted to publish a similar library at OuiCar based on our old blog post, but your library is much more mature. I especially like the way you handle the typing of the object primitive. Good to see, thanks!

One remark: the parameter of a decoder is of type any:

export type anything = any;

export type Guard<T> = anything => T;
export type Predicate<T> = T => boolean;
export type DecodeResult<T> = Result<Annotation, T>;
export type Decoder<T, F = anything> = F => DecodeResult<T>;

Could you use a type mixed instead? This would force Flow to check that the casting are correct.

Package requires nodejs 10 or newer, but this is not documented.

Description of the bug
The global URL was introduced in NodeJS in version 10, and is referred to in the url decoder.
Trying to use this package on an earlier version of NodeJS (e.g. 8.17.0) results in an error when importing the module.

Expected behavior
I expect NPM to prevent me from installing (version 2?) of this package, due to having {engines: { node: '>=10'}} in package.json

To Reproduce
index.js

require('decoders');

Command line:

node index.js

Additional context
Decoders version: 2.0.1
TypeScript version: 4.5.5
Node version: 8.17.0

I know Node10 is a relatively ancient version, so its reasonable to expect its features to exist, but it would be really appreciated if the minimum expected engine version was specified, ideally in both the README/documentation and in the package.json .

How to handle differences between property names

Here's my problem. The data comes from the backend with the compound names of the properties with underline, but in the frontend I want them to be camelCase.

How the data comes from backend:
{ id: 'some string', status: 'some string', token: 'some string', due_date: 'some string', bank_slip_url: 'some string', invoice_url: 'some string', created: 'some string' }

Here's my frontend implementation:
export class LicensePaymentModel { id?: string; status?: PaymentStatusEnum; token?: string; dueDate?: Date; bankSlipUrl?: string; invoiceUrl?: string; created?: Date; };

export const licenseDecoder: Decoder<LicensePaymentModel> = object({ id: string, status: string, token: string, due_date: date, bank_slip_url: string, invoice_url: string, created: date });

I realized that since I'm using Decoder in my decoder, the object must have the properties in camelCase, as in my LicensePaymentModel model. I was wondering if there is any way for me to associate the properties that comes from JSON underline with the model's camelCase.

Subtle gotcha with mapping decoder

Using rest spread with a javascript Map does not throw any runtime errors or flow errors, but it un-intuitively returns an empty object.

const someMapping = guard(mapping(string))({ foo: 'bar' });
console.log({ ...someMapping });
// -> logs {}

Can we consider change the output of a mapping to a object or add a new decoder that acts like a mapping decoder, but returns an object?

Flow Errors decoders/lib/string.js.flow

Running into flow errors, even when running the same version of flow as project.

Stevens-MacBook-Pro:labeltron stevefar$ flow --version
Flow, a static type checker for JavaScript, version 0.92.1
Stevens-MacBook-Pro:labeltron stevefar$ flow
Error ┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈ node_modules/decoders/lib/string.js.flow:61:5

Missing type annotation for V. V is a type parameter declared in function [1] and was implicitly instantiated at call of
compose [2].

     node_modules/decoders/lib/string.js.flow
     58│  * - url(['https', 'git+ssh'])  accepts both https:// and git+ssh:// URLs
     59│  */
     60│ export const url = (schemes: Array<string> = DEFAULT_SCHEMES) =>
 [2] 61│     compose(
     62│         string,
     63│         (value: string) => {
     64│             const matches = value.match(url_re);
     65│             if (!matches) {
     66│                 return Err(annotate(value, 'Must be URL'));
     67│             } else {
     68│                 const scheme = matches[1];
     69│                 if (schemes.length === 0 || schemes.includes(scheme.toLowerCase())) {
     70│                     return Ok(value);
     71│                 } else {
     72│                     return Err(annotate(value, `URL scheme must be any of: ${schemes.join(', ')}`));
     73│                 }
     74│             }
     75│         }
     76│     );
     77│

     node_modules/decoders/lib/utils.js.flow
 [1] 41│ export function compose<T, V>(decoder: Decoder<T>, next: Decoder<V, T>): Decoder<V> {



Found 1 error
Stevens-MacBook-Pro:labeltron stevefar$ 

Decoder with constants is throwing an exception. TypeError: (0 , _Result.Ok) is not a function

When trying to decode a string to one of 3 constants, am receiving an error TypeError: (0 , _Result.Ok) is not a function

Codepen example (open browser console to see error).

Snippet:

import { guard, either3, constant, Decoder } from "decoders";

const statusDecoder: Decoder<'complete' | 'closed' | 'failed'>
      = either3(constant<'complete'>('complete'), constant<'closed'>('closed'), constant<'failed'>('failed'));

const statusString = 'closed';

const status = guard(statusDecoder)(statusString);

console.log('Decoded:', status);

Typescript error with required properties

It seems like AllowImplicit is confusing Typescript into thinking that required properties are optional.

interface A {
  a: string;
}

const aDecoder: Decoder<A> = object({
  a: string
});

Output:

Type 'Decoder<{ a?: string; }, unknown>' is not assignable to type 'Decoder<A, unknown>'.
  Type '{ a?: string; }' is not assignable to type 'A'.
    Property 'a' is optional in type '{ a?: string; }' but required in type 'A'.ts(2322)

Live example: https://codesandbox.io/s/nostalgic-sinoussi-lkrq1?file=/src/index.ts

Decoder creates object property with undefined value

The actual behavior

const decoder = object({
    id: number,
    name: string,
    address: optional(string),
});
decoder({ id: 1, name: "test" }).unwrap();
// returns this
{ id: 1, name: "test", address: undefined };

Expected behavior

const decoder = object({
    id: number,
    name: string,
    address: optional(string),
});
decoder({ id: 1, name: "test" }).unwrap();
// returns this
{ id: 1, name: "test" };

Why it matters?

type Params = {|
  host?: string,
  port?: number,
|};
const decoder = exact({
  host: optional(string),
  port: optional(number),
});
const validate = guard(decoder);
const params = validate({});
const options = {
  host: "localhost",
  port: 3000,
  ...params,
};
console.log(options);
/*
{
  host: undefined,
  port: undefined
}
*/

Workaround

const decoder = exact({
  host: optional(string),
  port: optional(number),
});
const validate = guard(decoder);
const params = JSON.parse(JSON.stringify(validate({})));
console.log(params);
/*
{}
*/

How to avoid name clash with basic type keywords in TypeScript

Hi there,

I have only started with TypeScript and front-end development recently and I discovered the decoders package just now.
Quick question: What is the recommend way to import the basic decoders such as object, number, string, etc.? All these are built-in TypeScript keywords. Should I just rename the imports or is there any better technique?

Maybe this question could also become a feature request for export the basic decoders with some wrapper object called JsonDecoders which can then be imported and used like JsonDecoders.object, JsonDecoders.number, JsonDecoders.string, etc.
Just for convenience when using IDEs with auto-import, e.g. VSCode.

Typing for optional key does not allow for missing key

Hi, can you help me fix the below?
THANKS

This:

object({
  name: string,
  address: optional(string),
})

Should decode to type:

{
  name: string;
  address?: string;
}

But actually decodes to type:

{
  name: string;
  address: string | undefined;
}

Which according to my TypeScript (3.7.5) makes address a mandatory key that may have value undefined.

I extract the type like this, maybe not right?

export const propGuard = guard(
  object({
    name: string,
    address: optional(string),
  })
);
export type MyProps = ReturnType<typeof propGuard>;

// This doesn't compile:
const props: MyProps = {
  name: '123',
}

Runtime is fine and does allow for missing key.

Document optional/nullable in README

These are missing from the docs:

• optional
• nullable

[Bug] Builtin composite decoders cannot take decoders with non-`unknown` input types

Version

2.0.0-beta9

Steps to reproduce

1. Define a Decoder<O, I> where I is a type other than unknown
2. Attempt to pass the defined Decoder<O, I> to any of the builtin composite decoders

E.g.

import { Decoder, DecodeResult, optional } from 'decoders';
import { unknown } from 'decoders/annotate';
import { err, ok } from 'decoders/result';
import * as uuid from 'uuid';

const binaryUUIDFromStringUUID: Decoder<Uint8Array, string> = (uuidString: string): DecodeResult<Uint8Array> => {
  try {
    return ok(Uint8Array.from(uuid.parse(uuidString)));
  } catch (error) {
    return err(unknown(error));
  }
};

const optionalBinaryUUIDFromStringUUID = optional(binaryUUIDFromStringUUID);

Expected result

No type errors.

Actual result

TS2345: Argument of type 'Decoder<Uint8Array, string>' is not assignable to parameter of type 'Decoder<Uint8Array, unknown>'. 
  Type 'unknown' is not assignable to type 'string'.

Affected composite decoders/functions:

• guard
• compose
• predicate
• transform
• array
• nonEmptyArray
• set
• describe
• dict
• exact
• inexact
• mapping
• object
• either
• lazy
• maybe
• nullable
• optional
• taggedUnion
• tuple

Motivating use-cases

• The binary UUID example from above
• Decoding arrays with a known element type without having to re-assert the known type:

  const bigIntFromString: Decoder<bigint, string> = ...;
  const bigIntArrayFromStringArray: Decoder<bigint[], string[]> = array(bigIntFromString);

• Decoding objects with known value types without having to re-assert the known types:

  const booleanFromString: Decoder<boolean, string> = ...;
  const bigIntFromString: Decoder<bigint, string> = ...;
  const recordFromStringifiedValueRecord: Decoder<{ foo: boolean, bar: bigint }, { foo: string, bar: string }> =
    object({ foo: booleanFromString, bar: bigIntFromString });

• Piece-wise composition of decoders:

  const bigIntFromString: Decoder<bigint, string> = ...;
  const bigIntFromJSON: Decoder<bigint> = compose(compose(json, string), bigIntFromString);

Better support for non-empty string

First off, great library!

To guard against a non-empty string (in TypeScript) I have to do this now:

regex(/./, 'Field is required')

Q: Is there a cleaner way to do this?

I don't really like having to use a regexp for this - yes it is quite simple, but maybe there is (/can be invented) an even simpler/idiomatic way to do this. Empty strings are almost never useful in real-life JSON.

Note that the message is mandatory in the type definition of the regex guard, which is kinda annoying for my use, as I'm putting "Field is required" there (all my string fields use this regex guard, I never want empty strings in my JSON).

Maybe I'm missing something, interested to hear what you think.

Question: Are API-friendly error messages still available?

As per here: https://nvie.com/posts/introducing-decoders/#error-messages

Thanks for the library btw, it's great!

How to decode recursive data structures?

Hi there,

how would I decode recursive data structures, such as

type Folder = {
    name: string,
    subfolders: Folder[]
}

Elm has the lazy decoder for this:
https://package.elm-lang.org/packages/elm/json/latest/Json-Decode#lazy

How would I solve this using this decoders package?

nonEmptyString not exported in npm package

I see the export in the code on the repo but not in the package I pulled from npm. I believe all that's needed is a new release.

Flow is complaining "Cannot reassign constant"

Thanks for your work here! I love Elm Decoders/Encoders and I was missing it for Flow. I added this library in my project and flow is complaining about constants that are reassigned.

Error ┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈ node_modules/debrief/lib/serialize.js.flow:15:5

Cannot reassign constant s [1].

 [1]  6│ function serializeString(s: string, width: number = 80) {
       :
     12│     }
     13│
     14│     // Cut off a bit
     15│     s = s.substring(0, width - 15) + '...';
     16│     ser = JSON.stringify(s) + ' [truncated]';
     17│     return ser;
     18│ }


Error ┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈ node_modules/decoders/lib/either.js.flow:13:5

Cannot reassign constant s [1].

     10│  */
     11│ // istanbul ignore next
 [1] 12│ function itemize(s: string = ''): string {
     13│     s = indent(s);
     14│     return '-' + s.substring(1);
     15│ }
     16│



Found 2 errors

I am using Flow 0.72.0 in strict local mode.

Thoughts on a decoder "code generator" given a flow type.

One pain point is the decoder is more-or-less just the flowtype rewritten. The advantage of something like https://github.com/codemix/flow-runtime is that this is done automatically, but without the flexibility of making more complex runtime decoders and/or deciding where to apply the runtime check.

What are your thoughts about a tool that would generate decoders for all exported types? For example if you have a file ./src/myModule.js and you run $ decoder-codegen ./src/, it would create a new file ./src/__generated__/myModule.decoders.js. Then we can import the decoders into the source file.

myModule.js

import { guard } from 'decoders';
import { pointDecoder } from './__generated__/myModule.decoders';

const checkPointType = guard(pointDecoder);

export type Point = {
  x: number,
  y : number
}

export function moveUp(point: Point) {
  checkPointType(point);
  point.y = point.y - 1;
}

Inspired by https://github.com/apollographql/apollo-codegen#generate

Question: How to write your own decoders?

How could one write the following Elm decoder?

intFromStringDecoder : Decoder Int
intFromStringDecoder =
    Decode.string
        |> Decode.map
            (\str ->
               String.toInt str
                   |> Maybe.map Decode.succeed
                   |> Maybe.withDefault (Decode.fail "Not a number")
             )

Add Weakguard to decoders library

When adding decoders to an already functioning production environment, many times it is helpful to use a weakGuard temporarily. Rather than throwing an exception it would console.error the message (or maybe an option for a custom action?). This way the code can continue to function like it has, but the type can be refined as more variances come in. Eventually the weakGuard can be replaced with guard once one is confident the type is correct.

Decoding nested serialized JSON

I have a situation where I'm receiving a JSON event where one field contains a string containing serialized JSON. I would like to build a decoder that decodes the outer event as well as the nested serialized JSON string. Some pseudo-code to show what I mean:

const history = JSON.stringify({ employment: ["Captain of Costa Concordia", "Designer of the Vasa"] })
const event = {
  name: 'John Smith',
  age: 34,
  history
}

const decoder = object({
  name: nonEmptyString,
  age: positiveInteger,
  history: serializedJson(object({
    employment: array(string)
  }))
})

const parsed = decoder.verify(event)

console.log(parsed.history.employment[0]) // "Captain of Costa Concordia"

I went through the documentation on how to construct your own decoders, and I got it to work at runtime, but the resulting Typescript type is not quite right:

function serializedJson<T>(decoder: Decoder<T>): Decoder<T> {
    return nonEmptyString.transform(value => JSON.parse(value)).then(decoder.decode)
}

The result of this is that the event I defined above would complain about my history field being a string instead of an object, which makes sense since serializedJson<T> has a return type of Decoder<T>, but then I'm clearly missing how I can express that I expect the input type to be of type string but the return type from verify should be T.

Any guidance would be appreciated. 🙂

Is there a way to decode a field in source object into another field in target object

Having used Elm decoders, decoding a particular field or a combination of them into another field is a common use case for me.

An example could look something like this:

const decoder: Decoder<T> = object({ targetField: compose(field('sourceField'), string) })

So essentially it would be a 'mixed' decoder that takes its value from another field than its target field.

Is there currently a way to do this? If not, would this be possible to implement?

Variant of Dispatch

I would like to make a decoder similar to dispatch. Except instead of having a type field, I have a series of objects each with a single key:

type Foo =
  | { circle: { x: number; y: number; radius: number } }
  | { square: { x: number; y: number; size: number } }
  | { rectangle: { x: number; y: number; length: number; width: number } };

// Not sure how to get this line to correctly infer the type.
const fooDecoder: Decoder<Foo> = dispatch({
  circle: exact({ x: number, y: number, radius: number }),
  square: exact({ x: number, y: number, size: number }),
  rectangle: exact({ x: number, y: number, length: number, width: number }),
});

const verifyFoo = guard(fooDecoder);

verifyFoo({ circle: { radius: 5 } });
// ^^ Decoder error: Missing fields x and y

Composing these either9 get cumbersome and I would like easily digestible error messages. I am not sure how to formulate the types (typescript) so that inference works as expected.

Any Ideas or pointers? Much appreciated.

WeakGuard in 2.x

Please confirm:

• I have read the migration instructions

What is the problem?

• I found a bug in the migration instructions
• I ran into an issue that the migration instructions don't give clear guidance on
• Something else

Description
We use something like WeakGuard as the default. Is there some built-in mechanism that doesn't involve wrapping verify in a try/catch ?

See related issue.

// Paste your v1.x code here

Missing field in `inexact` doc

Description of the bug
Looks like the documentation of inexact decoder is missing field for y.
It is now

const decoder = inexact({
    x: number,
});

While should be

const decoder = inexact({
    x: number,
    y: number,
});

as in exact decoder above.

Or am I missing something?

Question: Type intersection composition

Is there an "intersection" composition available, such that

const a = object({a: number})
const b = object({b: boolean})
const c = ???(a, b)

would result in c being a Decoder<{a: number, b: boolean}>? Can this be achieved with some of the existing combinators?

io-ts equivalent: https://gcanti.github.io/io-ts/modules/index.ts.html#intersection

Is there a way to combine decoders?

For example:

• If I have 2 decoders that share a lot of values
• If I have a type like this:

export interface AnObject extends Record<string, string> {
  id: number;
  name: string;
}

I would need to find a way to combine

object({id: number, name: string})
//And
dict(string)

object and exact can't be called when exact_by_default=true

The error is:

Cannot call object with object literal bound to mapping because:
• property x is missing in object type [1] but exists in object literal [2].
• property y is missing in object type [1] but exists in object literal [2].

 [2] 18│ const pointDecoder = object({
     19│     x: number,
     20│     y: number,
     21│ });

     node_modules/decoders/object.js.flow
 [1] 78│ export function object<O: { +[field: string]: AnyDecoder }>(mapping: O): Decoder<$ObjMap<O, $DecoderType>> {