chrisgoddard / singer-discover

A simple utility to edit Singer-spec schemas in the command line.

Singer Discover Utility

Simply command line utility to take a Singer-specification JSON catalog file and select which streams and fields to include. Fields with "automatic" inclusion are included... er.. automatically.

Installation

pip install https://github.com/chrisgoddard/singer-discover/archive/master.zip

Use

Two ways of using this script.

First takes two filenames: input (--input, -i) and output (--output, -o). These can be the same

singer-discover --input catalog.json --output catalog.json

The second accepts the piped output of a tap in discover mode, making it a little more idiomatic with the rest of the Singer ecosystem.

tap-example --config config.json --discover | singer-discover -o catalog.json

NOTE: This uses a bit of a hack to allow the interactive interface to appear after the script has accepted input from stdin. It worked for me, but please let me know if you see errors. First method should work for all.

First select which streams to include

? Select Streams (<up>, <down> to move, <space> to select, <a> to toggle, <i> to invert)
> • stream_one
  ○ stream_two
  ○ stream_three

Hit enter. Then select the fields within each stream to include. Hit a to toggle select all. Automatically included fields are unselectable and annotated with (automatic)

? Select fields for stream: `stream_one` (<up>, <down> to move, <space> to select, <a> to toggle, <i> to invert)
> - id (automatic)
  • name
  ○ type
  ○ price

Hit enter, and the catalog file is updated with the field and stream selected properties updated.

singer-discover's Issues

stream not getting selected element

When running singer-discover (on tap-hubspot and on tap-jira) I find that the "selected": true does not get added at the top level of the stream.

If I select the stream "projects" from tap-jira I expect that the start of the json would be:

  "streams": [
    {
      "tap_stream_id": "projects",
      "key_properties": [
        "id"
      ],
      "schema": {
        "selected": true,
        "properties": {

What I do get is:

  "streams": [
    {
      "tap_stream_id": "projects",
      "key_properties": [
        "id"
      ],
      "schema": {
        "properties": {

so the streams.schema.selected doesn't exist.

The fields within each stream are getting set as selected as intended.

For me to get the two taps mentioned above to run I have to use the --properties argument rather than --catalog if that is a useful distinction in how singer-discover is determining where to add the "selected" element.

do not use hard version lock on singer-python

I'm having a small issue installing this tool alongside a tap with a newer version of the singer (5.8.1 to be exact).

Since this is really a tool that is used outside singer, would it make sense to use something like this to lock the version:

install_requires=[
        "singer-python>=5.4.1,<6.0",
        "PyInquirer==1.0.3"
    ],

I'll create a PR for the change, but wanted to have an issue to reference in it.