isabella232 / tap-duedil

tap-duedil

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

• Pulls raw data from DueDil
• Supports querying for companies that match given criteria
• Fetches data from the following endpoints:
  • company vitals
  • company industries
  • compay addresses
  • company descriptions
  • keywords
  • telephone numbers
  • websites
  • related names
  • company officers
  • social media
  • shareholders
  • company group parents
  • subsidiaries
  • portfolio
  • gazette notices
  • related companies
  • FCA authorisation
  • company filings
  • company charges
  • person of significant controls
  • company financials
• Outputs a schema for each resource

Quick Start

1. Create and source a virtualenv

$ virtualenv env
$ source env/bin/activate

2. Install from source

$ pip install .

3. Create a config file

You must create a JSON configuration file that looks like this:

{
    "api_key": "..."
}

You can find your API Key in the DueDil web interface.

4. Run the Tap in Discovery Mode

A script has been provided to generate a catalog.json file from the JSON Schemas provided for this tap. Run:

$ python get_catalog.py > catalog.json

to create a catalog file.

5. Run the tap in Query mode

First, create a JSON file containing your company search criteria.

# company-query.json
{
    "criteria": {
        "countryCodes": {
            "values": [
                "GB"
            ]
        }
    }
}

Next, run the tap in query mode:

$ tap-duedil query -q company-query.json -c config.json -p catalog.json --companies companies.txt

This command will create a text file called companies.txt, containing one company per line. When the tap is run in "sync" mode, the tap will use this file to sync each endpoint for each company.

6. Run the tap in Sync mode

$ tap-duedil sync -c config.json -p catalog.json --companies companies.txt

The output of tap-duedil sync .... can be piped to target-stitch to load the data into your warehouse.

A subset of streams can be selected using the --streams argument to tap-duedil sync. For more information on these parameters, run tap-duedil sync -h

---

Copyright © 2017 Fishtown Analytics