nickleomartin / tap-persistiq

A Singer.io tap for extracting data from the PersistIQ API

tap-persistiq

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

• Pulls raw data from the PersistIQ v1. 0 API
• Extracts the following resources:
  • Users
  • Leads
  • Campaigns
• Outputs the schema for each resource
• Incrementally pulls data based on the input state

Streams

users

• Endpoint: https://api.persistiq.com/v1/users
• Primary key fields: id
• Foreign key fields: none
• Replication strategy: FULL_TABLE
• Transformations: none

leads

• Endpoint: https://api.persistiq.com/v1/leads
• Primary key fields: id
• Foreign key fields: none
• Replication strategy: FULL_TABLE
• Transformations: none

campaigns

• Endpoint: https://api.persistiq.com/v1/campaigns
• Primary key fields: id
• Foreign key fields: lead_id > id, owner_id > id
• Replication strategy: FULL_TABLE
• Transformations: none

Authentication

Quick Start

1. Install

Clone this repository, and then install using setup. py. We recommend using a virtualenv:

    > virtualenv -p python3 venv
    > source venv/bin/activate
    > python setup.py install
    OR
    > cd .../tap-persistiq
    > pip install .

or alternatively:

    > virtualenv -p python3 venv
    > source venv/bin/activate
    > pip install git+https://github.com/NickLeoMartin/tap-persistiq#egg=tap-persistiq

2. Dependent libraries

The following dependent libraries were installed.

    > pip install singer-python
    > pip install singer-tools
    > pip install target-stitch
    > pip install target-json

• singer-tools
• target-stitch

3. Create your tap's config.json file.

    {
        "api_key": "YOUR_PERSISTIQ_API_KEY",
        "start_date": "2019-01-01T00:00:00Z",
        "user_agent": "tap-persistiq <api_user_email@your_company.com>"
    }

Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

    {
        "currently_syncing": "leads",
        "bookmarks": {
            "users": "2019-09-27T22:34:39.000000Z",
            "companies": "2019-09-28T18:23:53Z"
        }
    }

4. Run the Tap in Discovery Mode

This creates a catalog. json for selecting objects/fields to integrate:

    tap-persistiq --config config.json --discover > catalog.json

See the Singer docs on discovery mode here.

5. Run the Tap in Sync Mode (with catalog) and write out to state file

For Sync mode:

    > tap-persistiq --config tap_config.json --catalog catalog.json > state.json
    > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

To load to json files to verify outputs:

    > tap-persistiq --config tap_config.json --catalog catalog.json | target-json > state.json
    > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

To pseudo-load to Stitch Import API with dry run:

    > tap-persistiq --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
    > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json