isabella232 / tap-outreach

tap-outreach

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

• Pulls raw data from Outreach API
• Outputs the schema for each resource
• Incrementally pulls data based on the input state

Streams

accounts

call_dispositions

call_purposes

calls

content_categories

duties

events

mailboxes

mailings

opportunities

personas

prospects

stages

tasks

teams

users

Quick Start

1. Install

   Clone this repository, and then install using setup.py. We recommend using a virtualenv:

   > virtualenv -p python3 venv
   > source venv/bin/activate
   > python setup.py install
   OR
   > cd .../tap-outreach
   > pip install .

2. Here is what the config should look like

   {
       "start_date": "2019-01-01T00:00:00Z",
       "client_id": <CLIENT_ID>,
       "client_secret": <CLIENT_SECRET>,
       "redirect_uri": <REDIRECT_URI>,
       "refresh_token": <REFRESH_TOKEN>,
       "quota_limit": <QUOTA_LIMIT>,
       "page_size": <PAGE_SIZE>
   }

   The following are required values: start_date, client_id, client_secret, redirect_uri, refresh_token The following are optional values: quota_limit, page_size

   Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

     {
       "bookmarks": {
         "stream_1": "2020-01-04T00:00:00.000000Z",
         "stream_2": 1576266090,
         ...
       }
     }

3. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

   tap-outreach --config config.json --discover > catalog.json

   See the Singer docs on discovery mode here.

4. Run the Tap in Sync Mode (with catalog) and write out to state file

   For Sync mode:

   > tap-outreach --config tap_config.json --catalog catalog.json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To load to json files to verify outputs:

   > tap-outreach --config tap_config.json --catalog catalog.json | target-json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To pseudo-load to Stitch Import API with dry run:

   > tap-outreach --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

5. Test the Tap

   While developing the Outreach tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:

   > pylint tap_outreach -d broad-except,chained-comparison,empty-docstring,fixme,invalid-name,line-too-long,missing-class-docstring,missing-function-docstring,missing-module-docstring,no-else-raise,no-else-return,too-few-public-methods,too-many-arguments,too-many-branches,too-many-lines,too-many-locals,ungrouped-imports,wrong-spelling-in-comment,wrong-spelling-in-docstring

   The pylint disables that Stitch uses are:

   broad-except,chained-comparison,empty-docstring,fixme,invalid-name,line-too-long,missing-class-docstring,missing-function-docstring,missing-module-docstring,no-else-raise,no-else-return,too-few-public-methods,too-many-arguments,too-many-branches,too-many-lines,too-many-locals,ungrouped-imports,wrong-spelling-in-comment,wrong-spelling-in-docstring
   

   To check the tap and verify working:

   > tap-outreach --config tap_config.json --catalog catalog.json | singer-check-tap > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   Check tap resulted in the following:

   The output is valid.
   It contained 8240 messages for 16 streams.
   
       16 schema messages
   8108 record messages
       116 state messages
   
   Details by stream:
   +-----------------------------+---------+---------+
   | stream                      | records | schemas |
   +-----------------------------+---------+---------+
   | **ENDPOINT_A**              | 23      | 1       |
   +-----------------------------+---------+---------+

---

Copyright © 2020 Stitch