isabella232 / tap-helpscout

tap-helpscout

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

• Pulls raw data from the HelpScout Mailbox API
• Extracts the following resources:
  • Conversations
    • Conversation Threads
  • Customers
  • Mailboxes
    • Mailbox Fields
    • Mailbox Folders
  • Users
  • Workflows
• Outputs the schema for each resource
• Incrementally pulls data based on the input state

Streams

conversations

• Endpoint: https://api.helpscout.net/v2/conversations
• Primary keys: id
• Foreign keys: mailbox_id (mailboxes), assignee > id (users), created_by > id (users), primary_customer > id (customers), custom_fields > id (mailbox_fields)
• Replication strategy: Incremental (query all, filter results)
  • Filter: status = all
  • Sort by: modifiedAt ascending
  • Bookmark: updated_at
• Transformations: Fields camelCase to snake_case
• Children: conversation_threads

conversation_threads

• Endpoint: https://api.helpscout.net/v2/conversations/{conversation_id}/threads
• Primary keys: id
• Foreign keys: conversation_id (conversations), customer > id (customers), created_by > id (users), assigned_to > id (users)
• Replication strategy: Full table (ALL for each parent Conversation)
  • Bookmark: None
• Transformations: Fields camelCase to snake_case. De-nest attachments array node. Add parent conversation_id field.
• Parent: conversations

customers

• Endpoint: https://api.helpscout.net/v2/customers
• Primary keys: id
• Foreign keys: None
• Replication strategy: Incremental (query filtered)
  • Bookmark query parameter: modifiedSince
  • Sort by: modifiedAt ascending
  • Bookmark: updated_at (date-time)
• Transformations: Fields camelCase to snake_case. De-nest the following nodes: address, chats, emails, phones, social_profiles, websites.

mailboxes

• Endpoint: https://api.helpscout.net/v2/mailboxes
• Primary keys: id
• Foreign keys: None
• Replication strategy: Incremental (query all, filter results)
  • Bookmark: updated_at (date-time)
• Transformations: Fields camelCase to snake_case.
• Children: mailbox_fields, mailbox_folders

mailbox_fields

• Endpoint: https://api.helpscout.net/v2/mailboxes/{mailbox_id}/fields
• Primary keys: id
• Foreign keys: mailbox_id (mailboxes)
• Replication strategy: Full table (ALL for each parent Mailbox)
  • Bookmark: None
• Transformations: Fields camelCase to snake_case. Add parent mailbox_id field.
• Parent: mailboxes

mailbox_folders

• Endpoint: https://api.helpscout.net/v2/mailboxes/{mailbox_id}/folders
• Primary keys: id
• Foreign keys: mailbox_id (mailboxes)
• Replication strategy: Incremental (query all, filter results)
  • Bookmark: updated_at (date-time)
• Transformations: Fields camelCase to snake_case. Add parent mailbox_id field.
• Parent: mailboxes

users

• Endpoint: https://api.helpscout.net/v2/users
• Primary keys: id
• Foreign keys: None
• Replication strategy: Incremental (query all, filter results)
  • Bookmark: updated_at (date-time)
• Transformations: Fields camelCase to snake_case.

workflows

• Endpoint: https://api.helpscout.net/v2/users
• Primary keys: id
• Foreign keys: mailbox_id (mailboxes)
• Replication strategy: Incremental (query all, filter results)
  • Bookmark: modified_at (date-time)
• Transformations: Fields camelCase to snake_case.

Authentication

Refresh Access Token The tap should provides a refresh_token, client_id and client_secret to get an access_token when the tap starts. If/when the access_token expires in the middle of a run, the tap gets a new access_token and refresh_token. The refresh_token expires every use and new one is generated and persisted in the tap config.json until the next authentication. To generate the necessary API keys: client_id and client_secret, follow these instructions to Create My App in your User Profile of the HelpScout web console application.

• App Name: tap-helpscout
• Redirect URL: https://app.stitchdata.test:8080/v2/integrations/platform.helpscout/callback Record your credentials (for the tap config.json):
• App ID: client_id
• App Secret: client_secret

Authentication URL: https://secure.helpscout.net/authentication/authorizeClientApplication?client_id=`YOUR_CLIENT_ID`&state=`YOUR_CLIENT_SECRET` Adjust the above URL by replacing YOUR_CLIENT_ID and YOUR_CLIENT_SECRET. In your web browser, disable any ad/popup blockers, and navigate to the Authorize URL. Click Authorize and you will be redirected to your Redirect URL (from above). Record the code from the redirected browser URL in the browser header.

Authentication curl: Run the following curl command (from the command line or REST client) with the parameters replaced to return your access_token and refresh_token. This is a POST request.

> curl -0 -v -X POST https://api.helpscout.net/v2/oauth2/token
    -H "Accept: application/json"\
    -H "application/x-www-form-urlencoded"\
    -d "grant_type=authorization_code"\
    -d "code=YOUR_CODE"\
    -d "client_id=YOUR_CLIENT_ID"\
    -d "client_secret=YOUR_CLIENT_SECRET"

Record your client_id, client_secret, and the returned refresh_token into your tap config.json, which should look like the following:

```json
{
    "client_id": "OAUTH_CLIENT_ID",
    "client_secret": "OAUTH_CLIENT_SECRET",
    "refresh_token": "YOUR_OAUTH_REFRESH_TOKEN",
    "start_date": "2017-04-19T13:37:30Z"
}
```

Quick Start

1. Install

   Clone this repository, and then install using setup.py. We recommend using a virtualenv:

   > virtualenv -p python3 venv
   > source venv/bin/activate
   > python setup.py install
   OR
   > cd .../tap-helpscout
   > pip install .

2. Dependent libraries The following dependent libraries were installed.

   > pip install singer-python
   > pip install singer-tools
   > pip install target-stitch
   > pip install target-json
   

   • singer-tools
   • target-stitch

3. Create your tap's config.json file which should look like the following:

   {
       "client_id": "OAUTH_CLIENT_ID",
       "client_secret": "OAUTH_CLIENT_SECRET",
       "refresh_token": "YOUR_OAUTH_REFRESH_TOKEN",
       "start_date": "2017-04-19T13:37:30Z",
       "user_agent": "tap-helpscout <[email protected]>"
   }

   Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

   {
       "currently_syncing": "users",
       "bookmarks": {
           "customers": "2019-06-11T13:37:55Z",
           "mailbox_folders": "2019-06-19T19:48:42Z",
           "mailboxes": "2019-06-18T18:23:58Z",
           "users": "2019-06-20T00:52:46Z",
           "workflows": "2019-06-19T19:48:44Z",
           "conversation_threads": "2019-06-11T13:37:55Z",
           "conversations": "2019-06-11T13:37:55Z"
       }
   }
   

4. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

   tap-helpscout --config config.json --discover > catalog.json

   See the Singer docs on discovery mode here.

5. Run the Tap in Sync Mode (with catalog) and write out to state file

   For Sync mode:

   > tap-helpscout --config tap_config.json --catalog catalog.json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To load to json files to verify outputs:

   > tap-helpscout --config tap_config.json --catalog catalog.json | target-json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To pseudo-load to Stitch Import API with dry run:

   > tap-helpscout --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

6. Test the Tap

   While developing the HelpScout tap, the following utilities were run in accordance with Singer.io best practices: Pylint to improve code quality:

   > pylint tap_helpscout -d missing-docstring -d logging-format-interpolation -d too-many-locals -d too-many-arguments

   Pylint test resulted in the following score:

   Your code has been rated at 10.00/10 (previous run: 9.97/10, +0.03)

   To check the tap and verify working:

   > tap-helpscout --config tap_config.json --catalog catalog.json | singer-check-tap > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   Check tap resulted in the following:

   The output is valid.
   It contained 150 messages for 8 streams.
   
         8 schema messages
       124 record messages
       18 state messages
   
   Details by stream:
   +----------------------+---------+---------+
   | stream               | records | schemas |
   +----------------------+---------+---------+
   | conversations        | 17      | 1       |
   | customers            | 56      | 1       |
   | users                | 4       | 1       |
   | mailbox_folders      | 9       | 1       |
   | workflows            | 2       | 1       |
   | conversation_threads | 32      | 1       |
   | mailbox_fields       | 2       | 1       |
   | mailboxes            | 2       | 1       |
   +----------------------+---------+---------+

---

Copyright © 2020 Stitch