isabella232 / tap-ms-teams

tap-ms-teams

Python 3.5

This is a Singer tap that produces JSON-formatted data following the Singer spec.

This tap:

• Pulls raw data from the Microsoft Graph API
• Extracts the following resources:
  • users
  • groups
  • group_members
  • group_owners
  • channels
  • channel_members
  • channel_tabs
  • channel_messages
  • channel_message_replies
  • conversations
  • conversation_threads
  • conversation_posts
  • team_drives
  • team_device_usage_report
• Outputs the schema for each resource
• Incrementally pulls data based on the input state

Streams

• users
  • Data key: value
  • Primary keys: id
  • Replication strategy: Full Table
  • Transformations: camelCase to snake_case
• groups
  • Data key: value
  • Primary keys: id
  • Replication strategy: Full Table
  • Transformations: camelCase to snake_case
• group_members
  • Data key: value
  • Primary keys: id
  • Replication strategy: Full Table
  • Transformations: camelCase to snake_case
• group_owners
  • Data key: value
  • Primary keys: id
  • Replication strategy: Full Table
  • Transformations: camelCase to snake_case
• channels
  • Data key: value
  • Primary keys: id
  • Replication strategy: Full Table
  • Transformations: camelCase to snake_case
• channel_members
  • Data key: value
  • Primary keys: id
  • Replication strategy: Full Table
  • Transformations: camelCase to snake_case
• channel_tabs
  • Data key: value
  • Primary keys: id
  • Replication strategy: Full Table
  • Transformations: camelCase to snake_case
• channel_messages
  • Data key: value
  • Primary keys: id
  • Replication strategy: Incremental (query all, filter results)
  • Bookmark: ucreatedDateTime OR lastModifiedDateTime OR deletedDateTime
  • Transformations: camelCase to snake_case
• channel_message_replies
  • Data key: value
  • Primary keys: id
  • Replication strategy: Incremental (query all, filter results)
  • Bookmark: ucreatedDateTime OR lastModifiedDateTime OR deletedDateTime
  • Transformations: camelCase to snake_case
• conversations
  • Data key: value
  • Primary keys: id
  • Replication strategy: Incremental (query all, filter results)
  • Bookmark: lastDeliveredDateTime
  • Transformations: camelCase to snake_case
• conversation_threads
  • Data key: value
  • Primary keys: id
  • Replication strategy: Incremental (query all, filter results)
  • Bookmark: lastDeliveredDateTime
  • Transformations: camelCase to snake_case
• conversation_posts
  • Data key: value
  • Primary keys: id
  • Replication strategy: Incremental (query all, filter results)
  • Bookmark: lastDeliveredDateTime
  • Transformations: camelCase to snake_case
• team_drives
  • Data key: value
  • Primary keys: id
  • Replication strategy: Incremental (query all, filter results)
  • Bookmark: lastModifiedDateTime
  • Transformations: camelCase to snake_case
• team_device_usage_report
  • Primary keys: id
  • Replication strategy: Incremental (query all, filter results)
  • Bookmark: report_refresh_date
  • Transformations: camelCase to snake_case

App and Authentication

The Microsoft Graph API uses OAuth for authentication. Follow these instructions for creating an app in the Azure portal. This process will produce the client_id, client_secret, and tenant_id needed for the tap configuration file.

Register a new application with the Azure portal

1. Login to Azure Portal
2. Click upper-left menu icon and select Azure Active Directory
3. Select App Registrations
4. Click New registration
5. Register an application
   1. Name: tap-microsoft-teams
   2. Supported account types: Accounts in this organizational directory only
   3. Redirect URL: Web - https://YOURDOMAIN/callback
   4. Register (button)
6. Record the client_id, tenant_id, and application_id (Object ID) which will be used by the tap for authentication and API integration.
7. Select Branding
   1. Name: tap-microsoft-teams
   2. Logo: [singer-io-logo.png, 512x512 px]
   3. Homepage URL: https://YOURDOMAIN
   4. ToS URL: https://www.stitchdata.com/eula/
   5. Privacy URL: https://www.stitchdata.com/privacy/
   6. Save
8. Select Authentication
   1. Platform configurations: Web (already setup)
   2. Supported account types: (already setup)
   3. Logout URL: https://app.stitchdata.com/session/create
   4. Access tokens: checked
   5. Public client: No
   6. Save
9. Select Certificates & secrets
10. Provide Description and Expires
    1. Description: tap-microsoft-teams client secret
    2. Expires: Never
    3. Add
11. Copy the client secret Value, this will be the client_secret
12. Select API permissions
    1. Click Add a permission
13. Select Microsoft Graph
14. Select Application permissions
15. Select the following permissions:
    1. Application > Application.Read.All
    2. Calendars > Calendars.Read?
    3. Chat > Chat.Read.All
    4. Files > Files.Read.All
    5. Group > Group.Read.All
    6. Notes > Notes.Read.All
    7. OnlineMeetings > OnlineMeetings.Read.All
    8. Reports > Reports.Read.All
    9. Schedule > Schedule.Read.All
    10. Team > Team.ReadBasic.All
    11. User > User.Read.All
16. Click Add permissions

Quick Start

1. Install

   Clone this repository, and then install using setup.py. We recommend using a virtualenv:

   > virtualenv -p python3 venv
   > source venv/bin/activate
   > python setup.py install
   OR
   > cd .../tap-ms-teams
   > pip install .

2. Dependent libraries The following dependent libraries were installed.

   > pip install singer-python
   > pip install singer-tools
   > pip install target-stitch
   > pip install target-json
   

   • singer-tools
   • target-stitch

3. Create your tap's config.json file which should look like the following:

   {
       "client_id": "YOUR_CLIENT_ID",
       "client_secret": "YOUR_API_TOKEN",
       "tenant_id": "YOUR_TENANT_ID",
       "start_date": "2020-05-01T00:00:00Z",
       "user_agent": "tap-ms-teams<api_user_email@your_company.com>"
   }

   Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

   {
       "bookmarks": {
           "conversation_threads": "2020-08-03T14:21:40.000000Z",
           "team_drives": "2020-07-13T16:55:39.000000Z",
           "channel_messages": "2020-08-12T23:18:03.915000Z",
           "conversation_posts": "2020-08-03T14:21:40.000000Z",
           "conversations": "2020-08-03T14:21:40.000000Z",
           "channel_message_replies": "2020-08-12T23:38:57.582000Z"
       }
   }
   

4. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

   tap-ms-teams --config config.json --discover > catalog.json

   See the Singer docs on discovery mode here.

5. Run the Tap in Sync Mode (with catalog) and write out to state file

   For Sync mode:

   > tap-ms-teams --config tap_config.json --catalog catalog.json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To load to json files to verify outputs:

   > tap-ms-teams --config tap_config.json --catalog catalog.json | target-json > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   To pseudo-load to Stitch Import API with dry run:

   > tap-ms-teams --config tap_config.json --catalog catalog.json | target-stitch --config target_config.json --dry-run > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

6. Test the Tap

   To check the tap and verify working:

   > tap-ms-teams --config tap_config.json --catalog catalog.json | singer-check-tap > state.json
   > tail -1 state.json > state.json.tmp && mv state.json.tmp state.json

   Check tap resulted in the following:

       The output is valid.
       It contained 114 messages for 13 streams.
   
           13 schema messages
           75 record messages
           26 state messages
   
       Details by stream:
       +-------------------------+---------+---------+
       | stream                  | records | schemas |
       +-------------------------+---------+---------+
       | users                   | 10      | 1       |
       | groups                  | 3       | 1       |
       | group_members           | 13      | 1       |
       | group_owners            | 7       | 1       |
       | channels                | 5       | 1       |
       | channel_members         | 5       | 1       |
       | channel_tabs            | 5       | 1       |
       | channel_messages        | 8       | 1       |
       | channel_message_replies | 6       | 1       |
       | conversations           | 3       | 1       |
       | conversation_threads    | 3       | 1       |
       | conversation_posts      | 3       | 1       |
       | team_drives             | 4       | 1       |
       +-------------------------+---------+---------+

---

Copyright © 2020 Stitch