GithubHelp home page GithubHelp logo

bq-dts-partner-clients's Introduction

Getting Started with BigQuery Data Transfer API

Open in Cloud Shell

BigQuery Data Transfer Service features an API that allows developers to create transfer jobs from data sources to BigQuery. These sample Java applications demonstrate how to access the BigQuery Data Transfer API using the BigQuery Data Transfer Service Client Library for Java.

Quickstart

Before you begin

Please do the following, to set up GCP project with necessary APIs, service accounts, and permissions:

  1. Install the gcloud SDK

  2. Enable the following APIs in the Google Developers Console.

    • BigQuery Data Transfer API
    • Cloud Pub/Sub API
    gcloud services enable bigquerydatatransfer.googleapis.com
    gcloud services enable pubsub.googleapis.com
    
  3. Create an operational IAM Service Account and download credentials for running your source. Running these commands

    • Creates a new Service Account named bq-dts-[SOURCE]@[PROJECT_ID].iam.gserviceaccount.com
    • Grants roles/bigquery.admin, roles/pubsub.subscriber, roles/storage.objectAdmin
    • Downloads a Service-Account key called .gcp-service-account.json
    SOURCE="example-source"  # Change to your own name 
    PROJECT_ID=$(gcloud config get-value core/project)
    PARTNER_SA_NAME="bq-dts-${SOURCE}"
    PARTNER_SA_EMAIL="${PARTNER_SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
    
    # Creating a Service Account
    gcloud iam service-accounts create ${PARTNER_SA_NAME} --display-name ${PARTNER_SA_NAME}
    
     # Granting Service Account required roles
    gcloud projects add-iam-policy-binding ${PROJECT_ID} --member="serviceAccount:${PARTNER_SA_EMAIL}" --role='roles/bigquery.admin'
    gcloud projects add-iam-policy-binding ${PROJECT_ID} --member="serviceAccount:${PARTNER_SA_EMAIL}" --role='roles/pubsub.subscriber'
    gcloud projects add-iam-policy-binding ${PROJECT_ID} --member="serviceAccount:${PARTNER_SA_EMAIL}" --role='roles/storage.objectAdmin'
    
    # Create service account credentials and store it locally needed for starting/running data sources.
    gcloud iam service-accounts keys create --iam-account "${SERVICE_ACCOUNT_EMAIL}" .gcp-service-account.json
    
  4. Create an administrative IAM Service Account and store credentials locally for creating data source.

    • Creates a new Service Account named bq-dts-admin@[PROJECT_ID].iam.gserviceaccount.com
    • Grants roles/project.owner
    • Downloads a Service-Account key called .gcp-service-account.json
    PROJECT_ID=$(gcloud config get-value core/project)
    PARTNER_SA_NAME="bq-dts-admin"
    PARTNER_SA_EMAIL="${PARTNER_SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
    
    # Creating a Service Account
    gcloud iam service-accounts create ${PARTNER_SA_NAME} --display-name ${PARTNER_SA_NAME}
    gcloud projects add-iam-policy-binding ${PROJECT_ID} --member="serviceAccount:${PARTNER_SA_EMAIL}" --role='roles/owner'
    
    # Create service account credentials and store it locally needed for creating data source
    gcloud iam service-accounts keys create --iam-account "${PARTNER_SA_EMAIL}" .gcp-service-account-owner.json
    

    For more information on service account authentication, please refer to Authenticate using a service account.

  5. Grant permissions to a GCP-managed Service Account

    • Creates a custom role - bigquerydatatransfer.connector with permission clientauthconfig.clients.getWithSecret
    • Grants project-specific role bigquerydatatransfer.connector
    PROJECT_ID=$(gcloud config get-value core/project)
    GCP_SA_EMAIL="[email protected]"
    
    # Creating a custom role
    gcloud iam roles create bigquerydatatransfer.connector --project ${PROJECT_ID} --title "BigQuery Data Transfer Service Connector" --description "Custom role for GCP-managed Service Account for BQ DTS" --permissions "clientauthconfig.clients.getWithSecret" --stage ALPHA
    
    # Granting Service Account required roles
    gcloud projects add-iam-policy-binding ${PROJECT_ID} --member="serviceAccount:${GCP_SA_EMAIL}" --role="projects/${PROJECT_ID}/roles/bigquerydatatransfer.connector"
    
  6. Create an OAuth Consent Screen

  7. Join BigQuery Data Transfer Service Partner-level whitelists. Reach out to your Google Cloud Platform contact to get whitelisted for these APIs.

Run the Example

  1. Set up environment variables:

    export PROJECT_ID="<your project id>"
    export GOOGLE_APPLICATION_CREDENTIALS=.gcp-service-account-owner.json
    
  2. Build and run the example program that calls BigQuery Data Transfer Service APIs. The example program can be built using either Maven or Gradle. Gradle Wrapper is included in the repository so you don't need to install Gradle.

    • Using Maven

    Install Maven. The example program that makes

    mvn install -DskipTests
    cd partner-examples
    mvn exec:java -Dexec.classpathScope=compile \
      -Dexec.mainClass=com.example.bigquerydatatransfer.DataTransferServiceClientSample -e \
      -Dexec.args="${PROJECT_ID}" -Dexec.cleanupDaemonThreads=false
    
    • Using Gradle
    ./gradlew install
    cd partner-examples
    ../gradlew run -Pexec.args="${PROJECT_ID}"
    

Notes

For illustration purpose, the example calls the following two BigQuery Data Transfer API methods:

Note that the 2nd method is only visible to whitelisted partner projects. Plese talk to your Google Cloud Platform contact.

bq-dts-partner-clients's People

Contributors

cmm08 avatar lswu avatar

Stargazers

 avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.