precog / quasar-destination-gbq

Quasar connector providing support for Google BigQuery as a push destination

quasar-destination-gbq

Usage

libraryDependencies += "com.precog" %% "quasar-destination-gbq" % <version>

Configuration

{
  "name": <destination-name>,
  "type": {
    "name": "gbq",
    "version": 1
  },
  "config": {
    "authCfg": <service-account-json-auth-file-contents>,
    "datasetId": <dataset-name>,
    "maxFileSize": Number?
  }
}

• destination-name is what you would like to name the destination
• service-account-json-auth-file-contents is the contents of your service account authentication json file in string format
• dataset-name is the dataset name you would like your table to be pushed into
• maxFileSize is optional parameter, default to 2^30, determines size of streams uploaded to BigQuery as files, e.g. by default 30Gb stream will be uploaded as 30 files.

Testing

In order to run the tests, you must first decrypt the secret. You'll need to have the environment variable ENCRYPTION_PASSWORD set.

./sbt "decryptSecret core/src/test/resources/precog-ci-275718-e913743ebfeb.json.enc"

quasar-destination-gbq's Issues

Investigate completed datasets not showing up

Extend `TotalExceeded` to allow different metrics from `Offset`

precog/features#91

For OffsetPaginated we need to add additional capability to TotalExceeded. At the moment it is assumed that TotalExceeded operates with the same metric as Offset. This needs to be changed such that it would no longer be the case.

The changes are as thus

• Add additional totalRecordsExceeded and totalPagesExceeded options to config for stopCondition
• totalExceeded stop condition should select one of the above that matches offset
• Add additional methods to Offset to allow extracting either metric (pages or records)
• Compare the total exceeded against the right metric from Offset