thinkingmachines / bqup Goto Github PK

View Code? Open in Web Editor NEW

30.0 33.0 11.0 222 KB

(pronounced BACK-wup) BigQuery backup scripts

Home Page: https://pypi.org/project/bqup/

License: MIT License

Python 98.62% Makefile 1.38%

bqup's Introduction

Note If this is your first time checking out bqup, you might want to try out Dataform or dbt instead!

bqup

bqup is a backup tool for BigQuery projects. It can export a BigQuery project's structure and source code while mimicking the hierarchy of datasets and tables.

For the full story of why we made bqup, check out our blog post!

Installation

bqup can be installed using pip.

pip install bqup

Alternatively, you can also clone the repository then run install.

git clone https://github.com/thinkingmachines/bqup.git
cd bqup
python3 setup.py install

Usage

Command line options

You can list down the options by running bqup --help.

bqup [-p PROJECT_ID] [-d TARGET_DIR] [-fvxr] [-e REGEX]

Options:
  -p PROJECT_ID, --project PROJECT_ID  Project ID to load. If unspecified,
                                       defaults to current project in
                                       configuration.
  -d TARGET_DIR, --dir TARGET_DIR      The target directory where the project
                                       will be written. Defaults to current
                                       timestamp.
  -f --force                           Overwrite target directory if it exists.
  -v --verbose                         Print a summary of the loaded project.
  -x --schema                          Export table schemata as json.
  -r --routine                         Include routines in export.
  -e REGEX, --regex REGEX              Regex pattern to filter datasets to be exported.

Development

Set up gcloud to run with your personal account (aka run with scissors).
Set up application-default.
```
gcloud auth application-default login
```
Install wheel.
```
pip3 install wheel
```
Install bqup.
```
pip3 install -e .
```
Alternatively, you can also install it using:
```
python3 setup.py develop
```
Run bqup (see Usage).

Setting up regular backups

Check out bqup-starter to set up regular bqups using GitHub's workflows!

Distribution

Run make test to try a test upload.

Run make dist to upload a distribution.

Both of these will call make build, which rebuilds the package locally.

Contributors

Thanks to all these wonderful people who've helped out with bqup:

_Jess

_Ram

_{Pepe Berba}

_{Tim Pron}

_Enzo

_Ardie

Disclaimers

bqup is no longer being maintained

There are currently no plans to maintain this project.
Thinking Machines does not make any guarantees about the quality of the software.

bqup's People

Contributors

Stargazers

Watchers

Forkers

bhuesemann rbw-lpp niarepo gopinath678 bzieba-appfire spartez urtho ajkruse livtanong mkrso cgobb

bqup's Issues

Add CODE_OF_CONDUCT and CONTRIBUTING

You can check geomancer's CODE_OF_CONDUCT and CONTRIBUTING

ReadMe Link Broken

The link to the bqup-starter in the readme looks to be broken.

Update README.md

Minimize noise in README
https://www.makeareadme.com/

Add option/change view files to .SQL

Change view files to .SQL format for syntax highlighting in github. Mainly for code review purposes

Order of keys in json schemata is not consistent

If i'm not mistaken, it's just a matter of adding sort_keys=True to the json.dumps call.

Behavior confirmed in the ff. environment as per @profwacko :

Python 3.5.3
bqup==0.0.6

Use as cloud function

Useful for integrating with Slack

View definition doesn't include 'CREATE OR REPLACE VIEW' syntax

Add 'CREATE OR REPLACE VIEW ' in view definition.

Table schemas are empty

Sample:
https://github.com/thinkingmachines/cloversense-bqups/blob/master/projects/cloversense-checkmate/raw/oddslog201601.table

Save table schema as json in .table file

Routine definition includes 'AS (' syntax

Routine definition includes 'AS (' syntax so it cannot be run by bq query

bq query --use_legacy_sql=false < 'folder/proejct/dataset/name.procedure.sql'

Add support for service account keys

Add tests

Rename to bbq

Pronounced "bibiku"
Can't be confused with "backup"

Put deps in setup.py

setup(
  ...
  install_requires=[deps here],
)

Use python3.7 and use f-strings instead of format

Support for case insensitive filesystems

Work in https://github.com/thinkingmachines/bqup/tree/case-insensteve

Import view / table usage stats

First, figure out if this is even possible

One click deploy

What's the best way to set up a one-click deploy thing?

Use BigQuery Python Client Library

Add a way to log into the repo when the bqup was last updated

i.e. adding an "updated at" timestamp in README.md

Verbose flag is broken

script errors on "MODEL" type

ValueError is raised there is a BQ MODEL type in the BQ project. This occurs in projects with BQ ML models e.g. psc-demand.

sample stacktrace:

Traceback (most recent call last):
  File "/home/psc-bqup/.local/bin/bqup", line 11, in <module>
    load_entry_point('bqup', 'console_scripts', 'bqup')()
  File "/home/psc-bqup/bqup/bqup/main.py", line 33, in main
    p = Project(project_id or None, args['--schema'])
  File "/home/psc-bqup/bqup/bqup/project.py", line 16, in __init__
    map(partial(Dataset, self, export_schema), self.client.list_datasets()))
  File "/home/psc-bqup/bqup/bqup/dataset.py", line 22, in __init__
    map(partial(Table, self, export_schema), project.client.list_tables(bq_dataset.reference)))
  File "/home/psc-bqup/bqup/bqup/table.py", line 30, in __init__
    raise ValueError('Unrecognized table type: {}'.format(self.table_type))
ValueError: Unrecognized table type: MODEL

Error when running `bqup`

Traceback (most recent call last):
  File "/Users/jessicatiu/miniconda3/bin/bqup", line 11, in <module>
    load_entry_point('bqup', 'console_scripts', 'bqup')()
  File "/Users/jessicatiu/bqup/bqup/main.py", line 33, in main
    p = Project(project_id or None, args['--schema'])
  File "/Users/jessicatiu/bqup/bqup/project.py", line 16, in __init__
    map(partial(Dataset, self, export_schema), self.client.list_datasets()))
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 199, in _items_iter
    for page in self._page_iter(increment=False):
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 230, in _page_iter
    page = self._next_page()
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 349, in _next_page
    response = self._get_next_page_response()
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 401, in _get_next_page_response
    query_params=params)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/cloud/bigquery/client.py", line 311, in _call_api
    return call()
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/cloud/_http.py", line 290, in api_request
    headers=headers, target_object=_target_object)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/cloud/_http.py", line 183, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/cloud/_http.py", line 212, in _do_request
    url=url, method=method, headers=headers, data=data)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/auth/transport/requests.py", line 198, in request
    self._auth_request, method, url, request_headers)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/auth/credentials.py", line 122, in before_request
    self.refresh(request)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/oauth2/credentials.py", line 136, in refresh
    self._client_secret))
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/oauth2/_client.py", line 237, in refresh_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/oauth2/_client.py", line 111, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/Users/jessicatiu/miniconda3/lib/python3.6/site-packages/google/oauth2/_client.py", line 61, in _handle_error_response
    error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', '{\n  "error" : "invalid_grant",\n  "error_description" : "Bad Request"\n}')