Comments (17)
From @lukesneeringer on March 20, 2017 16:15
Hello @c0b. Thanks for reporting.
This seems feasible. We will look into it soon.
from nodejs-bigquery.
From @c0b on March 21, 2017 1:47
for better management of different jobs, to use a customized jobid is also something needed, from listJobs
I can get the jobId(s) look like these, I can see bqjob_
are from bq command line, bquijob_
are from bigqueryUI, I want all queries from my nodejs app to start with some special naming pattern,
job_ZhgN2K65s_XT1Yq7ooMJhG0TGcQ
job_xAN6iV-dcfb1XnCbpHkOdJ542JY
bquijob_74a81cc0_15aea476e69
bquijob_3c27ca25_15aea468ab4
bquijob_50d0e7e0_15aea45b2dc
job_qGBPt_F45gotmov1sdKobAgTEjk
job_gqUriVZhciWVHrSokbQ2AGIkGXo
job_oRykICSs6Tc7gm4dOQdt3nyQSpc
bqjob_r1f987f469a3e2a33_0000015
bqjob_r56059b08bb57b6ee_0000015
from the REST Reference, both sync and async job can start query with a specified jobReference
with customized jobId
https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs
I have tried put a jobReference
object on the startQuery but it seems ignored?
// jobReference need to be on top level of POST body object for job insert REST API
jobReference: {
projectId: 'my-project',
jobId: 'myjob_' + Date.now().toString(36) + '_' + Math.random().toString(36).substr(2),
},
this feature request could be another issue if you want me to file a separate issue#
from nodejs-bigquery.
From @lukesneeringer on March 21, 2017 19:32
Yes, please file distinct issues for distinct problems. :-)
from nodejs-bigquery.
From @lukesneeringer on March 21, 2017 19:46
As best as I can tell, the issue is that the labels are not actual query parameters, but they live one level up.
We take arbitrary query parameters as part of options.params
here:
if (options.params) {
options.useLegacySql = false;
options.parameterMode = is.array(options.params) ? 'positional' : 'named';
if (options.parameterMode === 'named') {
// Redacted; not relevant.
} else {
options.queryParameters = options.params
.map(BigQuery.valueToQueryParameter_);
}
...and then it looks like this in a request:
// Create a job.
self.request({
method: 'POST',
uri: '/queries',
json: options
}, responseHandler);
queryParameters
is not the same as configuration
, which is what I see in the JSON block that the command line tool sent. I am a little afraid to just shift the labels up to be a peer of queryParameters
since I am not sure what the effect will be.
@jmuk Please advise.
from nodejs-bigquery.
From @jmuk on March 21, 2017 22:8
@lukesneeringer -- it seems you're referring query
method, but I think the topic here is about startQuery
; and indeed it only accepts query
property for configuration
as https://github.com/GoogleCloudPlatform/google-cloud-node/blob/master/packages/bigquery/src/index.js#L959
But anyhow, the point is about other properties of the configuration
like labels
will not be specified. I have little ideas how to specify labels
through options
-- it will depend on which of the other properties in the configurations
we want to allow. Does anyone have ideas?
If it's labels
specific, we can special-handle options.labels
.
It's probably better to ask @stephenplusplus and @callmehiphop for the help.
from nodejs-bigquery.
From @c0b on March 23, 2017 18:53
ok, so, on the line I made this local changes to make it work, for both #2107 and #2121 to move labels
and jobReference
to their correct location
➸ diff -U7 -rNp -- ./node_modules/@google-cloud/bigquery/src/index.js{.orig,}
--- ./node_modules/@google-cloud/bigquery/src/index.js.orig 2017-03-23 18:48:53.730426569 +0000
+++ ./node_modules/@google-cloud/bigquery/src/index.js 2017-03-23 18:25:54.893332089 +0000
@@ -957,14 +957,23 @@ BigQuery.prototype.startQuery = function
var body = {
configuration: {
query: extend(true, defaults, options)
}
};
+ if (body.configuration.query.labels) {
+ body.configuration.labels = body.configuration.query.labels;
+ delete body.configuration.query.labels;
+ }
+ if (body.configuration.query.jobReference) {
+ body.jobReference = body.configuration.query.jobReference;
+ delete body.configuration.query.jobReference;
+ }
+
this.request({
method: 'POST',
uri: '/jobs',
json: body
}, function(err, resp) {
if (err) {
callback(err, null, resp);
although a full PR should cover query and others as well.
from nodejs-bigquery.
From @lukesneeringer on March 23, 2017 20:12
I would almost be inclined to go the other direction -- iterate over the keys we know should be under configuration
and move them out into a configuration
dict. That approach would be more future-proof.
from nodejs-bigquery.
From @c0b on March 23, 2017 20:17
sure, that should be a better and ultimate solution, an interface like below?
startQuery(queryOptions, configurationOptions, postBodyOptions)
labels would on configurationOptions, and jobReference on the postBodyOptions ?
from nodejs-bigquery.
Let's force a string-first argument for the query, then an options object in the format of a configuration
as described here:
var options = {
destination: bigquery.dataset('higher_education').table('institutions'),
labels: [...],
query: {}
}
bigquery.startQuery('SELECT url FROM ....', options, callback)
options.destination
is the only custom property we have, so just remove it from options
and send everything else straight through, after assigning the query string to the right place. Not tested, but something like:
BigQuery.prototype.startQuery = function(query, options, callback) {
var self = this;
if (!query) {
throw new Error('A SQL query string is required.');
}
options = options || {};
var configuration = extend(true, {}, options);
delete configuration.destination;
var defaultConfiguration = {
query: {
query: query
}
};
if (options.destination) {
if (!(options.destination instanceof Table)) {
throw new Error('Destination must be a Table object.');
}
defaultConfiguration.query.destinationTable = {
datasetId: options.destination.dataset.id,
projectId: options.destination.dataset.bigQuery.projectId,
tableId: options.destination.id
};
}
var body = {
configuration: extend(true, defaultConfiguration, options)
};
this.request({
method: 'POST',
uri: '/jobs',
json: body
}, ...
};
from nodejs-bigquery.
From @lukesneeringer on March 27, 2017 21:59
@stephenplusplus The problem is that the labels should not live in the configuration
object.
from nodejs-bigquery.
Are you sure? That's where it is in the docs: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.labels
I thought the problem is it shouldn't live in configuration.query
.
from nodejs-bigquery.
From @lukesneeringer on March 27, 2017 22:4
You are completely right and I am completely wrong. :-)
from nodejs-bigquery.
From @c0b on April 1, 2017 8:27
before you release the fixed @google-cloud/bigquery
on npm, I'm using this diff as workaround for this #2107 #2150 and #2121 waiting your better approach
--- ./node_modules/@google-cloud/bigquery/src/index.js.orig 2017-03-23 11:48:53.730426569 -0700
+++ ./node_modules/@google-cloud/bigquery/src/index.js 2017-03-28 17:30:51.561750235 -0700
@@ -957,14 +957,27 @@ BigQuery.prototype.startQuery = function
var body = {
configuration: {
query: extend(true, defaults, options)
}
};
+ if (body.configuration.query.dryRun) {
+ body.configuration.dryRun = body.configuration.query.dryRun;
+ delete body.configuration.query.dryRun;
+ }
+ if (body.configuration.query.labels) {
+ body.configuration.labels = body.configuration.query.labels;
+ delete body.configuration.query.labels;
+ }
+ if (body.configuration.query.jobReference) {
+ body.jobReference = body.configuration.query.jobReference;
+ delete body.configuration.query.jobReference;
+ }
+
this.request({
method: 'POST',
uri: '/jobs',
json: body
}, function(err, resp) {
if (err) {
callback(err, null, resp);
from nodejs-bigquery.
That works, and this should work as well, so you don't have to modify the source code:
var bigquery = require('@google-cloud/bigquery')({...})
bigquery.interceptors.push({
request: function(reqOpts) {
if (reqOpts.uri === '/jobs') {
var body = reqOpts.json;
if (body.configuration.query.dryRun) {
body.configuration.dryRun = body.configuration.query.dryRun;
delete body.configuration.query.dryRun;
}
if (body.configuration.query.labels) {
body.configuration.labels = body.configuration.query.labels;
delete body.configuration.query.labels;
}
if (body.configuration.query.jobReference) {
body.jobReference = body.configuration.query.jobReference;
delete body.configuration.query.jobReference;
}
}
return reqOpts;
}
});
from nodejs-bigquery.
From @c0b on April 2, 2017 18:23
thanks for the advanced usage; I just read the Interceptors from https://googlecloudplatform.github.io/google-cloud-node/#/docs/google-cloud
from nodejs-bigquery.
@stephenplusplus I think that BigQuery#createJob
should resolve this issue. WDYT?
from nodejs-bigquery.
Yep, that sounds right to me!
from nodejs-bigquery.
Related Issues (20)
- Authorized View Tutorial: should create an authorized view failed HOT 1
- Models: "before all" hook for "should retrieve a model if it exists" failed
- Queries: should run a query with legacy SQL and large results failed HOT 1
- Tables: should create failed HOT 1
- BigQuery BigQuery/Model: "before all" hook for "should get a list of models" failed
- BigQuery BigQuery/Model: "after all" hook for "should extract a model" failed
- JSON not a valid type for parameterized query
- PartialFailureError loading from storage lib instead of this lib HOT 2
- Table extract to gcs and keep the kmsKeyName HOT 1
- Querying on authorized view HOT 4
- 📦 pack-n-play test: TypeScript code failed HOT 1
- Support for IAM conditions in TypeScript types HOT 1
- 7.5.0 is a breaking change HOT 4
- Use Template Strings for GoogleSQL raw queries HOT 1
- Please update documentation to indicate which function calls, or if this entire package uses, legacy APIs
- Support for RANGE type
- Loading via write streams into Tables from Datasets with manually-specified project IDs returns invalid job
- GetRowsOptions Type Error for startIndex: Expects a string when it should expect a number HOT 5
- Library cannot parse timestamp value "0" - Cannot convert a BigInt value to a number as PreciseDate HOT 7
- Direct usage of `queryParameters` broke with 7.6.0 HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nodejs-bigquery.