treasure-data / td Goto Github PK
View Code? Open in Web Editor NEWCUI Interface
Home Page: https://toolbelt.treasuredata.com
CUI Interface
Home Page: https://toolbelt.treasuredata.com
but other commands like bulk_import or import uses just <db> <table>
arguments. Can't we use this way instead?
Can't it automatically create one?
Show storage volume at td status
. Requested by @y-ken.
want to show only success/error state jobs.
requested by @hotchpotch.
The following error occured to my environment.
% td job:list [~]
error NoMethodError: backtrace:
/usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/job.rb:55:in `job_list'
/usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/job.rb:28:in `each'
/usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/job.rb:28:in `job_list'
/usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/list.rb:133:in `call'
/usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/list.rb:133:in `get_method'
/usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/runner.rb:96:in `call'
/usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/runner.rb:96:in `run'
/usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/bin/td:9
/usr/local/bin/td:19:in `load'
/usr/local/bin/td:19
undefined method `localtime' for nil:NilClass
The ruby interpreter's version is here.
ruby 1.8.7 (2010-12-23 patchlevel 330) [x86_64-linux]
(centos5)
If given YAML file is broken, it should weird stack trace as below.
$ td connector:guess seed.yml -o test.yml
Error TypeError: backtrace:
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:23:in `[]'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:23:in `block in from_hash'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:22:in `map'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:22:in `from_hash'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:24:in `block in from_hash'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:22:in `map'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:22:in `from_hash'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:17:in `from_json'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/lib/td/command/connector.rb:266:in `prepare_bulkload_job_config'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/lib/td/command/connector.rb:29:in `connector_guess'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/lib/td/command/list.rb:163:in `call'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/lib/td/command/list.rb:163:in `block in get_method'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/lib/td/command/runner.rb:170:in `call'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/lib/td/command/runner.rb:170:in `run'
/Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/bin/td:29:in `<top (required)>'
/Users/takahashi/.rbenv/versions/2.0.0-p481/bin/td:23:in `load'
/Users/takahashi/.rbenv/versions/2.0.0-p481/bin/td:23:in `<main>'
Error: no implicit conversion of Symbol into Integer
td command should show proper error message.
The config_diff config can control what/how next execution should start. Some sort of users want to modify the config_diff.
T/O
By default, put configuration in ~/.config/td/
instead of in ~/.td
.
If the environment variable XDG_CONFIG_HOME
is set, use $XDG_CONFIG_HOME/td/
for configuration.
This is part of the XDG Base Directory Specification which is a relatively new standard but has a lot of momentum. Even git
implements it.
https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html#variables
http://unix.stackexchange.com/a/38871
https://github.com/git/git/blob/master/Documentation/git-config.txt#L241
connector:guess shows yaml file using this format:
config:
type: s3
parser:
columns:
- name: app_id
type: long
- name: dim_quarter_id
type: long
- name: date
type: timestamp
format: ! '%Y-%m-%d'
On the other hand, Embulk formats yaml file using this format:
config:
type: s3
parser:
columns:
- {name: app_id, type: long}
- {name: dim_quarter_id, type: long}
- {name: date, type: timestamp, format: '%Y-%m-%d'}
The latter format is easier to read especially if the file includes many columns. Can connector:guess formats the YAML file using this flow format instead?
An user had a problem to download a large result file (over 200MB) and convert it to TSV (this size could be over 2GB) using following command:
$ td job 5632435 -o /Users/anwarelbizanti/Downloads/results.tsv -f tsv
But it failed due because of network problem. Idea here is that job:result resumes downloading if it failed because of network problem.
I guess that you can add some code to td-client-ruby. If downloading failed due to network error, it can send GET request again with Range header.
On Linux, we provide td-agent for td command.
But this is bad because td-agent is large package and its release cycle doesn't depend on td command.
Even if td command supports new features, to deliver td command is delayed.
So providing single package without td-agent is better.
When connector:history
reads 'queued' connector jobs, it cannot convert the nil
start_at into Time
object and raises TypeError
like following:
Error TypeError: backtrace:
/Users/muga/works/workspace/td/lib/td/command/connector.rb:249:in `at'
/Users/muga/works/workspace/td/lib/td/command/connector.rb:249:in `block in connector_history'
/Users/muga/works/workspace/td/lib/td/command/connector.rb:239:in `map'
/Users/muga/works/workspace/td/lib/td/command/connector.rb:239:in `connector_history'
/Users/muga/works/workspace/td/lib/td/command/list.rb:171:in `call'
/Users/muga/works/workspace/td/lib/td/command/list.rb:171:in `block in get_method'
/Users/muga/works/workspace/td/lib/td/command/runner.rb:170:in `call'
/Users/muga/works/workspace/td/lib/td/command/runner.rb:170:in `run'
/Users/muga/.rbenv/versions/2.0.0-p643/lib/ruby/gems/2.0.0/gems/td-0.11.14/bin/td:29:in `<top (required)>'
/Users/muga/.rbenv/versions/2.0.0-p643/bin/td:23:in `load'
/Users/muga/.rbenv/versions/2.0.0-p643/bin/td:23:in `<main>'
Error: can't convert nil into an exact number
I think that the retrievals of start_at
and end_at
should be fixed.
https://github.com/treasure-data/td/blob/master/lib/td/command/connector.rb#L248-L249
$ ruby bin/trd drop-database
error NoMethodError: backtrace:
/Users/kzk/repos/trd/lib/trd/command/common.rb:27:in `cmd_opt'
/Users/kzk/repos/trd/lib/trd/command/database.rb:24:in `drop_database'
/Users/kzk/repos/trd/lib/trd/command/trd.rb:71:in `call'
/Users/kzk/repos/trd/lib/trd/command/trd.rb:71
bin/trd:6:in `require'
bin/trd:6
private method `split' called for nil:NilClass
td tables
shows the count
column number like "123,456" format.
It is readable but hard to parse.
So if format is specified, then count
number should not be formatted like "123456".
I'd like to use JSON objects with field names like this:
[
{"key1": "value", "key2" : "value2"},
...
]
However, td query --format json
does not include field names; instead, it returns the following format:
[
[ "value1", "value2", ... ],
...
]
How about labeld JSON format?
For sorting the db list.
When I execute connector:preview
command, rename
filter doesn't work even though rename
filter config was declared in the config.yml. The following workaround works for me but I don't know that it's good enough or not. Please take a look.
$ git diff lib/td/connector_config_normalizer.rb
diff --git a/lib/td/connector_config_normalizer.rb b/lib/td/connector_config_normalizer.rb
index e2531bb..877228f 100644
--- a/lib/td/connector_config_normalizer.rb
+++ b/lib/td/connector_config_normalizer.rb
@@ -8,9 +8,10 @@ module TreasureData
case
when @config['in']
{
- 'in' => @config['in'],
- 'out' => @config['out'] || {},
- 'exec' => @config['exec'] || {}
+ 'in' => @config['in'],
+ 'filters' => @config['filters'],
+ 'out' => @config['out'] || {},
+ 'exec' => @config['exec'] || {}
}
when @config['config']
if @config.size != 1
Users sometimes want to change a table name like:
$ td table:rename <db> <table> <new_name>
$ td table:rename --overwrite <db> <table> <new_name>
--overwrite
option allows users to replace (overwrite) existing table.
current dependency is ~> 0.3.21
.
but, latest td-logger version is 0.3.24
.
$ ruby bin/trd query "SELECT * FROM acccess" myapp
error ArgumentError: backtrace:
/Users/kzk/repos/trd/lib/trd/api.rb:218:in `initialize'
/Users/kzk/repos/trd/lib/trd/api.rb:103:in `new'
/Users/kzk/repos/trd/lib/trd/api.rb:103:in `query'
/Users/kzk/repos/trd/lib/trd/command/query.rb:16:in `query'
/Users/kzk/repos/trd/lib/trd/command/trd.rb:71:in `call'
/Users/kzk/repos/trd/lib/trd/command/trd.rb:71:in `<top (required)>'
<internal:lib/rubygems/custom_require>:29:in `require'
<internal:lib/rubygems/custom_require>:29:in `require'
bin/trd:6:in `<main>'
wrong number of arguments (3 for 4)
Let's remove 0.5.5 from dependency list.
% gem list msgpack
*** LOCAL GEMS ***
msgpack (0.5.5)
% td -c ~/.td/td_myui_development.conf job:show 104209
JobID : 104209
Status : success
Type : hive
Database : movielens1m
Priority : NORMAL
Retry limit : 0
Output :
Query : select v from training_fm limit 10
CPU time : 2s 740ms
Result size : 249 B
Result :
WARNING: the job result is being downloaded... Error RuntimeError: backtrace:
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api/job.rb:218:in `feed_each'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api/job.rb:218:in `block (2 levels) in job_result_each_with_compr_size'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:276:in `call'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:276:in `block in read_body'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/protocol.rb:411:in `call_block'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/protocol.rb:402:in `<<'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/protocol.rb:106:in `read'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http/response.rb:290:in `block in read_body_0'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http/response.rb:250:in `inflater'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http/response.rb:280:in `read_body_0'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http/response.rb:201:in `read_body'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:274:in `read_body'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:289:in `each_fragment'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api/job.rb:217:in `block in job_result_each_with_compr_size'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:337:in `call'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:337:in `block in do_get'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1421:in `block (2 levels) in transport_request'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http/response.rb:162:in `reading_body'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1420:in `block in transport_request'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1411:in `catch'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1411:in `transport_request'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1384:in `request'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1377:in `block in request'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:853:in `start'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1375:in `request'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:335:in `do_get'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:298:in `block in get'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:557:in `guard_no_sslv3'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:297:in `get'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api/job.rb:204:in `job_result_each_with_compr_size'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client.rb:238:in `job_result_each_with_compr_size'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/model.rb:497:in `result_each_with_compr_size'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/job.rb:457:in `render_result'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/job.rb:290:in `show_result'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/job.rb:263:in `show_result_with_retry'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/job.rb:201:in `get_and_show_result'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/job.rb:136:in `job_show'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/list.rb:171:in `call'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/list.rb:171:in `block in get_method'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/runner.rb:170:in `call'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/runner.rb:170:in `run'
/Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/bin/td:29:in `<top (required)>'
/Users/nahi/.rbenv/versions/2.2.3/bin/td:23:in `load'
/Users/nahi/.rbenv/versions/2.2.3/bin/td:23:in `<main>'
Error: can't modify frozen String
Do you have any plan to relax or update dependency restriction of parallel
gem?
The latest td
gem (0.10.77) has "parallel", "~> 0.5.19"
dependency, which prevents our app updating from 0.10.61 to the latest. parallel
0.6.x is there for 6 months --- I'd imagine it's matured and stable enough. http://rubygems.org/gems/parallel/versions
Thanks.
td job:show -o foo.csv -f csv
foo.csv.tmp (downloading) -> foo.csv (download finished)
When I run td command as a part of ETL workflow, I need to parse stdout message to know error or status. But it is hard and fragile. If the command writes the result into a file, integrating to ETL tools becomes much easier. And because ETL tools use shell script, the file format should be shell-friendly format.
For example,
td query ... --status status.sh # writes status.sh
source status.sh
echo $job_id
td job:show -w $job_id --status status.sh
source status.sh
echo $elapsed
echo $status
td import:show ... --status status.sh
source status.sh
echo $valid_records
echo $error_records
[2016-06-24 09:44:43] ERROR: TD Failed![/usr/bin/td -r -c /home/user001/.td/td.conf table:create table_test:/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/td-0.12.0/lib/td/command/runner.rb:194:in rescue in run': undefined method
split' for nil:NilClass (NoMethodError)
from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/td-0.12.0/lib/td/command/runner.rb:165:in run' from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/td-0.12.0/bin/td:29:in
<top (required)>'
from /usr/lib64/fluent/ruby/bin/td:23:in load' from /usr/lib64/fluent/ruby/bin/td:23:in
T/O
The current available version for OSX is 0.16.3 on toolbelt.treasuredata.com.
When you have multiple databases, current td tables
command call '/v3/table/list' API one by one for each database and it's actually slow. It would be nice if you can do it in parallel by threads.
Env variables override "td apikey:set -f", i.e., whatever new apikey you set would not replace the one set by the env variable.
This behavior is intended, but the associated message is kind of misleading:
$ td apikey:set <new_apikey>
API key is set.
Use 'td db:create <db_name>' to create a database.
When, in reality, API key is not set to <new_apikey>.
TreasureData::Helpers.format_with_delimiter uses `gsub!` to their argument
the argument (in our case, `number_raw`) will be rewritten by them
To avoid that behavior, pass `number_raw.dup` instead of `number_raw`
td/spec/td/command/table_spec.rb
Lines 133 to 135 in c4a9a8b
Since scheduled_time is not passed to td-client-ruby, connector:run is always executed at schedule time "1970-01-01 00:00:00 UTC".
https://github.com/treasure-data/td/blob/master/lib/td/command/connector.rb#L263
And I think:
Please update the dependent version.
$ bundle update rubocop
Fetching gem metadata from https://rubygems.org/..............
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Fetching gem metadata from https://rubygems.org/...........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies....
Bundler could not find compatible versions for gem "parallel":
In snapshot (Gemfile.lock):
parallel (= 1.8.0)
In Gemfile:
rubocop (~> 0.49.1) was resolved to 0.49.1, which depends on
parallel (~> 1.10)
td was resolved to 0.15.2, which depends on
parallel (~> 1.8.0)
Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.
Currently, td table:import
requires the users to create db and table before it. This requires additional steps at the quickstart. Please add --auto-create-table option to eliminate these steps.
In some environments, user accesses TD behind the proxy which doesn't support PUT method.
And someone says "td command should access TD over SSL by default!"
Default SSL access satisfies these requirements.
Changing default access to over SSL may cause the performance issue.
We should check some existence commands.
In addition, Option candidates to disable SSL:
connector:preview doesn't show full column name if they're too long.
But those information is very important to confirm the preview result.
% td connector:preview load1.yml
+--------+------------+------------+--------------+-------+------------+---------+-----------+------------+------------+
| "ca... | "catego... | "catego... | "category... | "s... | "ppppp... | "is_... | "is_pp... | "abc_in... | "abc_ou... |
+--------+------------+------------+--------------+-------+------------+---------+-----------+------------+------------+
...
+--------+------------+------------+--------------+-------+------------+---------+-----------+------------+------------+
It's better that users can see current config_diff of the scheduled execution. Related to #127.
It is useful!
For example -p / --priority as in:
λ ~SW/source/td-api/ PLT-1951* td_stag import:perform
usage:
$ td import:perform <name>
example:
$ td import:perform logs_201201
description:
Start to validate and convert uploaded files
options:
-w, --wait wait for finishing the job
-f, --force force start performing
-p, --priority job priority: -2: very low, -1: low, 0: normal, 1: high, +1 very high
bundle exec bin/td connector:show connector1
Name : connector1
Cron : 0 * * * *
Timezone : UTC
Delay : 0
Database :
Table :
Config
Error NoMethodError: backtrace:
/Users/nahi/git/td/lib/td/command/connector.rb:293:in `dump_connector_session'
/Users/nahi/git/td/lib/td/command/connector.rb:184:in `connector_show'
/Users/nahi/git/td/lib/td/command/list.rb:171:in `call'
/Users/nahi/git/td/lib/td/command/list.rb:171:in `block in get_method'
/Users/nahi/git/td/lib/td/command/runner.rb:170:in `call'
/Users/nahi/git/td/lib/td/command/runner.rb:170:in `run'
bin/td:29:in `<main>'
Error: undefined method `config' for #<Hash:0x007f9d41c79288>
connector:create also crashes at the end of command execution.
Patch:
diff --git a/lib/td/command/connector.rb b/lib/td/command/connector.rb
index 3110804..4cef6b6 100644
--- a/lib/td/command/connector.rb
+++ b/lib/td/command/connector.rb
@@ -283,14 +283,14 @@ private
end
def dump_connector_session(session)
- $stdout.puts "Name : #{session.name}"
- $stdout.puts "Cron : #{session.cron}"
- $stdout.puts "Timezone : #{session.timezone}"
- $stdout.puts "Delay : #{session.delay}"
- $stdout.puts "Database : #{session.database}"
- $stdout.puts "Table : #{session.table}"
+ $stdout.puts "Name : #{session["name"]}"
+ $stdout.puts "Cron : #{session["cron"]}"
+ $stdout.puts "Timezone : #{session["timezone"]}"
+ $stdout.puts "Delay : #{session["delay"]}"
+ $stdout.puts "Database : #{session["database"]}"
+ $stdout.puts "Table : #{session["table"]}"
$stdout.puts "Config"
- $stdout.puts YAML.dump(session.config)
+ $stdout.puts YAML.dump(session["config"])
end
def wait_connector_job(client, job_id, exclude)
Data connector can acquire 3 versions of config specifications from td CLI because td API server normalizes all of them. But we should move server-side normalization code to td CLI side. td CLI should send latest (3rd) config only to td API server. And td CLI can show warning messages if users still use old versioned config spec.
The following is 3 types config specification:
in
key like:type: s3
bucket: my_bucket
path_prefix: path/to/prefix
...
config
key like:config:
in:
type: s3
bucket: my_bucket
path_prefix: path/to/prefix
...
in:
type: s3
bucket: my_bucket
path_prefix: path/to/prefix
...
Current version of rubyzip (~> 1.1.7) is vulnerable:
Updated ruby-advisory-db
ruby-advisory-db: 283 advisories
Name: rubyzip
Version: 1.1.7
Advisory: CVE-2017-5946
Criticality: Unknown
URL: https://github.com/rubyzip/rubyzip/issues/315
Title: Directory traversal vulnerability in rubyzip
Solution: upgrade to >= 1.2.1
Vulnerabilities found!
Would be good to have a new released version of gem with the dependency updated.
rubyzip 1.2.x serise has a problem.
See https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-16892
connector:guess
is best-effort. Users need to confirm whether the guessed result is correct or not.
Embulk itself shows guessed result to stdout so that users can confirm the result immediately.
Why don't you change connector:guess to also show guessed yaml file to stdout?
Should it be consistent?
I used "td import:auto" command like below command,
$ td import:auto mysess test.csv --auto-create sampledb.sampletable ~
(I didn't create mysess: bulk_session. I created only sampledb, sampletable.)
Then, it shows the error like below.
Maybe, "--auto-create" option can't use with session name
------error------
td import:auto mysess ./test.csv --auto-create sampledb.sampletable --column-header -o ./ --time-value 1378439234
java version "1.6.0_51"
Java(TM) SE Runtime Environment (build 1.6.0_51-b11-456-11M4508)
Java HotSpot(TM) 64-Bit Server VM (build 20.51-b01-456, mixed mode)
Create sampledb_sampletable_2013_09_06_1378446034 bulk_import session
Uploading prepared files
Session name : sampledb_sampletable_2013_09_06_1378446034
File : mysess (0 bytes)
File : ./test.csv (59645903 bytes)
We want to monitor that data actually is received at Treasure Data. One way could be to use td
to tell us when the tables data was last updated. Another could be to expose it via the TD API, either way would be great for us.
connector commands (like connector:guess
) requires config
key. But it is unnecessary.
On the other hand, embulk will add multi-job feature (https://github.com/embulk/embulk/issues # 167). Once it is implemented to embulk, configuration file will be more complex. In this case, if both embulk and td CLI use the same config syntax, we can use the same document for users (although it is not the best solution, it's good for customer support if there is some documents at embulk.org). If only td
commands need config
key, we can't use the same document.
We can improve the progress indicator in common.rb with ruby-progressbar gem's progress indicator which looks quite nicer.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.