GithubHelp home page GithubHelp logo

td's Introduction

Treasure Data command line tool

<img src=“https://circleci.com/gh/treasure-data/td.svg?style=svg” alt=“Build Status” /> <img src=“https://ci.appveyor.com/api/projects/status/github/treasure-data/td?branch=master&svg=true” alt=“appveyor” /> <img src=“https://coveralls.io/repos/treasure-data/td/badge.svg?branch=master&service=github” alt=“Coverage Status” />

This CUI utility wraps the Ruby Client Library td-client-ruby to interact with the REST API in managing databases and jobs on the Treasure Data Cloud.

For more info about Treasure Data, see <www.treasuredata.com/>.

For full documentation see <docs.treasuredata.com/>.

Getting Started

Install td command as a gem.

> gem install td

See help message for details.

> td

You need to authorize the account, before executing any other commands.

> td account

Sample Workflow

> td account -f                       # authorize an account
user: [email protected]
password: **********
> td database:create mydb             # create a database
> td table:create mydb www_access     # create a table

Packaging

Mac OS X

Disable RVM or rbenv and use ruby.pkg’s ruby (/usr/local/td/ruby/bin/ruby). And then run following commands:

$ /usr/local/td/ruby/bin/gem install bundler rubyzip
$ /usr/local/td/ruby/bin/bundle install
$ /usr/local/td/ruby/bin/rake pkg:build

It uses github.com/treasure-data/ruby-osx-packager

Windows

Install following binary packages:

* MinGW with MSYS Basic System and using mingw-get-inst
* Git for Windows, with Windows Command Prompt support
* Ruby ruby-1.9.3p327 using RubyInstaller for Windows, with PATH update
* Inno Setup 5

Then run following commands on MinGW Shell:

$ mingw-get install msys-vim
$ mingw-get install msys-wget
$ gem install bundler rubyzip
$ bundle install  # don't use "--path" option
$ rake exe:build  # don't use "bundle exec"

Bulk Import

Some of the td commands prefixed with td import leverages the Java Bulk Import CLI td-import-java to process and Bulk load data in the Treasure Data Cloud.

The Bulk Import CLI is downloaded automatically at the first call to any of the command that require it; the use will need internet connectivity in order to fetch the Bulk Import CLI JAR file from the Central Maven repository and take advantage of these advanced features. If you need to setup a proxy, please consult this documentation page.

The log levels and properties of the Bulk Import CLI can be configured in a logging.properties file. A default configuration is provided in a file within the gem or Toolbelt folder root, in the java/ folder. If you wish to customize it, please make a copy of this file and store it in the:

~/.td/java             folder on Mac OSX or Linux
%USERPROFILE%\.td\java folder on Windows

Testing Hooks

The CLI implements several hooks to enable/disable/trigger special behaviors. These hooks are expressed as environment variables and can therefore be provided in several ways:

How to Use

  • Unix / Linux / MacOSX

    • environment variable export in the shell the command is executed. The setting remains active until the shell is closed. E.g.:

      $ export TD_TOOLBELT_DEBUG=1
    • in the shell configuration file, to be active in any new shell that is opened. E.g.: add

      export TD_TOOLBELT_DEBUG=1
      

      to ~/.bashrc or equivalent shell configuration file. To make the setting active in the current shell, source the configuration file, e.g.:

      $ source ~/.bashrc
    • on the command line at runtime (active only for the duration of the command). E.g.:

      $ TD_TOOLBELT_DEBUG=1 td ....
    • as alias on in the current shell. The setting remains active until the shell is closed. E.g.:

      $ alias td='TD_TOOLBELT_DEBUG=1 td'
    • as alias in configuration file, to be active in any new shell that is opened. E.g.:

      alias td='TD_TOOLBELT_DEBUG=1 td'`

      to ~/.bashrc or equivalent shell configuration file. To make the setting active in the current shell, source the configuration file, e.g.:

      $ source ~/.bashrc
  • Windows

    • in the command prompt the command is executed. The setting remains active until the command prompt window is closed. E.g.:

      cmd> set TD_TOOLBELT_DEBUG=1
    • as a global environment variable in the system settings. It will be active for all new command prompt windows.

These are the available hooks:

  • Enable debugging mode:

    $ TD_TOOLBELT_DEBUG=1
  • JAR auto update (enabled by default is not specified). This setting does not affect import:jar_update:

    • Enable:

      $ TD_TOOLBELT_JAR_UPDATE=1
    • Disable:

      $ TD_TOOLBELT_JAR_UPDATE=0
  • Specify an alternative endpoint to use updating the toolbelt (default: toolbelt.treasuredata.com):

    $ TD_TOOLBELT_UPDATE_ROOT="http://toolbelt.treasuredata.com"
  • Specify an alternative endpoint to use updating the JAR file (default: repo1.maven.org):

    $ TD_TOOLBELT_JARUPDATE_ROOT="https://repo1.maven.org"

Copyright

Copyright

Copyright © 2015 Treasure Data Inc.

License

Apache License, Version 2.0

td's People

Contributors

akito19 avatar danielnorberg avatar drshanegreen avatar eddsteel avatar frsyuki avatar guilleiguaran avatar jwyuan avatar k0kubun avatar kietdo360 avatar komamitsu avatar kzk avatar makimoto avatar mcaramello avatar mickeey2525 avatar muga avatar nahi avatar naritta avatar nurse avatar repeatedly avatar swapnil-bhasale-td avatar tagomoris avatar takkanm avatar toru-takahashi avatar trung-huynh avatar tung-vu avatar tung-vu-td avatar uu59 avatar y-ken avatar yakitorii avatar yui-knk avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

td's Issues

undefined method `localtime' for nil:NilClass

The following error occured to my environment.

% td job:list                                                                                                                                       [~]
error NoMethodError: backtrace:
  /usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/job.rb:55:in `job_list'
  /usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/job.rb:28:in `each'
  /usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/job.rb:28:in `job_list'
  /usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/list.rb:133:in `call'
  /usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/list.rb:133:in `get_method'
  /usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/runner.rb:96:in `call'
  /usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/lib/td/command/runner.rb:96:in `run'
  /usr/local/lib/ruby/gems/1.8/gems/td-0.9.11/bin/td:9
  /usr/local/bin/td:19:in `load'
  /usr/local/bin/td:19

undefined method `localtime' for nil:NilClass

The ruby interpreter's version is here.

ruby 1.8.7 (2010-12-23 patchlevel 330) [x86_64-linux]
(centos5)

connector:preview doesn't show full column name

connector:preview doesn't show full column name if they're too long.
But those information is very important to confirm the preview result.

% td connector:preview load1.yml
+--------+------------+------------+--------------+-------+------------+---------+-----------+------------+------------+
| "ca... | "catego... | "catego... | "category... | "s... | "ppppp... | "is_... | "is_pp... | "abc_in... | "abc_ou... |
+--------+------------+------------+--------------+-------+------------+---------+-----------+------------+------------+
...
+--------+------------+------------+--------------+-------+------------+---------+-----------+------------+------------+

Using threads for `td tables`

When you have multiple databases, current td tables command call '/v3/table/list' API one by one for each database and it's actually slow. It would be nice if you can do it in parallel by threads.

columns: field is easy to read if connector:guess formats yaml file using flow syntax

connector:guess shows yaml file using this format:

config:
  type: s3
  parser:
    columns:
    - name: app_id
      type: long
    - name: dim_quarter_id
      type: long
    - name: date
      type: timestamp
      format: ! '%Y-%m-%d'

On the other hand, Embulk formats yaml file using this format:

config:
  type: s3
  parser:
    columns:
    - {name: app_id, type: long}
    - {name: dim_quarter_id, type: long}
    - {name: date, type: timestamp, format: '%Y-%m-%d'}

The latter format is easier to read especially if the file includes many columns. Can connector:guess formats the YAML file using this flow format instead?

connector:run's scheduled time argument is not passed to td-client-ruby

Since scheduled_time is not passed to td-client-ruby, connector:run is always executed at schedule time "1970-01-01 00:00:00 UTC".
https://github.com/treasure-data/td/blob/master/lib/td/command/connector.rb#L263

And I think:

  • Since api requires Integer value (unix timestamp) as scheduled_time, "2016-01-01 00:00:00" like string should be converted at td CLI side.
  • It's better that scheduled_time is changed as optional argument I think. Actually scheduled_time is optional in td-client-ruby.

connector:* normalize connector config and show warning messages if users use old config specification

Data connector can acquire 3 versions of config specifications from td CLI because td API server normalizes all of them. But we should move server-side normalization code to td CLI side. td CLI should send latest (3rd) config only to td API server. And td CLI can show warning messages if users still use old versioned config spec.

The following is 3 types config specification:

  • 1st version of config specification: doesn't include in key like:
type: s3
bucket: my_bucket
path_prefix: path/to/prefix
...
  • 2nd version: includes config key like:
config:
  in:
    type: s3
    bucket: my_bucket
    path_prefix: path/to/prefix
...
  • #79 3rd version: Embulk config specification like:
in:
  type: s3
  bucket: my_bucket
  path_prefix: path/to/prefix
...

Can't update rubocop ver0.49 because of the parallel gem conflict

Please update the dependent version.

$ bundle update rubocop
Fetching gem metadata from https://rubygems.org/..............
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Fetching gem metadata from https://rubygems.org/...........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies....
Bundler could not find compatible versions for gem "parallel":
  In snapshot (Gemfile.lock):
    parallel (= 1.8.0)

  In Gemfile:
    rubocop (~> 0.49.1) was resolved to 0.49.1, which depends on
      parallel (~> 1.10)

    td was resolved to 0.15.2, which depends on
      parallel (~> 1.8.0)

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

feature request: labeld JSON format

I'd like to use JSON objects with field names like this:

[
  {"key1": "value", "key2" : "value2"},
  ...
]

However, td query --format json does not include field names; instead, it returns the following format:

[
  [ "value1", "value2", ... ],
   ...
]

How about labeld JSON format?

Connector show "can't convert symbol to integer" when config file was failed to guess

If given YAML file is broken, it should weird stack trace as below.

$ td connector:guess seed.yml -o test.yml
Error TypeError: backtrace:
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:23:in `[]'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:23:in `block in from_hash'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:22:in `map'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:22:in `from_hash'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:24:in `block in from_hash'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:22:in `map'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:22:in `from_hash'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-client-0.8.69/lib/td/client/api/to_hash_struct.rb:17:in `from_json'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/lib/td/command/connector.rb:266:in `prepare_bulkload_job_config'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/lib/td/command/connector.rb:29:in `connector_guess'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/lib/td/command/list.rb:163:in `call'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/lib/td/command/list.rb:163:in `block in get_method'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/lib/td/command/runner.rb:170:in `call'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/lib/td/command/runner.rb:170:in `run'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/td-0.11.9/bin/td:29:in `<top (required)>'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/bin/td:23:in `load'
  /Users/takahashi/.rbenv/versions/2.0.0-p481/bin/td:23:in `<main>'

Error: no implicit conversion of Symbol into Integer

td command should show proper error message.

Add resuming functionality to job:result

An user had a problem to download a large result file (over 200MB) and convert it to TSV (this size could be over 2GB) using following command:

$ td job 5632435 -o /Users/anwarelbizanti/Downloads/results.tsv -f tsv

But it failed due because of network problem. Idea here is that job:result resumes downloading if it failed because of network problem.
I guess that you can add some code to td-client-ruby. If downloading failed due to network error, it can send GET request again with Range header.

Priority options for Bulk Import Perform command: td import:perform

For example -p / --priority as in:

λ ~SW/source/td-api/ PLT-1951* td_stag import:perform
usage:
  $ td import:perform <name>

example:
  $ td import:perform logs_201201

description:
  Start to validate and convert uploaded files

options:
  -w, --wait                       wait for finishing the job
  -f, --force                      force start performing
  -p, --priority                   job priority: -2: very low, -1: low, 0: normal, 1: high, +1 very high

connector:guess: showing guessed result to stdout

connector:guess is best-effort. Users need to confirm whether the guessed result is correct or not.
Embulk itself shows guessed result to stdout so that users can confirm the result immediately.
Why don't you change connector:guess to also show guessed yaml file to stdout?

Relax parallel gem dependency

Do you have any plan to relax or update dependency restriction of parallel gem?

The latest td gem (0.10.77) has "parallel", "~> 0.5.19" dependency, which prevents our app updating from 0.10.61 to the latest. parallel 0.6.x is there for 6 months --- I'd imagine it's matured and stable enough. http://rubygems.org/gems/parallel/versions

Thanks.

Show when a table was last updated

We want to monitor that data actually is received at Treasure Data. One way could be to use td to tell us when the tables data was last updated. Another could be to expose it via the TD API, either way would be great for us.

Upgrade rubyzip dependency

Current version of rubyzip (~> 1.1.7) is vulnerable:

Updated ruby-advisory-db
ruby-advisory-db: 283 advisories
Name: rubyzip
Version: 1.1.7
Advisory: CVE-2017-5946
Criticality: Unknown
URL: https://github.com/rubyzip/rubyzip/issues/315
Title: Directory traversal vulnerability in rubyzip
Solution: upgrade to >= 1.2.1

Vulnerabilities found!

Would be good to have a new released version of gem with the dependency updated.

Writing a status file to make it easy to integrate with workflow/ETL engines

When I run td command as a part of ETL workflow, I need to parse stdout message to know error or status. But it is hard and fragile. If the command writes the result into a file, integrating to ETL tools becomes much easier. And because ETL tools use shell script, the file format should be shell-friendly format.
For example,

td query ... --status status.sh  # writes status.sh
source status.sh
echo $job_id

td job:show -w $job_id --status status.sh
source status.sh
echo $elapsed
echo $status

td import:show ... --status status.sh
source status.sh
echo $valid_records
echo $error_records

trd query shows the backtrace

$ ruby bin/trd query "SELECT * FROM acccess" myapp
error ArgumentError: backtrace:
  /Users/kzk/repos/trd/lib/trd/api.rb:218:in `initialize'
  /Users/kzk/repos/trd/lib/trd/api.rb:103:in `new'
  /Users/kzk/repos/trd/lib/trd/api.rb:103:in `query'
  /Users/kzk/repos/trd/lib/trd/command/query.rb:16:in `query'
  /Users/kzk/repos/trd/lib/trd/command/trd.rb:71:in `call'
  /Users/kzk/repos/trd/lib/trd/command/trd.rb:71:in `<top (required)>'
  <internal:lib/rubygems/custom_require>:29:in `require'
  <internal:lib/rubygems/custom_require>:29:in `require'
  bin/trd:6:in `<main>'

wrong number of arguments (3 for 4)

td should respect $XDG_CONFIG_HOME

By default, put configuration in ~/.config/td/ instead of in ~/.td.

If the environment variable XDG_CONFIG_HOME is set, use $XDG_CONFIG_HOME/td/ for configuration.

This is part of the XDG Base Directory Specification which is a relatively new standard but has a lot of momentum. Even git implements it.

https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html#variables
http://unix.stackexchange.com/a/38871
https://github.com/git/git/blob/master/Documentation/git-config.txt#L241

td doesn't work with msgpack 0.5.5 on OS X

Let's remove 0.5.5 from dependency list.

% gem list msgpack

*** LOCAL GEMS ***

msgpack (0.5.5)
% td -c ~/.td/td_myui_development.conf job:show 104209
JobID       : 104209
Status      : success
Type        : hive
Database    : movielens1m
Priority    : NORMAL
Retry limit : 0
Output      :
Query       : select v from training_fm limit 10
CPU time    : 2s 740ms
Result size : 249 B
Result      :
WARNING: the job result is being downloaded...          Error RuntimeError: backtrace:
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api/job.rb:218:in `feed_each'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api/job.rb:218:in `block (2 levels) in job_result_each_with_compr_size'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:276:in `call'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:276:in `block in read_body'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/protocol.rb:411:in `call_block'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/protocol.rb:402:in `<<'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/protocol.rb:106:in `read'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http/response.rb:290:in `block in read_body_0'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http/response.rb:250:in `inflater'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http/response.rb:280:in `read_body_0'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http/response.rb:201:in `read_body'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:274:in `read_body'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:289:in `each_fragment'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api/job.rb:217:in `block in job_result_each_with_compr_size'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:337:in `call'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:337:in `block in do_get'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1421:in `block (2 levels) in transport_request'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http/response.rb:162:in `reading_body'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1420:in `block in transport_request'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1411:in `catch'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1411:in `transport_request'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1384:in `request'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1377:in `block in request'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:853:in `start'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/2.2.0/net/http.rb:1375:in `request'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:335:in `do_get'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:298:in `block in get'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:557:in `guard_no_sslv3'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api.rb:297:in `get'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/api/job.rb:204:in `job_result_each_with_compr_size'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client.rb:238:in `job_result_each_with_compr_size'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-client-0.8.75/lib/td/client/model.rb:497:in `result_each_with_compr_size'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/job.rb:457:in `render_result'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/job.rb:290:in `show_result'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/job.rb:263:in `show_result_with_retry'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/job.rb:201:in `get_and_show_result'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/job.rb:136:in `job_show'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/list.rb:171:in `call'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/list.rb:171:in `block in get_method'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/runner.rb:170:in `call'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/lib/td/command/runner.rb:170:in `run'
  /Users/nahi/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/td-0.12.0/bin/td:29:in `<top (required)>'
  /Users/nahi/.rbenv/versions/2.2.3/bin/td:23:in `load'
  /Users/nahi/.rbenv/versions/2.2.3/bin/td:23:in `<main>'

Error: can't modify frozen String

td import:auto --auto-create option can't create own session_name.

I used "td import:auto" command like below command,
$ td import:auto mysess test.csv --auto-create sampledb.sampletable ~
(I didn't create mysess: bulk_session. I created only sampledb, sampletable.)
Then, it shows the error like below.
Maybe, "--auto-create" option can't use with session name

------error------
td import:auto mysess ./test.csv --auto-create sampledb.sampletable --column-header -o ./ --time-value 1378439234
java version "1.6.0_51"
Java(TM) SE Runtime Environment (build 1.6.0_51-b11-456-11M4508)
Java HotSpot(TM) 64-Bit Server VM (build 20.51-b01-456, mixed mode)
Create sampledb_sampletable_2013_09_06_1378446034 bulk_import session
Uploading prepared files
Session name : sampledb_sampletable_2013_09_06_1378446034
File : mysess (0 bytes)
File : ./test.csv (59645903 bytes)

connector: remove unnecessary `config` key

connector commands (like connector:guess) requires config key. But it is unnecessary.
On the other hand, embulk will add multi-job feature (https://github.com/embulk/embulk/issues # 167). Once it is implemented to embulk, configuration file will be more complex. In this case, if both embulk and td CLI use the same config syntax, we can use the same document for users (although it is not the best solution, it's good for customer support if there is some documents at embulk.org). If only td commands need config key, we can't use the same document.

connector:history raises TypeError when it reads connector jobs that are 'queued' status.

When connector:history reads 'queued' connector jobs, it cannot convert the nil start_at into Time object and raises TypeError like following:

Error TypeError: backtrace:
  /Users/muga/works/workspace/td/lib/td/command/connector.rb:249:in `at'
  /Users/muga/works/workspace/td/lib/td/command/connector.rb:249:in `block in connector_history'
  /Users/muga/works/workspace/td/lib/td/command/connector.rb:239:in `map'
  /Users/muga/works/workspace/td/lib/td/command/connector.rb:239:in `connector_history'
  /Users/muga/works/workspace/td/lib/td/command/list.rb:171:in `call'
  /Users/muga/works/workspace/td/lib/td/command/list.rb:171:in `block in get_method'
  /Users/muga/works/workspace/td/lib/td/command/runner.rb:170:in `call'
  /Users/muga/works/workspace/td/lib/td/command/runner.rb:170:in `run'
  /Users/muga/.rbenv/versions/2.0.0-p643/lib/ruby/gems/2.0.0/gems/td-0.11.14/bin/td:29:in `<top (required)>'
  /Users/muga/.rbenv/versions/2.0.0-p643/bin/td:23:in `load'
  /Users/muga/.rbenv/versions/2.0.0-p643/bin/td:23:in `<main>'

Error: can't convert nil into an exact number

I think that the retrievals of start_at and end_at should be fixed.
https://github.com/treasure-data/td/blob/master/lib/td/command/connector.rb#L248-L249

trd drop-database ends with backtrace

$ ruby bin/trd drop-database 
error NoMethodError: backtrace:
  /Users/kzk/repos/trd/lib/trd/command/common.rb:27:in `cmd_opt'
  /Users/kzk/repos/trd/lib/trd/command/database.rb:24:in `drop_database'
  /Users/kzk/repos/trd/lib/trd/command/trd.rb:71:in `call'
  /Users/kzk/repos/trd/lib/trd/command/trd.rb:71
  bin/trd:6:in `require'
  bin/trd:6

private method `split' called for nil:NilClass

TD command to create new table

[2016-06-24 09:44:43] ERROR: TD Failed![/usr/bin/td -r -c /home/user001/.td/td.conf table:create table_test:/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/td-0.12.0/lib/td/command/runner.rb:194:in rescue in run': undefined methodsplit' for nil:NilClass (NoMethodError)
from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/td-0.12.0/lib/td/command/runner.rb:165:in run' from /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/td-0.12.0/bin/td:29:in<top (required)>'
from /usr/lib64/fluent/ruby/bin/td:23:in load' from /usr/lib64/fluent/ruby/bin/td:23:in

'
**** Please help me check this problem! Thank you

Helpers.format_with_delimiter modifies given parameter with `gsub!`

TreasureData::Helpers.format_with_delimiter uses `gsub!` to their argument
the argument (in our case, `number_raw`) will be rewritten by them
To avoid that behavior, pass `number_raw.dup` instead of `number_raw`

# NOTE: TreasureData::Helpers.format_with_delimiter uses `gsub!` to their argument
# the argument (in our case, `number_raw`) will be rewritten by them
# To avoid that behavior, pass `number_raw.dup` instead of `number_raw`

td command should be installed without td-agent.

On Linux, we provide td-agent for td command.
But this is bad because td-agent is large package and its release cycle doesn't depend on td command.
Even if td command supports new features, to deliver td command is delayed.
So providing single package without td-agent is better.

Add table:rename

Users sometimes want to change a table name like:

$ td table:rename <db> <table> <new_name>
$ td table:rename --overwrite <db> <table> <new_name>

--overwrite option allows users to replace (overwrite) existing table.

connector_config_normalizer ignores `filters:` config

When I execute connector:preview command, rename filter doesn't work even though rename filter config was declared in the config.yml. The following workaround works for me but I don't know that it's good enough or not. Please take a look.

$ git diff lib/td/connector_config_normalizer.rb
diff --git a/lib/td/connector_config_normalizer.rb b/lib/td/connector_config_normalizer.rb
index e2531bb..877228f 100644
--- a/lib/td/connector_config_normalizer.rb
+++ b/lib/td/connector_config_normalizer.rb
@@ -8,9 +8,10 @@ module TreasureData
       case
       when @config['in']
         {
-          'in'   => @config['in'],
-          'out'  => @config['out']  || {},
-          'exec' => @config['exec'] || {}
+          'in'      => @config['in'],
+          'filters' => @config['filters'],
+          'out'     => @config['out']  || {},
+          'exec'    => @config['exec'] || {}
         }
       when @config['config']
         if @config.size != 1

connector:show crashes

bundle exec bin/td connector:show connector1
Name     : connector1
Cron     : 0 * * * *
Timezone : UTC
Delay    : 0
Database :
Table    :
Config
Error NoMethodError: backtrace:
  /Users/nahi/git/td/lib/td/command/connector.rb:293:in `dump_connector_session'
  /Users/nahi/git/td/lib/td/command/connector.rb:184:in `connector_show'
  /Users/nahi/git/td/lib/td/command/list.rb:171:in `call'
  /Users/nahi/git/td/lib/td/command/list.rb:171:in `block in get_method'
  /Users/nahi/git/td/lib/td/command/runner.rb:170:in `call'
  /Users/nahi/git/td/lib/td/command/runner.rb:170:in `run'
  bin/td:29:in `<main>'

Error: undefined method `config' for #<Hash:0x007f9d41c79288>

connector:create also crashes at the end of command execution.

Patch:

diff --git a/lib/td/command/connector.rb b/lib/td/command/connector.rb
index 3110804..4cef6b6 100644
--- a/lib/td/command/connector.rb
+++ b/lib/td/command/connector.rb
@@ -283,14 +283,14 @@ private
   end

   def dump_connector_session(session)
-    $stdout.puts "Name     : #{session.name}"
-    $stdout.puts "Cron     : #{session.cron}"
-    $stdout.puts "Timezone : #{session.timezone}"
-    $stdout.puts "Delay    : #{session.delay}"
-    $stdout.puts "Database : #{session.database}"
-    $stdout.puts "Table    : #{session.table}"
+    $stdout.puts "Name     : #{session["name"]}"
+    $stdout.puts "Cron     : #{session["cron"]}"
+    $stdout.puts "Timezone : #{session["timezone"]}"
+    $stdout.puts "Delay    : #{session["delay"]}"
+    $stdout.puts "Database : #{session["database"]}"
+    $stdout.puts "Table    : #{session["table"]}"
     $stdout.puts "Config"
-    $stdout.puts YAML.dump(session.config)
+    $stdout.puts YAML.dump(session["config"])
   end

   def wait_connector_job(client, job_id, exclude)

Change default access to over SSL?

In some environments, user accesses TD behind the proxy which doesn't support PUT method.
And someone says "td command should access TD over SSL by default!"
Default SSL access satisfies these requirements.

Changing default access to over SSL may cause the performance issue.
We should check some existence commands.

In addition, Option candidates to disable SSL:

  • --disable-ssl
  • --insecure (from google app engine)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.