catprintlabs / bq_stream

Streaming data to BigQuery

BqStream

BqStream can be used Google's BigQuery to send selected model attribute values when records are created, updated and destroyed. It was created to help better analyze ever-changing records within a project's database.

Installation

Add the following lines to your application's Gemfile:

gem 'bq_stream'

For the servers that dequeue items need the following

gem 'bigquery', '~>0.9.0'
gem 'bulk_insert'

And then execute:

$ bundle

Or install it yourself as:

$ gem install bq_stream

Usage

TODO: Write usage instructions here

Development

After checking out the repo, run bin/setup to install dependencies.

You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install.

Testing

An RSpec test suite is available to ensure proper functionality. Run rake spec to run the tests.

There is also a test with a live connection to BigQuery that be called by using:

$ rspec spec/bq_example.rb

Before live testing you should be sure to have created an .env file in the root directory with the following Google BigQuery credentials:

• Client Id
• Service Email
• KEY
• Project Id
• Dataset

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/catprintlabs/bq_stream. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

The gem is available as open source under the terms of the MIT License.

bq_stream's Issues

undefined method `[]' for nil:NilClass

Message from application: undefined method `[]' for nil:NilClass (NoMethodError)
  /var/www/v4/shared/bundle/ruby/2.3.0/bundler/gems/bq_stream-9374c2955b8b/lib/bq_stream/bq_stream.rb:61:in `block in initialize_old_records'
  /var/www/v4/shared/bundle/ruby/2.3.0/bundler/gems/bq_stream-9374c2955b8b/lib/bq_stream/bq_stream.rb:58:in `each'
  /var/www/v4/shared/bundle/ruby/2.3.0/bundler/gems/bq_stream-9374c2955b8b/lib/bq_stream/bq_stream.rb:58:in `initialize_old_records'
  /var/www/v4/shared/bundle/ruby/2.3.0/bundler/gems/bq_stream-9374c2955b8b/lib/bq_stream/bq_stream.rb:49:in `config_initialized'
  /var/www/v4/shared/bundle/ruby/2.3.0/bundler/gems/bq_stream-9374c2955b8b/lib/bq_stream/configuration.rb:7:in `configuration'
  /var/www/v4/releases/20170426185235/config/initializers/bq_stream.rb:4:in `<top (required)>'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/activesupport-4.1.16/lib/active_support/dependencies.rb:241:in `load'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/activesupport-4.1.16/lib/active_support/dependencies.rb:241:in `block in load'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/activesupport-4.1.16/lib/active_support/dependencies.rb:232:in `load_dependency'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/activesupport-4.1.16/lib/active_support/dependencies.rb:241:in `load'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/engine.rb:648:in `block in load_config_initializer'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/activesupport-4.1.16/lib/active_support/notifications.rb:161:in `instrument'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/engine.rb:647:in `load_config_initializer'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/engine.rb:612:in `block (2 levels) in <class:Engine>'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/engine.rb:611:in `each'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/engine.rb:611:in `block in <class:Engine>'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/initializable.rb:30:in `instance_exec'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/initializable.rb:30:in `run'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/initializable.rb:55:in `block in run_initializers'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:228:in `block in tsort_each'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:431:in `each_strongly_connected_component_from'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/initializable.rb:44:in `each'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/initializable.rb:44:in `tsort_each_child'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:415:in `call'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:415:in `each_strongly_connected_component_from'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:349:in `block in each_strongly_connected_component'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:347:in `each'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:347:in `call'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:347:in `each_strongly_connected_component'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:226:in `tsort_each'
  /home/catprint/.rbenv/versions/2.3.1/lib/ruby/2.3.0/tsort.rb:205:in `tsort_each'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/initializable.rb:54:in `run_initializers'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/application.rb:300:in `initialize!'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/railtie.rb:194:in `public_send'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/railties-4.1.16/lib/rails/railtie.rb:194:in `method_missing'
  /var/www/v4/releases/20170426185235/config/environment.rb:5:in `<top (required)>'
  config.ru:3:in `require'
  config.ru:3:in `block in <main>'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/rack-1.5.5/lib/rack/builder.rb:55:in `instance_eval'
  /var/www/v4/shared/bundle/ruby/2.3.0/gems/rack-1.5.5/lib/rack/builder.rb:55:in `initialize'
  config.ru:1:in `new'
  config.ru:1:in `<main>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:110:in `eval'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:110:in `preload_app'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:156:in `<module:App>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
  /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<main>'