Comments (4)
@joker1007 what you say about this?
from fluent-plugin-bigquery.
I think that this feature is substitutable by other filter plugin.
For example, fluent-plugin-record-transformer or fluent-plugin-record-reformer or fluent-plugin-record-modifier.
If other plugin can realize your demand, it is better to use the other plugin.
Because record transforming is not main purpose of this plugin.
How about this?
from fluent-plugin-bigquery.
The problem is that if use another plugin we need to duplicate the field definition on multiple places. Practically I need to hardcode all the fields that exists so a plugin can detect what's missing. And that needs constant maintenance. It was extremely handful and great that here with auto table schema we should not describe the fields, only some the meta
field.
On top of the above reasons, it's a very known feature of hybrid columnar storage system to have this functionality in place, place rest of the fields as json in a data/meta column.
Does this change your thinking? Is this something complex to achieve?
from fluent-plugin-bigquery.
I see your probrem.
Please give me time to think about it.
from fluent-plugin-bigquery.
Related Issues (20)
- Dataset not found (asia-northeast1)
- Error streaming to GBQ HOT 1
- Date partitioned table error HOT 1
- Gem googleauth breaks support for Ruby 2.2 HOT 1
- Column based partitioning cause BadRequest HOT 2
- requirePartitionFilter of TimePartitioning is deprecated HOT 1
- Can't push legacy buffer files automatic after fluentd restart HOT 1
- I can't output data to Bigqueiry when the contents cotains Chinese
- How can I output the entire json object as a string to BigQuery HOT 7
- Cannot see error message for insert
- BigQuery Plugin Not Recognized
- Bug in retry logic ? HOT 1
- Can use strftime-formatting with fluentd 1.x.x? HOT 3
- "wrong number of arguments (given 5, expected 3..4)" while insert into Bigquery HOT 1
- UnRetryableError bigquery_load HOT 5
- Invalid field type: GEOGRAPHY HOT 2
- loss of data
- Too short time to wait for creating tables
- JSON type support HOT 4
- [Suggest Deprecation] Deprecate auto_create_table
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluent-plugin-bigquery.