Dynamic table name assign support about fluent-plugin-bigquery

[nagachika]

I think assigning a table name by record value has difficulties about validation (field existence, type, contained characters).
How about assign table name by fluent tag? In such a case, fluent-plugin-forest is useful. https://github.com/tagomoris/fluent-plugin-forest

[GLStephen]

I know that exploring a data/field driven approach to tables would help many use cases tremendously. It is worth exploring and is not quite the same as what is provided by plugin forest. For instance, in a multi-tenant system data may be ingested with a value indicating the customer. Moving that data into tables specifically for that customer is one way to reduce data usage in BigQuery.

BigQuery also supports creating tables from a schema table. A table that uses the schema of the named table, but is named by the passed in value. This is definitely worth exploring I'm just not yet knowledgeable enough with this plugin to start implementing it.

[joker1007]

Thank you for comments.
master branch of fluent-plugin-bigquery (released as v0.5.0.beta) has already supported this feature. Because fluentd-0.14.x can use any keys as buffer chunk key, and this plugin supports the feature.
But on fluentd-0.12, it is not easy to support dynamic name assigning.
If this feature is very important, Please use fluentd-0.14.x and v0.5.0.beta.