How can I output the entire json object as a string to BigQuery about fluent-plugin-bigquery

I really appreciate you help.
It works now. I have replaced all the double quotes in the filed and then it works.

The wired thing is still showing the same error log, however, it has been inserted into BigQuery successfully.

[joker1007]

Probably, you can use record_transformer plugin.

For example, record_transformer converts entire record to {"payload": <jsonized record>} and fluent-plugin-bigquery inserts to table that has only payload:STRING column.

<filter sample.*>
  @type record_transformer
  enable_ruby
  <record>
    payload ${record.to_json}
  </record>
</filter>

<match sample.*>
  @type bigquery_insert

  auth_method private_key   # default
  email [e-mail]
  private_key_path /home/username/.keys/00000000000000000000000000000000-privatekey.p12
  # private_key_passphrase notasecret # default

  project yourproject_id
  dataset yourdataset_id
  table   tablename

  schema [
    {"name": "payload", "type": "STRING"},
  ]
</match>

(This configuration is just sample)

Thanks for the quick response. I have tried this and it shows error as below

insert errors project_id="xxxx" dataset="xxx" table="edge_server" insert_errors="[#<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x000055814a749778 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x000055814a748940 @debug_info="", @location="topic", @message="no such field.", @Reason="invalid">], @Index=0>,

My json object is a nested json, so it seems the output plugin just parse the nested json and then try to load each field to Bigquery.

My config looks like below

<source>
    @type kafka_group
    brokers "kafka-1:9092,kafka-2:9092,kafka-3:9092"
    consumer_group "fluentd"
    topics "edge"
    format "text"
  </source>
  <filter edge>
    @type record_transformer
    enable_ruby
    renew_record true
    keep_keys payload
    <record>
      payload ${record.to_json}
    </record>
  </filter>
  <match edge>
    @type bigquery_insert
    auth_method compute_engine
    project "xxxx"
    dataset "xxxx"
    table "edge_server_aaa"
    fetch_schema true
    <buffer>
      @type "file"
      path "/data/bigquery.edge.buffer"
      flush_interval 0.1
      queue_length_limit 4096
      flush_thread_count 16
      chunk_records_limit 1000
    </buffer>
  </match>

My json data looks like
{ {\\\"@timestamp\\\":\\\"2020-02-25T13:04:49.853Z\\\",\\\"@metadata\\\":{\\\"beat\\\":\\\"filebeat\\\",\\\"type\\\":\\\"doc\\\",\\\"version\\\":\\\"6.7.2\\\",\\\"topic\\\":\\\"edge-server\\\"} The json schema is keep changing, that's why I want to load the entire json as a string in 1 column.

Wondering if this is related how does the plugin generate the rows in multi_json

[joker1007]

Does edge_server_aaa has payload column?
It seems not to have the column for me.

Or, extra fields may cause this error.
How about using keep_keys or remove_keys at record_transformer configuration.

Yes, we use the record_transformer configuration to add the payload column. I mean it may be due to how does BigQuery load the nested JSON. It is trying to load ['payload]['xxx'] into BigQuery which results in the error. As the configuration listed

  <filter edge>
    @type record_transformer
    enable_ruby
    renew_record true
    keep_keys payload
    <record>
      payload ${record.to_json}
    </record>
  </filter>

I renew the entire record as one field 'payload' to include the entire JSON object. Much appreciate for your help. It always shows an error no such field' which means it tries to load each field into BigQuery instead of loading the payload` as a single column.

[joker1007]

Hmm...
I'm very sorry, I can't guess what causes the error.
If the payload column becomes a string value, bigquery plugin should be able to send it properly.
I suggest stdout filter in order to see actual records sent by bigquery plugin.
It's my last resort.