Data Type Change in Adapter Causes Data Storage Failure in InfluxDB about streampipes

[tenthe]

When data is written to the data lake, an entry is created in CouchDB with the schema information and the data. If a user adjusts the event schema of an adapter, this is also updated in CouchDB, but the old data remains unchanged in InfluxDB.

Problem

• Adding new properties or renaming a property works without issues.
• Changing the data type of a property causes a runtime error when writing the data, due to the column in the database having a different data type.

Potential Solutions

• Migrate old entries in the database to match the new schema.
• Adapt the schema internally to prevent the issue. This requires checking for pipelines that write data to the DataLake when editing the adapter.

Current Workaround for Users

• For now we will show a warning when the user changes the data type
• Change the name of the data type.
• Write data to a new measurement.
• Truncate the old data in the database.