valmi-io / valmi-activation

⚡ valmi.io reverse ETL (data activation) is the open source ( OSS ) data activation platform to load data from warehouses into Webhooks and SaaS tools like Klaviyo, Facebook Ads, Salesforce, Braze etc. Valmi.io Customer Data Platform (CDP) helps track and ingest user activity events from websites, shopify, serverside events. https://cloud.valmi.io

• Trusted By

  

• We have launched EVENT Streams (powered by Jitsu). Try free @ cloud.valmi.io both User Activity Events Tracking and Data Activation

• See it in Action

  Quickly witness a One Million (1M) Rows Sync at Live Sync.

  Events	Shopify App

Website · Slack Community · Documentation · Blog . Changelog . Roadmap

valmi.io Customer Data Platform (CDP) provides open-source Reverse ETL (data activation) platform to load data from warehouses into SaaS platforms such as Advertising (Facebook Ads, Google Ads), Marketing (Klaviyo, Braze), CRM (Salesforce, Hubspot), Accounting (Xero, Netsuite), Customer Support (Zendesk), Messaging (Slack), Webhook Apis etc.

✨ Checkout docs at - https://www.valmi.io/docs/

✨ Read the variety of usecases valmi.io can enable for your organization - https://www.valmi.io/blog/

3 ways to start with valmi.io

👉 1. Use it in the cloud

• You can immediately create a sync at https://cloud.valmi.io.

• Watch a demo video to create a sync.

  
  

  

👉 2. Run it locally or in your Cloud

• Prerequisites: valmi.io relies heavily on both Docker and Docker-compose. Install Docker Desktop to get both docker and docker-compose.

• Clone this repo and recursively clone submodules.

git clone [email protected]:valmi-io/valmi-activation.git
cd valmi-activation
git submodule update --init --recursive

• Setup the environment.

cp .env-example .env

cd valmi-app-backend
cp .env-example .env

cd ../valmi-app
`For macos`
cp .env-example.macos .env
`For linux`
cp .env-example.linux .env

• Intermediate storage, We are adding support for object stores like S3, GCS. Until then, Local storage is used.

sudo mkdir -p /tmp/shared_dir/intermediate_store
sudo chmod -R 777 /tmp/shared_dir/intermediate_store

• Launch the reverse-etl service.

./valmi prod --with-jitsu

• To stop the service, run the following.

./valmi prod --with-jitsu down

• Please wait for about 2 minutes before you access the service, since valmi-app builds an optimized compiled version of the app UI. To access the service, please check the 'Accessing the service' section for local deployments.

👉 3. Develop a connector locally to customize valmi.io as per your needs. You can just contact us too.

• Clone, setup environment variables and create intermediate storage (see above section).
• Create a new connector (Optional).

# Copy code base from any existing connectors from valmi-integrations folder (ex. destination-webhook)

cd valmi-integrations/connectors
cp -r destination-webhook destination-awesome_connector

# Make necessary changes and build the connector
cd destination-awesome_connector
make build_docker version=latest

# Add the new connector information to "valmi-app-backend/init_db/connector_def.json"

• Run the service.

./valmi dev --with-jitsu

• To access the service, please check the 'Accessing the service' section for local deployments.

• To Stop the service, run the following.

./valmi dev --with-jitsu down

• Accessing the service for local deployments

  Syncs http://localhost:3000	Sync Runs http://localhost:3000

  UI Backend Server API http://localhost:4000/api/docs	Activation Server API http://localhost:8000/docs

valmi-activation's Issues

bug: If one connector fails, other connector gets stuck.

Story: Shopify Event Tracking App for valmi.io

Story: Gong Connector

• Finalize Connector with just API keys - not Oauth

Story - OAuth configurations

Story: "valmi figures" for Customer-360

Story: CI/CD pipeline

Connectors:

• Docker build for linux/amd64 and linux/arm64 platforms
• Add commands for building and pushing dockers images for connectors in Makefile

Valmi Activation:

• Makefile rules for building valmi-repo, valmi-dagit and valmi-dagster-daemon

Valmi App:

• Makefile rules

Valmi App Backend:

• Makefile rules

---

• Docker compose files for prod and dev environments

Story: Monitoring stack for valmi-activation

• view Logs, Traces and Metrics in New Relic

Story: Logs collection for the UI display

• Collect rejected samples from sync runs and store them in intermediate store
• API call to retrieve rejected samples

Story: On update mapping in Sync UI, trigger sync full_refresh

Bug: Metrics are not finalised when a sync is aborted. The UI showed empty metrics as expected.

Hubspot upsert mode needs query for existence first and then insert/update accordingly. Also needs support for just update

valmi-activation Repository Shutdown Exception

05/25/2023 03:21:38 PM INFO JobCreator requests::execute >>> {"query": "mutation ($repositoryLocationName: String!) {\n shutdownRepositoryLocation(repositoryLocationName: $repositoryLocationName) {\n __typename\n ... on PythonError {\n message\n }\n ... on RepositoryLocationNotFound {\n message\n }\n }\n}", "variables": {"repositoryLocationName": "valmi-repo"}}
05/25/2023 03:21:38 PM DEBUG JobCreator connectionpool::_new_conn Starting new HTTP connection (1): dagit:3000
05/25/2023 03:21:38 PM ERROR SyncRunner run_manager::_run Error while handling syncs in run manager
Traceback (most recent call last):
File "/opt/valmi-activation/lib/python3.10/site-packages/dagster_graphql/client/client.py", line 98, in _execute
return self._client.execute(gql(query), variable_values=variables)
File "/opt/valmi-activation/lib/python3.10/site-packages/gql/client.py", line 403, in execute
return self.execute_sync(
File "/opt/valmi-activation/lib/python3.10/site-packages/gql/client.py", line 220, in execute_sync
with self as session:
File "/opt/valmi-activation/lib/python3.10/site-packages/gql/client.py", line 702, in enter
return self.connect_sync()
File "/opt/valmi-activation/lib/python3.10/site-packages/gql/client.py", line 678, in connect_sync
self.transport.connect()
File "/opt/valmi-activation/lib/python3.10/site-packages/gql/transport/requests.py", line 106, in connect
raise TransportAlreadyConnected("Transport is already connected")
gql.transport.exceptions.TransportAlreadyConnected: Transport is already connected

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/workspace/src/./orchestrator/run_manager.py", line 106, in _run
dagster_run_id = self.dc.submit_job_execution(
File "/opt/valmi-activation/lib/python3.10/site-packages/dagster_graphql/client/client.py", line 319, in submit_job_execution
return self._core_submit_execution(
File "/opt/valmi-activation/lib/python3.10/site-packages/dagster_graphql/client/client.py", line 149, in _core_submit_execution
pipeline_info_lst = self._get_repo_locations_and_names_with_pipeline(pipeline_name)
File "/opt/valmi-activation/lib/python3.10/site-packages/dagster_graphql/client/client.py", line 106, in _get_repo_locations_and_names_with_pipeline
res_data = self._execute(CLIENT_GET_REPO_LOCATIONS_NAMES_AND_PIPELINES_QUERY)
File "/opt/valmi-activation/lib/python3.10/site-packages/dagster_graphql/client/client.py", line 100, in _execute
raise DagsterGraphQLClientError(
dagster_graphql.client.utils.DagsterGraphQLClientError: Exception occured during execution of query

query {
repositoriesOrError {
__typename
... on RepositoryConnection {
nodes {
name
location {
name
}
pipelines {
name
}
}
}
... on PythonError {
message
}
}
}

with variables
None

05/25/2023 03:21:38 PM ERROR SyncRunner retry_decorators::wrapper Failure _run due to error: Exception occured during execution of query

query {
repositoriesOrError {
__typename
... on RepositoryConnection {
nodes {
name
location {
name
}
pipelines {
name
}
}
}
... on PythonError {
message
}
}
}

with variables
None

Story: Google Ads Connector

• Creating separate offline jobs for delete and create actions
• Implementing back-off for rate limit error
• Documentation

Bug: Long live access token not retrieving from fb when trying to authenticate while configuring facebook ads

use dummy keys all across.

Story: Jitsu Integration

Doc: what will happend on click on deployment link or looks like there is no deployment section

Story: Edit Sync API

Run not found error

Connector: Gong - Use Batching to Upsert and update records and use /v2/crm/request-status for success. Now using one request for each record.

Bug: Source connector got stopped on ./valmi prod down and the resulting UI shows Success for intermediate results.

Syncs Api response.

[
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "7736700b-9609-4cdc-953f-4cae79dc8e77",
        "run_at": "2023-08-12T12:05:42.872152",
        "status": "running",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "succeeded": {
                    "value": 92000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 66300,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 306
                            }
                        },
                        "emitted_at": 1691842378000
                    }
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 300
                                },
                                "chunk_id": 219,
                                "finished": false,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                }
            }
        },
        "run_time_args": null,
        "run_end_at": null
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "b037da8b-833a-42f7-97c5-c323e4b3701a",
        "run_at": "2023-08-12T11:58:30.637649",
        "status": "stopped",
        "metrics": {
            "src": {
                "succeeded": {
                    "value": 90900,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                }
            },
            "dest": {
                "succeeded": {
                    "value": 52600,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 302
                            }
                        },
                        "emitted_at": 1691841862000
                    }
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 300
                                },
                                "chunk_id": 174,
                                "finished": false,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T12:05:40.844190"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "5d3706da-e213-4d27-91ab-c036916ece3a",
        "run_at": "2023-08-12T10:22:36.348108",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691839850000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T11:58:29.392701"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "4fc24fbd-965d-4fdf-91fc-ac035954f348",
        "run_at": "2023-08-12T08:47:15.037642",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691833978000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T10:22:35.073529"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "fd84da71-633f-42e2-9544-33bc947e16ba",
        "run_at": "2023-08-12T07:32:22.202988",
        "status": "stopped",
        "metrics": {
            "src": {
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691826490000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T08:47:13.480052"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "04aca962-a428-47c8-91ab-eae5a45ad394",
        "run_at": "2023-08-12T06:13:53.778872",
        "status": "stopped",
        "metrics": {
            "src": {
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691822039000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T07:32:20.962417"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "d9e8e850-d4a8-4093-b0df-f6f9951bc5fd",
        "run_at": "2023-08-12T04:44:42.612574",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691818126000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T06:13:51.787631"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "bfb1ad26-7478-443e-8bcd-03910403bab7",
        "run_at": "2023-08-12T03:11:57.037993",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691813318000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T04:44:40.819042"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "4f001824-d3a6-4bd2-8d21-c6ea2ebe2df5",
        "run_at": "2023-08-12T01:36:28.291449",
        "status": "stopped",
        "metrics": {
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691808098000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T03:11:55.166514"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "284de56d-2541-4a12-a19f-ed5a3d8dc12d",
        "run_at": "2023-08-12T00:05:37.547739",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691801646000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T01:36:26.813729"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "d36d1060-5806-4a3f-9fe0-f63e44ab1dad",
        "run_at": "2023-08-11T22:35:00.997702",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691796204000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-12T00:05:36.057233"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "14babb64-a8b4-4682-92c3-eac60f3d0c7c",
        "run_at": "2023-08-11T21:09:50.813134",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691790143000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T22:34:59.409940"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "b0193b98-d979-46ac-8123-a1e220e0e897",
        "run_at": "2023-08-11T19:35:19.264918",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691786181000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T21:09:48.764788"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "4fdca72e-6f0e-4de8-8765-70b15d8f8429",
        "run_at": "2023-08-11T18:00:08.094616",
        "status": "stopped",
        "metrics": {
            "src": {
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691780710000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T19:35:17.881522"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "11cc2feb-a063-416a-97eb-05cc3300f5d4",
        "run_at": "2023-08-11T16:36:24.129758",
        "status": "stopped",
        "metrics": {
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "src": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691773543000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T18:00:05.992098"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "1ed518d9-eefc-4aea-9539-93dcdeca4dc4",
        "run_at": "2023-08-11T15:08:04.803818",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691768969000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T16:36:22.249576"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "4ed455c3-fd85-454d-b643-e9b1b62f4995",
        "run_at": "2023-08-11T13:32:49.986874",
        "status": "stopped",
        "metrics": {
            "src": {
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691764637000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T15:08:03.285073"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "078a3be3-aab6-4e19-b2e1-c61a7cf782bb",
        "run_at": "2023-08-11T12:12:59.446682",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691757278000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T13:32:48.065335"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "93a14e7f-78e7-4fc7-945c-29796eb70554",
        "run_at": "2023-08-11T10:37:05.447218",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691754255000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T12:12:57.274907"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "67dea90b-8ce4-4f3b-9ebb-5756b590efdf",
        "run_at": "2023-08-11T09:01:08.931101",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691748500000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T10:37:04.009195"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "e90b950b-32c7-43ac-996e-201d42ca7140",
        "run_at": "2023-08-11T07:37:06.545194",
        "status": "stopped",
        "metrics": {
            "src": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691741271000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T09:01:07.277233"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "2f769413-2abd-4da5-8e8f-a25b9c73566e",
        "run_at": "2023-08-11T06:05:46.042595",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691737026000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T07:37:05.208067"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "a27707df-56a7-4102-af49-9998ee49fd20",
        "run_at": "2023-08-11T04:30:52.383389",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691732110000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T06:05:44.411593"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "9e845439-1b5f-4f7a-af36-2063c59c19cb",
        "run_at": "2023-08-11T02:55:18.467970",
        "status": "stopped",
        "metrics": {
            "src": {
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                },
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691726456000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T04:30:50.338333"
    },
    {
        "sync_id": "d69cf9f9-0e20-4e2c-a683-2649404f52ed",
        "run_id": "1dc717c9-661e-41f7-9ba2-31cc6c47a38d",
        "run_at": "2023-08-11T01:27:25.689896",
        "status": "stopped",
        "metrics": {
            "src": {
                "total$$0": {
                    "value": 1000000,
                    "display_order": 0
                },
                "valid$$0": {
                    "value": 1000000,
                    "display_order": 8
                },
                "succeeded": {
                    "value": 1000000,
                    "display_order": 20
                },
                "new$$0": {
                    "value": 1000000,
                    "display_order": 4
                }
            },
            "dest": {
                "succeeded": {
                    "value": 1000000,
                    "display_order": 0
                }
            }
        },
        "extra": {
            "src": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "chunk_id": 3334
                            }
                        },
                        "emitted_at": 1691719659000
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "dest": {
                "state": {
                    "state": {
                        "type": "STATE",
                        "state": {
                            "type": "STREAM",
                            "data": {
                                "records_delivered": {
                                    "upsert": 100
                                },
                                "chunk_id": 3333,
                                "finished": true,
                                "commit_state": true,
                                "commit_metric": true
                            }
                        }
                    }
                },
                "status": {
                    "status": "success"
                }
            },
            "run_manager": {
                "status": {
                    "status": "success",
                    "message": null
                }
            }
        },
        "run_time_args": null,
        "run_end_at": "2023-08-11T02:55:16.821350"
    }
]```

Story: Send certain failed/rejected records on the destination back to the source and they need to be retried on the subsequent syncs

Story: Inject Checkpoint state to connectors

#TODO:

• on connector launch, pass stored checkpoint state.

Bug: snowflake mirror mode duplicates issue.

Snowflake mirror causes metrics and delivery issues, when there are duplicates.
Reproduce.

1. Sync once
2. Add a duplicate key
3. Now sync.
4. All the metrics are garbled because of the duplicate key.
   Also check in upsert mode,

Bug: Production dockers are not working

Story: Google Sheets

• Change the code to use new discover & write functionality
• Verify Google App Application

Story: Android push notifications connector

• Finalise Connector
• Documentation

Story: Helm charts to deploy on k8s

bug: duplicate runs on making a sync active:inactive

Story: Separate Metric-Service into a different process : do it by passing a flag --metric_server embedded or --metric_server remote --metric_server_url http://xx/

Bug: Dagster terminate run is failing

When will it occur?

• Update the sync's status from "active" to "inactive," resulting in the termination of any currently running jobs associated with valmi-activation.
• The failure of the Dagster client to terminate the job creates an endless loop of termination calls.

Why is this happening?

• The default Dagster TerminateRunPolicy is set to SAFE_TERMINATE, causing failure when no container is available for termination.
• Containers are self-terminated by examining the flag in syn_runs.extra.run_manager.

Story - Show Sample data for reverse ETL syncs

• UI call to retrieve rejected samples & successful samples

Story: Ability to build models in the app?

• hi I wanted to start by thanking the team for this great piece of software
• I was wondering if there is any plan to build models in valmi, i.e., ala grouparoo/census
• intention would be to build groups/segments/audiences based off models which can be rETL'd and also shared across the org
  • eg: marketing build a segment for LTV>100, which can also be targetted by product for a different product experience
• thanks!

bug: making a sync inactive must first confirm abort from activation engine , then only set the app-backend to inactive. order matters

Story: Stripe Connector

• Finalise upserts and updates
• Customise backoff & retry behavior
• Change the code to use new discover & write functionality

Bug: Sync runs got stopped when switching from active to Inactive

Stripe Finalise these issues

#TODO:

• finalize upserts and updates

• Handle errors along with rate limits - customize back-off behavior code

Story: Hubspot Connector

• Finalize upserts and updates - Check for existence before creating
• Change the code to use new discover & write functionality

Story: Customer IO

Issues

• x out of n in batch mode rejections to be handled & counted as failed

Enhancements

• Change the code to use new discover & write functionality

bug: add privileged to docker- compose for docker-in-docker runs

bug: mount metrics db for prod docker compose on a volume

Task: CI/CD pipeline

• Remove .version files
• Add github actions

Story - [valmi-app] bugs

• Slack - oAuth2 redirecting to login page.
• Login - Not able to login from 1 million sync page.
• Live events - logs - json formatting.
• Sync runs - logs - json formatting.
• Icons - update icons for Activate , Streams & Destinations in events - connections page.
• Logout - Logs out - when clicked on the sidebar routes after logged in from 1 million sync page.

Support Templated messages for Slack

without jinja templates , slack destination will be useless. It also means extending ValmiSink to contain templated messages in addition to mapping.

Story: Documentation

Connectors

• Hubspot
• Customer.io
• Slack
• Stripe
• Facebook Ads

Introduction Topics

• Intro
• Concepts
• Architecture
• Deployment

Story: Facebook Ads

• Oauth Testing
• Verify Facebook App
• Change the code to use new discover & write functionality

Task: Sync runs api output change

Currently there is no order for displaying metrics in sync runs page
Need to add display_order for each metric

Old format:

{
  "src": {
    "total$$0": 4935,
    "new$$0": 4935,
    "success": 4935,
    "valid$$0": 4935
  },
  "dest": {
    "succeeded": 290
  }
}

New format:

{
  "src": {
    "total$$0": { "value": 4935, "display_order": 0},
    "new$$0":  { "value": 4935, "display_order": 4},
    "success":  { "value": 4935, "display_order": 16},
    "valid$$0":  { "value": 4935, "display_order": 8},
  },
  "dest": {
    "succeeded":  { "value": 290, "display_order": 0},
  }
}

Story: Start stop functionality for syncs

• Fast API end points in valmi activation
• Backend api points

Story: Slack Connector

• Template Messages
• Change the code to use new discover & write functionality
• Documentation

Task: Adding `run_end_at` column to `sync_runs` table