faros-ai / faros-community-edition Goto Github PK

View Code? Open in Web Editor NEW

403.0 15.0 55.0 12.66 MB

BI, API and Automation layer for your Engineering Operations data

Home Page: https://community.faros.ai

License: Apache License 2.0

Shell 7.81% Dockerfile 1.09% JavaScript 5.45% TypeScript 84.74% Makefile 0.63% HTML 0.28%

devops analytics metrics space dora business-intelligence cicd incident-management version-control tasks-management

faros-community-edition's People

Contributors

Stargazers

Watchers

faros-community-edition's Issues

Cant add my github repo to airbyte

I filled in my personal auth token and github repo to airbyte. But i get the error in the screenshot attached

Please how i can resolve this

Build to Deployment dbt transform example

Is your feature request related to a problem? Please describe.
Many CI/CD sources cannot distinguish a build pipeline from a deployment one. However, in practice it is possible to distinguish based on pipeline, build or step name.

Describe the solution you'd like
dbt transform with simple controls that will select the proper subset of builds that are in fact deployments, and convert them into deployments. It is acceptable if the original entity is still present post application of the transform.

Describe alternatives you've considered
Saved Questions in Metabase. While that is possible, it may be desirable to get the data automatically converted as part of the ingestion process, so that canned dashboards just work out of the box.

Additional context
This transform is a fundamental entry in our transform catalog. We will automate the orchestration and execution of those dbt transforms as part of #71 .

Release Action shall fail if an init fails

Describe the bug
We introduced an issue in #151 that made the dashboard import fail (see here). However, neither the PR nor the Release action caught the issue.

To Reproduce

Checkout 71dafdc
Start Faros CE; notice the failure
However, the Release workflow for that commit passed

Expected behavior
At least the Release workflow (I think the PR workflow does not build the init code) shall fail.

kube deployment kustomize seem not working properly

Describe the bug
kube deployment kustomize seem not working properly. with the current kustomize

To Reproduce
Steps to reproduce the behavior:

Install using the kustomize
Follow the step by step initiation
In the step of airbyte - github sources the save changes and test will always error, ref: https://community.faros.ai/docs/quickstart-step-2

I'm using GKE 1.27

Expected behavior
the step should have no error

Screenshots and logs
When I retry multiple times below is the result :

After retry multiple time seem success but I got stuck at triggering the github sync now

Any Idea on whats going on?

Deployment

If cloud, GKE 1.27 and latest version

Task to Incident dbt transform example

Is your feature request related to a problem? Please describe.
Some engineering organization track incidents in their task management systems (like Jira). Hence, those incidents will be stored as tasks in our canonical schema.

Describe the solution you'd like
dbt transform with simple controls that will select the proper subset of tasks that are in fact incidents, and convert them into incidents. It is acceptable if the original entity is still present post application of the transform.

Additional context
This transform is a fundamental entry in our transform catalog. We will automate the orchestration and execution of those dbt transforms as part of #71 .

Jira Dashboards

Is your feature request related to a problem? Please describe.
When a Faros CE user adds Projects to a JIRA source, they do not have a great Dashboard to go to that leverages most Jira idiosyncrasies (Kanban/Sprint, Status, Releases, ...).
The Task Dashboard is bare-bone.

Describe the solution you'd like
There are 5 new dashboards.

Jira Kanban Release Dashboard supports filters on modification date, boards and releases, and on top of basic issues stats for the selected release focuses on Work-In-Progress
Jira Sprint Dashboard supports filters on modification date, boards and sprint, and on top of basic issues stats for the selected sprint focuses on Burn-down and velocity
Jira Kanban Board Dashboard supports filters on modification date, boards, and shows stats (incl. Work-in-progress) broken down by Releases
Jira Sprint Board Dashboard supports filters on modification date, boards, and shows stats (incl. velocity) broken down by Sprints
Jira Epic Dashboard supports filters on epics, and show epics-related metrics.

Additional context
This is similar in intent to the GitHub dashboard we created, even though we already had a PR dashboard. The GitHub dashboard leverages GitHub idiosyncrasies, in this instance: repository filtering; display of both issues and pull requests charts.

Add CFR to DORA dashboard

Sync worker failed

Describe the bug

Airbyte version: example is .39.37-alpha
OS Version / Ubuntu 20.04
Deployment: Docker
Source Connector and version: Github(cloud)
Destination Connector and version: Faros Destination
Step where error happened: Sync job
After adding Github to the Source item, I get a "Sync worker failed" error when I run Sync now.

`2023-03-08 01:16:11 �[1;31mERROR�[m i.a.w.g.DefaultReplicationWorker(run):180 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:175) ~[io.airbyte-airbyte-workers-0.39.37-alpha.jar:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:65) ~[io.airbyte-airbyte-workers-0.39.37-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.37-alpha.jar:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
	Suppressed: io.airbyte.workers.exception.WorkerException: Destination process exit with code 1. This warning is normal if the job was cancelled.
		at io.airbyte.workers.internal.DefaultAirbyteDestination.close(DefaultAirbyteDestination.java:119) ~[io.airbyte-airbyte-workers-0.39.37-alpha.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:137) ~[io.airbyte-airbyte-workers-0.39.37-alpha.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:65) ~[io.airbyte-airbyte-workers-0.39.37-alpha.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.39.37-alpha.jar:?]
		at java.lang.Thread.run(Thread.java:833) [?:?]`

Logs
server-logs.txt
logs-5.txt

Expected behavior
clear and concise description of what you expected to happen.

Screenshots and logs
If applicable, add screenshots and logs to help explain your problem.

Deployment

If local, docker and OS versions
If cloud, cluster type and version

Cycle Time query is broken

Describe the bug
Cycle Time query is broken

To Reproduce
Steps to reproduce the behavior:

Start Faros CE
Open Metabase UI, navigate to GitHub dashboard, click on Cycle Time
Open editor
See invalid query "repositoryNam" and "mergedA" are missing a letter each.

Expected behavior
The query should reference "repositoryName" and "mergedAt"

Screenshots and logs

Discussion
https://faroscommunity.slack.com/archives/C0335HMN2NQ/p1650509861394399

Set the same timezone for all services

Bitbucket source

Support for Kubernetes Deployment

Support for vcs_PullRequestCommit in the schema

The Events CLI now accepts pull request number to enrich commit information with the corresponding PR number.

We enrich the schema to support that association.

The model should be:

"A commit that is part of a pull request"
type vcs_PullRequestCommit @model(key: ["commit", "pullRequest"]) {
  "Commit in a pull request"
  commit: vcs_Commit @reference(back: "pullRequests")

  "Pull request the commit is part of"
  pullRequest: vcs_PullRequest @reference(back: "commits")
}

Add vcs_Team and vcs_TeamMembership models

Syncing from Github never completes

Describe the bug
Hi there,

I've been trying to setup a connection between Github and Faros and haven't been able to get it to complete. We have a fairly large mono repo and the sync gets stuck on Syncing stream: pull_requests .

I actually tried copying some of the files from airbyte using the instructions from https://docs.airbyte.com/operator-guides/browsing-output-logs/. There's a pull_requests.yml file that's 321MB. When I inspect it, it contains the request and response data for pull requests that were created back in 2015. I suspect that the sync is trying to get the full history of pulls requests and is not using the start data the way I expect it to.

To Reproduce
Steps to reproduce the behavior:

Create a Github source (private repo)
Set the start date to 2023-09-01T00:00:00Z
Create a connection with the Github source and Faros destination
Start sync
Nothing happens for 20 hours
The logs show that the job was canceled because my computer shut itself down

Expected behavior
I expect the sync to finish given the start date

Screenshots and logs

Logs
9305eb98_eff2_4933_980a_3ec54522104b_logs_25_txt.txt

Deployment
Local deployment
M1 Mac
Docker version 24.0.5, build ced0996

Faros Performance Fork of Github connector does not work on M1 Mac

Describe the bug
Related to the Faros performant fork of GitHub connector that was made default in (#140) and released as part of v0.2.0.

The Faros fork of the Airbyte Github Source does not appear to be built for M1 Mac architectures (ie. linux/arm64). I looked around to see where I might make a PR to update it but I cannot find the source of that fork. Additionally, the same error occurs for the faros fork of the airbyte-faros-destination connector.

To Reproduce
Steps to reproduce the behavior:

Follow the QuickStart guide
In Step 2, configure the Github source
Run a sync and check the logs
Relevant log pasted below
Operative errors are the logline below which occurs for both farosai/airbyte-faros-destination and farosai/airbyte-github-source connectors
Error:
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Expected behavior
A full sync of the configured Github repos

Screenshots and logs

2022-09-25 16:11:47 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-25 16:11:47 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-25 16:11:47 �[32mINFO�[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. 
errors: $.access_token: is missing but it is required, $.option_title: must be a constant value OAuth Credentials
2022-09-25 16:11:47 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-25 16:11:47 �[33mWARN�[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-09-25 16:11:47 �[32mINFO�[m i.a.a.LoggingTrackingClient(track):43 - track. version: 0.39.37-alpha, userId: 00000000-1111-2222-3333-444444444444, action: Connector Jobs, metadata: {job_type=sync, config.source.credentials.option_title=PAT Credentials, config.destination.source_specific_configs.bitbucket.application_mapping=set, namespace_definition=source, config.destination.edition_configs.segment_user_id=set, config.source.credentials.personal_access_token=set, config.destination.dry_run=false, frequency=manual, connector_source_definition_id=00000000-1111-2222-3333-444444444445, workspace_id=00000000-1111-2222-3333-444444444446, config.destination.source_specific_configs.jira.use_board_ownership=true, attempt_stage=STARTED, config.destination.edition_configs.hasura_url=set, config.destination.source_specific_configs.squadcast.application_mapping=set, attempt_id=1, connector_destination=Faros Destination, catalog.sync_mode.incremental=set, connector_destination_docker_repository=farosai/airbyte-faros-destination, table_prefix=true, config.destination.jsonata_mode=FALLBACK, workspace_name=00000000-1111-2222-3333-444444444446, config.source.start_date=set, connector_source=GitHub (fork by Faros AI), config.destination.source_specific_configs.jira.exclude_fields=set, connector_source_docker_repository=farosai/airbyte-github-source, config.destination.source_specific_configs.pagerduty.application_mapping=set, config.destination.source_specific_configs.victorops.application_mapping=set, config.source.repository=set, config.destination.invalid_record_strategy=SKIP, catalog.sync_mode.full_refresh=set, connection_id=00000000-1111-2222-3333-444444444447, config.destination.edition_configs.edition=community, job_id=1, connector_source_version=0.0.2, config.destination.source_specific_configs.victorops.application_field=set, catalog.destination_sync_mode.append=set, connector_destination_version=0.3.2, operation_count=0, config.source.page_size_for_large_streams=set, config.destination.source_specific_configs.statuspage.application_mapping=set, config.destination.edition_configs.hasura_admin_secret=set, connector_destination_definition_id=00000000-1111-2222-3333-444444444448}
2022-09-25 16:11:47 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/1/0/logs.log
2022-09-25 16:11:47 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.39.37-alpha
2022-09-25 16:11:47 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - Checking if farosai/airbyte-github-source:0.0.2 exists...
2022-09-25 16:11:48 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - farosai/airbyte-github-source:0.0.2 was found locally.
2022-09-25 16:11:48 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 1
2022-09-25 16:11:48 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name airbyte-github-source-check-1-0-dkhgz --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=farosai/airbyte-github-source:0.0.2 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=1 farosai/airbyte-github-source:0.0.2 check --config source_config.json
2022-09-25 16:11:48 �[1;31mERROR�[m i.a.c.i.LineGobbler(voidCall):82 - WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-09-25 16:11:51 �[33mWARN�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):96 - Syncing `RepositoryStats` stream isn't available for repository `github/gitignore`.
2022-09-25 16:11:51 �[32mINFO�[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):97 - Check succeeded
2022-09-25 16:11:51 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2022-09-25 16:11:51 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/1/0/logs.log
2022-09-25 16:11:51 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.39.37-alpha
2022-09-25 16:11:51 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - Checking if farosai/airbyte-faros-destination:0.3.2 exists...
2022-09-25 16:11:51 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - farosai/airbyte-faros-destination:0.3.2 was found locally.
2022-09-25 16:11:51 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 1
2022-09-25 16:11:51 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name airbyte-faros-destination-check-1-0-ppsxq --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=farosai/airbyte-faros-destination:0.3.2 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=1 farosai/airbyte-faros-destination:0.3.2 check --config source_config.json
2022-09-25 16:11:51 �[1;31mERROR�[m i.a.c.i.LineGobbler(voidCall):82 - WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-09-25 16:11:55 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2022-09-25 16:11:55 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/1/0/logs.log
2022-09-25 16:11:55 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.39.37-alpha
2022-09-25 16:11:55 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):115 - start sync worker. job id: 1 attempt id: 0
2022-09-25 16:11:55 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):127 - configured sync modes: {null.workflows=incremental - append, null.pull_requests=incremental - append, null.commits=incremental - append, null.issue_labels=full_refresh - append, null.assignees=full_refresh - append, null.organizations=full_refresh - append, null.issues=incremental - append, null.branches=full_refresh - append, null.issue_milestones=incremental - append, null.reviews=incremental - append, null.review_comments=incremental - append, null.pull_request_stats=incremental - append, null.releases=incremental - append, null.workflow_runs=full_refresh - append, null.users=full_refresh - append, null.collaborators=full_refresh - append, null.tags=full_refresh - append, null.repositories=full_refresh - append}
2022-09-25 16:11:55 �[32mINFO�[m i.a.w.i.DefaultAirbyteDestination(start):69 - Running destination...
2022-09-25 16:11:55 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - Checking if farosai/airbyte-faros-destination:0.3.2 exists...
2022-09-25 16:11:56 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - farosai/airbyte-faros-destination:0.3.2 was found locally.
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 1
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name airbyte-faros-destination-write-1-0-qzsue --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=farosai/airbyte-faros-destination:0.3.2 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=1 farosai/airbyte-faros-destination:0.3.2 write --config destination_config.json --catalog destination_catalog.json
2022-09-25 16:11:56 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - Checking if farosai/airbyte-github-source:0.0.2 exists...
2022-09-25 16:11:56 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):82 - farosai/airbyte-github-source:0.0.2 was found locally.
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 1
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0 --log-driver none --name airbyte-github-source-read-1-0-nbdpg --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_CONNECTOR_IMAGE=farosai/airbyte-github-source:0.0.2 -e WORKER_JOB_ATTEMPT=0 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=1 farosai/airbyte-github-source:0.0.2 read --config source_config.json --catalog source_catalog.json
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$7):405 - Destination output thread started.
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):169 - Waiting for source and destination threads to complete.
2022-09-25 16:11:56 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):298 - Replication thread started.
2022-09-25 16:11:56 �[43mdestination�[0m > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-09-25 16:11:56 �[44msource�[0m > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2022-09-25 16:11:59 �[44msource�[0m > Starting syncing SourceGithub
2022-09-25 16:11:59 �[44msource�[0m > Syncing `RepositoryStats` stream isn't available for repository `github/gitignore`.
2022-09-25 16:11:59 �[44msource�[0m > Syncing `Branches` stream isn't available for repository `github/gitignore`.
2022-09-25 16:11:59 �[44msource�[0m > 'github/gitignore'
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 129, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 120, in run
    for message in generator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 101, in read
    stream_instances = {s.name: s for s in self.streams(config)}
  File "/airbyte/integration_code/source_github/source.py", line 186, in streams
    default_branches, branches_to_pull = self._get_branches_data(config.get("branch", ""), repository_args)
  File "/airbyte/integration_code/source_github/source.py", line 133, in _get_branches_data
    repo_branches = [default_branches[repo]]
KeyError: 'github/gitignore'
2022-09-25 16:11:59 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):335 - Total records read: 1 (0 bytes)

Deployment

Local Deployment
M1 MacBookPro 2022
MacOS 12.6 - Monterey
Docker Engine - 20.10.17
Docker Desktop - 4.12.0

GitHub diff stats

Describe the bug
The GitHub canned source does not get PR diff stats.

The pull_request_stats stream (which exists in the community source AND for which we have a converter that updates the PR entity is deactivated because we get those errors:

2022-10-31 05:42:56 source > Syncing stream: pull_request_stats 
2022-10-31 05:43:13 source > Backing off _send(...) for 5.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Request URL: https://api.github.com/graphql, Response Code: 502, Response Text: {
   "data": null,
   "errors":[
      {
         "message":"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `F8DD:188C:158A262:164FCC8:635F6064` when reporting this issue."
      }
   ]
})
2022-10-31 05:43:13 source > Caught retryable error 'Request URL: https://api.github.com/graphql, Response Code: 502, Response Text: {
   "data": null,
   "errors":[
      {
         "message":"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `F8DD:188C:158A262:164FCC8:635F6064` when reporting this issue."
      }
   ]
}
' after 1 tries. Waiting 5 seconds then retrying...
2022-10-31 05:43:28 source > Backing off _send(...) for 10.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Request URL: https://api.github.com/graphql, Response Code: 502, Response Text: {
   "data": null,
   "errors":[
      {
         "message":"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `F8DD:188C:158B878:1651373:635F6076` when reporting this issue."
      }
   ]
})
2022-10-31 05:43:28 source > Caught retryable error 'Request URL: https://api.github.com/graphql, Response Code: 502, Response Text: {
   "data": null,
   "errors":[
      {
         "message":"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `F8DD:188C:158B878:1651373:635F6076` when reporting this issue."
      }
   ]
}
' after 2 tries. Waiting 10 seconds then retrying...

One shall investigate the nature of those errors in the source and figure out if this is a configuration issue, a transient issue, or something wrong with the source itself. Until then, the stream remains disabled and that data will be missing.

Error with the getting started at Airbyte.findFarosSource (/home/node/cli/lib/airbyte/airbyte-client.js:52:81)

Describe the bug
using the latest getitng started , walking in to the issue with all connectors

at Airbyte.findFarosSource (/home/node/cli/lib/airbyte/airbyte-client.js:52:81)
To Reproduce
Steps to reproduce the behavior:

Go to /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/faros-ai/faros-community-edition/main/install.sh)" _ --source doc_essentials
set a connector for Jira or Bitbucket
see the issue

Expected behavior
No error

Screenshots and logs
If applicable, add screenshots and logs to help explain your problem.

Deployment

If local, docker and OS versions
Ubuntu

Airbyte GitHub Sync Fails: Something went wrong within the destination connector

Describe the bug

I am experiencing an issue were GitHub Sync always fails. It happens at vcs_PullRequestComment stage. Other data gets synced properly and is available via Metabase and Hasura.

I tried it 5 times, sync always fails after 25 minutes to 1h.

It appears as if newly synced items violate unique constraints but I have no idea why that happens.

To Reproduce
Steps to reproduce the behavior:

Go to http://localhost:8000
Click on 'Launch' next to GitHub connections
Go to details and see failed logs

Expected behavior
Sync succeeds and data is available via Metabase and Hasura.

Screenshots and logs

0 Bytesno recordsno records22m 55sSync
Failure Origin: destination, Message: Something went wrong within the destination connector

2023-08-22 10:46:17 - Additional Failure Information: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed

I'm not able to provide unreacted logs because they contain pull request comments with AWS deployment details (from pull request comments).

023-08-22 10:45:46 �[43mdestination�[0m > Error while saving batch: [{"extensions":{"internal":{"statement":"WITH \"vcs_PullRequestComment__mutation_result\" AS (INSERT INTO \"public\".\"vcs_PullRequestComment\" ( \"origin\"
 \"createdAt\"
 \"uid\"
 \"pullRequest\"
 \"author\"
 \"id\"
 \"updatedAt\"
 \"number\"
 \"comment\"
 \"refreshedAt\" ) VALUES (('github_source')::text
 ('2023-02-08T15:26:36Z')::timestamptz
 ('1100296384')::text


')::text
 DEFAULT) ON CONFLICT ON CONSTRAINT "vcs_PullRequestComment_pkey" DO UPDATE SET "origin" = EXCLUDED."origin"
"createdAt" = EXCLUDED."createdAt"
"author" = EXCLUDED."author"
"updatedAt" = EXCLUDED."updatedAt"
"number" = EXCLUDED."number"
"comment" = EXCLUDED."comment" WHERE (('true') AND ('true')) RETURNING * 
 ('true')::boolean AS "check__constraint") SELECT  json_build_object('affected_rows'
 (SELECT  COUNT(*)  FROM "vcs_PullRequestComment__mutation_result"      )
 'returning_columns'
 (SELECT  coalesce(json_agg("root" )
 '[]' ) AS "root" FROM  (SELECT  row_to_json((SELECT  "_1_e"  FROM  (SELECT  ("_0_root.base"."origin")::text AS "origin"
 ("_0_root.base"."createdAt")::text AS "createdAt"
 ("_0_root.base"."uid")::text AS "uid"
 ("_0_root.base"."pullRequest")::text AS "pullRequest"
 ("_0_root.base"."author")::text AS "author"
 ("_0_root.base"."id")::text AS "id"
 ("_0_root.base"."updatedAt")::text AS "updatedAt"
 ("_0_root.base"."number")::text AS "number"
 ("_0_root.base"."comment")::text AS "comment"
 ("_0_root.base"."refreshedAt")::text AS "refreshedAt"       ) AS "_1_e"      ) ) AS "root" FROM  (SELECT  *  FROM "vcs_PullRequestComment__mutation_result" WHERE ('true')     ) AS "_0_root.base"      ) AS "_2_root"      ) ) 
 (SELECT  coalesce(bool_and("check__constraint" )
 'true' )  FROM "vcs_PullRequestComment__mutation_result"      )        \"
\"prepared\":false
\"error\":{\"exec_status\":\"FatalError\"
\"hint\":null
\"message\":\"index row requires 9352 bytes
 maximum size is 8191\"
\"status_code\":\"54000\"
\"description\":null}
\"arguments\":[]}
\"path\":\"$.selectionSet.insert_vcs_PullRequestComment_one.args.object[0]\"
\"code\":\"unexpected\"}
\"message\":\"database query error\"}] - {\"jse_shortmsg\":\"Failed to write vcs_PullRequestComment record with query 'mutation { insert_vcs_PullRequestComment_one (object: {number: 1100296384
 uid: "1100296384"
 comment: "

2023-08-22 10:47:17 �[33mWARN�[m i.a.c.i.LineGobbler(voidCall):119 - airbyte-source gobbler IOException: Stream closed. Typically happens when cancelling a job.
2023-08-22 10:47:17 �[1;31mERROR�[m i.a.w.g.DefaultReplicationWorker(run):188 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:181) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
	Suppressed: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled.
		at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:145) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at java.lang.Thread.run(Thread.java:833) ~[?:?]
	Suppressed: java.io.IOException: Stream closed
		at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:445) ~[?:?]
		at java.io.OutputStream.write(OutputStream.java:162) ~[?:?]
		at java.io.BufferedOutputStream.write(BufferedOutputStream.java:123) ~[?:?]
		at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234) ~[?:?]
		at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:313) ~[?:?]
		at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:318) ~[?:?]
		at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:160) ~[?:?]
		at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:248) ~[?:?]
		at java.io.BufferedWriter.flush(BufferedWriter.java:257) ~[?:?]
		at io.airbyte.workers.internal.DefaultAirbyteDestination.notifyEndOfInput(DefaultAirbyteDestination.java:98) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.internal.DefaultAirbyteDestination.close(DefaultAirbyteDestination.java:111) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:145) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getDestinationOutputRunnable$7(DefaultReplicationWorker.java:450) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	... 1 more
2023-08-22 10:47:17 �[33mWARN�[m i.a.w.g.DefaultReplicationWorker(run):220 - Could not reliably determine committed record counts, committed record stats will be set to null
2023-08-22 10:47:17 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):279 - Source output at least one state message
2023-08-22 10:47:18 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):285 - State capture: Updated state to: Optional[io.airbyte.config.State@38bf5de4[state=[{"type":"STREAM","stream":{"stream_descriptor":{"name":"review_comments"},"stream_state":
2023-08-22 10:47:18 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):300 - sync summary: {
  "status" : "failed",
  "recordsSynced" : 35177,
  "bytesSynced" : 39556741,
  "startTime" : 1692699895332,
  "endTime" : 1692701237993,
  "totalStats" : {
    "recordsEmitted" : 35177,
    "bytesEmitted" : 39556741,
    "sourceStateMessagesEmitted" : 1739,
    "destinationStateMessagesEmitted" : 1717,
    "meanSecondsBeforeSourceStateMessageEmitted" : 3,
    "maxSecondsBeforeSourceStateMessageEmitted" : 250,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 5,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0
  },
  "streamStats" : [ {
    "streamName" : "github_source__github__pull_requests",
    "stats" : {
      "recordsEmitted" : 194,
      "bytesEmitted" : 2803965
    }
  }, {
    "streamName" : "github_source__github__collaborators",
    "stats" : {
      "recordsEmitted" : 14080,
      "bytesEmitted" : 15532594
    }
  }, {
    "streamName" : "github_source__github__repositories",
    "stats" : {
      "recordsEmitted" : 228,
      "bytesEmitted" : 1253144
    }
  }, {
    "streamName" : "github_source__github__releases",
    "stats" : {
      "recordsEmitted" : 16,
      "bytesEmitted" : 35073
    }
  }, {
    "streamName" : "github_source__github__issues",
    "stats" : {
      "recordsEmitted" : 195,
      "bytesEmitted" : 1168345
    }
  }, {
    "streamName" : "github_source__github__commits",
    "stats" : {
      "recordsEmitted" : 524,
      "bytesEmitted" : 1986803
    }
  }, {
    "streamName" : "github_source__github__review_comments",
    "stats" : {
      "recordsEmitted" : 266,
      "bytesEmitted" : 1116811
    }
  }, {
    "streamName" : "github_source__github__organizations",
    "stats" : {
      "recordsEmitted" : 1,
      "bytesEmitted" : 1775
    }
  }, {
    "streamName" : "github_source__github__branches",
    "stats" : {
      "recordsEmitted" : 2608,
      "bytesEmitted" : 774681
    }
  }, {
    "streamName" : "github_source__github__issue_labels",
    "stats" : {
      "recordsEmitted" : 2985,
      "bytesEmitted" : 826570
    }
  }, {
    "streamName" : "github_source__github__assignees",
    "stats" : {
      "recordsEmitted" : 14080,
      "bytesEmitted" : 14056980
    }
  } ]
}
2023-08-22 10:47:18 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):301 - failures: [ {
  "failureOrigin" : "destination",
  "internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed",
  "externalMessage" : "Something went wrong within the destination connector",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 8,
    "connector_command" : "write"
  },
  "stacktrace" : "java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:346)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\nCaused by: java.io.IOException: Broken pipe\n\tat java.base/java.io.FileOutputStream.writeBytes(Native Method)\n\tat java.base/java.io.FileOutputStream.write(FileOutputStream.java:349)\n\tat java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)\n\tat java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234)\n\tat java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:304)\n\tat java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)\n\tat java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:132)\n\tat java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:205)\n\tat java.base/java.io.BufferedWriter.flushBuffer(BufferedWriter.java:120)\n\tat java.base/java.io.BufferedWriter.write(BufferedWriter.java:233)\n\tat java.base/java.io.Writer.write(Writer.java:249)\n\tat io.airbyte.workers.internal.DefaultAirbyteDestination.accept(DefaultAirbyteDestination.java:90)\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:343)\n\t... 4 more\n",
  "timestamp" : 1692701177835
}, {
  "failureOrigin" : "destination",
  "internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1",
  "externalMessage" : "Something went wrong within the destination connector",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 8,
    "connector_command" : "write"
  },
  "stacktrace" : "java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getDestinationOutputRunnable$7(DefaultReplicationWorker.java:450)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\n",
  "timestamp" : 1692701177835
} ]
2023-08-22 10:47:18 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - 
2023-08-22 10:47:18 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - ----- END REPLICATION -----
2023-08-22 10:47:18 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - 
2023-08-22 10:47:18 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling...
2023-08-22 10:47:18 �[32mINFO�[m i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$3):149 - sync summary: io.airbyte.config.StandardSyncOutput@59107209[standardSyncSummary=io.airbyte.config.StandardSyncSummary@608888a8[status=failed,recordsSynced=35177,bytesSynced=39556741,startTime=1692699895332,endTime=1692701237993,totalStats=io.airbyte.config.SyncStats@16490e94[recordsEmitted=35177,bytesEmitted=39556741,sourceStateMessagesEmitted=1739,destinationStateMessagesEmitted=1717,recordsCommitted=<null>,meanSecondsBeforeSourceStateMessageEmitted=3,maxSecondsBeforeSourceStateMessageEmitted=250,maxSecondsBetweenStateMessageEmittedandCommitted=5,meanSecondsBetweenStateMessageEmittedandCommitted=0,additionalProperties={}],streamStats=[io.airbyte.config.StreamSyncStats@72431d5f[streamName=github_source__github__pull_requests,stats=io.airbyte.config.SyncStats@2899b924[recordsEmitted=194,bytesEmitted=2803965,sourceStateMessagesEmitted=<null>,destinationStateMessagesEmitted=<null>,recordsCommitted=<null>,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,additionalProperties={}]], 
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:346)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	... 3 more
Caused by: java.io.IOException: Broken pipe
	at java.base/java.io.FileOutputStream.writeBytes(Native Method)
	at java.base/java.io.FileOutputStream.write(FileOutputStream.java:349)
	at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)
	at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234)
	at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:304)
	at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
	at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:132)
	at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:205)
	at java.base/java.io.BufferedWriter.flushBuffer(BufferedWriter.java:120)
	at java.base/java.io.BufferedWriter.write(BufferedWriter.java:233)
	at java.base/java.io.Writer.write(Writer.java:249)
	at io.airbyte.workers.internal.DefaultAirbyteDestination.accept(DefaultAirbyteDestination.java:90)
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:343)
	... 4 more
,retryable=<null>,timestamp=1692701177835], io.airbyte.config.FailureReason@32058b7[failureOrigin=destination,failureType=<null>,internalMessage=io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1,externalMessage=Something went wrong within the destination connector,metadata=io.airbyte.config.Metadata@3c77b2c0[additionalProperties={attemptNumber=0, jobId=8, connector_command=write}],stacktrace=java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getDestinationOutputRunnable$7(DefaultReplicationWorker.java:450)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	... 3 more
,retryable=<null>,timestamp=1692701177835]]]
2023-08-22 10:47:18 �[32mINFO�[m i.a.w.t.TemporalUtils(withBackgroundHeartbeat):279 - Stopping temporal heartbeating...

Deployment

Local
MacOS 13.4 (22F66), MacBook Air M1
Docker version 23.0.5, build bc4487a
Docker Compose version v2.17.3

Thank you for providing us with a great tool and help with this bug!

Support running Faros CE on Mac M1 chips

Airbyte does not currently run on M1 Chips: airbytehq/airbyte#2017
Metabase does currently not support M1 either: metabase/metabase#13119

Canned charts that drill into json are broken

Describe the bug
Canned Tasks charts that use ->>jsonb are broken.
It appears they are not imported properly.

To Reproduce
Steps to reproduce the behavior:

Download and start Faros CE
Go to the Tasks dashboard (http://localhost:3000/dashboard/4)
Many tasks charts are broken because the SQL query is incorrect

Expected behavior
Non-broken SQL query

Screenshots and logs
This query:
https://github.com/faros-ai/faros-community-edition/blob/main/init/resources/metabase/dashboards/tasks.json#L72
is imported as

SELECT count(*) AS "count"
FROM "public"."tms_Task"
WHERE ("public"."tms_Task"."updatedAt" >= CAST((CAST(now() AS timestamp) + (INTERVAL '-30 day')) AS date)
   AND "public"."tms_Task"."updatedAt" < CAST((CAST(now() AS timestamp) + (INTERVAL '1 day')) AS date))
   AND "public"."tms_Task"."status"::json-}}'category' = 'Done'

This behavior is likely an unintended side effect of #77

Update Airbyte to new 44 version

Update Airbyte to version 44 or newer because support gitlab self hosted version

Upgrade to Node.js 16

Is your feature request related to a problem? Please describe.
Upgrade to Node.js 16

Describe the solution you'd like
Upgrade to Node.js 16

Describe alternatives you've considered
N/A

Error during sync from Gitlab connector: Found multiple possible origins from catalog

Describe the bug
Unable to sync from Gitlab to Faros destination.

After setting up Faros CE locally, starting it all up using ./start.sh, I set up a Source (Gitlab) and Destination (Faros, which I assume is the safest to start with, just to try things out).
This appeared to have been successful.

But when I try to sync in Airbyte, it keeps throwing me this error: Encountered an error while writing to destination: VError: Found multiple possible origins from catalog

Log entry:

2022-10-18 20:15:41 �[43mdestination�[0m > Encountered an error while writing to destination: VError: Found multiple possible origins from catalog: branches,commits,epic_issues,epics,group_issue_boards,group_labels,group_members,group_milestones,groups,issues,jobs,merge_request_commits,merge_requests,pipelines,pipelines_extended,project_labels,project_members,project_milestones,projects,releases,tags,users - {"jse_shortmsg":"Found multiple possible origins from catalog: branches,commits,epic_issues,epics,group_issue_boards,group_labels,group_members,group_milestones,groups,issues,jobs,merge_request_commits,merge_requests,pipelines,pipelines_extended,project_labels,project_members,project_milestones,projects,releases,tags,users","jse_info":{},"message":"Found multiple possible origins from catalog: branches,commits,epic_issues,epics,group_issue_boards,group_labels,group_members,group_milestones,groups,issues,jobs,merge_request_commits,merge_requests,pipelines,pipelines_extended,project_labels,project_members,project_milestones,projects,releases,tags,users"}
2022-10-18 20:15:41 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(lambda$getDestinationOutputRunnable$7):415 - State in DefaultReplicationWorker from destination: io.airbyte.protocol.models.AirbyteMessage@35819db0[type=TRACE,log=<null>,spec=<null>,connectionStatus=<null>,catalog=<null>,record=<null>,state=<null>,trace=io.airbyte.protocol.models.AirbyteTraceMessage@3b466570[type=ERROR,emittedAt=1.666124141733E12,error=io.airbyte.protocol.models.AirbyteErrorTraceMessage@3f79f6c0[message=Found multiple possible origins from catalog: branches,commits,epic_issues,epics,group_issue_boards,group_labels,group_members,group_milestones,groups,issues,jobs,merge_request_commits,merge_requests,pipelines,pipelines_extended,project_labels,project_members,project_milestones,projects,releases,tags,users,internalMessage=VError,stackTrace=VError: Found multiple possible origins from catalog: branches,commits,epic_issues,epics,group_issue_boards,group_labels,group_members,group_milestones,groups,issues,jobs,merge_request_commits,merge_requests,pipelines,pipelines_extended,project_labels,project_members,project_milestones,projects,releases,tags,users
    at FarosDestination.getOrigin (/home/node/airbyte/destinations/airbyte-faros-destination/lib/destination.js:269:19)
    at FarosDestination.write (/home/node/airbyte/destinations/airbyte-faros-destination/lib/destination.js:277:29)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Command.<anonymous> (/home/node/airbyte/faros-airbyte-cdk/lib/destinations/destination-runner.js:67:34)
    at async Command.parseAsync (/home/node/airbyte/node_modules/commander/lib/command.js:917:5),failureType=system_error,additionalProperties={jse_shortmsg=Found multiple possible origins from catalog: branches,commits,epic_issues,epics,group_issue_boards,group_labels,group_members,group_milestones,groups,issues,jobs,merge_request_commits,merge_requests,pipelines,pipelines_extended,project_labels,project_members,project_milestones,projects,releases,tags,users, jse_info={}}],additionalProperties={}],additionalProperties={}]
2022-10-18 20:15:41 �[1;31mERROR�[m i.a.w.g.DefaultReplicationWorker(run):180 - Sync worker failed.

In the destination's replication settings, all streams are all enabled (was by default, I didn't change anything):

To Reproduce
Steps to reproduce the behavior:

As per Quickstart, I cloned the repo and ran ./start.sh
Set up Gitlab source (self-hosted, v15.4.1) in Airbyte
Select Faros Destination for the connection.
Navigate to Connection and try to sync.
See error and be sad.

Expected behavior
Successful sync, so I can see my data in a Metabase dashboard.

Deployment

local, docker on Linux (WSL2), as per Quickstart

Azure DevOps support in upcoming release

I just wanted to ask when the next release is scheduled for? There doesn’t appear to be any info about this.

I’ve seen that there will be Azure DevOps / Pipelines support in upcoming release and really want to test out your project.

Buildkite Source

Support dbt transforms in Airbyte

Is your feature request related to a problem? Please describe.
As a Faros CE user, I want to apply transformations on what Airbyte writes against the Faros Schema to capture my idiosyncrasies.

Describe the solution you'd like
Support for dbt transforms in Airbyte: https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-airbyte
This would allow the ability to write lightweight transforms in SQL for one's own consumption, but also publish and share those with the community for common use cases. For example, transforming tasks that have a certain label or type into incidents.

Describe alternatives you've considered
It is possible for Analysts in Metabase to transform the data with SQL, but they cannot write it back to the database. dbt transforms during ingestion allows users to leverage canned dashboards better, and provide better separation of concerns between the analyst and data engineer personas.

Additional context

Currently blocked by airbytehq/airbyte#7229 on the Airbyte side.
If possible, the user should have the option in Airbyte to have the non-transformed data written to Faros (desirable when your dbt transform creates a new metric table like #116) or discarded (desirable when your dbt transform swaps entity types, like #114 and #115 ).

Faros init build issue on M1 chips

Some M1 machines are unable to build our "faros-init" docker image that initializes the other services -- create Airbyte connection templates, track data tables in Hasura, add initial Metabase dashboards. The "docker build" step fails when compiling the typescript code, but the root cause is still unknown.

We solved this on for an M1 test machine by building and pushing the image to Docker Hub from a non-M1 laptop. The test machine successfully pulled and launched this published "faros-init" image.

Tasks:
Create GitHub Actions workflow to automatically build and push faros-init image to Docker Hub.
Update docker-compose yaml to use versioned image from Docker Hub.
Create documentation and necessary tooling to enable contributors to quickly update the image version in their pull requests.

Feed mock data fails to load store on Hasura due to lack of permission

Describe the bug
Commands outlined in the documentation regarding loading mock data from the FarosAI community doesn't work due to lack of permission in the Hasura API.

To Reproduce
Steps to reproduce the behavior:

Follow the Quickstart guide from FarosAI
Attempt to load mock-data as indicated here

Expected behavior
mock data should be loaded in the store to be visualized in Metabase

Screenshots and logs

Deployment

If local, docker and OS versions

Docker

Client:
 Cloud integration: v1.0.22
 Version:           20.10.13
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 10 14:08:43 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Desktop 4.6.1 (76265)
 Engine:
  Version:          20.10.13
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       906f57f
  Built:            Thu Mar 10 14:05:37 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.5.10
  GitCommit:        2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

MacOS

ProductName:	macOS
ProductVersion:	12.3
BuildVersion:	21E230

Custom Metric Table dbt transform example

Is your feature request related to a problem? Please describe.
Users may want to generate custom metric tables to be leveraged in Metabase.

Describe the solution you'd like
dbt transform that creates a Task Status Duration table that for each task and non-final status in the Task table, computes the duration the task was in that status.

Describe alternatives you've considered
Saved Questions in Metabase. While that is possible, it may be desirable to transform the data outside of the query path, for performance and UX considerations (sparing the users to browse both Saved Questions AND the tables).

Additional context
This transform is a fundamental entry in our transform catalog. We will automate the orchestration and execution of those dbt transforms as part of #71 .

Automated integration tests

We currently have a GitHub Action (ci-cd) start a local Faros CE with docker-compose and runs automated tests. Those tests are currently testing services in isolation. We want to augment the scope of those test to capture how those services interact.

In particular:

Airbyte Faros Destination pushing data into the Faros DB through the Hasura GraphQL endpoint
Metabase being able to read that data

Add better tests for faros-ce-init docker image

Describe the solution you'd like
Add better tests for faros-ce-init docker image

Describe alternatives you've considered
N/A

Additional context
Faros CE init image changes are fragile

Update canned connections to use Overwrite instead of Append sync mode

Describe the bug
The Faros Destination does not follow the sync mode spec (see faros-ai/airbyte-connectors#575). Hence, all sync mode right now are either Full Refresh | Append or Incremental | Append which works as intended but not as an Airbyte user might expect. Once the Faros Destination is fixed, we will have to update the connections to use Overwrite.

Jira Issues not linking to Epics

Describe the bug
The tms_Task converted from a Jira issue that belongs to a Jira epic does not link to the tms_Epic (or to the parent tms_Task). Introduced in destination 0.4.49.

To Reproduce
Steps to reproduce the behavior (for Faros employees):

Deploy Faros CE
Set Faros destination version to 0.4.48 in the Airbyte Setting screen
Sync the TEST jira project (12 months)
check TEST-20 in tms_Task; it does link to the TEST-18 corresponding epic
Set Faros destination version to 0.4.49 in the Airbyte Setting screen
Reset the Jira connection
Sync the TEST jira project (12 months)
check TEST-20 in tms_Task; it does NOT link to the TEST-18 corresponding epic

Better import of new canned dashboards

Is your feature request related to a problem? Please describe.
When a new canned dashboard is added to the main line, existing users only have a clunky way to have it deployed:

pull latest code
archive all canned dashboard
call the import script that re-adds all dashboards

Describe the solution you'd like
At the very least, the ability to import a specific dashboard only.
Better would be the automatic import of new canned dashboards; if an existing canned dashboard has been updated, something (some logic or a human) would need to figure out if an overwrite or duplicate dashboard shall be created.

Describe alternatives you've considered
Alternate is the current procedure.

n8n initialization error - "/home/node/.n8n/config". It does not seem to be valid JSON."

Missing Jira Additional Fields

Describe the bug
Jira Additional Fields are not written to Faros.

To Reproduce

Configure your Jira source Additional Fields and save
Sync
In Hasura, do the following query:

query MyQuery {
  tms_Task(where: {uid: {_eq: "TEST-1"}}) {
    additionalFields
    name
  }
}

The additional field is missing.
Note that in Metabase, the additionalField column in tms_Task will only display in the detail view (when you look at the record).

Can be reproduced on our TEST project with custom field customfield_10049 (testcustom field). TEST-1 should have foo in there. Jira source is the community one on 0.2.20.

Use Hash Index wherever appropriate

`refreshedAt` is not updated when a record is modified

Describe the bug
refreshedAt is not updated when a record is modified.

To Reproduce
Steps to reproduce the behavior:

Insert a record through GraphQL and check refreshedAt value
Update some record field and check refreshedAt value again

Expected behavior
refreshedAt should be updated automatically on every record change.

GitLab sync fails if project doesn't have CI/CD enabled

Describe the bug
When a project does not have CI/CD activated, the sync will fail because of 403 API errors during the pipeline stream.

Screenshots and logs
If applicable, add screenshots and logs to help explain your problem.

403 Client Error: Forbidden for url: https://gitlab.com/api/v4/projects/[redacted]/pipelines?per_page=50&updated_after=2023-01-03T02%3A49%3A24Z

[doubt] update airbyte

How can I update airbyte's version through faros .env?

I'm already tried to change by enviroment variables putting VERSION=0.53.0.

If can someone help me, I'd be thankfull

Unable to install faros on windows OS

Error:
Error response from daemon: create tmp\airbyte_local: "tmp\airbyte_local" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed. If you intended to pass a host directory, use absolute path

fatal: destination path 'faros-community-edition' already exists and is not an empty directory.

Describe the bug

fatal: destination path 'faros-community-edition' already exists and is not an empty directory.

To Reproduce

run /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/faros-ai/faros-community-edition/main/install.sh)" _ --source doc_welcome
stop it
run it again $ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/faros-ai/faros-community-edition/main/install.sh)" _ --source doc_welcome

Expected behavior

If the destination directory exists, stop the script or cd git pull latest

Weekly Release / Deployment Frequency (Avg) computation is incorrect on low volume

Describe the bug
In DORA dashboards, for the top left gauge chart Weekly Release / Deployment Frequency (Avg), we use a UI built query to compute the average. It is incorrect because weeks with no releases / deployments are simply ignored in the final averaging because of how the query is built. Indeed, the first aggregation that counts per week will have no entries for week with no release / deployment.

Crafting the proper query may (?) require changing the card to a SQL query.

Expected behavior
If you have the following:

week 1 - 1 release
week 2 - 1 release
week 4 - 1 release
The Weekly Release frequency past 30 days should be 0.75, instead it is 1.

Screenshots and logs
Notice how some weeks do not have releases.

Metabase v43

Cannot refresh schema on canned connections

Describe the bug
Airbyte fails to fetch an updated schema for canned connections. It works fine if one create a NEW source, link it to the EXISTING destination.

This is likely caused by one of 2 mismatches: our canned source def mentions GitHub 0.3.8 but:

Airbyte changes it to 0.2.42, likely because of a version matching thing.
It is possible that the source config we pass is NOT 0.3.8 (and also NOT 0.2.42) - I can tell because the source config screen looks different between the canned one and when you create a new GitHub source

Suggested course of action: standardize on 0.2.42 for now, and redo the source config.

Or, move to airbyte 40+ and redo the init.

To Reproduce
Steps to reproduce the behavior:

Start Faros from scratch
Put valid settings in the EXISTING GitHub source
Go to its connection
Click on refresh schema

Note that:

step 2 is important, as Airbyte will refuse to update schema if the source has not been marked as "discoverable", which happens after a check if I am not mistaken
this behavior happens on the Jira canned source as well; I suspect it happens on ALL canned sources

Expected behavior
All streams are displayed.

Screenshots and logs
Worker is fine:

2023-03-03 17:10:30 2023-03-04 01:10:30 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/c74afff5-d342-47bc-b2e3-9884381a1ec3/0/logs.log
2023-03-03 17:10:30 2023-03-04 01:10:30 INFO i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.39.37-alpha
2023-03-03 17:10:30 2023-03-04 01:10:30 INFO i.a.c.i.LineGobbler(voidCall):82 - Checking if airbyte/source-github:0.2.42 exists...
2023-03-03 17:10:30 2023-03-04 01:10:30 INFO i.a.c.i.LineGobbler(voidCall):82 - airbyte/source-github:0.2.42 was found locally.
2023-03-03 17:10:30 2023-03-04 01:10:30 INFO i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: c74afff5-d342-47bc-b2e3-9884381a1ec3
2023-03-03 17:10:30 2023-03-04 01:10:30 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/c74afff5-d342-47bc-b2e3-9884381a1ec3/0 --log-driver none --name source-github-discover-c74afff5-d342-47bc-b2e3-9884381a1ec3-0-naayk --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=false -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/source-github:0.2.42 -e AIRBYTE_VERSION=0.39.37-alpha -e WORKER_JOB_ID=c74afff5-d342-47bc-b2e3-9884381a1ec3 airbyte/source-github:0.2.42 discover --config source_config.json
2023-03-03 17:10:32 2023-03-04 01:10:32 INFO i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...

but server complains:

2023-03-03 16:50:09 2023-03-04 00:50:09 ERROR i.a.s.e.UncaughtExceptionMapper(toResponse):22 - Uncaught exception
2023-03-03 16:50:09 java.lang.IllegalArgumentException: malformed JsonSchema object type, must have one of the following fields: properties, oneOf, allOf, anyOf in {}
2023-03-03 16:50:09     at io.airbyte.commons.json.JsonSchemas.traverseJsonSchemaInternal(JsonSchemas.java:213) ~[io.airbyte-airbyte-commons-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.commons.json.JsonSchemas.traverseJsonSchemaInternal(JsonSchemas.java:194) ~[io.airbyte-airbyte-commons-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.commons.json.JsonSchemas.traverseJsonSchemaInternal(JsonSchemas.java:206) ~[io.airbyte-airbyte-commons-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.commons.json.JsonSchemas.traverseJsonSchema(JsonSchemas.java:107) ~[io.airbyte-airbyte-commons-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.commons.json.JsonSchemas.traverseJsonSchemaWithFilteredCollector(JsonSchemas.java:142) ~[io.airbyte-airbyte-commons-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.commons.json.JsonSchemas.traverseJsonSchemaWithCollector(JsonSchemas.java:125) ~[io.airbyte-airbyte-commons-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.protocol.models.CatalogHelpers.getFullyQualifiedFieldNamesWithTypes(CatalogHelpers.java:224) ~[io.airbyte.airbyte-protocol-protocol-models-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.protocol.models.CatalogHelpers.getStreamDiff(CatalogHelpers.java:320) ~[io.airbyte.airbyte-protocol-protocol-models-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.protocol.models.CatalogHelpers.lambda$getCatalogDiff$7(CatalogHelpers.java:309) ~[io.airbyte.airbyte-protocol-protocol-models-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
2023-03-03 16:50:09     at io.airbyte.protocol.models.CatalogHelpers.getCatalogDiff(CatalogHelpers.java:305) ~[io.airbyte.airbyte-protocol-protocol-models-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.server.handlers.ConnectionsHandler.getDiff(ConnectionsHandler.java:263) ~[io.airbyte-airbyte-server-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.server.handlers.WebBackendConnectionsHandler.webBackendGetConnection(WebBackendConnectionsHandler.java:261) ~[io.airbyte-airbyte-server-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.server.apis.ConfigurationApi.lambda$webBackendGetConnection$88(ConfigurationApi.java:812) ~[io.airbyte-airbyte-server-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.server.apis.ConfigurationApi.execute(ConfigurationApi.java:873) ~[io.airbyte-airbyte-server-0.39.37-alpha.jar:?]
2023-03-03 16:50:09     at io.airbyte.server.apis.ConfigurationApi.webBackendGetConnection(ConfigurationApi.java:812) ~[io.airbyte-airbyte-server-0.39.37-alpha.jar:?]

Dashboard export script does not correctly handle Metabase template tags

The dashboard export script double escapes template tags in native sql charts, which in turn causes errors in the import.

Kustomize doesn't support --load-restrictor LoadRestrictionsNone for remote resources

Describe the bug
When trying to follow these instructions for deploying FarosCE in Kubernetes I'm getting the following issue:

Error: accumulating resources: accumulation err='accumulating resources from './faros': '/private/var/folders/vx/gbhv_rk12vq1zdf76bpn0nhh0000gq/T/kustomize-1233669672/kube/base/faros' must resolve to a file': recursed accumulation of path '/private/var/folders/vx/gbhv_rk12vq1zdf76bpn0nhh0000gq/T/kustomize-1233669672/kube/base/faros': loading KV pairs: env source files: [./config/.env]: security; file '/private/var/folders/vx/gbhv_rk12vq1zdf76bpn0nhh0000gq/T/kustomize-1233669672/kube/base/faros/config/.env' is not in or below '/private/var/folders/vx/gbhv_rk12vq1zdf76bpn0nhh0000gq/T/kustomize-1233669672/kube/base/faros'

This is caused by the fact that we are trying to re-utilize the existing .env configuration file in the root by symlinking it to kube/faros/config/.env. When kustomize parses this files, it attempts to load a resource that is above the path of the kustomization.yaml file in question which is typically prohibited for security reasons. However this can be bypassed with the flag --load-restrictor LoadRestrictionsNone and that often works but as we have just found out it doesn't seem to hold true for when resources are being referenced remotely like the command above.

Here's a reference to a kustomize Github issue touching on the unexpected behavior.

If one clones the repository and runs

kustomize build --load-restrictor LoadRestrictionsNone kube/base

things work properly, and the --load-restrictor flag is respected

To Reproduce
Steps to reproduce the behavior:

Install kustomize
From a terminal execute kustomize build --load-restrictor LoadRestrictionsNone https://github.com/faros-ai/faros-community-edition/kube/base

Expected behavior
FarosCE K8s manifests are rendered correctly

Screenshots and logs

Deployment

Kustomize 4.5.5

Use of JSON fields in template fails dashboard import

Describe the bug
The use of field derived from a JSON-containing column makes import fail on field-level table sync check.

To Reproduce
Steps to reproduce the behavior:

Start Faros
Import some tasks from Jira/GH/...
Trigger a database schema sync and field value scan
Check that tms_Task now has 2 versions of status category for tms_Task: the generated field statusCategory and the newly metabase-derived one status -> category
Create a dashboard with a non-SQL question using that status->category field as aggregation
Export the dashboard.
Try to re-import it.
Import fails on syncTable because the check can not find that field in the metadata.

Expected behavior
Dashboard import from a template using such field is successful.

Screenshots and logs
If applicable, add screenshots and logs to help explain your problem.
Example of bad template that had to be fixed here

Notice the difference between the 2 field types:

Encountered during #231 .

Configmap "faros-config" breaks Kustomize installation

Describe the bug
When I try to install using Kustomize (by referring to the /kube/base directory) I encounter the following error which breaks the installation:

kustomize build .kube/base` failed exit status 1: # Warning: 'bases' is deprecated. Please use 'resources' instead. 
Run 'kustomize edit fix' to update your Kustomization automatically. 
Error: accumulating resources: accumulation err='accumulating resources from './faros': '.kube/base/faros' must resolve to a file': recursed accumulation of path '.kube/base/faros': configmap faros-config illegally repeats the key `JOB_ERROR_REPORTING_STRATEGY`

Upon looking at kube/base/faros/config/.env I indeed see that the JOB_ERROR_REPORTING_STRATEGY env var is repeated a second time in the file on line #174

I'm using ArgoCD, and tried to install using the v0.6.0 tag

To Reproduce
Try installing using the official documented way

Expected behavior
Installation is completed successfully

faros-ai / faros-community-edition Goto Github PK

faros-community-edition's People

Contributors

Stargazers

Watchers

Forkers

faros-community-edition's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs