Topic: data-integration Goto Github

Some thing interesting about data-integration

👇 Here are 379 public repositories matching this topic...

airbytehq / airbyte

data-integration,The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

Organization: airbytehq

Home Page: https://airbyte.com

data pipeline data-analysis data-engineering java python etl change-data-capture data-collection data-integration

apache / airflow

data-integration,Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Organization: apache

Home Page: https://airflow.apache.org/

airflow apache apache-airflow python scheduler workflow automation dag data-engineering data-integration

apache / flink-cdc

data-integration,Flink CDC is a streaming data integration tool

Organization: apache

Home Page: https://nightlies.apache.org/flink/flink-cdc-docs-stable

change-data-capture cdc batch data-integration data-pipeline distributed elt etl flink kafka mysql paimon postgresql real-time schema-evolution

apache / hop

data-integration,Hop Orchestration Platform

Organization: apache

Home Page: https://hop.apache.org/

java streaming hop apache data-integration etl orchestration pipeline workflow

apache / hudi

data-integration,Upserts, Deletes And Incremental Processing on Big Data.

Organization: apache

Home Page: https://hudi.apache.org/

hudi apachehudi datalake bigdata apachespark incremental-processing stream-processing data-integration apacheflink

data-integration,Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

Organization: apache

Home Page: https://devlake.apache.org/

data data-analysis data-engineering data-integration data-transfers devops domain-layer etl golang integration jira open-source user-friendly dashboard-friendly dora hacktoberfest

apache / seatunnel

data-integration,SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.

Organization: apache

Home Page: https://seatunnel.apache.org/

data-integration high-performance offline real-time apache batch cdc change-data-capture data-ingestion elt

apache / seatunnel-web

data-integration,SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

Organization: apache

Home Page: https://seatunnel.apache.org/

apache data-integration data-pipeline etl-framework high-performance offline real-time seatunnel sql-engine

artie-labs / transfer

data-integration,Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.

Organization: artie-labs

Home Page: https://artie.so

snowflake cdc change-data-capture golang bigquery kafka apache-kafka data-integration data-pipelines database

avaiga / taipy

data-integration,Turns Data and AI algorithms into production-ready web applications in no time.

Organization: avaiga

Home Page: https://www.taipy.io

automation data-engineering data-ops data-visualization datascience developer-tools mlops orchestration pipeline pipelines

bruin-data / ingestr

data-integration,ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

Organization: bruin-data

Home Page: https://bruin-data.github.io/ingestr/

bigquery copy-database data-ingestion data-integration data-pipeline duckdb ingestion-pipeline mssql postgresql snowflake

bytedance / bitsail

data-integration,BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

Organization: bytedance

Home Page: https://bytedance.github.io/bitsail/

flink big-data data-integration data-lake data-pipeline data-synchronization high-performance real-time

categoricaldata / cql

data-integration,Categorical Query Language IDE

Organization: categoricaldata

Home Page: http://categoricaldata.net

cql category-theory functional-programming data-migratation etl data-integration data-science data-exchange

ceumicrodata / metl

data-integration,mito ETL tool

Organization: ceumicrodata

python data-integration etl etl-framework pipeline

cloudquery / cloudquery

data-integration,The open source high performance ELT framework powered by Apache Arrow

Organization: cloudquery

Home Page: https://cloudquery.io

aws gcp azure sql data-integration elt etl etl-framework bigquery data-collection

commoncoreontology / commoncoreontologies

data-integration,The Common Core Ontology Repository holds the current released version of the Common Core Ontology suite.

Organization: commoncoreontology

cco applied-ontology bfo semantics ontologies ontology-suite owl-ontology interoperability data-integration semantic-consistency

conduitio / conduit

data-integration,Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

Organization: conduitio

Home Page: https://conduit.io

go data-integration etl data-pipeline data-engineering data-stream conduit kafka kafkaconnect

cuebook / cuelake

data-integration,Use SQL to build ELT pipelines on a data lakehouse.

Organization: cuebook

Home Page: https://cuelake.cuebook.ai

apache-iceberg delta lakehouse datalake data-lake elt etl data-engineering data-integration data-ingestion

dagster-io / dagster

data-integration,An orchestration platform for the development, production, and observation of data assets.

Organization: dagster-io

Home Page: https://dagster.io

data-pipelines dagster workflow data-science workflow-automation python scheduler data-orchestrator etl analytics

dataplane-app / dataplane

data-integration,Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.

Organization: dataplane-app

Home Page: https://dataplane.app

golang airflow kubernetes data scheduler workflows workflow-automation data-pipelines data-science data-engineering

dtstack / chunjun

data-integration,A data integration framework

Organization: dtstack

Home Page: https://dtstack.github.io/chunjun/

flink bigdata data-integration framework java

genular / pandora

data-integration,PANDORA - Predictive Analytics aNd Data Oriented Research Applications :computer:

Organization: genular

Home Page: https://genular.org

machine-learning systems-biology data-science data-visualization data-mining pandora bioinformatics biomarkers clinical-data clustering

google / megalista

data-integration,First Party data integration solution built for marketing teams to enable audience and conversion onboarding into Google Marketing products (Google Ads, Campaign Manager, Google Analytics).

Organization: google

google googleads data-integration googleanalytics dataflow python bigquery customermatch conversions audience-targeting

hetio / hetionet

data-integration,Hetionet: an integrative network of disease

Organization: hetio

Home Page: https://neo4j.het.io

hetionet rephetio network neo4j hetnet drug-repurposing data-integration

immunogenomics / harmony

data-integration,Fast, sensitive and accurate integration of single-cell data with Harmony

Organization: immunogenomics

Home Page: https://portals.broadinstitute.org/harmony/

algorithm data-integration r scrna-seq

infinyon / fluvio

data-integration,Lean and mean distributed stream processing system written in rust and web assembly.

Organization: infinyon

Home Page: https://www.fluvio.io/

cloud-native streaming rust real-time serverless stateful stream-processing webassembly data-integration data-flow distributed-systems event-driven-architecture stream-processing-engine data-pipelines streaming-data streaming-data-pipelines streaming-data-processing

jitsucom / jitsu

data-integration,Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

Organization: jitsucom

Home Page: https://jitsu.com

data-integration clickhouse golang bigquery data-collection data-connectors redshift snowflake postgres

jupyter-naas / drivers

data-integration,Low-code Python library enabling access to APIs, tools, data sources in seconds.

Organization: jupyter-naas

data data-science etl integration lowcode open-source opensource pipeline pipelines data-analysis

kestra-io / kestra

data-integration,Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

Organization: kestra-io

Home Page: https://kestra.io

workflow workflow-engine orchestration scheduler data-pipeline elt etl data data-engineering data-orchestration

kuwala-io / kuwala

data-integration,Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times

Organization: kuwala-io

Home Page: https://kuwala.io

data data-integration data-science open-data spatial-analysis elt kuwala open-source scraping dbt

leesf / hudi-resources

data-integration,汇总Apache Hudi相关资料

User: leesf

hudi apachehudi apache hudi-resources datalake bigdata stream-processing incremental-processing data-integration

mage-ai / mage-ai

data-integration,🧙 Build, run, and manage data pipelines for integrating and transforming data.

Organization: mage-ai

Home Page: https://www.mage.ai/

machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines

mara / mara-example-project-2

data-integration,An example mini data warehouse for python project stats, template for new projects

Organization: mara

etl pypi bigquery data-integration sql

mara / mara-pipelines

data-integration,A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

Organization: mara

etl data-integration python postgresql pipeline data

mims-harvard / scikit-fusion

data-integration,scikit-fusion: Data fusion via collective latent factor models

Organization: mims-harvard

matrix-factorization data-fusion latent-features data-integration embeddings knowledge-graphs

morph-kgc / morph-kgc

data-integration,Powerful RDF Knowledge Graph Generation with RML Mappings

Organization: morph-kgc

Home Page: https://morph-kgc.readthedocs.io

knowledge-graph data-integration rml r2rml rdf database python etl data-engineering rdf-star

olehmberg / winter

data-integration,WInte.r is a Java framework for end-to-end data integration. The WInte.r framework implements well-known methods for data pre-processing, schema matching, identity resolution, data fusion, and result evaluation.

User: olehmberg

data-fusion identity-resolution schema-matching tabular-data data-integration

opensanctions / nomenklatura

data-integration,Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources

Organization: opensanctions

data-integration deduplication record-link

recap-build / recap

data-integration,Work with your web service, database, and streaming schemas in a single format.

Organization: recap-build

Home Page: https://recap.build

data-catalog metadata data-discovery recap data-engineering data-integration data-pipelines etl

rudderlabs / rudder-server

data-integration,Privacy and Security focused Segment-alternative, in Golang and React

Organization: rudderlabs

Home Page: https://www.rudderstack.com/

golang hybrid-cloud privacy security warehouse-management data-warehouse rudderstack customer-data customer-data-pipeline customer-data-platform

runprism / prism

data-integration,Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python.

Organization: runprism

Home Page: https://runprism.com

data data-engineering data-science etl etl-pipeline machine-learning orchestration pipeline python data-orc

saeyslab / nichenetr

data-integration,NicheNet: predict active ligand-target links between interacting cells

Organization: saeyslab

single-cell-omics intercellular-communication ligand-receptor ligand-target data-integration single-cell-rna-seq rna-seq gene-expression network-inference cell-cell-communication

saezlab / cosmosr

data-integration,COSMOS (Causal Oriented Search of Multi-Omic Space) is a method that integrates phosphoproteomics, transcriptomics, and metabolomics data sets.

Organization: saezlab

Home Page: https://saezlab.github.io/cosmosR/

metabolomic-data proteomics phosphoproteomics data-integration transcriptomics network-modelling