Topic: dataops Goto Github

👇 Here are 142 public repositories matching this topic...

alibaba / sreworks
dataops,Cloud Native DataOps & AIOps Platform | 云原生数智运维平台
Organization: alibaba
Home Page: https://sreworks.cn
kubernetes sre application saas cloudnative dataops aiops oam engineering maintenance

awesome-mlops / awesome-ml-experiment-management
dataops, A curated list of awesome open source tools and commercial products for ML Experiment Tracking and Management 🚀
Organization: awesome-mlops
aiops awesome awesome-list data-science dataops datascience deep-learning experiment-management experiment-tracking kubernetes machine-learning ml-lifecycle mlops model-management model-registry orchestration

aws-samples / dataops-platform-airflow-dbt
dataops,Build DataOps platform with Apache Airflow and dbt on AWS
Organization: aws-samples
Home Page: https://aws.amazon.com/tr/blogs/big-data/build-a-dataops-platform-to-break-silos-between-engineers-and-analysts/?nc1=h_ls&advocacy_source=everyonesocial&trk=global_employee_advocacy&sc_channel=sm&es_id=ab2ae17684
aws airflow dbt dataops

awslabs / aws-ddk
dataops,An open source development framework to help you build data workflows and modern data architecture on AWS.
Organization: awslabs
Home Page: https://awslabs.github.io/aws-ddk/
aws dataops dataengineering python

azure-samples / modern-data-warehouse-dataops
dataops,DataOps for the Modern Data Warehouse on Microsoft Azure. https://aka.ms/mdw-dataops.
Organization: azure-samples
devops dataops data mdw azure databricks datafactory cicd automatedtesting

beneath-hq / beneath
dataops,Beneath is a serverless real-time data platform ⚡️
Organization: beneath-hq
Home Page: https://beneath.dev
dataops mlops data-science data-engineering data-pipelines developer-tools data-warehouse streaming etl analytics

c-3lab / dim
dataops,📦 dim: Manage the open data in your project like a package manager.
Organization: c-3lab
cli command-line-tool commads opendata dim package-manager data dataops public-data public-dataset

cleanlab / cleanlab
dataops,The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
Organization: cleanlab
Home Page: https://cleanlab.ai
weak-supervision data-cleaning data-quality data-science noisy-labels data-centric-ai out-of-distribution-detection outlier-detection active-learning data-labeling

datavane / tis
dataops,Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI
Organization: datavane
Home Page: https://tis.pub
java datax etl flink flink-streaming cdc dataops chunjun

elementary-data / elementary
dataops,The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
Organization: elementary-data
Home Page: https://www.elementary-data.com/
data-lineage data-governance data-warehouse snowflake bigquery data-analysis data-pipelines data-pipeline lineage data-reliability

flowerfine / scaleph
dataops,Open data platform based on Kubernetes. Scaleph supports SeaTunnel、Flink and Doris backended by SeaTunnel on Flink engine、Flink Kubernetes Operator and Doris operator.
Organization: flowerfine
Home Page: https://flowerfine.github.io/scaleph-repress-site/
data-platform dataops dag seatunnel flink flink-sql flink-sql-gateway flink-kubernetes flink-kubernetes-operator doris

flyteorg / flyte
dataops,Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
Organization: flyteorg
Home Page: https://flyte.org
flyte machine-learning golang scale workflow data-science data-analysis data kubernetes-operator kubernetes orchestration-engine mlops dataops grpc python production production-grade declarative fine-tuning llm

garystafford / tickit-data-lake-demo
dataops,Resources for video demonstrations and blog posts related to DataOps on AWS
User: garystafford
dataops aws devops airflow redshift data-lake

gojekfarm / beast
dataops,[Deprecated] Load data from Kafka to any data warehouse. BQ sink is being supported in Firehose now. https://github.com/odpf/firehose
Organization: gojekfarm
kafka bigquery beast warehouse dataops

google / grizzly
dataops,End-to-end DataOps platform deployed by Terraform.
Organization: google
dataops dataops-platform gcp terraform airflow bigquery cloud-storage composer data-catalog data-loss-prevention

google / space
dataops,Unified storage framework for the entire machine learning lifecycle
Organization: google
dataset lakehouse machine-learning mlops olap ray tensorflow tensorflow-dataset apache-arrow apache-parquet

jacopotagliabue / paas-data-ingestion
dataops,Ingesting data with Pulumi, AWS lambdas and Snowflake in a scalable, fully replayable manner
User: jacopotagliabue
Home Page: https://towardsdatascience.com/tagged/mlops-without-much-ops
mlops dataops snowflake severless dbt

kelvins / awesome-dataops
dataops,:sunglasses: A curated list of awesome DataOps tools
User: kelvins
awesome awesome-list dataops data-engineering data-engineer

lancedb / lance
dataops,Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..
Organization: lancedb
Home Page: https://lancedb.github.io/lance/
machine-learning computer-vision data-format deep-learning python apache-arrow duckdb mlops data-analysis data-analytics data-science dataops data-centric embeddings rust llms

lensesio / fast-data-dev
dataops,Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors
Organization: lensesio
Home Page: https://lenses.io
kafka-rest-proxy schema-registry kafka docker dataops

lensesio / kafka-helm-charts
dataops,In-deprecation. For Lenses please check lensesio/lenses-helm-charts. Soon Stream Reactor will also get its own Helm repository.
Organization: lensesio
Home Page: https://lenses.io
kubernetes helm kafka kafka-connect prometheus dataops

lensesio / lenses-docker
dataops,❤for real-time DataOps - where the application and data fabric blends - Lenses
Organization: lensesio
Home Page: https://docs.lenses.io
kafka kubernetes openshift docker governance security enterprise dataops

meltano / meltano
dataops,Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
Organization: meltano
Home Page: https://meltano.com/
dataops dataops-platform elt open-source opensource data pipelines extract-data meltano meltano-sdk

merantix-momentum / squirrel-core
dataops,A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way :chestnut:
Organization: merantix-momentum
Home Page: https://squirrel-core.readthedocs.io/
python ml machine-learning data-science computer-vision cv nlp natural-language-processing ai pytorch

noahgift / data-engineering-and-dataops
dataops,Duke MIDS: Data Engineering and DataOps Course
User: noahgift
Home Page: https://noahgift.github.io/data-engineering-and-dataops/
duke cloud dataops dataengineering data data-science software-engineering mlops book course

noronha-dataops / noronha
dataops,DataOps framework for Machine Learning projects.
User: noronha-dataops
Home Page: https://noronha-dataops.readthedocs.io/
machine-learning dataops mlops docker kubernetes

opendatadiscovery / awesome-data-catalogs
dataops,📙 Awesome Data Catalogs and Observability Platforms.
Organization: opendatadiscovery
data-catalog data-discovery metadata dataops awesome observability data-engineering data-quality big-data opensource

pbi-tools / pbi-tools
dataops,Power BI DevOps & Source Control Tool
Organization: pbi-tools
Home Page: https://pbi.tools/cli
power-bi pbix source-control devops dataops

polyaxon / traceml
dataops,Engine for ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.
Organization: polyaxon
pandas pandas-summary dataframes data-science spark dask plotly statistics matplotlib data-profiling

raptor-ml / raptor
dataops,Transform your pythonic research to an artifact that engineers can deploy easily.
Organization: raptor-ml
Home Page: https://raptor.ml
ai-infra data-science feature-engineering feature-extraction kubernetes machine-learning ml production kubeflow mlops

raystack / compass
dataops,Compass is an enterprise data catalog that makes it easy to find, understand, and govern data.
Organization: raystack
Home Page: https://compass-raystack.vercel.app/
data dataops discovery lineage metadata

raystack / cosmos
dataops,Cosmos is an operational analytics server to build custom apps with embedded analytics that deliver data experiences as unique as your business.
Organization: raystack
analysis cubejs dataops framework metrics

raystack / dagger
dataops,Dagger is an easy-to-use, configuration over code, cloud-native framework built on top of Apache Flink for stateful processing of real-time streaming data.
Organization: raystack
Home Page: https://raystack.github.io/dagger/
apache-flink apache-kafka dataops framework influxdb prometheus real-time-analytics real-time-processing stream-processing

raystack / firehose
dataops,Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.
Organization: raystack
Home Page: https://raystack.github.io/firehose/
apache-kafka bigquery dataops firehose influxdb kafka postgresql prometheus sink streaming

raystack / frontier
dataops,Frontier is an all-in-one user management platform that provides identity, access and billing management to help organizations secure their systems and data. (Open source alternative to Clerk)
Organization: raystack
Home Page: https://raystack-frontier.vercel.app/
authentication authorization billing clerkauth dataops golang rbac spicedb user-management

raystack / guardian
dataops,Guardian is universal data access management tool with automated access workflows and security controls across data stores, analytical systems, and cloud products.
Organization: raystack
Home Page: https://guardian.vercel.app/
access compliance control data dataops

raystack / meteor
dataops,Meteor is an easy-to-use, plugin-driven metadata collection framework to extract data from different sources and sink to any data catalog.
Organization: raystack
Home Page: https://raystack.github.io/meteor/
metadata collector scraper extractors sinks bigdata data-catalog data-management dataops

raystack / optimus
dataops,Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Organization: raystack
Home Page: https://raystack.github.io/optimus
airflow analytics analytics-engineering automation bigquery business-intelligence data-modelling data-pipelines data-transformation data-warehouse dataops elt etl golang workflows

raystack / raccoon
dataops,Raccoon is a high-throughput, low-latency service to collect events in real-time from your web, mobile apps, and services using multiple network protocols.
Organization: raystack
Home Page: https://raystack.github.io/raccoon/
clickstream dataops eventsourcing kafka

raystack / siren
dataops,Siren provides an easy-to-use universal alert, notification, channels management framework for the entire observability infrastructure.
Organization: raystack
Home Page: https://odpf.github.io/siren/
alerting dataops influx monitoring prometheus

raystack / stencil
dataops,Stencil is a schema registry that provides schema management and validation dynamically, efficiently, and reliably to ensure data compatibility across applications.
Organization: raystack
Home Page: https://raystack.github.io/stencil/
cli clojure clojure-library dataops descriptor golang javascript javascript-library js protobuf protocol-buffers protocol-buffers-library protocol-buffers-parsing schema-registry schema-validation

redpanda-data / console
dataops,Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging.
Organization: redpanda-data
Home Page: https://redpanda.com
apache-kafka dataops react typescript kafka-ui kafka-gui web-ui go kafka

sernst / cauldron
dataops,Interactive computing for complex data processing, modeling and analysis in Python 3
User: sernst
Home Page: http://www.unnotebook.com
data-science python-3 notebook notebooks data-engineering dataops

taivop / awesome-data-annotation
dataops,A list of tools for annotating data, managing annotations, etc.
User: taivop
Home Page: https://github.com/taivop/awesome-data-annotation
data-annotation image-annotation computer-vision annotations nlp awesome-list awesome dataops

tenzir / tenzir
dataops,Open source security data pipelines.
Organization: tenzir
Home Page: https://docs.tenzir.com
incident-response threathunting siem soc security dataops investigation pcap secdataops netflow

titan-systems / titan
dataops,Snowflake infrastructure-as-code. Provision environments, automate deploys, CI/CD. Manage RBAC, users, roles, and data access. Declarative Python Resource API. Change Management tool for the Snowflake data warehouse.
Organization: titan-systems
data-engineering data-warehouse snowflake data-governance dataops secops compliance-as-code devops rbac

tobikodata / sqlmesh
dataops,Efficient data transformation and modeling framework that is backwards compatible with dbt.
Organization: tobikodata
Home Page: https://sqlmesh.com
dataops elt etl sql python dataengineering transformation dbt

vmware / versatile-data-kit
dataops,One framework to develop, deploy and operate data workflows with Python and SQL.
Organization: vmware
data-science data-engineering sql trino data-lineage etl elt data-pipelines data-engineer data-warehouse

waterdipai / datachecks
dataops,Open Source Data Quality Monitoring.
Organization: waterdipai
Home Page: https://datachecks.io
data-engineering data-validation dataops dataquality metrics mlops postgresql python data-governance data-observability

whylabs / whylogs
dataops,An open-source data logging library for machine learning models and data pipelines. 📚 Provides visibility into data quality & model performance over time. 🛡️ Supports privacy-preserving data collection, ensuring safety & robustness. 📈
Organization: whylabs
Home Page: https://whylogs.readthedocs.io/
ai-pipelines approximate-statistics statistical-properties data-quality calculate-statistics python logging mlops dataops ml-pipelines