hartl3y94 / database-lab-engine

Thin PostgreSQL clones. DLE provides blazing fast database cloning to build powerful development, test, QA, staging environments. Follow to stay updated.

Database Lab Engine (DLE)

⚡ Blazing-fast cloning of PostgreSQL databases 🐘
Thin clones of PostgreSQL to build powerful development, test, QA, staging environments.
_{Available for any PostgreSQL, including AWS RDS, GCP CloudSQL, Heroku, Digital Ocean, and self-managed instances.}

Features | Documentation | Blog | Community & Support | Contributing

Why DLE?

• Build dev/QA/staging environments based on full-size production-like databases.
• Provide temporary full-size database clones for SQL query analysis and optimization (see also: SQL optimization chatbot Joe).
• Automatically test database changes in CI/CD pipelines to avoid incidents in production.

For example, cloning a 1 TiB PostgreSQL database takes ~10 seconds. Dozens of independent clones are up and running on a single machine, supporting lots of development and testing activities, not increasing costs for hardware.

How it works

Thin cloning is fast because it uses Copy-on-Write (CoW). DLE supports two technologies to enable CoW and thin cloning: ZFS (default) and LVM.

With ZFS, Database Lab Engine periodically creates a new snapshot of the data directory and maintains a set of snapshots, cleaning up the old and unused ones. When requesting a new clone, users can choose which snapshot to use.

Read more:

• How it works
• Database Migration Testing
• SQL Optimization with Joe Bot
• Questions and answers

Where to start

• Database Lab tutorial for any PostgreSQL database
• Database Lab tutorial for Amazon RDS
• Terraform module template (AWS)

Case studies

• Qiwi: How Qiwi Controls the Data to Accelerate Development
• GitLab: How GitLab iterates on SQL performance optimization workflow to reduce downtime risks

Features

• Blazing-fast cloning of Postgres databases – a few seconds to create a new clone ready to accept connections and queries, regardless of the database size.
• The theoretical maximum number of snapshots and clones is 2⁶⁴ (ZFS, default).
• The theoretical maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2¹²⁸ bytes (ZFS, default).
• PostgreSQL major versions supported: 9.6–14.
• Two technologies are supported to enable thin cloning (CoW): ZFS and LVM.
• All components are packaged in Docker containers.
• UI to make manual work more convenient.
• API and CLI to automate the work with DLE snapshots and clones.
• By default, PostgreSQL containers include many popular extensions (docs).
• PostgreSQL containers can be customized (docs).
• Source database can be located anywhere (self-managed Postgres, AWS RDS, GCP CloudSQL, Azure, Timescale Cloud, and so on) and does NOT require any adjustments. There are NO requirements to install ZFS or Docker to the source (production) databases.
• Initial data provisioning can be at both physical (pg_basebackup, backup / archiving tools such as WAL-G or pgBackRest), or logical (dump/restore directly from the source or from files stored at AWS S3) levels.
• For the logical mode, partial data retrieval is supported (specific databases, specific tables).
• For the physical mode, a continuously updated state is supported ("sync container"), making DLE a specialized version of standby Postgres.
• For the logical mode, periodical full refresh is supported, automated, and controlled by DLE. It is possible to use multiple disks containing different versions of the database, so full refresh won't require downtime.
• Fast Point in Time Recovery (PITR) to the points available in DLE snapshots.
• Unused clones are automatically deleted.
• "Deletion protection" flag can be used to block automatic or manual deletion of clones.
• Snapshot retention policies supported in DLE configuration.
• Persistent clones: clones survive DLE restarts (including full VM reboots).
• The "reset" command can be used to switch to a different version of data.
• DB Migration Checker component collects various artifacts useful for DB testing in CI (docs).
• SSH port forwarding for API and Postgres connections.
• Docker container config parameters can be specified in the DLE config.
• Resource usage quotas for clones: CPU, RAM (container quotas, supported by Docker)
• Postgres config parameters can be specified in the DLE config (separately for clones, the "sync" container, and the "promote" container).
• Monitoring: auth-free /healthz API endpoint, extended /status (requires auth), Netdata module.

How to contribute

Give the project a star

The easiest way to contribute is to give the project a GitHub/GitLab star:

Mention that you use DLE

Please post a tweet mentioning @Database_Lab or share the link to this repo in your favorite social network.

If you are actively using DLE at work, think about where you could mention it. The best way of mentioning it is using graphics with a link. Brand assets can be found in the ./assets folder. Feel free to put them in your documents, slide decks, application, and website interfaces to show that you use DLE.

HTML snippet for lighter backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-light-background.svg" />
</a>

Fro darker backgrounds:

<a href="http://databaselab.io">
  <img width="400" src="https://postgres.ai/assets/powered-by-dle-for-dark-background.svg" />
</a>

Propose an idea or report a bug

Check out our contributing guide for more details.

Participate in development

Check out our contributing guide for more details.

Reference guides

• DLE components
• DLE configuration reference
• DLE API reference
• Client CLI reference

How-to guides

• How to install Database Lab with Terraform on AWS
• How to install and initialize Database Lab CLI
• How to manage DLE
• How to work with clones

More you can found in the "How-to guides" section of the docs.

Miscellaneous

• DLE Docker images
• Extended Docker images for PostgreSQL (with plenty of extensions)
• SQL Optimization chatbot (Joe Bot)
• DB Migration Checker

License

DLE source code is licensed under the OSI-approved open source license GNU Affero General Public License version 3 (AGPLv3).

Reach out to the Postgres.ai team if you want a trial or commercial license that does not contain the GPL clauses: Contact page.

Community & Support

• "Database Lab Engine Community Covenant Code of Conduct"
• Where to get help: Contact page
• Community Slack
• If you need to report a security issue, follow instructions in "Database Lab Engine security guidelines".