Thin clones of PostgreSQL to build powerful development, test, QA, and staging environments.
Available for any PostgreSQL, including AWS RDS, GCP CloudSQL, Heroku, Digital Ocean, and self-managed instances.
- Build dev/QA/staging environments based on full-size production-like databases.
- Provide temporary full-size database clones for SQL query analysis and optimization (see also: SQL optimization chatbot Joe).
- Automatically test database changes in CI/CD pipelines to avoid incidents in production.
For example, cloning a 1 TiB PostgreSQL database takes ~10 seconds. Dozens of independent clones are up and running on a single machine, supporting lots of development and testing activities, without increasing costs for hardware.
Try it yourself right now:
- enter the Database Lab Platform, join the "Demo" organization, and test cloning of ~1 TiB demo database, or
- check out another demo setup, DLE CE: https://nik-tf-test.aws.postgres.ai:446/instance, use the token
demo
to enter (this setup has self-signed certificates, so ignore browser's complaints)
Thin cloning is fast because it uses Copy-on-Write (CoW). DLE supports two technologies to enable CoW and thin cloning: ZFS (default) and LVM.
With ZFS, Database Lab Engine periodically creates a new snapshot of the data directory and maintains a set of snapshots, cleaning up old and unused ones. When requesting a new clone, users can choose which snapshot to use.
Read more:
- Database Lab tutorial for any PostgreSQL database
- Database Lab tutorial for Amazon RDS
- Terraform module template (AWS)
- Qiwi: How Qiwi Controls the Data to Accelerate Development
- GitLab: How GitLab iterates on SQL performance optimization workflow to reduce downtime risks
- Blazing-fast cloning of Postgres databases โ a few seconds to create a new clone ready to accept connections and queries, regardless of database size.
- The theoretical maximum number of snapshots and clones is 264 (ZFS, default).
- The theoretical maximum size of PostgreSQL data directory: 256 quadrillion zebibytes, or 2128 bytes (ZFS, default).
- PostgreSQL major versions supported: 9.6โ14.
- Two technologies are supported to enable thin cloning (CoW): ZFS and LVM.
- All components are packaged in Docker containers.
- UI to make manual work more convenient.
- API and CLI to automate the work with DLE snapshots and clones.
- By default, PostgreSQL containers include many popular extensions (docs).
- PostgreSQL containers can be customized (docs).
- Source database can be located anywhere (self-managed Postgres, AWS RDS, GCP CloudSQL, Azure, Timescale Cloud, and so on) and does NOT require any adjustments. There are NO requirements to install ZFS or Docker to the source (production) databases.
- Initial data provisioning can be done at either the physical (pg_basebackup, backup / archiving tools such as WAL-G or pgBackRest) or logical (dump/restore directly from the source or from files stored at AWS S3) level.
- For logical mode, partial data retrieval is supported (specific databases, specific tables).
- For physical mode, a continuously updated state is supported ("sync container"), making DLE a specialized version of standby Postgres.
- For logical mode, periodic full refresh is supported, automated, and controlled by DLE. It is possible to use multiple disks containing different versions of the database, so full refresh won't require downtime.
- Fast Point in Time Recovery (PITR) to the points available in DLE snapshots.
- Unused clones are automatically deleted.
- "Deletion protection" flag can be used to block automatic or manual deletion of clones.
- Snapshot retention policies supported in DLE configuration.
- Persistent clones: clones survive DLE restarts (including full VM reboots).
- The "reset" command can be used to switch to a different version of data.
- DB Migration Checker component collects various artifacts useful for DB testing in CI (docs).
- SSH port forwarding for API and Postgres connections.
- Docker container config parameters can be specified in the DLE config.
- Resource usage quotas for clones: CPU, RAM (container quotas, supported by Docker)
- Postgres config parameters can be specified in the DLE config (separately for clones, the "sync" container, and the "promote" container).
- Monitoring: auth-free
/healthz
API endpoint, extended/status
(requires auth), Netdata module.
The easiest way to contribute is to give the project a GitHub/GitLab star:
Post a tweet mentioning @Database_Lab or share the link to this repo in your favorite social network.
If you are actively using DLE, tell others about your experience. You can use the logo referenced below and stored in the ./assets
folder. Feel free to put them in your documents, slide decks, application, and website interfaces to show that you use DLE.
HTML snippet for lighter backgrounds:
<a href="http://databaselab.io">
<img width="400" src="https://postgres.ai/assets/powered-by-dle-for-light-background.svg" />
</a>
For darker backgrounds:
<a href="http://databaselab.io">
<img width="400" src="https://postgres.ai/assets/powered-by-dle-for-dark-background.svg" />
</a>
Check out our contributing guide for more details.
Check out our contributing guide for more details.
Making Database Lab Engine more accessible to engineers around the Globe is a great help for the project. Check details in the translation section of contributing guide.
- How to install Database Lab with Terraform on AWS
- How to install and initialize Database Lab CLI
- How to manage DLE
- How to work with clones
More you can find in the "How-to guides" section of the docs.
- DLE Docker images
- Extended Docker images for PostgreSQL (with plenty of extensions)
- SQL Optimization chatbot (Joe Bot)
- DB Migration Checker
DLE source code is licensed under the OSI-approved open source license GNU Affero General Public License version 3 (AGPLv3).
Reach out to the Postgres.ai team if you want a trial or commercial license that does not contain the GPL clauses: Contact page.
- "Database Lab Engine Community Covenant Code of Conduct"
- Where to get help: Contact page
- Community Slack
- If you need to report a security issue, follow instructions in "Database Lab Engine security guidelines".