mgramin / database-as-code

Treat your database as Code

Hi there 👋

My name is Maksim Gramin. I’m a software engeneer, consultant and researcher passionate about database technologies and SQL.

I have spent a lot of time working with various database systems. I was a SQL-developer on big distributed teams, implementing tons of complex business logic on a database side. And I worked as a classical Java developer and a team leader where we built monolith and microservices applications and interacted with any databases mainly through ORM or something like that.

And I do want to share my experience and create tools and systems for simpler and more efficient work with data.

database-as-code's Issues

Add malewicz

Add Liquibase

Add "Programmatic APIs" item

Add pigsty to tools: Database as Code for PostgreSQL / Redis / Greenplum

Pigsty: Describe database with code, then create them in one-click

Examples:

#----------------------------------#
# pgsql cluster: pg-meta (CMDB)    #
#----------------------------------#
# EXAMPLE pg-meta is the default SINGLE-NODE pgsql cluster deployed on meta node (10.10.10.10)
pg-meta:
  hosts: { 10.10.10.10: { pg_seq: 1, pg_role: primary , pg_offline_query: true } }
  vars:
    pg_cluster: pg-meta
    patroni_watchdog_mode: off
    pg_databases:
      - { name: meta, baseline: cmdb.sql , schemas: [ pigsty ]   , extensions: [ { name: adminpack, schema: pg_catalog } ,  { name: postgis, schema: public } , { name: timescaledb } ] ,comment: pigsty meta database }
      - { name: grafana     , owner: dbuser_grafana     , revokeconn: true , comment: grafana primary database }
      - { name: bytebase    , owner: dbuser_bytebase    , revokeconn: true , comment: bytebase primary database }
      - { name: kong        , owner: dbuser_kong        , revokeconn: true , comment: kong the api gateway database }
      - { name: confluence  , owner: dbuser_confluence  , revokeconn: true, connlimit: 100 }
      - { name: gitlab      , owner: dbuser_gitlab      , revokeconn: true, connlimit: 100 }
      - { name: jira        , owner: dbuser_jira        , revokeconn: true, connlimit: 100 }

    pg_users:
      - { name: dbuser_meta       ,password: DBUser.Meta       , pgbouncer: true ,roles: [ dbrole_admin ]    ,comment: pigsty cmdb admin user }
      - { name: dbuser_view       ,password: DBUser.Viewer     , pgbouncer: true ,roles: [ dbrole_readonly ] ,comment: read-only viewer for meta database }
      - { name: dbuser_grafana    ,password: DBUser.Grafana    , pgbouncer: true ,roles: [ dbrole_admin ]    ,comment: admin user for grafana database }
      - { name: dbuser_bytebase   ,password: DBUser.Bytebase   , pgbouncer: true ,roles: [ dbrole_admin ]    ,comment: admin user for bytebase database }
      - { name: dbuser_kong       ,password: DBUser.Kong       , pgbouncer: true ,roles: [ dbrole_admin ]    ,comment: admin user for kong api gateway }
      - { name: dbuser_confluence ,password: DBUser.Confluence , pgbouncer: true, roles: [ dbrole_admin ] }
      - { name: dbuser_gitlab     ,password: DBUser.Gitlab     , pgbouncer: true, roles: [ dbrole_readwrite ] }
      - { name: dbuser_jira       ,password: DBUser.Jira       , pgbouncer: true, roles: [ dbrole_admin ] }

    pg_hba_rules_extra:
      - title: allow grafana bytebase kong intranet access
        role: common
        rules:
          - host    kong            dbuser_kong         10.0.0.0/8          md5
          - host    bytebase        dbuser_bytebase     10.0.0.0/8          md5
          - host    grafana         dbuser_grafana      10.0.0.0/8          md5

    vip_mode: l2
    vip_address: 10.10.10.2
    vip_cidrmask: 8
    vip_interface: eth1
    node_crontab:
      - '00 01 * * * postgres /pg/bin/pg-backup 2>>/pg/log/backup.log'

# pg-test is the original cluster
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-test
    pg_users: [ { name: test , password: test , pgbouncer: true , roles: [ dbrole_admin ] , comment: test user } ]
    pg_databases: [ { name: test , extensions: [ { name: postgis, schema: public } ] } ]

# pg-testdelay is a standby cluster with 1h apply delay
pg-testdelay:
  hosts:
    10.10.10.13: { pg_seq: 1, pg_role: primary , pg_upstream: 10.10.10.11 , pg_delay: 1h }
  vars:
    pg_cluster: pg-testdelay

#----------------------------------#
# redis sentinel example           #
#----------------------------------#
redis-meta:
  hosts:
    10.10.10.10:
      redis_node: 1
      redis_instances:  { 6001 : {} ,6002 : {} , 6003 : {} }
  vars:
    redis_cluster: redis-meta
    redis_mode: sentinel
    redis_max_memory: 64MB

#----------------------------------#
# redis cluster example            #
#----------------------------------#
redis-test:
  hosts:
    10.10.10.11:
      redis_node: 1
      redis_instances: { 6501 : {} ,6502 : {} ,6503 : {} ,6504 : {} ,6505 : {} ,6506 : {} }
    10.10.10.12:
      redis_node: 2
      redis_instances: { 6501 : {} ,6502 : {} ,6503 : {} ,6504 : {} ,6505 : {} ,6506 : {} }
  vars:
    redis_cluster: redis-test           # name of this redis 'cluster'
    redis_mode: cluster                 # standalone,cluster,sentinel
    redis_max_memory: 32MB              # max memory used by each redis instance
    redis_mem_policy: allkeys-lru       # memory eviction policy

#----------------------------------#
# redis standalone example         #
#----------------------------------#
redis-common:
  hosts:
    10.10.10.13:
      redis_node: 1
      redis_instances:
        6501: {}
        6502: { replica_of: '10.10.10.13 6501' }
        6503: { replica_of: '10.10.10.13 6501' }
  vars:
    redis_cluster: redis-common         # name of this redis 'cluster'
    redis_mode: standalone              # standalone,cluster,sentinel
    redis_max_memory: 64MB              # max memory used by each redis in

Then create them & apply change with built-in ansible playbooks.

Add Soda.io

https://www.soda.io - Data Reliability Engineering as Code

Collect all related links

Add repo examples

Add aiosql

https://github.com/nackjicholson/aiosql