BattDB is a database based on Timescaledb (Postgres 15 extension) to store battery-related field and experimental (physical and virtual) data, as well as the metatada. Detailed database documentation and a few deployment options are provided.
A .dbml file for the database is available in the diagrams directory. A condensed version in PDF form here. The condensed version only includes the tables and columns that are used by the ETL process.
BattDB can be deployed in following ways: locally using Docker-compose or using the migration scripts on self-hosted or managed Timescaledb server or remotely using Ansible on bare-metal.
A Docker Compose configuration file is provided to facilitate local deployment of the BattDB database.
In order to use Docker Compose, you will need to have the following software installed on your machine:
-
Docker - a platform for developing, shipping, and running applications in containers.
-
Docker Compose - a tool for defining and running multi-container Docker applications.
-
Enter the compose directory:
cd assets/battdb_docker/ -
Modify the
.envfile, change the environment variables:POSTGRES_USER=YOUR_USERNAME POSTGRES_PASSWORD=YOUR_PASSWORD POSTGRES_DATABASE=YOUR_DATABASE POSTGRES_PORT=YOUR_PORT FLYWAY_SQL=../migration_scriptsUse
FLYWAY_SQL=../migration_scripts_quickfor quick mode. -
Run Docker Compose:
docker network create battsoft-net docker-compose --env-file .env up -d
-
Wait for Docker Compose to complete, and the
BattDBdatabase has been successfully deployed. The data for the database will be stored in the directoryassets/Docker/data.
Please note that if you need to update your database version, you can simply re-run the Ansible or Docker Compose deployment commands. This will apply any necessary updates using Docker & Flyway. However, if you need to manually migrate your database, you can use Flyway as described below.
If you need to manually migrate your database, follow these steps:
- Download and install Flyway from https://documentation.red-gate.com/fd/command-line-184127404.html.
- Navigate to the Flyway folder and edit the conf/flyway.conf file with the following parameters:
flyway.url=jdbc:postgresql://[URL]:[PORT]/[DATABASE]
# Example: flyway.url=jdbc:postgresql://localhost:5432/battdb
flyway.user=[USERNAME]
flyway.password=[PASSWORD]
- Update to latest version of the database
flyway migrate -locations=filesystem:./assets/migration_scriptsor update to specific version
flyway -target="[VERSION]" migrate -locations=filesystem:./assets/migration_scriptsAn Ansible playbook is provided to quickly deploy the BattDB database remotely.
- The remote host needs to have SSH service enabled.
- The remote host needs to have root privileges.
In order to use Ansible, you will need to have the following software installed on your machine:
-
Ansible - a configuration management tool that allows you to automate the deployment and configuration of software and services.
sudo apt install ansible
-
sshpass - a tool that allows you to provide a password for ssh connections.
sudo apt install sshpass
Please note that if you are using SSH key authentication to log in to the client machines, you do not need to install
sshpass.sshpassis a tool used to automate password input during SSH connections. However, it is not recommended to use sshpass as it may compromise the security of the system. Instead, it is recommended to use SSH key authentication which provides a more secure way of authentication.If you have already set up SSH key authentication for your client machines, you can skip the installation of sshpass and proceed with your Ansible playbook. You can refer to the
hostsfile for examples on how to usessh_passandssh_private_key_filefor password and SSH key authentication respectively.
battdb.yml:
- Install Python3
- Install Install Docker and Docker Compose
- Clone
db_bg_cell_testing_main - Run Docker Compose to create database
-
Modify the
hostsfile, change the IP address, username, and password of the remote host- Refer to the
hostsfile for an example. If you are using a private key to log in, add your private key to theassets/ssh/directory.
- Refer to the
-
Modify the
variables/postgresql.ymlfile, change the credentials:--- db_user: YOUR_USERNAME db_password: YOUR_PASSWORD db_name: YOUR_DATABASE db_port: YOUR_PORT
-
Run the playbook using the following command:
ansible-playbook battdb.yml --extra-vars "variable_host=dev" --ask-become-passPlease note that --ask-become-pass is used to prompt for the sudo password. However, if you are using AWS EC2 instances, you may not need to use this option as the instances may be configured to allow passwordless sudo access.
-
Wait for the playbook to complete, and the
BattDBdatabase has been successfully deployed. The data for the database will be stored in the directory~/battdb/data.
To establish an SSL connection with the database, please refer to the PostgreSQL official documentation:
https://www.postgresql.org/docs/current/ssl-tcp.html
You will need at least five files:
- root.crt
- server.crt
- server.key
- client.crt
- client.key
Please place root.crt, server.crt and server.key in the BattDB/assets/battdb_docker/data/battdb directory.
You might need to modify the following content in postgresql.conf:
ssl = on
ssl_ca_file = '/var/lib/postgresql/data/root.crt'
ssl_cert_file = '/var/lib/postgresql/data/server.crt'
ssl_key_file = '/var/lib/postgresql/data/server.key'
ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers
ssl_prefer_server_ciphers = on
We recommend adding the following settings:
Allow password login for containers under the same battsoft-net Docker network:
host all all 172.0.0.0/8 scram-sha-256
External connections should use SSL:
hostssl all all all cert
Restart the docker container to apply the SSL settings.
Please adjust these configurations based on your requirements and ensure that the specified files and paths are accurate for your setup.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent