credativ / elephant-shed

PostgreSQL Management Appliance

Intro

Elephant Shed is a web-based PostgreSQL management front-end that bundles several utilities and applications for use with PostgreSQL. It currently manages single-node Linux PostgreSQL servers and appliances.

The main components are:

• PostgreSQL - https://www.postgresql.org/
• OmniDB - https://omnidb.org/
• postgresql-common - https://salsa.debian.org/postgresql/postgresql-common
• pgBadger - http://dalibo.github.io/pgbadger/
• pgBackRest - http://www.pgbackrest.org/
• Grafana - https://grafana.com/
• Prometheus - https://prometheus.io/
• Cockpit - http://cockpit-project.org/
• Shell In A Box - https://github.com/shellinabox/shellinabox
• tmate - https://tmate.io/

All PostgreSQL versions are supported via https://apt.postgresql.org.

Supported distributions:

• Debian Bullseye (11)
• Debian Buster (10)
• Debian Stretch (9)
• Ubuntu Focal (20.04)
• Ubuntu Bionic (18.04)
• RHEL/CentOS (7)

Download

Packages, Vagrant boxes, images: https://elephant-shed.io/#download

Documentation

The documentation is located at https://elephant-shed.io/doc/

Screenshots

Portal

The main portal webpage has links to all bundled applications. It also shows the PostgreSQL status, and allows starting/stopping the clusters, backup, and log reports.

Cockpit

For managing clusters and services, Cockpit is used.

Monitoring - Prometheus

Prometheus is a metric based monitoring system for servers and services. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.

Monitoring - Grafana

Grafana bundles metrics collected by Prometheus and presents them in a dashboard.

DBA Tool - OmniDB

OmniDB is a management tool for PostgreSQL to help DBAs execute many different tasks. It provides user management, DDL functionality, an interactive SQL shell, and more.

Backup - pgBackRest

The Elephant Shed comes with a preinstalled backup solution, pgBackRest.

Reporting - pgBadger

A pgBadger report is created for each PostgreSQL instance.

Web Terminal - Shell In A Box

Shell In A Box is a convenient web based terminal.

Remote Control - tmate

tmate is a fork of the popular terminal multiplexer tmux, tweaked for instant terminal sharing. It is preconfigured to connect to a relay server and enables the user to share the current terminal with a third party using a secret token.

License

The Elephant Shed itself is licensed under the GPLv3 (https://www.gnu.org/licenses/gpl-3.0.html).

All bundled components are Free/Open-Source software with a known and approved open source license.

Support

• Documentation: https://elephant-shed.io/doc/
• Web-Chat: #elephant-shed
• IRC #elephant-shed on irc.oftc.net

Elephant Shed is an open source project, developed and maintained by credativ.

For the Elephant Shed PostgreSQL appliance, credativ offers comprehensive technical support with service level agreements, which are also available on 365 days a year and 24 hours a day as an option.

Installation and integration support, as well as an introduction in Elephant Shed PostgreSQL appliance is of course also part of credativ's services. If you are interested, please feel free to contact us.

• Web credativ.de
• E-Mail: [email protected]
• Phone: [phone]

elephant-shed's Issues

OmniDB fails to start because it cannot initialize logging

I am getting this traceback when selecting omnidb on buster (manual package install):

[Tue Jun 15 07:55:11.059448 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118] mod_wsgi (pid=22714): Failed to exec Python script file '/usr/share/elephant-shed/omnidb/wsgi.py'.
[Tue Jun 15 07:55:11.059557 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118] mod_wsgi (pid=22714): Exception occurred processing WSGI script '/usr/share/elephant-shed/omnidb/wsgi.py'.
[Tue Jun 15 07:55:11.059964 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118] Traceback (most recent call last):
[Tue Jun 15 07:55:11.060036 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]   File "/usr/lib/python3.7/logging/config.py", line 562, in configure
[Tue Jun 15 07:55:11.060046 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]     handler = self.configure_handler(handlers[name])
[Tue Jun 15 07:55:11.060057 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]   File "/usr/lib/python3.7/logging/config.py", line 735, in configure_handler
[Tue Jun 15 07:55:11.060065 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]     result = factory(**kwargs)
[Tue Jun 15 07:55:11.060074 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]   File "/usr/lib/python3.7/logging/handlers.py", line 147, in __init__
[Tue Jun 15 07:55:11.060082 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]     BaseRotatingHandler.__init__(self, filename, mode, encoding, delay)
[Tue Jun 15 07:55:11.060092 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]   File "/usr/lib/python3.7/logging/handlers.py", line 54, in __init__
[Tue Jun 15 07:55:11.060099 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]     logging.FileHandler.__init__(self, filename, mode, encoding, delay)
[Tue Jun 15 07:55:11.060110 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]   File "/usr/lib/python3.7/logging/__init__.py", line 1092, in __init__
[Tue Jun 15 07:55:11.060117 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]     StreamHandler.__init__(self, self._open())
[Tue Jun 15 07:55:11.060128 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]   File "/usr/lib/python3.7/logging/__init__.py", line 1121, in _open
[Tue Jun 15 07:55:11.060135 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]     return open(self.baseFilename, self.mode, encoding=self.encoding)
[Tue Jun 15 07:55:11.060172 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118] FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/omnidb/.omnidb/omnidb-server/omnidb.log'
[Tue Jun 15 07:55:11.060186 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]
[Tue Jun 15 07:55:11.060193 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118] The above exception was the direct cause of the following exception:
[Tue Jun 15 07:55:11.060199 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]
[Tue Jun 15 07:55:11.060209 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118] Traceback (most recent call last):
[Tue Jun 15 07:55:11.060253 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]   File "/usr/share/elephant-shed/omnidb/wsgi.py", line 10, in <module>
[Tue Jun 15 07:55:11.060262 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]     application = get_wsgi_application()
[Tue Jun 15 07:55:11.060272 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]   File "/usr/lib/python3/dist-packages/django/core/wsgi.py", line 12, in get_wsgi_application
[Tue Jun 15 07:55:11.060279 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]     django.setup(set_prefix=False)
[Tue Jun 15 07:55:11.060288 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]   File "/usr/lib/python3/dist-packages/django/__init__.py", line 19, in setup
[Tue Jun 15 07:55:11.060295 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]     configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
[Tue Jun 15 07:55:11.060304 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]   File "/usr/lib/python3/dist-packages/django/utils/log.py", line 76, in configure_logging
[Tue Jun 15 07:55:11.060311 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]     logging_config_func(logging_settings)
[Tue Jun 15 07:55:11.060321 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]   File "/usr/lib/python3.7/logging/config.py", line 799, in dictConfig
[Tue Jun 15 07:55:11.060329 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]     dictConfigClass(config).configure()
[Tue Jun 15 07:55:11.060338 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]   File "/usr/lib/python3.7/logging/config.py", line 570, in configure
[Tue Jun 15 07:55:11.060344 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118]     '%r' % name) from e
[Tue Jun 15 07:55:11.060363 2021] [wsgi:error] [pid 22714:tid 140436323346176] [remote 192.168.122.1:60118] ValueError: Unable to configure handler 'logfile_django'

When I create /var/lib/omnidb/.omnidb/omnidb-server manually for the omnisb user, it loads fine

Support for Ubuntu Server

Hey,

this product looks really good. Do you also plan to support Ubuntu Server?

pgadmin4-apache2 is dead

and also
sudo apt install elephant-shed
ends with error:
elephant-shed : Depends: elephant-shed-pgadmin4 but it is not going to be installed

repository: deb https://packages.credativ.com/public/postgresql/ bionic-stable main

Vagrant image contains incorrect apt keys

When running apt-get update on a freshly built vagrant image:

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.credativ.com/public/postgresql stretch-test InRelease: The following signatures were invalid: EXPKEYSIG C86768840A59F867 aptly test repository (credativ) [email protected]
W: Failed to fetch https://packages.credativ.com/public/postgresql/dists/stretch-test/InRelease The following signatures were invalid: EXPKEYSIG C86768840A59F867 aptly test repository (credativ) [email protected]
W: Some index files failed to download. They have been ignored, or old ones used instead.

Installation-Docs are broken. Not working on Ubuntu or Centos...

I would like to inform you that the installation instructions for Elephant-shed on Ubuntu and Centos do not work.

We have now tried seven times with Ubuntu 20.04 and three times with 18.04. Without success. The installation of Ubuntu was fresh each time and it followed their guide 1:1.

Grafana has no dashboard when invoked.
Also, OmniDB sometimes doesn't work at all, sometimes it doesn't accept the password .

Are these errors known? Is there possibly a work-around?

On Centos we can not even load the repository. Credativ is not responding for E-Mails.

yum install https://packages.credativ.com/public/postgresql/yum/credativ-repo.rpm
Elephant Shed for Red Hat/CentOS 8 - x86_64 3.4 kB/s | 268 B 00:00
Errors during downloading metadata for repository 'credativ-arch':

• Status code: 404 for https://packages.credativ.com/public/postgresql/yum/el8/x86_64/repodata/repomd.xml (IP: 93.94.130.77)
  Fehler: Failed to download metadata for repo 'credativ-arch': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Show backup status in cluster overview

I have 8 databases of my clients to back up and check if the copy is not corrupted. Would Elephant Shed help me with this?

If not, could you point me to a panel that would facilitate the management of database backups?

Support for k8s (Docker)

Hey,

Do you also plan to support Docker/k8s installations?

Best,
Stanislav

Installation on debian 10 with postgres 11

Hi,
I found a few problems with installing elephant-shed on debian 10 following https://elephant-shed.io/doc/chapter/installation.html

1. every 30s I get: " postgres@postgres ERROR: pg_stat_statements must be loaded via shared_preload_libraries" in postgresql-Log

I think this comes from prometheus - adding "shared_preload_libraries = 'pg_stat_statements'" to postgres-conf fixes the issue

2. Activating archiving is not possible via Elephant-Shed Dashboard
   the script " /usr/share/elephant-shed/pgbackrest-toggle-archiving" only sets archive_command, archive_mode is not activated, WAL-Parameters are also not set

I fixed it by adding the parameters myself - but I think that is not the intention of elephant-shed. So my complete "fix" is a conf-file within /etc/postgresql/11/main/conf.d/
listen_addresses='*'
wal_level = replica
fsync = on
synchronous_commit = on
wal_sync_method = fsync
archive_mode = on
archive_timeout = 900
shared_preload_libraries = 'pg_stat_statements'

node-exporter fails to run on bullseye

The system metrics are missing and I see this in the journal:

Dez 21 21:18:43 elephant-shed systemd[1]: Started Prometheus exporter for machine metrics.
Dez 21 21:18:43 elephant-shed prometheus-node-exporter[27066]: prometheus-node-exporter: error: unknown long flag '--collector.netdev.ignored-devices', try --help
Dez 21 21:18:43 elephant-shed systemd[1]: prometheus-node-exporter.service: Main process exited, code=exited, status=1/FAILURE
Dez 21 21:18:43 elephant-shed systemd[1]: prometheus-node-exporter.service: Failed with result 'exit-code'.

Support for rhel8 and rhel9

Hi,
are there any plans to add support for rhel8 and rhel9?

End of Maintenance Support 1 was on August 6, 2020 and End of life will be on June 30, 2024

Thanks,
Ruben

Custom Port

How to change the web port?

Grafana etc. not working on CentOS 7

Hello,

I've just installed elephant shed on a fresh CentOS 7 system with PostgreSQL 12, but I can't see anything in Grafana: "Server" and "Cluster" are both showing "none" as the only entry, and all graphs are just showing "N/A".
But I can see my cluster 12/main on the start page, and there are also some values in cockpit etc., so, I'd suppose it's not a general problem.
Additionally, pgadmin is also not working, when I try to access it the browser just shows a http 500 error.
Any ideas what's wrong with my installation? Here's what I did:

1. yum install postgresql12-server
2. yum install https://packages.credativ.com/public/postgresql/yum/credativ-repo.rpm
3. yum install postgresql-common
4. pg_createcluster 12 main --start
5. yum install elephant-shed
6. vi /etc/group, added my user to elephant-shed group

Many thanks and best regards from Germany,

Harald

Consider to use a manger configurator to manage external or multiple cluster

Hi, I think this tools are awsome, and i'm interested in elephant-shed, but like you said in the note of 4.1.2. Cluster Administration section, "Note: Future versions of The Elephant Shed will include cluster management in the portal." This means I can't manage clusters out of the main Elephant-Shed enviroment (PostgreSQL, pgAdmin4, postgresql-common, pgBadger, pgBackRest, Grafana, Prometheus, Cockpit, Shell In A Box), I can just manage postgresql instance in the same cluster where all of Elephant-Shed tools are, isn't it?, because I want to connect all of them (Elephant-Shed Tools) to manage another postgresql cluster, like configurate postgresql connection to manage different postgresql cluster o instances and control them, like pgAdmin do it, but I suppose it would be an enterprise edition o Non-free edition, I said all of above because having (every and each of them -tools-) in one server, for security and performance it's a bit dangerous, i would like to receive notices and news from you to follow the advance and in one moment implement in my infrastructure, thanks a lot and regards.

pgbackrest-toggle-archiving is not idempotent

AFAICT, systemctl start pgbackrest-toggle-archiving@<cluster-name> turns archiving on or off, depending on the previous state.

That makes it really difficult to handle with Ansible, as running something like

- name: Enable pgBackrest archiving
  systemd:
    name: pgbackrest-toggle-archiving@{{ postgresql_cluster }}
    state: started

Will turn archiving on or off for each run, making the Ansible runs non-idempotent.

Is there a way to circumvent that? Conceptually, running systemctl enable/disable pgbackrest-archiving@<cluster-name> would be the best from a UX perspective, but systemd won't invoke the /usr/share/elephant-shed/pgbackrest-toggle-archiving script in this case but bail out.

ubuntu 20.04 instructions are not creating the omnidb config.py file by default

Reported on IRC:

22 07:03 <web-user-9> the ubuntu 20.04 instructions are not creating the omnidb config.py file by default
22 07:10 <web-user-9> im not sure what the real problem here is, but after clean ubuntu 20.04 install + running 
                      the instructions, i cant auth on omnidb, other services work fine
22 07:38 <web-user-9> seems like the app is never getting a proper init
22 07:51 <web-user-9> trying to figure it out but i cant find the documentation for omnidb in wsgi mode
22 08:19 <web-user-9> fixed it by creating config.py in /var/lib/omnidb/.omnidb/omnidb-server and running 
                      omnidb-config-server -r
22 08:19 <web-user-9> sqlite db was empty, somethings broken

how to join a new node in cluster already exists ?

I know it might be a simple/stupid question , but I can not find related documents.
let's said I create a new node , install elephant shed + postgresql , so there's only one node in main cluster now . so far so good .
then I create another new postgresql node , and I wanna join this node to the main cluster , how could I do that ?

Support for RPM-based Linux distributions

Hi,
This project looks great! Do you plan to support RPM based Linux distros (Fedora, Centos, Redhat)?

Apache config depends on omnidb user even if elephant-shed-omnidb is not installed

elephant-shed/portal/elephant-shed.conf

Line 83 in c349efd

WSGIDaemonProcess omnidb user=omnidb group=omnidb

This should be moved to the elephant-shed-omnidb package.

Vagrant up fails

I've tried to use the Vagrant variant, but there's an issue with the guest additions installation requiring the kernel module rebuild:

Err:2 http://security.debian.org/debian-security stretch/updates/main amd64 linux-libc-dev amd64 4.9.168-1+deb9u3
  404  Not Found [IP: 151.101.120.204 80]
Err:3 http://security.debian.org/debian-security stretch/updates/main amd64 linux-compiler-gcc-6-x86 amd64 4.9.168-1+deb9u3
  404  Not Found [IP: 151.101.120.204 80]
Err:4 http://security.debian.org/debian-security stretch/updates/main amd64 linux-headers-4.9.0-9-common all 4.9.168-1+deb9u3
  404  Not Found [IP: 151.101.120.204 80]
Err:5 http://security.debian.org/debian-security stretch/updates/main amd64 linux-kbuild-4.9 amd64 4.9.168-1+deb9u3
  404  Not Found [IP: 151.101.120.204 80]
Err:6 http://security.debian.org/debian-security stretch/updates/main amd64 linux-headers-4.9.0-9-amd64 amd64 4.9.168-1+deb9u3
  404  Not Found [IP: 151.101.120.204 80]
...
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

apt-get install -y linux-headers-`uname -r` dkms

Stdout from the command:

Reading package lists...
Building dependency tree...
Reading state information...


Stderr from the command:

E: Unable to locate package linux-headers-4.9.0-9-amd64
E: Couldn't find any package by glob 'linux-headers-4.9.0-9-amd64'
E: Couldn't find any package by regex 'linux-headers-4.9.0-9-amd64'

Dunno if this is temporary or a known problem with strech archive :-/

ferm not installed on Debian 10

Hi,
I did the package based installation on Debian 10.
Contrary to the docs, there is no trace of ferm to be found.

Using Elephant Shed with custom data directories

Hi,

I am currently trying to use the current stable release of Elephant Shed with custom PostgreSQL data directories. My data directories are named like /pgdata/pg1_1000 (1 & 1000 are variable) but the bin-directories are the default ones.
I already tried the solution from issue #7 .
I ran:
ln -s /pgdata/pg1_1000 /var/lib/postgresql/12/pg1_1000
ln -s /pgdata/pg1_1000 /etc/postgresql/12/pg1_1000

Then I also tried linking our custom service unit for systemd:
ln -s /etc/systemd/system/pg1_1000.service /lib/systemd/system/postgresql@12-pg1_1000.service

The cluster is correctly shown typing pg_lsclusters.
Then I implemented the recommended postgresql.conf for elephant-shed, but it still does not show up in the Elephant-Shed GUI.
But I recognized that pgBadger was showing a report for the cluster I created. So the link looks to be working.

What am I supposed to do to get my custom Postgres data directories to work with Elephant Shed?
Am I missing someting?

Thank you for your help.

question about getting elephant-shed to work with a customized PostgreSQL

Hi Support Team,

I have configured a customized PostgreSQL (Fujitsu Enterprise Postgres) in which all the binaries and libraries are in a non-default location. I wanted to know what files do i need to edit in order to update all the locations to have a completely working elephant shed. I have currently installed it on a host, I can only see the grafana-server and prometheus processes active while the rest of the services (systemd unit files) are either not started or not configured.

I can login to the portal but i could not see the clusters information and I take it that it is due to the fact that postgresql binaries are in a non-default location. I could navigate to grafana portal but all the meters and graphs are empty with not data in it.

Please help me to configure it so i can have a fully working setup.

Regards,
Nawaz Ahmed