Atena server is primarily intended to support environmental projects (NAIADES, Water4Cities, PerceptiveSentinel and enviroLENS) and any other projects that develop solutions related to processing sensor and other time-series data, data fusion, time-series prediction and stream mining.
It might be that the default terminal shell of a new user is set to sh
. This shell works different that for instance bash
(the source
command does not work, etc). To change the default terminal shell of the user to bash
run the following command and restart the terminal:
chsh -s /usr/bin/bash
The 6.9T disk is mounted on /mnt/data
.
Use this to:
- store your files/data
- docker images
- backups
- project-dedicated repos
- running web services
To store personal data, use /mnt/data/users/username
.
To store project-related stuff, use /mnt/data/projects/projectname
.
To store data for running services, use /mnt/data/services/servicename
.
For file transfer use scp (e. g. WinSCP). For development environment use VS Code with MS extension Remote Development.
For compressing and decompressing files use gzip
.
To compress the file or folders:
# -k: keep the uncompressed files (optional)
# -l: get stats how much space was saved (optional)
gzip -k -l file
To decompress the zip file:
# -k: keep the compressed files (optional)
# -l: get stats of decompression (optional)
gunzip -k -l file.gz
Any intended new installation of a software should be reported to Slack.
Some utilities are already installed (tmux
, git
).
Installed is Apache2 web server.
Config location: /etc/apache2
.
Use sudo apache2ctl
for controlling the component (i.e. restarting).
Use Apache2 to create proxies and similar.
Version | Python Shell | Pip command |
---|---|---|
3.8.2 | python3 |
pip3 |
Virtual environment is installed. Use it! Documentation is here.
Anaconda3 is installed in /opt/anaconda3
.
To use it, you have to change your .bashrc
, so that you add the following lines to the end:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/opt/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/opt/anaconda3/etc/profile.d/conda.sh" ]; then
. "/opt/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/opt/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
After this you either re-login into your account or source the .bashrc with source .bashrc
. Example of this file is also available on /mnt/data/sw/anaconda.rc
, which you can also source with source /mnt/data/sw/anaconda.rc
.
You can then use conda create -n myenv
. For further details check conda documentation. Virtual environments will be setup within your user. When activated, python
and python3
will point to the Anaconda's version of Python. To deactivate conda or a specific conda environment use conda deactivate
.
We are using NodeJS Version Manager (nvm) for managing node. Each user has his NodeJS distributions installed on a local account. nvm
itself is installed globally and you can invoke it by running source /mnt/data/sw/nvm.rc
. Alternatively you can copy the content of /mnt/data/sw/nvm.rc
at the end your ~/.bashrc
file.
Note: Some users have problems using globally installed
nvm
. In this case, follow the this link and install it locally for you account.
Table of frequently used nvm
commands.
Command | Description |
---|---|
nvm ls |
Lists all currently installed NodeJS versions. |
nvm install 12 |
Installs the latest version of 12.x.y of NodeJS. |
nvm install 8.0 |
Installs the latest version of 8.0.x of NodeJS. |
nvm use v12.18.0 |
Uses the selected version of NodeJS (as listed in nvm ls ). |
nvm uninstall v12.18.0 |
Uninstalls the selected version of NodeJS. |
nvm deactivate |
Deactivates the current version of NodeJS. |
nvm install --lts |
Install the latest long-term-support (LTS) version of NodeJS. |
Be aware to only install as little NodeJS versions on the system as needed!
Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large codebases across multiple repositories, and large numbers of users. (src: https://docs.bazel.build/versions/4.0.0/bazel-overview.html)
On Atena Bazel 3.7.2. is installed and was used to build custom tensorflow wheel since Atena does not support AVX2.
Additional documentation is available here. PostgreSQL is installed on a default port. Ask for a username/password or to install additional modules on Slack.
- Data directory:
/mnt/data/postgresql/12/main
- Start command:
sudo systemctl start postgresql
- Status command:
sudo systemctl status postgresql
- Stop command:
sudo systemctl stop postgresql
- Running psql as default user:
sudo -u postgres psql
Docker is an open platform for developing, shipping, and running applications. Docker was installed using this guide (install using repository). The current docker version installed is 20.10.6.
Running images with docker is described here
Docker compose: In addition docker compose 1.29.2 was also installed using this instructions.
Influxdb is a time series database. On Atena a 2.0.6 version is deployed with docker. The docker compose file is located at /mnt/data/docker_services/naiades_influxdb_grafana/docker-compose.yml. The storage is mounted to docker volumes influxdb_storage and configurations at influxdb_configurations.
List of buckets for organization naiades:
- braila
Grafana is the open source analytics & monitoring solution for every database. On Atena it is run a a docker container, created from the latest image available on docker hub. The docker compose file is located at /mnt/data/docker_services/naiades_influxdb_grafana/docker-compose.yml. The grafana storage directory is mounted at docker volumes grafana.
Elasticsearch is a search engine service. Additional documentation is available here. Elasticsearch is installed on a default port (9200
). No username and password is required.
Before using elasticsearch, check which indexes are already in use via the following command:
curl http://localhost:9200/_aliases?pretty=true
- Data directory:
/mnt/data/elasticsearch/data
- Logs directory:
/mnt/data/elasticsearch/logs
- Configuration directory:
/etc/elasticsearch
- Start command:
sudo systemctl start elasticsearch.service
- Status command:
sudo systemctl status elasticsearch.service
- Stop command:
sudo systemctl stop elasticsearch.service
Kafka is a distributed streaming platform, mainly used as a high-performance messaging system. Kafka was installed using this guide. It uses default ports: 9092
for Kafka and 2181
for Zookeeper.
- Logs (data) directory:
/mnt/data/kafka/logs
- Installation directory:
/home/kafka/kafka/
- Start command:
sudo systemctl start kafka
- Status command:
sudo systemctl status kafka
- Stop command:
sudo systemctl stop kafka
Logging in as kafka
user to use command-line utilities: sudo su - kafka
.
Try running command ~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TutorialTopic --from-beginning
Supervisor is a client/server system that allows its users to monitor and control a number of processes. It use used to run the services in production mode and restarts the service if it crashes. It provides various features and can be used through the linux terminal.
Each process requires its own config file in /etc/supervisor/conf.d/
folder (examples of existing services are already available in the folder).
The config files documentation is available here.
Once the config file is setup, one can start the process with the following terminal commands (lets say the servies name is service
):
- Reread processes:
sudo supervisorctl reread
- reads which processes have its own config files - Update processes:
sudo supervisorctl update
- updates the supervisor to take into account new and changed config files - Start process:
sudo supervisorctl start service
- End process:
sudo supervisorctl stop service
- Log process:
sudo supervisorctl tail -{number} service
where{number}
is the number of bytes one would like to output, e.g. 5000. Example:sudo supervisorctl tail -5000 service
- Start all processes:
sudo supervisorctl start all
- End all processes:
sudo supervisorctl stop all
Here are listed all the services running on Atena (including URL/port).
Service Name | Description | Virtual host | Port | Caretaker |
---|---|---|---|---|
iot-rapids | IoT platform for multiple EU projects (NAIADES, Water4Cities) | atena.?.? | 80 | Klemen |
GPS tracking | Tracking of GPS devices | atena.?.? | 8888 | Klemen |
Zookeeper | Kafka Zookeeper | localhost | 2181 | Klemen |
Kafka | Kafka | localhost | 9092 | Klemen |
eLENS miner system | Connecting eLENS services | atena.?.? | 4300 | Erik |
eLENS text embeddings | Creates Text Embeddings Interface | atena.?.? | 4000 | Erik |
eLENS text embeddings | Creates English Text Embeddings | atena.?.? | 4001 | Erik |
eLENS text embeddings | Creates Slovene Text Embeddings | atena.?.? | 4002 | Erik |
eLENS text embeddings | Creates German Text Embeddings | atena.?.? | 4003 | Erik |
eLENS text embeddings | Creates Greek Text Embeddings | atena.?.? | 4004 | Erik |
eLENS document search | Searches through legal documents | atena.?.? | 4100 | Erik |
eLENS document similarity | Calculates document similarity | atena.?.? | 4200 | Erik |
NAIADES WatchDog API | End-point for NAIADES WatchDog | atena.?.? | 5001 | Mark |
NAIADES WatchDog GUI | GUI for NAIADES WatchDog | atena.?.? | 5002 | Mark |
NAIADES FIWARE adapter | End-point for FIWARE for NAIADES | atena.?.? | 5003 | Gal |
NAIADES Data Docs server | Server for carrying out requests to FIWARE api for NAIADES | atena.?.? | 5004 | Mark |
NAIADES Data Docs app | GUI for FIWARE data for NAIADES | atena.?.? | 5005 | Mark |
NAIADES Grafana | End-point for Grafana for NAIADES | atena.?.? | 5006 | Gal |