Unable to setup CLP in Debian OS about clp

[bossjie]

@kirkrodrigues I met similar issue when running cmd "python3 start-clp"
here are my server info.

RAM: 64GB
Docker version 24.0.7
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1160.49.1.el7.x86_64
Architecture: x86-64
clp last commit 2b884bd ( I believe so as the package tool pulls newest code from github)

error log:
Traceback (most recent call last):
File "start-clp", line 462, in main
start_queue(instance_id, clp_config)
File "start-clp", line 271, in start_queue
subprocess.run(cmd, stdout=subprocess.DEVNULL, check=True)
File "/usr/lib64/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker', 'exec', '-it', 'clp-queue-938f', 'rabbitmqctl', 'wait', '/tmp/rabbitmq.pid']' returned non-zero exit status 137.

And I consider the compress is seperate with clp process(Hope I'm wrong). so I run "python3 compress xxl-service.log"
and got below error. I'm heading to install using single node. the experience is like a blackbox touching by me. If we have better doc to explain the components( I see it trys to run docker images to support it) and install flow, it would be great. As the log info is limited.

docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /vdb/clp-package-ubuntu-focal-x86_64-v0.0.2/var/data/archives.
See 'docker run --help'.
Traceback (most recent call last):
File "compress", line 168, in
sys.exit(main(sys.argv))
File "compress", line 159, in main
subprocess.run(cmd, check=True)
File "/usr/lib64/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker', 'run', '-i', '--rm', '--network', 'host', '-w', '/opt/clp', '-u', '0:0', '--name', 'clp-compressor-502b', '--mount', 'type=bind,src=/vdb/clp-package-ubuntu-focal-x86_64-v0.0.2,dst=/opt/clp', '--mount', 'type=bind,src=/,dst=/mnt/logs,readonly', '--mount', 'type=bind,src=/vdb/clp-package-ubuntu-focal-x86_64-v0.0.2/var/data/somehostname.company.com,dst=/opt/clp/var/data', '--mount', 'type=bind,src=/vdb/clp-package-ubuntu-focal-x86_64-v0.0.2/var/log/somehostname.company.com,dst=/opt/clp/var/log', '--mount', 'type=bind,src=/vdb/clp-package-ubuntu-focal-x86_64-v0.0.2/var/data/archives,dst=/mnt/archive-output', 'ghcr.io/y-scope/clp/clp-execution-x86-ubuntu-focal:main', '/opt/clp/sbin/native/compress', '--config', '/opt/clp/var/log/.clp-compressor-502b-config.yml', '--remove-path-prefix', '/mnt/logs', '/mnt/logs/vdb/clp-package-ubuntu-focal-x86_64-v0.0.2/sbin/xxl-service.log']' returned non-zero exit status 125.

[kirkrodrigues]

Hi mantrimahitha,

Sorry, I haven't seen that issue before. From my own tests, the container in your error message uses a maximum of about 200MB during startup.

Could you give us the following details about your system to help us debug?

• Amount of RAM
• Docker version
• OS version
• clp commit hash that you're building from

Thanks!

[bossjie]

by downloading from https://github.com/y-scope/clp/releases/download/v0.0.2/clp-package-ubuntu-focal-x86_64-v0.0.2.tar.gz and install in same type of centos server, run pip3 install -r requirements-pre-3.7.txt successfully but
it's not working

[root@hostname clp-package-ubuntu-focal-x86_64-v0.0.2]# sbin/start-clp

Status: Downloaded newer image for rabbitmq:3.9.8
2023-11-16 09:17:04,807 [INFO] [clp] Stopping clp-db-bffd...
2023-11-16 09:17:05,254 [INFO] [clp] Stopped clp-db-bffd.
2023-11-16 09:17:05,283 [ERROR] [clp] Failed to start CLP.
Traceback (most recent call last):
File "sbin/start-clp", line 462, in main
start_queue(instance_id, clp_config)
File "sbin/start-clp", line 271, in start_queue
subprocess.run(cmd, stdout=subprocess.DEVNULL, check=True)
File "/usr/lib64/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker', 'exec', '-it', 'clp-queue-bffd', 'rabbitmqctl', 'wait', '/tmp/rabbitmq.pid']'

[bossjie]

I even upgrade python3 to 3.7 and retry, it fails also.
2023-11-16 09:44:10,263 [INFO] [clp] Starting database...
2023-11-16 09:44:11,904 [INFO] [clp] Started database.
2023-11-16 09:44:11,904 [INFO] [clp] Creating database tables...
2023-11-16 09:44:12,617 [INFO] [clp] Created database tables.
2023-11-16 09:44:12,617 [INFO] [clp] Starting queue...
2023-11-16 09:44:16,752 [INFO] [clp] Stopping clp-db-0862...
2023-11-16 09:44:17,159 [INFO] [clp] Stopped clp-db-0862.
2023-11-16 09:44:17,179 [ERROR] [clp] Failed to start CLP.
Traceback (most recent call last):
File "sbin/start-clp", line 462, in main
start_queue(instance_id, clp_config)
File "sbin/start-clp", line 271, in start_queue
subprocess.run(cmd, stdout=subprocess.DEVNULL, check=True)
File "/usr/local/lib/python3.7/subprocess.py", line 468, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker', 'exec', '-it', 'clp-queue-0862', 'rabbitmqctl', 'wait', '/tmp/rabbitmq.pid']' returned non-zero exit status 137.

[jackluo923]

After chatting offline with @bossjie, several issues were identified which prevented the successful startup of CLP package.

1. Docker bug in CentOS 7 which prevents the MariaDB SQL database and rabbitmq container image from being pulled and started from docker-hub

• solution: upgrade to a recent docker release such as docker-ce-24.0.7-1

2. Permission issues with rabbitmq container's log file when started with the root user

• temporary solution: use another user to start CLP package
• permanent solution: a PR will be pushed out to resolve the underlying permission issue

After the two issues above are addressed, we were able to successfully start CLP package on CentOS 7.