Error while pushing images about sregistry

[vsoch]

The way to debug this is to docker-compose logs for the minio /uwsgi containers and see what the error message(s) are.

[imerelli]

Thank you for the suggestion. But this time logs are not helping me in understanding what is happening:

docker compose logs for uwsgi
sregistry-uwsgi-1 | [pid: 54|app: 0|req: 18/184] 155.253.6.100 () {34 vars in 497 bytes} [Sat Sep 10 15:18:05 2022] GET /v1/entities/imerelli => generated 305 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 1)
sregistry-uwsgi-1 | GET GetNamedCollectionView
sregistry-uwsgi-1 | [pid: 60|app: 0|req: 61/185] 155.253.6.100 () {34 vars in 517 bytes} [Sat Sep 10 15:18:05 2022] GET /v1/collections/imerelli/rnaseq => generated 342 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 1)
sregistry-uwsgi-1 | GET GetNamedContainerView
sregistry-uwsgi-1 | [pid: 60|app: 0|req: 62/186] 155.253.6.100 () {34 vars in 561 bytes} [Sat Sep 10 15:18:05 2022] GET /v1/containers/imerelli/rnaseq/centos7-python3-r4-deg => generated 522 bytes in 26 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 2)
sregistry-uwsgi-1 | GET PushNamedContainerView
sregistry-uwsgi-1 | [pid: 54|app: 0|req: 19/187] 155.253.6.100 () {34 vars in 718 bytes} [Sat Sep 10 15:18:05 2022] GET /v1/images/imerelli/rnaseq/centos7-python3-r4-deg:sha256.49bfd3dcac50e3fbcfbb4b535b1924c98ced1fc49a16446bfb4af5ee7da0b08c?arch=amd64 => generated 563 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 2)
sregistry-uwsgi-1 | [pid: 54|app: 0|req: 20/188] 155.253.6.100 () {34 vars in 471 bytes} [Sat Sep 10 15:18:05 2022] GET /version => generated 58 bytes in 2 msecs (HTTP/1.1 200) 5 headers in 141 bytes (1 switches on core 3)
sregistry-uwsgi-1 | POST RequestMultiPartPushImageFileView
sregistry-uwsgi-1 | Start multipart upload 13e3176e-156d-4422-9be2-cdeb41b1e69e
sregistry-uwsgi-1 | [pid: 60|app: 0|req: 63/189] 155.253.6.100 () {36 vars in 535 bytes} [Sat Sep 10 15:18:05 2022] POST /v2/imagefile/3/_multipart => generated 96 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 132 bytes (1 switches on core 3)
sregistry-uwsgi-1 | PUT RequestMultiPartPushImageFileView
sregistry-uwsgi-1 | [pid: 55|app: 0|req: 39/190] 155.253.6.100 () {36 vars in 536 bytes} [Sat Sep 10 15:18:07 2022] PUT /v2/imagefile/3/_multipart => generated 499 bytes in 19 msecs (HTTP/1.1 200) 5 headers in 133 bytes (1 switches on core 0)
sregistry-uwsgi-1 | PUT RequestMultiPartAbortView
sregistry-uwsgi-1 | Delete imagefile signal running.
sregistry-uwsgi-1 | [pid: 54|app: 0|req: 21/191] 155.253.6.100 () {36 vars in 546 bytes} [Sat Sep 10 15:18:07 2022] PUT /v2/imagefile/3/_multipart_abort => generated 0 bytes in 31 msecs (HTTP/1.1 200) 4 headers in 93 bytes (1 switches on core 1)

docker compose logs for the minio
sregistry-minio-1 | WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
sregistry-minio-1 | Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
sregistry-minio-1 | MinIO Object Storage Server
sregistry-minio-1 | Copyright: 2015-2022 MinIO, Inc.
sregistry-minio-1 | License: GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html
sregistry-minio-1 | Version: RELEASE.2022-09-07T22-25-02Z (go1.18.6 linux/amd64)
sregistry-minio-1 |
sregistry-minio-1 | Status: 1 Online, 0 Offline.
sregistry-minio-1 | API: http://172.18.0.3:9000 http://127.0.0.1:9000
sregistry-minio-1 | Console: http://172.18.0.3:45563 http://127.0.0.1:45563
sregistry-minio-1 |
sregistry-minio-1 | Documentation: https://docs.min.io

[vsoch]

But I think they are? This seems important:

sregistry-minio-1 | WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
sregistry-minio-1 | Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
sregistry-minio-1 | MinIO Object Storage Server

[vsoch]

Minio is on port 9000 so logically the error (and that message) hints that the server is not correctly starting. Notice toward the bottom the status is Offline.

[imerelli]

If I put MINIO_ROOT_USER and MINIO_ROOT_PASSWORD in the .minio-env file the warning goes away. But the error is still present. Moreover, if I remove MINIO_ACCESS_KEY and MINIO_SECRET_KEY the site does not start because I think uwsgi needs these parameters. So in the end I think MINIO_ACCESS_KEY and MINIO_SECRET_KEY are ok and the warning can be ignored for now.

I also think that minio is up and running because I installed mc for logging and trying a push here what is reported:

./mc admin trace -v myminio
minio:9000 [REQUEST s3.NewMultipartUpload] [2022-09-12T07:07:18.125] [Client IP: 172.18.0.5]
minio:9000 POST /sregistry/rnaseq/centos7-python3-r4-deg%3Asha256.49bfd3dcac50e3fbcfbb4b535b1924c98ced1fc49a16446bfb4af5ee7da0b08c?uploads
minio:9000 Proto: HTTP/1.1
minio:9000 Host: minio:9000
minio:9000 X-Amz-Date: 20220912T070718Z
minio:9000 Accept-Encoding: identity
minio:9000 Amz-Sdk-Request: attempt=1
minio:9000 Authorization: AWS4-HMAC-SHA256 Credential=minio/20220912/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3b1bcb0d7f09fa064e7fed0fdac0d1e26b441e047f4d832b35864d91974bf7a4
minio:9000 Content-Length: 0
minio:9000 User-Agent: Boto3/1.21.46 Python/3.6.13 Linux/4.18.0-240.1.1.el8_3.x86_64 Botocore/1.24.46
minio:9000 Amz-Sdk-Invocation-Id: 8f9e4967-291e-4bcf-a3c3-36aff45dcd24
minio:9000 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
minio:9000
minio:9000 [RESPONSE] [2022-09-12T07:07:18.127] [ Duration 1.562ms ↑ 131 B ↓ 345 B ]
minio:9000 200 OK
minio:9000 Content-Type: application/xml
minio:9000 Vary: Origin,Accept-Encoding
minio:9000 X-Content-Type-Options: nosniff
minio:9000 X-Xss-Protection: 1; mode=block
minio:9000 Accept-Ranges: bytes
minio:9000 Content-Length: 345
minio:9000 Content-Security-Policy: block-all-mixed-content
minio:9000 Server: MinIO
minio:9000 Strict-Transport-Security: max-age=31536000; includeSubDomains
minio:9000 X-Amz-Request-Id: 17140B2A29DF7F22
minio:9000
< InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">sregistry< Key>rnaseq/centos7-python3-r4-deg:sha256.49bfd3dcac50e3fbcfbb4b535b1924c98ced1fc49a16446bfb4af5ee7da0b08caa9764b3-f62a-4f5a-9d68-4b5f4a0d247c< /UploadId>< /InitiateMultipartUploadResult>
minio:9000

It seems that for some reasons the upload starts, but then it gets interrupted.
No idea of what is happening.

[vsoch]

That’s correct! You’ll need to update the two variables here:

sregistry/shub/apps/library/views/minio.py

Line 49 in 49e644b

access_key=os.environ.get("MINIO_ACCESS_KEY"),

. Also with a change in environment variables I’ve found I often need to stop, remove, and recreate the containers.

[vsoch]

These docs might help debug, these variables are new: https://docs.min.io/minio/baremetal/reference/minio-server/minio-server.html#envvar.MINIO_ROOT_USER

[imerelli]

I modified the .minio-env file, which now contains only MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
I also modified the shub/apps/library/views/minio.py file changing all the occurrence of MINIO_ACCESS_KEY and MINIO_SECRET_KEY, with MINIO_ROOT_USER and MINIO_ROOT_PASSWORD, respectively.
I stopped and cancelled everything and restarted the whole system brand new.

Results: the warning message disappeared and the system is up, but still get exactly the same error while pushing the image. I really think the problem is somewhere else...

[vsoch]

Can you please show me the logs for each of minio, uwsgi, and nginx? If minio wasn't running because of those credentials it's likely that was at least part of the problem (or in other words, you wouldn't get a working push in that state).

[imerelli]

Below the logs related to a push attempt.
For minio there are no logs apart from the startup signal.
In case I can contact you also by mail to organize a quick chat.

docker compose logs for the minio
sregistry-minio-1 | MinIO Object Storage Server
sregistry-minio-1 | Copyright: 2015-2022 MinIO, Inc.
sregistry-minio-1 | License: GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html
sregistry-minio-1 | Version: RELEASE.2022-09-07T22-25-02Z (go1.18.6 linux/amd64)
sregistry-minio-1 |
sregistry-minio-1 | Status: 1 Online, 0 Offline.
sregistry-minio-1 | API: http://172.18.0.4:9000 http://127.0.0.1:9000
sregistry-minio-1 | Console: http://172.18.0.4:41575 http://127.0.0.1:41575
sregistry-minio-1 |
sregistry-minio-1 | Documentation: https://docs.min.io

docker compose logs for the nginx
sregistry-nginx-1 | 155.253.6.100 - - [12/Sep/2022:13:23:13 +0000] "GET /v1/entities/imerelli HTTP/1.1" 200 305 "-" "Singularity/3.8.7 (Linux amd64) Go/1.16.13" "-"
sregistry-nginx-1 | 155.253.6.100 - - [12/Sep/2022:13:23:13 +0000] "GET /v1/collections/imerelli/rnaseq HTTP/1.1" 200 342 "-" "Singularity/3.8.7 (Linux amd64) Go/1.16.13" "-"
sregistry-nginx-1 | 155.253.6.100 - - [12/Sep/2022:13:23:14 +0000] "GET /v1/containers/imerelli/rnaseq/centos7-python3-r4-deg HTTP/1.1" 200 522 "-" "Singularity/3.8.7 (Linux amd64) Go/1.16.13" "-"
sregistry-nginx-1 | 155.253.6.100 - - [12/Sep/2022:13:23:14 +0000] "GET /v1/images/imerelli/rnaseq/centos7-python3-r4-deg:sha256.49bfd3dcac50e3fbcfbb4b535b1924c98ced1fc49a16446bfb4af5ee7da0b08c?arch=amd64 HTTP/1.1" 200 563 "-" "Singularity/3.8.7 (Linux amd64) Go/1.16.13" "-"
sregistry-nginx-1 | 155.253.6.100 - - [12/Sep/2022:13:23:14 +0000] "GET /version HTTP/1.1" 200 58 "-" "Singularity/3.8.7 (Linux amd64) Go/1.16.13" "-"
sregistry-nginx-1 | 155.253.6.100 - - [12/Sep/2022:13:23:14 +0000] "POST /v2/imagefile/3/_multipart HTTP/1.1" 200 96 "-" "Singularity/3.8.7 (Linux amd64) Go/1.16.13" "-"
sregistry-nginx-1 | 155.253.6.100 - - [12/Sep/2022:13:23:16 +0000] "PUT /v2/imagefile/3/_multipart HTTP/1.1" 200 496 "-" "Singularity/3.8.7 (Linux amd64) Go/1.16.13" "-"
sregistry-nginx-1 | 155.253.6.100 - - [12/Sep/2022:13:23:16 +0000] "PUT /v2/imagefile/3/_multipart_abort HTTP/1.1" 200 0 "-" "Singularity/3.8.7 (Linux amd64) Go/1.16.13" "-"

docker compose logs for the uwsgi
sregistry-uwsgi-1 | GET NamedEntityView
sregistry-uwsgi-1 | <QueryDict: {}>
sregistry-uwsgi-1 | imerelli
sregistry-uwsgi-1 | [pid: 56|app: 0|req: 14/49] 155.253.6.100 () {34 vars in 497 bytes} [Mon Sep 12 08:23:13 2022] GET /v1/entities/imerelli => generated 305 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 0)
sregistry-uwsgi-1 | GET GetNamedCollectionView
sregistry-uwsgi-1 | [pid: 56|app: 0|req: 15/50] 155.253.6.100 () {34 vars in 517 bytes} [Mon Sep 12 08:23:13 2022] GET /v1/collections/imerelli/rnaseq => generated 342 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 1)
sregistry-uwsgi-1 | GET GetNamedContainerView
sregistry-uwsgi-1 | [pid: 55|app: 0|req: 10/51] 155.253.6.100 () {34 vars in 561 bytes} [Mon Sep 12 08:23:13 2022] GET /v1/containers/imerelli/rnaseq/centos7-python3-r4-deg => generated 522 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 2)
sregistry-uwsgi-1 | GET PushNamedContainerView
sregistry-uwsgi-1 | [pid: 55|app: 0|req: 11/52] 155.253.6.100 () {34 vars in 718 bytes} [Mon Sep 12 08:23:14 2022] GET /v1/images/imerelli/rnaseq/centos7-python3-r4-deg:sha256.49bfd3dcac50e3fbcfbb4b535b1924c98ced1fc49a16446bfb4af5ee7da0b08c?arch=amd64 => generated 563 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 142 bytes (1 switches on core 0)
sregistry-uwsgi-1 | [pid: 55|app: 0|req: 12/53] 155.253.6.100 () {34 vars in 471 bytes} [Mon Sep 12 08:23:14 2022] GET /version => generated 58 bytes in 2 msecs (HTTP/1.1 200) 5 headers in 141 bytes (1 switches on core 1)
sregistry-uwsgi-1 | POST RequestMultiPartPushImageFileView
sregistry-uwsgi-1 | Start multipart upload 3047bf5b-cbf1-42f9-9faa-5c1d52652f1d
sregistry-uwsgi-1 | [pid: 55|app: 0|req: 13/54] 155.253.6.100 () {36 vars in 535 bytes} [Mon Sep 12 08:23:14 2022] POST /v2/imagefile/3/_multipart => generated 96 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 132 bytes (1 switches on core 3)
sregistry-uwsgi-1 | PUT RequestMultiPartPushImageFileView
sregistry-uwsgi-1 | [pid: 56|app: 0|req: 16/55] 155.253.6.100 () {36 vars in 536 bytes} [Mon Sep 12 08:23:16 2022] PUT /v2/imagefile/3/_multipart => generated 496 bytes in 24 msecs (HTTP/1.1 200) 5 headers in 133 bytes (1 switches on core 3)
sregistry-uwsgi-1 | PUT RequestMultiPartAbortView
sregistry-uwsgi-1 | Delete imagefile signal running.
sregistry-uwsgi-1 | [pid: 56|app: 0|req: 17/56] 155.253.6.100 () {36 vars in 546 bytes} [Mon Sep 12 08:23:16 2022] PUT /v2/imagefile/3/_multipart_abort => generated 0 bytes in 32 msecs (HTTP/1.1 200) 4 headers in 93 bytes (1 switches on core 2)

[imerelli]

Hi, I changed this line in the config.py file (Before it was 127.0.0.1:9000) and the error changed

MINIO_EXTERNAL_SERVER = (
"sregistry.bioinfotiget.it:9000" # minio server for Singularity to interact with
)

FATAL: Unable to push image to library: Put "http://sregistry.bioinfotiget.it:9000/sregistry/rnaseq/centos7-python3-r4-deg%3Asha256.49bfd3dcac50e3fbcfbb4b535b1924c98ced1fc49a16446bfb4af5ee7da0b08c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minio%2F20220917%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220917T144406Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host%3Bx-amz-content-sha256&partNumber=1&uploadId=f5e68e33-2b7b-467c-ab13-63b07cb32999&X-Amz-Signature=e1cdbf1296a1a4cd2fe9d4fa0a4b7a096f06e05bfb163eabdca942ce76ea4638": dial tcp 131.175.207.216:9000: i/o timeout

Instead of connection refused I have a timeout.
I want to point out that the only minio I have is the one that starts with docker compose so, according to the documentation, it is not clear to me if I have to change this line or not.

Also, have I to change this?
MINIO_SERVER = "minio:9000"
I don't think so because changing this variable the system does not start correctly

What about this?
MINIO_SSL = False
I'm using SSL for the web site, but I not sure if I have to change this.
I don't think so because changing this variable the system does not start correctly.

Final questions:
connecting to 131.175.207.216:9000 should I see something?
How can I connect to the minio interface?
My server is on openstack, so I opened port 9000. Should I open other ports?

Please help me in solving this, I really think I'm close to the solution, but still I don't see how to solve the problem.
Since there are no other options for a private storage of singularity images, this is very important for me...

[vsoch]

MINIO_EXTERNAL_SERVER = (
"sregistry.bioinfotiget.it:9000" # minio server for Singularity to interact with
)

Okay so let's step back a second. There are two mini server URLs, the first (above) is what the external URL should look like. If there is a timeout, you can try testing this locally with ping and various network tools. I would read the mini docs to see what basic interactions with a deployed registry should look like, and verify they work before coming back to sregistry. You can tell that some aspect is working because in the payload you showed me, you got as far as to have an upload id. If I were in your shoes, I'd engage with other minio users and the developer - for example this looks similar to what you are experiencing minio/console#1818 and could have resulted from a new Minio update.

Also, have I to change this?
MINIO_SERVER = "minio:9000"

This is the server that the other containers see, essentially internal networking. Try shelling into any container and looking at /etc/hosts. You should be able to ping minio from the inside of the other containers.

MINIO_SSL = False
I'm using SSL for the web site, but I not sure if I have to change this.
I don't think so because changing this variable the system does not start correctly.

This means you aren't using https. When I'm working on this I traditionally start without https, and then when I've confirmed it works I will add SSL and change this or True. You might consider removing ssl temporary while you debug this further because you will eventually need this. It's driving the logic here

sregistry/shub/apps/library/views/minio.py

Lines 44 to 80 in 49e644b

	MINIO_HTTP_PREFIX = "https://" if MINIO_SSL else "http://"
	minioClient = Minio(
	MINIO_SERVER,
	region=MINIO_REGION,
	access_key=os.environ.get("MINIO_ACCESS_KEY"),
	secret_key=os.environ.get("MINIO_SECRET_KEY"),
	secure=MINIO_SSL,
	)
	minioExternalClient = Minio(
	MINIO_EXTERNAL_SERVER,
	region=MINIO_REGION,
	access_key=os.environ.get("MINIO_ACCESS_KEY"),
	secret_key=os.environ.get("MINIO_SECRET_KEY"),
	secure=MINIO_SSL,
	)
	if not minioClient.bucket_exists(MINIO_BUCKET):
	minioClient.make_bucket(MINIO_BUCKET)
	session = Session(
	aws_access_key_id=os.environ.get("MINIO_ACCESS_KEY"),
	aws_secret_access_key=os.environ.get("MINIO_SECRET_KEY"),
	region_name=MINIO_REGION,
	)
	# https://github.com/boto/boto3/blob/develop/boto3/session.py#L185
	s3 = session.client(
	"s3",
	verify=MINIO_SSL,
	use_ssl=MINIO_SSL,
	endpoint_url=MINIO_HTTP_PREFIX + MINIO_SERVER,
	region_name=MINIO_REGION,
	config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
	)

if you are interested.

connecting to 131.175.207.216:9000 should I see something?

Yes you should. Check out the tutorial. https://docs.min.io/docs/minio-quickstart-guide.html

Please help me in solving this, I really think I'm close to the solution, but still I don't see how to solve the problem. Since there are no other options for a private storage of singularity images, this is very important for me...

I'm doing my best, of course without being able to debug it myself.

[vsoch]

I'm going to start preparing a PR with some of the Minio changes we've discovered in this issue! Hopefully if I can reproduce your issue I can help too.

[vsoch]

okay I've tested the entire thing from a fresh build to login -> push -> pull, and updated the docs accordingly. #405. I did the entire workflow without https and couldn't reproduce your issue (using the new minio) so I think my advice (if you can't get the above working) is to step back and test first without https, and add when you are sure that is working. If you test from my branch it should also be using a newer Python and have the bugs fixed that we chat about, so it might be a good idea too.

And a few more checks:

• always test first with a small image so you can ensure it's not a size issue (e.g., I usually pull docker://busybox)
• when you add remote to singularity, there is now an --insecure option that dictates if singularity should use https when pushing. You'll still need --no-https when pulling.

[imerelli]

I tested the new version of your software, recreating everything from scratch and without using https (for doing this I had to update singularity to 3.10). The problem is still the same. My feeling is that there is a problem with ports. When I try from a browser to connect to http://131.175.207.216:9000 I'm pushed to http://131.175.207.216:33187 which is clearly not available.
I disabled firewalld and, although the server is on an openstack based virtual machine, all ports are open (this was one of the first things I tried).

[vsoch]

Ah so we actually have been debugging your deployment and not an issue with (the expected) deployment of sregistry. That's a bit out of scope, no?

You need to get Minio running on OpenStack first it seems. I think perhaps you should start with that https://github.com/minio/minio/issues?q=openstack. I've never used OpenStack so I can't offer experience, but I agree with you it sounds like you haven't properly deployed a service on your cluster.

[vsoch]

Another strategy is to look at similar services and do what they do, e.g., https://docs.openstack.org/install-guide/environment-sql-database-ubuntu.html. This seems important:

set the bind-address key to the management IP address of the controller node to enable access by other nodes via the management network.

[imerelli]

Sorry, but this time I don't see the point. Openstack is only used to manage virtual machine.
There should be no difference between a Centos7 on a physical server and a Centos7 on a virtualized server.
Minio is delivered with your docker compose, right? I don't think I have to install a separated one.

[vsoch]

Yes but I don't think it's the fault of the docker-compose here, it's something about this custom setup. That's all I'm saying.

[vsoch]

There also appears to be something called "nova" https://www.edureka.co/community/65178/how-can-i-configure-docker-service-inside-openstack. and controlling ingress https://ovh.github.io/cds/hosting/ready-to-run/docker-compose/full-example/