APP Credentials disapper in webapp about clearml-server

[jingli-wtbox]

I delete all data under /opt/clearml, and redeploy it following Instruction. Then, I add one line in /opt/clearml/config/apiserver.conf because i saw my account is deleted. i did not find out the reason but it works now.

[yur1xpp]

Even after redeploy, I believe any modification to the apiserver.conf or setting new environment variables, the same issue will still occur? At least that's what is happening to me. It means I can't add new user once it's up since I have to purge all the data, which is not practical at all.

[yur1xpp]

@jkhenning Update:
I can now confirm that setting delete_missing_autocreated_users: false will fix this issue (redeploy server required). I can add new users and server will run as expected and I can now login. Seems like a bug that clearml server delete the old users (hence why the missing "create new credentials" button)?

[jkhenning]

Hi @jingli-wtbox,

A few questions:

1. What operating system are you using to run the docker compose in?
2. How did you set up the docker-compose? Did you create the local mounted folders? Do they exist with the data in them after shutting down the docker containers?
3. When does step #4 mean in your description? Where are you setting the environment variables?

[jingli-wtbox]

Hi @jkhenning , thank you for you reply.

please find answers below:

1. What operating system are you using to run the docker compose in?

I deploy clearml-server to one AWS ec2 instance, and the os is Amazon Linux 2.

2. How did you set up the docker-compose? Did you create the local mounted folders? Do they exist with the data in them after shutting down the docker containers?

I set up docker-compose by following the steps in documentation about clear-server deployment.
https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac
But I change the volume of elastic search in docker-compose.yml. if i use default value, my elastic container always restart, and i found this solution in clearml repo issues.

3. When does step #4 mean in your description? Where are you setting the environment variables?

I set the variables at clearml-server machine. I stop all containers using "docker-compose -f /opt/clearml/docker-compose.yml down" then i set these variables, and start containers again using "docker-compose -f /opt/clearml/docker-compose.yml up -d". After that, I login webUI, finding that the created credentials were not there and I also could not create a new one because there was no "+" icon anymore.

thank you

[jkhenning]

It seems like after the restart the server is not starting properly (or at all) - check the apiserver docker container logs and see if there are any errors

[yur1xpp]

I'm having the same issue, either setting new environments variables or modifying apiserver.conf, the button is missing from the page, and I can no longer login to the webserver using the exactly same user credentials after clearing cookies. This is my docker apiserver logs:

[2024-04-25 08:22:08,419] [9] [INFO] [clearml.redis_manager] Using override redis host redis
[2024-04-25 08:22:08,419] [9] [INFO] [clearml.redis_manager] Using override redis port 6379
[2024-04-25 08:22:08,468] [9] [INFO] [clearml.es_factory] Using override elastic host elasticsearch
[2024-04-25 08:22:08,468] [9] [INFO] [clearml.es_factory] Using override elastic port 9200
[2024-04-25 08:22:08,712] [9] [WARNING] [clearml.schema_reader] failed loading cache: [Errno 2] No such file or directory: '/opt/clearml/apiserver/schema/services/_cache.json'
[2024-04-25 08:22:08,712] [9] [INFO] [clearml.schema_reader] regenerating schema cache
[2024-04-25 08:22:19,258] [9] [INFO] [clearml.app_sequence] ################ API Server initializing #####################
[2024-04-25 08:22:19,259] [9] [INFO] [clearml.database] Initializing database connections
[2024-04-25 08:22:19,259] [9] [INFO] [clearml.database] Using override mongodb host mongo
[2024-04-25 08:22:19,259] [9] [INFO] [clearml.database] Using override mongodb port 27017
[2024-04-25 08:22:19,261] [9] [INFO] [clearml.database] Registering connection to auth-db (mongodb://mongo:27017/auth)
[2024-04-25 08:22:19,263] [9] [INFO] [clearml.database] Registering connection to backend-db (mongodb://mongo:27017/backend)
[2024-04-25 08:22:19,271] [9] [WARNING] [elastic_transport.node_pool] Node <Urllib3HttpNode(http://elasticsearch:9200)> has failed for 1 times in a row, putting on 1 second timeout
[2024-04-25 08:22:19,272] [9] [WARNING] [elastic_transport.transport] Retrying request after failure (attempt 0 of 3)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_node/_http_urllib3.py", line 202, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa274b34250>: Failed to establish a new connection: [Errno 111] Connection refused)
[2024-04-25 08:22:19,277] [9] [WARNING] [elastic_transport.node_pool] Node <Urllib3HttpNode(http://elasticsearch:9200)> has failed for 2 times in a row, putting on 2 second timeout
[2024-04-25 08:22:19,278] [9] [WARNING] [elastic_transport.transport] Retrying request after failure (attempt 1 of 3)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_node/_http_urllib3.py", line 202, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa274b19e50>: Failed to establish a new connection: [Errno 111] Connection refused)
[2024-04-25 08:22:19,284] [9] [WARNING] [elastic_transport.node_pool] Node <Urllib3HttpNode(http://elasticsearch:9200)> has failed for 3 times in a row, putting on 4 second timeout
[2024-04-25 08:22:19,285] [9] [WARNING] [elastic_transport.transport] Retrying request after failure (attempt 2 of 3)
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_node/_http_urllib3.py", line 202, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fa274b19ac0>: Failed to establish a new connection: [Errno 111] Connection refused)
[2024-04-25 08:22:19,287] [9] [WARNING] [elastic_transport.node_pool] Node <Urllib3HttpNode(http://elasticsearch:9200)> has failed for 4 times in a row, putting on 8 second timeout
[2024-04-25 08:22:19,287] [9] [WARNING] [clearml.initialize] Could not connect to ElasticSearch Service. Retry 1 of 4. Waiting for 30sec
[2024-04-25 08:22:49,415] [9] [INFO] [clearml.initialize] Applying mappings to ES host: [ConfigTree([('host', 'elasticsearch'), ('port', 9200), ('scheme', 'http')])]
[2024-04-25 08:22:49,492] [9] [INFO] [clearml.initialize] [{'component_template': 'events_common', 'result': ObjectApiResponse({'acknowledged': True})}, {'index_template': 'events_training_stats_scalar', 'result': ObjectApiResponse({'acknowledged': True})}, {'index_template': 'events_training_debug_image', 'result': ObjectApiResponse({'acknowledged': True})}, {'index_template': 'events_plot', 'result': ObjectApiResponse({'acknowledged': True})}, {'index_template': 'events_log', 'result': ObjectApiResponse({'acknowledged': True})}]
[2024-04-25 08:22:49,493] [9] [INFO] [clearml.initialize] Applying mappings to ES host: [ConfigTree([('host', 'elasticsearch'), ('port', 9200), ('scheme', 'http')])]
[2024-04-25 08:22:49,515] [9] [INFO] [clearml.initialize] [{'index_template': 'queue_metrics', 'result': ObjectApiResponse({'acknowledged': True})}, {'index_template': 'worker_stats', 'result': ObjectApiResponse({'acknowledged': True})}]
[2024-04-25 08:22:49,516] [9] [INFO] [clearml.apiserver.mongo.initialize.migration] Started mongodb migrations
[2024-04-25 08:22:49,532] [9] [INFO] [clearml.apiserver.mongo.initialize.migration] Finished mongodb migrations
[2024-04-25 08:22:49,548] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: webserver
[2024-04-25 08:22:49,559] [9] [INFO] [clearml.apiserver.mongo.initialize] Creating user: tests
[2024-04-25 08:22:49,564] [9] [INFO] [clearml.apiserver.mongo.initialize] Fixed users mode is enabled
[2024-04-25 08:22:49,566] [9] [INFO] [clearml.apiserver.mongo.initialize] Updating user name: admin
[2024-04-25 08:22:49,569] [9] [INFO] [clearml.apiserver.mongo.initialize] Updating user name: admin
[2024-04-25 08:22:49,572] [9] [INFO] [clearml.service_repo] Loading services from /opt/clearml/apiserver/services
[2024-04-25 08:22:49,791] [9] [INFO] [clearml.app_sequence] Exposed Services: auth.create_credentials auth.create_user auth.edit_credentials auth.edit_user auth.fixed_users_mode auth.get_credentials auth.get_token_for_user auth.login auth.logout auth.revoke_credentials auth.validate_token debug.ping events.add events.add_batch events.clear_scroll events.clear_task_log events.debug_images events.delete_for_model events.delete_for_task events.download_task_log events.get_debug_image_sample events.get_multi_task_metrics events.get_multi_task_plots events.get_plot_sample events.get_scalar_metric_data events.get_scalar_metrics_and_variants events.get_task_events events.get_task_latest_scalar_values events.get_task_log events.get_task_metrics events.get_task_plots events.get_task_single_value_metrics events.get_vector_metrics_and_variants events.multi_task_scalar_metrics_iter_histogram events.next_debug_image_sample events.next_plot_sample events.plots events.scalar_metrics_iter_histogram events.scalar_metrics_iter_raw events.vector_metrics_iter_histogram login.logout login.supported_modes models.add_or_update_metadata models.archive_many models.create models.delete models.delete_many models.delete_metadata models.edit models.get_all models.get_all_ex models.get_by_id models.get_by_id_ex models.get_by_task_id models.get_frameworks models.make_private models.make_public models.move models.publish_many models.set_ready models.unarchive_many models.update models.update_for_task models.update_tags organization.download_for_get_all organization.get_entities_count organization.get_tags organization.get_user_companies organization.prepare_download_for_get_all pipelines.delete_runs pipelines.start_pipeline projects.create projects.delete projects.get_all projects.get_all_ex projects.get_by_id projects.get_hyper_parameters projects.get_hyperparam_values projects.get_model_metadata_keys projects.get_model_metadata_values projects.get_model_tags projects.get_project_tags projects.get_task_parents projects.get_task_tags projects.get_unique_metric_variants projects.get_user_names projects.make_private projects.make_public projects.merge projects.move projects.update projects.validate_delete queues.add_or_update_metadata queues.add_task queues.create queues.delete queues.delete_metadata queues.get_all queues.get_all_ex queues.get_by_id queues.get_default queues.get_next_task queues.get_num_entries queues.get_queue_metrics queues.move_task_backward queues.move_task_forward queues.move_task_to_back queues.move_task_to_front queues.peek_task queues.remove_task queues.update reports.archive reports.create reports.delete reports.get_all_ex reports.get_tags reports.get_task_data reports.move reports.publish reports.unarchive reports.update server.config server.endpoints server.get_stats server.info server.report_stats_option tasks.add_or_update_artifacts tasks.add_or_update_model tasks.archive tasks.archive_many tasks.clone tasks.close tasks.completed tasks.create tasks.delete tasks.delete_artifacts tasks.delete_configuration tasks.delete_hyper_params tasks.delete_many tasks.delete_models tasks.dequeue tasks.dequeue_many tasks.edit tasks.edit_configuration tasks.edit_hyper_params tasks.enqueue tasks.enqueue_many tasks.failed tasks.get_all tasks.get_all_ex tasks.get_by_id tasks.get_by_id_ex tasks.get_configuration_names tasks.get_configurations tasks.get_hyper_params tasks.get_types tasks.make_private tasks.make_public tasks.move tasks.ping tasks.publish tasks.publish_many tasks.reset tasks.reset_many tasks.set_requirements tasks.started tasks.stop tasks.stop_many tasks.stopped tasks.unarchive_many tasks.update tasks.update_batch tasks.update_tags tasks.validate users.create users.delete users.get_all users.get_all_ex users.get_by_id users.get_current_user users.get_preferences users.set_preferences users.update workers.get_activity_report workers.get_all workers.get_count workers.get_metric_keys workers.get_stats workers.register workers.status_report workers.unregister
Loading config from /opt/clearml/apiserver/config/default
Loading config from file /opt/clearml/apiserver/config/default/apiserver.conf
Loading config from file /opt/clearml/apiserver/config/default/logging.conf
Loading config from file /opt/clearml/apiserver/config/default/secure.conf
Loading config from file /opt/clearml/apiserver/config/default/hosts.conf
Loading config from file /opt/clearml/apiserver/config/default/services/projects.conf
Loading config from file /opt/clearml/apiserver/config/default/services/events.conf
Loading config from file /opt/clearml/apiserver/config/default/services/tasks.conf
Loading config from file /opt/clearml/apiserver/config/default/services/storage_credentials.conf
Loading config from file /opt/clearml/apiserver/config/default/services/async_urls_delete.conf
Loading config from file /opt/clearml/apiserver/config/default/services/queues.conf
Loading config from file /opt/clearml/apiserver/config/default/services/auth.conf
Loading config from file /opt/clearml/apiserver/config/default/services/models.conf
Loading config from file /opt/clearml/apiserver/config/default/services/_mongo.conf
Loading config from file /opt/clearml/apiserver/config/default/services/organization.conf
Loading config from /opt/clearml/config
Loading config from file /opt/clearml/config/services.conf
Loading config from file /opt/clearml/config/apiserver.conf
 * Serving Flask app 'server'
 * Debug mode: off
[2024-04-25 08:22:55,218] [9] [INFO] [clearml.service_repo] Returned 200 for debug.ping in 0ms

[jkhenning]

What do you see when you go to the WebUI? Also, can you include the full docker log for the elasticsearch service?

[yur1xpp]

After clearing the cache/cookies on my web browser, I will be landed on the login page. There, all login attempts will fail as it complains invalid username/password.

sudo docker logs clearml-elastic:

Apr 25, 2024 9:47:59 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
{"type": "server", "timestamp": "2024-04-25T09:48:02,702Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "version[7.17.18], pid[7], build[default/docker/8682172c2130b9a411b1bd5ff37c9792367de6b0/2024-02-02T12:04:59.691750271Z], OS[Linux/6.5.0-1018-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/21.0.2/21.0.2+13-58]" }
{"type": "server", "timestamp": "2024-04-25T09:48:02,725Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]" }
{"type": "server", "timestamp": "2024-04-25T09:48:02,727Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-1875177940112166198, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Des.cgroups.hierarchy.override=/, -Xms2g, -Xmx2g, -Dlog4j2.formatMsgNoLookups=true, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,809Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [aggs-matrix-stats]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,809Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [analysis-common]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,810Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [constant-keyword]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,810Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [frozen-indices]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,810Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [ingest-common]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,815Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [ingest-geoip]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,816Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [ingest-user-agent]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,816Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [kibana]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,817Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [lang-expression]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,817Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [lang-mustache]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,817Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [lang-painless]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,817Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [legacy-geo]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,818Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [mapper-extras]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,818Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [mapper-version]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,818Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [parent-join]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,819Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [percolator]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,819Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [rank-eval]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,820Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [reindex]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,820Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [repositories-metering-api]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,821Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [repository-encrypted]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,821Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [repository-url]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,822Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [runtime-fields-common]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,822Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [search-business-rules]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,823Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [searchable-snapshots]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,823Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [snapshot-repo-test-kit]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,824Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [spatial]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,824Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [transform]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,824Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [transport-netty4]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,825Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [unsigned-long]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,825Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [vector-tile]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,825Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [vectors]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,826Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [wildcard]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,826Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-aggregate-metric]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,826Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-analytics]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,826Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-async]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,827Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-async-search]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,827Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-autoscaling]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,827Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ccr]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,827Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-core]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,828Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-data-streams]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,828Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-deprecation]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,828Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-enrich]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,829Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-eql]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,829Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-fleet]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,829Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-graph]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,830Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-identity-provider]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,830Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ilm]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,830Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-logstash]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,831Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ml]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,831Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-monitoring]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,832Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-ql]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-rollup]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-security]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-shutdown]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-sql]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,833Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-stack]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,834Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-text-structure]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,834Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-voting-only-node]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,834Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "loaded module [x-pack-watcher]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,835Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "clearml", "node.name": "clearml", "message": "no plugins loaded" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,895Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "clearml", "node.name": "clearml", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/root)]], net usable_space [22gb], net total_space [28.8gb], types [ext4]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,896Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "clearml", "node.name": "clearml", "message": "heap size [2gb], compressed ordinary object pointers [true]" }
{"type": "server", "timestamp": "2024-04-25T09:48:07,995Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "node name [clearml], node ID [eUjO7QLtRcS4JGNoVbUwhg], cluster name [clearml], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
{"type": "server", "timestamp": "2024-04-25T09:48:15,080Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "clearml", "node.name": "clearml", "message": "[controller/201] [Main.cc@122] controller (64 bit): Version 7.17.18 (Build a1014b051a9d63) Copyright (c) 2024 Elasticsearch BV" }
{"type": "server", "timestamp": "2024-04-25T09:48:15,703Z", "level": "INFO", "component": "o.e.i.g.ConfigDatabases", "cluster.name": "clearml", "node.name": "clearml", "message": "initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/usr/share/elasticsearch/config/ingest-geoip] for changes" }
{"type": "server", "timestamp": "2024-04-25T09:48:15,706Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "initialized database registry, using geoip-databases directory [/tmp/elasticsearch-1875177940112166198/geoip-databases/eUjO7QLtRcS4JGNoVbUwhg]" }
{"type": "server", "timestamp": "2024-04-25T09:48:16,482Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "clearml", "node.name": "clearml", "message": "creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]" }
{"type": "server", "timestamp": "2024-04-25T09:48:16,528Z", "level": "INFO", "component": "o.e.i.r.RecoverySettings", "cluster.name": "clearml", "node.name": "clearml", "message": "using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]" }
{"type": "server", "timestamp": "2024-04-25T09:48:16,586Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "clearml", "node.name": "clearml", "message": "using discovery type [single-node] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2024-04-25T09:48:17,379Z", "level": "INFO", "component": "o.e.g.DanglingIndicesState", "cluster.name": "clearml", "node.name": "clearml", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
{"type": "server", "timestamp": "2024-04-25T09:48:18,340Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "initialized" }
{"type": "server", "timestamp": "2024-04-25T09:48:18,341Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "starting ..." }
{"type": "server", "timestamp": "2024-04-25T09:48:18,362Z", "level": "INFO", "component": "o.e.x.s.c.f.PersistentCache", "cluster.name": "clearml", "node.name": "clearml", "message": "persistent cache index loaded" }
{"type": "server", "timestamp": "2024-04-25T09:48:18,363Z", "level": "INFO", "component": "o.e.x.d.l.DeprecationIndexingComponent", "cluster.name": "clearml", "node.name": "clearml", "message": "deprecation component started" }
{"type": "server", "timestamp": "2024-04-25T09:48:18,496Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "clearml", "node.name": "clearml", "message": "publish_address {192.168.80.4:9300}, bound_addresses {[::]:9300}" }
{"type": "server", "timestamp": "2024-04-25T09:48:19,011Z", "level": "INFO", "component": "o.e.c.c.Coordinator", "cluster.name": "clearml", "node.name": "clearml", "message": "cluster UUID [rt68OCO8RuOFNKwTibAVbQ]" }
{"type": "server", "timestamp": "2024-04-25T09:48:19,134Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "clearml", "node.name": "clearml", "message": "elected-as-master ([1] nodes joined)[{clearml}{eUjO7QLtRcS4JGNoVbUwhg}{BIkQ9CFXTy2qugYr9eNjLw}{192.168.80.4}{192.168.80.4:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 2, version: 67, delta: master node changed {previous [], current [{clearml}{eUjO7QLtRcS4JGNoVbUwhg}{BIkQ9CFXTy2qugYr9eNjLw}{192.168.80.4}{192.168.80.4:9300}{cdfhilmrstw}]}" }
{"type": "server", "timestamp": "2024-04-25T09:48:19,251Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "clearml", "node.name": "clearml", "message": "master node changed {previous [], current [{clearml}{eUjO7QLtRcS4JGNoVbUwhg}{BIkQ9CFXTy2qugYr9eNjLw}{192.168.80.4}{192.168.80.4:9300}{cdfhilmrstw}]}, term: 2, version: 67, reason: Publication{term=2, version=67}" }
{"type": "server", "timestamp": "2024-04-25T09:48:19,379Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "clearml", "node.name": "clearml", "message": "publish_address {192.168.80.4:9200}, bound_addresses {[::]:9200}", "cluster.uuid": "rt68OCO8RuOFNKwTibAVbQ", "node.id": "eUjO7QLtRcS4JGNoVbUwhg"  }
{"type": "server", "timestamp": "2024-04-25T09:48:19,380Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "clearml", "node.name": "clearml", "message": "started", "cluster.uuid": "rt68OCO8RuOFNKwTibAVbQ", "node.id": "eUjO7QLtRcS4JGNoVbUwhg"  }
{"type": "server", "timestamp": "2024-04-25T09:48:19,920Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "clearml", "node.name": "clearml", "message": "license [f62b8fc5-383c-48c0-92e8-a6a5804b2bc0] mode [basic] - valid", "cluster.uuid": "rt68OCO8RuOFNKwTibAVbQ", "node.id": "eUjO7QLtRcS4JGNoVbUwhg"  }
{"type": "server", "timestamp": "2024-04-25T09:48:19,929Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "clearml", "node.name": "clearml", "message": "recovered [7] indices into cluster_state", "cluster.uuid": "rt68OCO8RuOFNKwTibAVbQ", "node.id": "eUjO7QLtRcS4JGNoVbUwhg"  }
{"type": "server", "timestamp": "2024-04-25T09:48:20,668Z", "level": "ERROR", "component": "o.e.i.g.GeoIpDownloader", "cluster.name": "clearml", "node.name": "clearml", "message": "exception during geoip databases update", "cluster.uuid": "rt68OCO8RuOFNKwTibAVbQ", "node.id": "eUjO7QLtRcS4JGNoVbUwhg" ,
"stacktrace": ["org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active",
"at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:137) ~[ingest-geoip-7.17.18.jar:7.17.18]",
"at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:284) [ingest-geoip-7.17.18.jar:7.17.18]",
"at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-7.17.18.jar:7.17.18]",
"at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-7.17.18.jar:7.17.18]",
"at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.18.jar:7.17.18]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.18.jar:7.17.18]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]",
"at java.lang.Thread.run(Thread.java:1583) [?:?]"] }
{"type": "server", "timestamp": "2024-04-25T09:48:20,683Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "clearml", "node.name": "clearml", "message": "Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[events-training_debug_image-][0], [.geoip_databases][0], [queue_metrics_d1bd92a3b039400cbafc60a7a5b1e52b_2024-04][0], [worker_stats_d1bd92a3b039400cbafc60a7a5b1e52b_2024-04][0], [events-log-][0], [events-training_stats_scalar-][0], [events-plot-][0]]]).", "cluster.uuid": "rt68OCO8RuOFNKwTibAVbQ", "node.id": "eUjO7QLtRcS4JGNoVbUwhg"  }
{"type": "server", "timestamp": "2024-04-25T09:48:20,724Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-1875177940112166198/geoip-databases/eUjO7QLtRcS4JGNoVbUwhg/GeoLite2-Country.mmdb.tmp.gz]", "cluster.uuid": "rt68OCO8RuOFNKwTibAVbQ", "node.id": "eUjO7QLtRcS4JGNoVbUwhg"  }
{"type": "server", "timestamp": "2024-04-25T09:48:20,726Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-1875177940112166198/geoip-databases/eUjO7QLtRcS4JGNoVbUwhg/GeoLite2-City.mmdb.tmp.gz]", "cluster.uuid": "rt68OCO8RuOFNKwTibAVbQ", "node.id": "eUjO7QLtRcS4JGNoVbUwhg"  }
{"type": "server", "timestamp": "2024-04-25T09:48:20,726Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-1875177940112166198/geoip-databases/eUjO7QLtRcS4JGNoVbUwhg/GeoLite2-ASN.mmdb.tmp.gz]", "cluster.uuid": "rt68OCO8RuOFNKwTibAVbQ", "node.id": "eUjO7QLtRcS4JGNoVbUwhg"  }
{"type": "server", "timestamp": "2024-04-25T09:48:21,466Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-1875177940112166198/geoip-databases/eUjO7QLtRcS4JGNoVbUwhg/GeoLite2-Country.mmdb]", "cluster.uuid": "rt68OCO8RuOFNKwTibAVbQ", "node.id": "eUjO7QLtRcS4JGNoVbUwhg"  }
{"type": "server", "timestamp": "2024-04-25T09:48:21,528Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-1875177940112166198/geoip-databases/eUjO7QLtRcS4JGNoVbUwhg/GeoLite2-ASN.mmdb]", "cluster.uuid": "rt68OCO8RuOFNKwTibAVbQ", "node.id": "eUjO7QLtRcS4JGNoVbUwhg"  }
{"type": "server", "timestamp": "2024-04-25T09:48:22,036Z", "level": "INFO", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "clearml", "node.name": "clearml", "message": "successfully reloaded changed geoip database file [/tmp/elasticsearch-1875177940112166198/geoip-databases/eUjO7QLtRcS4JGNoVbUwhg/GeoLite2-City.mmdb]", "cluster.uuid": "rt68OCO8RuOFNKwTibAVbQ", "node.id": "eUjO7QLtRcS4JGNoVbUwhg"  }

[jkhenning]

From the apiserver log it looks like you have two admin users defined?

[yur1xpp]

I do, but those are name rather than username, I guess it wouldn't matter? It actually did those just for test purposes as no matter what unique username I choose, I still end up the same. The apiserver.conf:

auth {
     # Fixed users login credentials
     # No other user will be able to login
     fixed_users {
         enabled: true
         pass_hashed: true
         users: [
             {
                 username: "admin"
                 password: "xxxxxxx"
                 name: "admin"
             },
             {
                 username: "admin2"
                 password: "xxxxxxx"
                 name: "admin"
             },
         ]
     }
 }

Even if I don't modify the apiserver.conf, but just instead just do what OP did (that shutdown the server, export variable=XXXX, and start the server again), I end up with same problem. It seems I couldn't modify anything once the server is up. Also I'm using HTTP instead of HTTPS if that matters.

[jingli-wtbox]

Hi @yur1xpp ,

there are some things you could try:

1. check if the MongoDB or elastic docker is always restarting (this happened on my side because the port (9200 or 27017 is used by other service)
2. try to set pass_hashed as false. set it true again when you fix the login issue.
3. delete all things and re-deploy

[yur1xpp]

Thanks @jingli-wtbox.

1. I don't believe either my mongo or elastic docker restarting in my case
   

2. This doesn't seems to help in my case as well.

3. I'm trying to avoid having to resort to purging the data as there are already experimentations going on.

How did you end up solving your case? Re-deploy?

[lpbarto]

Hi, I'm experiencing the same issue. From the logs, it appears that the users get deleted after the restart.
[2024-04-26 09:11:40,264] [9] [INFO] [clearml.apiserver.mongo.initialize] Fixed users mode is enabled [2024-04-26 09:11:40,266] [9] [INFO] [clearml.apiserver.mongo.initialize] Updating user name: Jane Doe [2024-04-26 09:11:40,269] [9] [INFO] [clearml.apiserver.mongo.initialize] Updating user name: John Doe [2024-04-26 09:11:40,271] [9] [INFO] [clearml.apiserver.mongo.initialize] Removing user that is no longer in configuration: 4f1bc81ea609a47e3b3b8b5e2f92efd2 [email protected] Jane Doe [2024-04-26 09:11:40,276] [9] [INFO] [clearml.apiserver.mongo.initialize] Removing user that is no longer in configuration: 5882019dfbb54341621c8b0ab8f6ace5 [email protected] John Doe

I think the problem occurs during restart: if the user already exists, the 'ensure_fixed_user' function returns before adding the email to 'internal_user_emails.' Consequently, in 'apiserver/mongo/initialize/init.py' the user's email isn't in 'internal_user_emails' and thus gets deleted.

for user in AuthUser.objects( company=company_id, autocreated=True, email__nin=internal_user_emails ): log.info( f"Removing user that is no longer in configuration: {user['id']}\t{user['email']}\t{user['name']}" ) user.delete()

[jkhenning]

@lpbarto, @jingli-wtbox, @yur1xpp, we've identified the issue - this is indeed a bug in the latest version, and we will release a patch version as soon as possible and will notify here, thanks for raising this issue!

[NathanClearML]

@yur1xpp , we are in the final stages of testing the fix, to verify this will be applicable, what version of the server are you running?

[yur1xpp]

@NathanClearML Is the version is the one stated in /opt/clearml/docker-compose.yml? If yes, it's version: "3.6".

[jkhenning]

Hi @yur1xpp, I think @NathanClearML means the server version that appears in the bottom right part of the ClearML profile page (the docker image tag used for the apiserver service will provide this info)

[jingli-wtbox]

for me:
WebApp: 1.15.0-472 • Server: 1.15.0-472 • API: 2.29

[yur1xpp]

Ah in that case, it's WebApp: 1.15.0-472 • Server: 1.15.0-472 • API: 2.29.