Bug Report: vttestserver log files in $VTDATAROOT/logs are missing. about vitess

[frouioui]

@yoheimuta, I am still looking on finding a solution to the issue. In the meantime you can see all the logs in ls tmp/ if you are on Docker, on Mac it would be ls $TMPDIR/.

vitess@35eecd00828c:/$ ls tmp/
mysqlctl.35eecd00828c.vitess.log.ERROR.20240207-220021.18    vtcombo.ERROR
mysqlctl.35eecd00828c.vitess.log.INFO.20240207-220021.18     vtcombo.INFO
mysqlctl.35eecd00828c.vitess.log.WARNING.20240207-220021.18  vtcombo.WARNING
mysqlctl.ERROR                                               vttestserver.35eecd00828c.vitess.log.ERROR.20240207-220021.1
mysqlctl.INFO                                                vttestserver.35eecd00828c.vitess.log.INFO.20240207-220021.1
mysqlctl.WARNING                                             vttestserver.35eecd00828c.vitess.log.WARNING.20240207-220021.1
vtcombo.35eecd00828c.vitess.log.ERROR.20240207-220024.715    vttestserver.ERROR
vtcombo.35eecd00828c.vitess.log.INFO.20240207-220024.715     vttestserver.INFO
vtcombo.35eecd00828c.vitess.log.WARNING.20240207-220024.715  vttestserver.WARNING

[deepthi]

@yoheimuta is it possible for you track down the commit where this behavior changed?

[yoheimuta]

@deepthi I've discovered that running the vttestserver binary directly on my M2 Mac doesn't cause the issue, regardless of whether it's version v18.0.0, v18.0.1, v18.0.2, or v19.0.0-SNAPSHOT ebf0106 (the latest main).

Therefore, the issue seems specific to running the binary in Docker on an M2 Mac. I've updated the Operating System and Environment details section of this issue accordingly. I'm in the process of identifying the problematic commit, but it may take some time since it involves building a Docker image.

[mattlord]

Thanks, @yoheimuta ! So perhaps the current implementation and/or docs assume that the files can be accessed via the local filesystem? With Docker on macOS a LinuxKit VM is used via the xhyve hypervisor or the (newish) Virtualization framework and that assumption would break.

[frouioui]

I am still trying to figure out what happened, but in the vttestserver logs it is important to not that we can see we are starting vtcombo with the proper --logs_dir CLI flag:

I0206 17:23:57.608488       1 vtprocess.go:150] /vt/bin/vtcombo --port 33574 --bind-address localhost --log_dir /vt/vtdataroot/vttest445905202/logs --alsologtostderr --grpc_port 33575 --db_charset utf8mb4 --db_app_user vt_dba --db_app_password  --db_dba_user vt_dba --db_dba_password  --proto_topo keyspaces:{name:"test" shards:{name:"-80"} shards:{name:"80-"} replica_count:2 rdonly_count:1} keyspaces:{name:"unsharded" shards:{name:"0"} replica_count:2 rdonly_count:1} cells:"test" --mycnf_server_id 1 --mycnf_socket_file /vt/vtdataroot/vttest445905202/vt_0000000001/mysql.sock --normalize_queries --dbddl_plugin vttest --foreign_key_mode allow --planner-version  --enable_online_ddl=true --enable_direct_ddl=true --enable_system_settings=true --tablet_refresh_interval=10s --queryserver-config-pool-size 4 --queryserver-config-query-timeout 300s --queryserver-config-schema-reload-time 60s --queryserver-config-stream-pool-size 4 --queryserver-config-transaction-cap 4 --queryserver-config-transaction-timeout 300s --queryserver-config-txpool-timeout 300s --service_map grpc-vtgateservice,grpc-vtctl,grpc-vtctld --transaction_mode MULTI --tablet_hostname localhost --mysql_server_version 8.0.30-Vitess --db_socket /vt/vtdataroot/vttest445905202/vt_0000000001/mysql.sock --mysql_auth_server_impl none --mysql_server_port 33577 --mysql_server_bind_address 0.0.0.0 %!v(MISSING)

[frouioui]

I can trace back the apparition of this issue to commit 35dd1b4, which updates the version of glog from v1.0.0 to v1.1.2. I will investigate what changed from the previous and current version. Below is a table of the bisection I performed. A full list of commits that were pushed between v18.0.0 and v18.0.1 is available here.

SHA	Date	Repro passes
v18.0.0	Nov 6	✅
5a3ca083dbd6167819a2344a34a17ae4777393c5	Nov 10	✅
7e90aa2540b25313d6fc37b5379a1752c1111873	Nov 10	✅
35dd1b429e21a1034cbde000b765d00ee9349238	Nov 10	⛔️
v18.0.1	Nov 22	⛔️
d76f47992aca9801858e8b54a6f6e5912c0239e3	Dec 5	⛔️
2f43160047102ae728018ab03c61e0cfd9761033	Dec 15	⛔️

[frouioui]

After forking the glog Go package and modifying the code that handles the creation of the log files, I am able to see that locally logDirs (which is a slice containing the value of the --log_dir and /tmp) is set to:

panic: /Users/florentpoinsard/Code/vitess/vtdataroot/vttest3115060181/logs,/var/folders/gh/3x61bzxj00ncmq5_j8wnqy2w0000gn/T

We can see the logs folder inside VTDATAROOT first, followed by my Mac's temporary directory ($TMPDIR). On the other hand, when running inside the Docker container, here is what we see:

panic: /tmp

We can only see the /tmp directory, not the VTDATAROOT's logs folder. Even though the --log_dir flag is properly set when starting vtcombo:

vitess/go/vt/vttest/vtprocess.go

Line 133 in 574162f

"--log_dir", vtp.LogDirectory,

[frouioui]

Hello @yoheimuta! This bug should now be fixed, the fix has been merged on release-18.0 already: #15185.

[yoheimuta]

@frouioui Thank you for the fix 👍