"CREATE ROLE" commands issued from non-maindbs via non-superuser might fail about citus

[JelteF]

Helpful logs:

2024-01-22 17:25:53.959 CET [74752] LOG:  statement: BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 19, '2024-01-22 17:25:53.951188+01');
2024-01-22 17:25:53.961 CET [74752] LOG:  statement: SET citus.enable_metadata_sync TO 'off';create user new_user;;SET citus.enable_metadata_sync TO 'on'
2024-01-22 17:25:53.998 CET [74755] LOG:  statement: BEGIN TRANSACTION ISOLATION LEVEL READ COMMITTED;SELECT assign_distributed_transaction_id(0, 19, '2024-01-22 17:25:53.951188+01');
2024-01-22 17:25:54.001 CET [74755] LOG:  execute <unnamed>: WITH distributed_object_data(typetext, objnames, objargs, distargumentindex, colocationid, force_delegation)  AS (VALUES ('role', ARRAY['new_user']::text[], ARRAY[]::text[], -1, 0, false)) SELECT citus_internal_add_object_metadata(typetext, objnames, objargs, distargumentindex::int, colocationid::int, force_delegation::bool) FROM distributed_object_data;

The CREATE USER is executed on a different connection than the citus_internal_add_object_metadata call. This only happens when using a non-superuser to create the new user. So, I think the reason for this is that we fallback to the regular_user when using running the command on other workers using execute_command_on_remote_nodes_as_user, but I guess we still execute citus_internal_add_object_metadata on a superuser connection.

I see two options:

1. we fallback to the regular user for our call to citus_internal_add_object_metadata
2. we run mark_object_distributed on the remote nodes too, instead of using citus_internal_add_object_metadata.

I'd prefer 1. since it's always good to do as few things as possible as superuser. But if that results in hard to understand code or something, I think 2. would be okay too.