Also Add Password Changing Guidelines about supabase

[IcedTea2K]

I'm able to repro the error. If I changed POSTGRES_PASSWORD in .env only, running docker compose up -d won't start up docker, since supabase_analytics will fail with the password authentication failed for user "supabase_admin" error above. This would prevent me from accessing the dashboard and other services, unless I remove the database rm -rf volumes/db/data/. It makes sense because the database postgres is created supabase_admin having the initial password. Trying to change password in .env won't change the role's password in the database.

So, I tried changing supabase_admin password directly (as well as postgres role). And then, I changed the password in .env. This will allow you to run supabase somewhat normally.

supabase/docker
⇡3% ➜ docker compose up -d

supabase/docker
⇡4% ➜ psql -h 127.0.0.1 -p 5432 -d postgres -U supabase_admin
Password for user supabase_admin:
psql (16.2 (Homebrew), server 15.1 (Ubuntu 15.1-1.pgdg20.04+1))
Type "help" for help.

postgres=# alter user supabase_admin with password 'changed_password';
ALTER ROLE
postgres=# alter user postgres with password 'changed_password';
ALTER ROLE
postgres=# \q

supabase/docker
⇡7% ➜ docker compose down


supabase/docker
⇡7% ➜ docker compose up

The reason why this only "somewhat" works is because the error still appears in the supabase_anlytics logs. But, it appears less (though, still frequent), and you could access other services (at least the dashboard since that's the only thing i tested). I've yet to find a way to cleanly change password :(

[Ziinc]

It's a bug due to the stored pg url, there is currently no updating done on startup .
Thanks @encima for getting this on my radar.

[Ziinc]

Recommended fix for now if you don't want to remove your docker volume is to update the stored pg url in the backends table under the analytics schema

[IcedTea2K]

Confirmed that it works as the current work around

UPDATE _analytics.source_backends
SET config = jsonb_set(config, '{url}', '"postgresql://supabase_admin:$new_passwd@db:5432/postgres"', 'false')
WHERE type='postgres';

[encima]

Thanks for opening and sorry for the hours you spent!

It is probably good to add this for most configs here as changing any of these will often require a restart. PRs welcome!

[IcedTea2K]

It is probably good to add this for most configs here as changing any of these will often require a restart. PRs welcome!

On that note though, maybe we should recommend storing POSTGRES_PASSWORD and sensitive data with docket secrets instead of using .env for better security. Check the tip note here

[IcedTea2K]

@InfoSecMastermind I agree that the current method I'm using is not entirely ideal yet, since the error still lingers (debugging to find a better way rn). But I wouldn't conclude that it's a bug yet, because docker takes the password in the .env to set database password initially, and then uses it to log into the database subsequently. For example, this is how supabase_analytics "logging" into the postgres database.

POSTGRES_BACKEND_URL: postgresql://supabase_admin:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}

So, changing the password in the environment variable only will not be picked up automatically, and result in invalid password. We need to reset the password for the database if we want to keep it or rm -rf volumes/db/data/ if we don't.

@encima I totally agree with your highlights. Once we find a clean way to reset password, documenting it and adding a CLI command will allow the users to do it both manually or automatically. And I'd assume for the CLI command, it'd be part of the supabase CLI right? I'd love to contribute to that

[IcedTea2K]

Seems like I have to reset the password for the following users:

authenticator,dashboard_user,pgbouncer,postgres,supabase_admin,supabase_auth_admin,supabase_functions_admin,supabase_read_only_user,supabase_replication_admin,supabase_storage_admin

But, I'm still running into the password authentication failed for user "supabase_admin" for db, and analytics (the rest of the services run fine).

Would anyone have any clue why? It's recurrent and continuous logs, and doesn't fail any thing. I could still connect to the database, dashboard, etc.

[encima]

@vvvyyynet I would wait until there are docs for this if you are not super comfortable as it can break things.

However, running the command above when connected to the database (i.e. from the SQL Editor or using psql) will work

[InfoSecMastermind]

As I also commented on the latest commit for this guide.

After updating the POSTGRES_PASSWORD in .env , it results in an error with docker compose down and then docker compose up -d instructions. The changes only took effect for me by first docker compose down -v then rm -rf volumes/db/data/ and then docker compose up -d. The screenshot for log (docker logs supabase-analytics) of the error is attached.

I think its more of an issue, maybe someone else should also test it.

[IcedTea2K]

Looking through the docs again, there's already a section on restarting services after making changes to the configs (https://supabase.com/docs/guides/self-hosting/docker#restarting-all-services). No changes maybe required

[InfoSecMastermind]

Its not working for me as per those instructions in the docs, especially when dealing with changing POSTGRES_PASSWORD in .env.

As I also commented on the latest commit for this guide.

After updating the POSTGRES_PASSWORD in .env , it results in an error with docker compose down and then docker compose up -d instructions. The changes only took effect for me by first docker compose down -v then rm -rf volumes/db/data/ and then docker compose up -d. The screenshot for log (docker logs supabase-analytics) of the error is attached.

I think its more of an issue, maybe someone else should also test it.

[IcedTea2K]

The link to your log might be broken or private. Do you mind taking screenshot or sending another link?

[InfoSecMastermind]

[InfoSecMastermind]

It is more likely a bug.

[encima]

Thanks for this @IcedTea2K ! This is a pretty clean guide and could be integrated into the docs for sure!

As I see it, I think this issue highlights 2 things:

1. This is a non-trivial action and should be documented in more detail
2. The steps outlined above could be rolled up into a CLI command to allow easy resetting of the environment (not just passwords but volumes as well)

Is that right or do you all see it differently?

[InfoSecMastermind]

@encima In my opinion, The issue is that the above commands are also not suitable for production. They still contain errors. Putting them as a workaround in documentation will not be viable. They should also not be rolled up into a CLI command, including resetting volumes, because volumes would be better if they remained persistent, and that will only make it more broken and complex. Rather, the program needs to be fixed to handle changing passwords.

[IcedTea2K]

The problem narrows down to just supabase_analytics. All the services are working fine, except for the analytics. From the logs, it tries to connect to the postgres database as supabase_admin over and over again, but it keeps failing. @encima, would you know what's making it to continuously reconnect like that? I'm trying to look around, but couldn't find it, so I can't really identify what's performing the authentications, and with what credentials.

[vvvyyynet]

@Ziinc, @IcedTea2K can you please give some more advice on how to access the backends from CLI? For me, removing volumes/db/data/ worked just fine for now, however later I may rely on your workaround. I'm very new to docker and supabase. Thanks!

[IcedTea2K]

@vvvyyynet If you'd like, here's a little script that I used to work around the bug. Just put it into a bash file (e.g., change_password.sh) and run ./change_password.sh new_pass, and everything should be automated for you.

#!/bin/bash

old_passwd=`grep POSTGRES_PASSWORD= .env | sed "s/.*=\(.*\)/\1/"`
if [ -z $1 ]; then
    echo "Please specify a new password"
    exit 1
fi
new_passwd=$1

PGPASSWORD=$old_passwd psql -h 127.0.0.1 -p 5432 -d postgres -U supabase_admin << EOT
    alter user anon with password '$new_passwd';
    alter user authenticated with password '$new_passwd';
    alter user authenticator with password '$new_passwd';
    alter user dashboard_user with password '$new_passwd';
    alter user pgbouncer with password '$new_passwd';
    alter user pgsodium_keyholder with password '$new_passwd';
    alter user pgsodium_keyiduser with password '$new_passwd';
    alter user pgsodium_keymaker with password '$new_passwd';
    alter user postgres with password '$new_passwd';
    alter user service_role with password '$new_passwd';
    alter user supabase_admin with password '$new_passwd';
    alter user supabase_auth_admin with password '$new_passwd';
    alter user supabase_functions_admin with password '$new_passwd';
    alter user supabase_read_only_user with password '$new_passwd';
    alter user supabase_replication_admin with password '$new_passwd';
    alter user supabase_storage_admin with password '$new_passwd';

    UPDATE _analytics.source_backends
    SET config = jsonb_set(config, '{url}', '"postgresql://supabase_admin:$new_passwd@db:5432/postgres"', 'false')
    WHERE type='postgres';
EOT

if [ $? -eq 0 ]; then
    sed -i -e "s/POSTGRES_PASSWORD=.*/POSTGRES_PASSWORD=$new_passwd/g" .env
fi