db_import with keyauth_credentials fails on re-run about kong

[hanshuebner]

We'll see how we can address this issue. (KAG-4044)

[hanshuebner]

@ahalay Thank you for reporting this issue. As you have seen in PR #12597, the underlying issue is difficult to fix. We think of db_import primarily as an import tool, with DecK being the solution for updates. As you're using DecK already, can you maybe avoid running the db_import step multiple times or reset the database before you run it?

[ahalay]

Thanks for the info @hanshuebner

Unfortunately this is something we'd really like to avoid, as we haven't found a more convenient way than db_import for our needs. We use it to initially configure route and keyauth for Kong's admin-api without exposing admin port, and then configure everything else through the admin-api route with decK. Dropping records from the database in our case can have unexpected consequences if there are multiple replicas (admin-api could left without keyauth or will not be recreated at all), as we run db_import as an initContainer before Kong's own container for each replica.

[hanshuebner]

Given that a complete solution is not in sight, would it be possible to perform a check on the database to see if it needs to receive its initial payload using db_import in your initContainer?

[ahalay]

Yes, it is possible, but it deprives us of the possibility of updating the value of this keyauth, whose uniqueness the function complains about.

[BastiNBG85]

we have exactly the same problem. We also use db_import after the automated migration steps in order to overwrite the admin-api route and the admin credentials (this will avoid locking out the user on accidently removing the route / user / api-key). I just wanted to upgrade from 3.3.0 to 3.6.1 and had the same issues. Using deck would not be a solution as we would have a "chicken-and-egg-problem" when something is accidently removed. so this is a kind of bypass ensuring that we always have access to our productive system (we just need to reboot the container and have access again. we have no other access to the commandline in order to change something as this is managed by our oc4 provider).

[hanshuebner]

We have discussed this internally once more and also reviewed #12597. It is unfortunately not straightforward to completely fix the issue, given that PostgreSQL only allows one ON CONFLICT clause and furthermore, IDs of rows that are being inserted are created in Lua land. The latter, however, shows a way how the problem can be avoided: When the IDs of the entities that are inserted are specified, the existing ON CONFLICT clauses will fire and correctly convert the INSERT into an upsert if the entity already exists.

Would you be able to specify the IDs of your entities in your YAML files?

consumers:
- username: admin-api-user
  acls:
  - group: admin-api-group
  keyauth_credentials:
  - id: 01E83305-9489-4232-8E4E-D7DD5FC6901F
    key: very-secret-key

[ahalay]

Yes, I think this option should work, as far as I understand in case an existing record will have a different id it can be safely updated in the database itself before db_import. Thanks for your help.

[StarlightIbuki]

Closing as this is a non-issue and the reporter seems to have a workaround.