Resource processor fails to deploy first workspace on fresh TRE deployment about azuretre

[marrobi]

Docker is passed through the container so the creds should pass through. I'm sure I tested it.

But, yes, fix if it is an issue is to add the login commands to to the explain command.

[jonnyry]

You can recreate this bug without having to deploy a fresh TRE by:

• SSHing into the resource processor VM (via the jump box):
• logging in to the running resource processor docker container shell
• clearing down the porter cache (which holds the manifests & parameters) by issuing cd /root/.porter.cache then rm -rf *
• issuing a docker logout <TRE_ID>acr.azurecr.io which will remove the credentials docker holds to connect to the container register (resetting it to as it was when it was first deployed).
• then try and deploy a workspace
• you will get the error on the first deploy
• the second deploy will succeed because the credentials have now been cached

[marrobi]

I can remember looking at this at the time. Weird you have seen it as was a while ago and don't think I've come across the issue and our E2E PR tests would fail. So I'm confused why seeing this now, and not in the tests.

Looking at the code needs running once on RP start-up and is done here -

AzureTRE/core/terraform/resource_processor/vmss_porter/cloud-config.yaml

Line 91 in ddbbffe

- az acr login --name ${docker_registry_server}

Looking at your logs I think your actual error is Error message: parameter "tre_id" is required. Is this a custom bundle, if so think you are missing passing tre_id somewhere.

[Danny-Cooke-CK]

I've see this too recently

[tim-allen-ck]

@jonnyry I think I've seen this before when deploying a workspace, as as you say subsequent deploys work, thats been our "workaround".

[jonnyry]

@jonnyry I think I've seen this before when deploying a workspace, as as you say subsequent deploys work, thats been our "workaround".

yes - also our workaround :-) just thought i'd get it logged as seen it several times now

[jonnyry]

Looking at the code needs running once on RP start-up and is done here -

AzureTRE/core/terraform/resource_processor/vmss_porter/cloud-config.yaml

Line 91 in ddbbffe

- az acr login --name ${docker_registry_server}

I notice the az acr login is run on the VM itself rather than inside the resource processor docker container - is the az "session" shared inside the docker container?

  - az acr login --name ${docker_registry_server}
  - docker run -d -p 8080:8080 -v /var/run/docker.sock:/var/run/docker.sock
    --restart always --env-file .env
    --name resource_processor1
    --log-driver local
    ${docker_registry_server}/${resource_processor_vmss_porter_image_repository}:${resource_processor_vmss_porter_image_tag}

Looking at your logs I think your actual error is Error message: parameter "tre_id" is required. Is this a custom bundle, if so think you are missing passing tre_id somewhere.

The logs in the issue description are for a custom bundle, however it also happens for standard bundles, this is from a test I ran just now after resetting the cache & docker credentials inside the resource processor container -

1) Main step for 28b8b4b2-8840-4eac-89d9-ab6294ac1aa2
28b8b4b2-8840-4eac-89d9-ab6294ac1aa2: Error message: parameter "address_spaces" is required ; Command executed: porter install "28b8b4b2-8840-4eac-89d9-ab6294ac1aa2" --reference XXXXX.azurecr.io/tre-workspace-airlock-import-review:v0.12.16 --force --credential-set arm_auth --credential-set aad_auth

[jonnyry]

It looks like az login & az acr login are called when running a constructed porter command (install etc):

AzureTRE/resource_processor/vmss_porter/runner.py

Lines 100 to 109 in ddbbffe

	async def run_porter(command, config: dict):
	"""
	Run a Porter command
	"""
	command = [
	f"{azure_login_command(config)} && ",
	f"{azure_acr_login_command(config)} && ",
	f"{apply_porter_credentials_sets_command(config)} && ",
	*command
	]

But not when calling porter explain, prior to the above code running:

AzureTRE/resource_processor/resources/commands.py

Lines 106 to 107 in ddbbffe

	async def get_porter_parameter_keys(config, msg_body):
	command = [f"porter explain --reference {config['registry_server']}/{msg_body['name']}:v{msg_body['version']} --output json"]

[jonnyry]

OK just checking the creds on the VM and inside the resource processor container... the two are not the same, at least on my instance :-D