Single user pipeline run is having error about mlx

[Tomcli]

/assign @ckadner

[ckadner]

@Tomcli -- on your single-user cluster (connecting from the outside) I get this error when trying to create (or get) an experiment:

Invalid input error: In single-user mode, ListExperiment cannot filter by namespace.

This happens regardless of setting the namespace to None, "", leaving the parameter out, since the KFP client will default to populate the kubeflow namespace (client.get_user_namespace() --> kubeflow for me) in it's own call to the KFP server API to get_experiment.

Interestingly I can run the same MLX API code snippet directly from inside the MLX API pod (client.get_user_namespace() --> ''). The kfp SDK versions inside vs outside the cluster are the same.

---

@yhwang -- I am getting this error (from inside the demo cluster) multi-user:

Internal error: Unauthenticated: Request header error: there is no user identity header.: Request header error: there is no user identity header.
Failed to authorize with API resource references

I did look up the issues 4440 and 4377 on KFP regarding this error, but it was not clear to me how to solve it. Can you help?

[Tomcli]

@ckadner so I use the KFP-Tekton 0.8.1 SDK to run pipelines on single user mode. If you follow this instruction it should able to submit the pipeline with the run_pipeline function.
https://github.com/kubeflow/kfp-tekton/tree/master/guides/kfp-user-guide#2-run-pipelines-using-the-kfp_tektontektonclient-in-python

[ckadner]

@Tomcli -- the MLX API deployed on your single-user cluster is still trying to use the ml-pipeline-ui service. I will switch to using the ml-pipeline service, or, rather I will let KFP Client figure that out.

[Tomcli]

@ckadner as long as the API is able to run pipelines, it doesn't really matter how you provision the client. But I do recommend to use the ml-pipeline service endpoint if we want to keep things consistent.

[Tomcli]

you can add a flag during deployment time to tell whether MLX is deployed with single user or multi-user KFP. I will update the manifest accordingly.

[yhwang]

For the error in multi-user env, I guess the solution is that mlx-api pod needs to carry the kubeflow-userid header when calling ml-pipeline APIs. can you try out this mlx-ui image: yihongwang/mlx-ui:dev?
In that image, I injected the kubeflow-userid header into the request at the proxy middleware.

In mlx-api, if it talks to ml-pipeline-ui, then adding kubeflow-userid header that received from mlx-ui (with my image above), it should works. However, if it talks to ml-pipeline, then I need to study the code to know how it get the userid information. Before I study the code, I assume that ml-pipeline also rely on kubeflow-userid header.

[yhwang]

@ckadner I checked the code in both ml-pipeline-ui and ml-pipeline. For ml-pipeline-ui, it just forwards the apis calls to ml-pipeline which contain kubeflow-userid header for user id. For ml-pipeline, it uses two authenticators to retrieve user id:

• http_header: uses kubeflow-userid header
• token_review: uses authorization header

So the bottom line is you have to pass kubeflow-userid header to ml-pipeline. However, I don't know how can you do that by using the kfp.Client().

[ckadner]

Thank @yhwang for your findings. Curiously the pipeline upload and pipeline delete work just fine. Just the pipeline run/launch does run into the Internal error: Unauthenticated: Request header error: there is no user identity header. Failed to authorize with API resource references

If there is no obvious solution for doing this authentication using the kfp.Client, should we open a new issue on the kubeflow/pipelines repo?