Default role kubeflow-edit not permitted to create/update pytorchjob about training-operator

[kimwnasptd]

Hey @canonikamal, first of this is indeed a bug in Charmed Kubeflow. The bug is that we've missed the Training Operator roles in our Charm that creates Aggregation ClusterRoles. This is why the default-editor ends up not having permissions for PyTorch job CRs.

We'll keep this issue open and work on fixing this in the next release.

To unblock you I'd suggest you to apply the upstream ClusterRoles directly
https://github.com/kubeflow/training-operator/blob/v1.5-branch/manifests/overlays/kubeflow/kubeflow-training-roles.yaml

Lastly, regarding the recommended approach: It's always preferred to create ClusterRoles that get aggregated to Kubeflow's edit/view/admin ClusterRoles. The reason is that this way these permissions will get propagated to all user namespaces (since in the user namespace the default-editor will RoleBinding the kubeflow-edit ClusterRole in each Profile namespace)

[canonikamal]

Thanks for the response and confirmation on the bug @kimwnasptd .

I've shared the workaround with the customer and thanks again for sharing the recommended approach on the aggregated roles.

Best regards
Kamal Bhaskar

[canonikamal]

Got closed erroneously while adding last comment, Re-opened as the bug is still being tracked.

[kimwnasptd]

Small clarification, after my updated understanding, that this AggregatedRole from our side will need to live inside the Training Operator repo https://github.com/canonical/training-operator/tree/main/src/templates

The pattern is that these Roles should live alongside the Charm. But for this to happen we'll need to ensure the Charm is not PodSpec but follows the Sidecar pattern.

For the Training Operator though it is a Sidecar Charm so we can just put the template in the correct folder.