Blocker: customUserAcls - Plan command - Marks the action for every entry on the state.yaml as "REMOVE". about kafka-gitops

[jaykatti]

I am using the latest version 0.2.15.
This seems to be a complete blocker. Can this please be taken-up on priority ?

[devshawn]

Hi @jaykatti, can you provide an example state.yaml file to help reproduce?

Do note that ACLs do not update, they only add/remove. (so a change to an ACL should have one create and one delete).

[jaykatti]

Hello Shawn @devshawn , Thanks for your response !

For the below sample state.yaml, if I have to change the permission for describe_configs_topic to "DENY". I change the permission to "DENY" for that block and then run the kafka-gitops command for validate, plan.
When I run the plan, I see that, the plan indicates "Delete" or "Remvove" for all the ACL's present in my system and not just the ones in the yaml file.

customUserAcls:
User:kattijay:
describe_configs_topic:
name: topic1
pattern: LITERAL
host: ""
permission: ALLOW
type: TOPIC
operation: DESCRIBE_CONFIGS
describe_topic:
name: topic1
pattern: LITERAL
host: ""
permission: ALLOW
type: TOPIC
operation: DESCRIBE
write_topic:
name: topic1
pattern: LITERAL
host: ""
permission: ALLOW
type: TOPIC
operation: WRITE
read_topic:
name: topic1
pattern: LITERAL
host: ""
permission: ALLOW
type: TOPIC
operation: READ

[jaykatti]

host above is * for all the records.

[jaykatti]

Hello @devshawn , Any further update for me on this please ?

[jaykatti]

Hi @jaykatti, can you provide an example state.yaml file to help reproduce?

Do note that ACLs do not update, they only add/remove. (so a change to an ACL should have one create and one delete).

I have generated a yaml (initial state) for my existing cluster. Now, this includes a large list of users and ACL's. I will use this as my base configuration and start modifying it based on the requirement. Either update an ACL or delete or Add a new one.
Say, I want to add a new ACL to the existing list, How do I go about this ? I do not want any of my existing ACL's to be deleted.

Kindly advise at the earliest as I need to get this wrapped up and start using it on our cluster.

Thanks,
Jay.

[devshawn]

Hi @jaykatti,

You have to have every ACL listed in your state file. So you need to take your existing cluster and make an initial state file with all topics / ACLs until you run a plan and it says there are no changes. Then, you can start adding/removing ACLs and it will only have the changes you want.

Or am I misunderstanding the situation?

[jaykatti]

Hello Shawn, Yes, I have already completed the below steps - - Generate the state.yaml for my cluster ( This has all the current ACL's for my cluster ). I am running the plan on this yaml and it marks all the existing ACL's for deletions. This is where, I was confused. Ideally, I expected the plan to have nil deletions. Can you pls. guide me on this ? Thanks, Jay.

…

On Tue, Sep 21, 2021 at 6:17 PM Shawn Seymour ***@***.***> wrote: Hi @jaykatti <https://github.com/jaykatti>, You have to have every ACL listed in your state file. So you need to take your existing cluster and make an initial state file with all topics / ACLs until you run a plan and it says there are no changes. Then, you can start adding/removing ACLs and it will only have the changes you want. Or am I misunderstanding the situation? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#77 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI5B75XOUJZD6Z5NNLEBEWDUDCVX5ANCNFSM5C5NFEPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Regards, - Jay

[fireydagostino]

Hi @jaykatti

You're stating a customUserAcls in the above text - do you have a matching users in your state.yaml file as well?

Every customUserAcls and customServiceAcls is connected to a matching users and service respectively.

If you're only claiming the customUserAcls in your state file without the matching users definition, then the plan will disregard the customUserAcls listed in your state file.

For example, if I were to add a customUserAcls named Access_Bob then I'd have in my state file:

users:
  Access_Bob:
    principal: User:Bob
customUserAcls:
  Access_Bob:
    read-kstream-topics:
      name: kstream-
      type: TOPIC
      pattern: PREFIXED
      host: "*"
      operation: READ
      permission: ALLOW

[jaykatti]

Hi @jaykatti

You're stating a customUserAcls in the above text - do you have a matching users in your state.yaml file as well?

Every customUserAcls and customServiceAcls is connected to a matching users and service respectively.

If you're only claiming the customUserAcls in your state file without the matching users definition, then the plan will disregard the customUserAcls listed in your state file.

For example, if I were to add a customUserAcls named Access_Bob then I'd have in my state file:

users:
  Access_Bob:
    principal: User:Bob
customUserAcls:
  Access_Bob:
    read-kstream-topics:
      name: kstream-
      type: TOPIC
      pattern: PREFIXED
      host: "*"
      operation: READ
      permission: ALLOW

Oh !! Okay, Let me try that.

Thanks !, I will keep you posted.

[jaykatti]

Sorry, closed by mistake

[jaykatti]

Hi @jaykatti
You're stating a customUserAcls in the above text - do you have a matching users in your state.yaml file as well?
Every customUserAcls and customServiceAcls is connected to a matching users and service respectively.
If you're only claiming the customUserAcls in your state file without the matching users definition, then the plan will disregard the customUserAcls listed in your state file.
For example, if I were to add a customUserAcls named Access_Bob then I'd have in my state file:

users:
  Access_Bob:
    principal: User:Bob
customUserAcls:
  Access_Bob:
    read-kstream-topics:
      name: kstream-
      type: TOPIC
      pattern: PREFIXED
      host: "*"
      operation: READ
      permission: ALLOW

Oh !! Okay, Let me try that.

Thanks !, I will keep you posted.

---

Hello @fireydagostino @devshawn

I tried with that and it seems to be working. So many thanks !

Do we also need to list all the topics as part of the state.yaml ?
I see, when I run the plan, it is marking the topics for deletion, even though, there are ACL's for those topics.
Kindly confirm !

Thanks,
Jay.

[fireydagostino]

Yeah @jaykatti

The state file literally claims everything that should be present within the current state of your Kafka cluster.

Any topics/ACLs that are on Kafka, but not defined for in your state will be privy to deletion upon execution - and respectively the opposite for topics/Acls defined in your state but not present on Kafka.

Your state.yaml file will need

• default options / blacklisted topic prefixes
• topics
• users + customUserAcl
• services + customServiceAcl

In my production environment, we have some of these separated into specific directories/files pertaining to different teams/technologies and then utilize yq4 to merge all the information into the state.yaml file.

[jaykatti]

Thanks a lot for the clarification. Would you please share a sample state.yaml from ur environment ? Thanks and regards, Jay

…

On Tue, 28 Sep 2021, 7:13 pm fireydagostino, ***@***.***> wrote: Yeah @jaykatti <https://github.com/jaykatti> The state file literally claims everything that should be present within the current state of your Kafka cluster. Any topics/ACLs that are on Kafka, but not defined for in your state will be privy to deletion upon execution - and respectively the opposite for topics/Acls defined in your state but not present on Kafka. Your state.yaml file will need - default options / blacklisted topic prefixes - topics - users + customUserAcl - services + customServiceAcl In my production environment, we have some of these separated into specific directories/files pertaining to different teams/technologies and then utilize yq4to merge all the information into thestate.yaml` file. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#77 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AI5B75VRYZ4HD2T3OMI7YQ3UEHBBZANCNFSM5C5NFEPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.