Account Index Generator Crashes in Organizations with Nested OUs about starfleet

[ericwestfall]

Feel free to assign this one to me, I will raise a PR with the fix shortly.

[mikegrima]

Sounds great! It's all you :)

[mikegrima]

@ericwestfall I don't have a great way of testing for this. Can you supply the same API result from AWS that causes the failure? With that, I should be able to make the necessary adjustments.

[mikegrima]

@ericwestfall Interesting... so basically I need to... for each OU... make a call to the list_organizational_units_for_parent() API to see if there are parents in there?

I would just like to add: and please pass this along to the requisite teams: I absolutely detest the Orgs API! The fact that we can't easily get a list of all the OUs and their parental relationships is utterly ridiculous!

I'm going to need to figure out how to make these calls recursively... I might need to do some more refactoring to have yet another async call to get these details first and then collect the details about the tags and regions 🤮

[ericwestfall]

I would just like to add: and please pass this along to the requisite teams: I absolutely detest the Orgs API! The fact that we can't easily get a list of all the OUs and their parental relationships is utterly ridiculous!

I hear you on that for sure @mikegrima 🤣

If you can give me a few days, I will send you a PR that refactors the AWS Organizations integration in Starfleet to use an AWS Organizations library I developed that provides discovery of accounts and OUs with recursion, fully-qualified path resolution for accounts back to the organization root, and supports attribute-based search (e.g., account name, tags, path, etc.). I have this integration work mostly done over here, just need to clean it up a bit and finish writing tests.

Let me know if you want the PR or prefer to tackle it yourself!

[mikegrima]

That works too!

Another thing I could do (and I hate this) is that in this code here:

starfleet/src/starfleet/worker_ships/plugins/account_index_generator/utils.py

Line 168 in 3d495f1

parent_ous.append({"Id": parent["Id"], "Type": parent["Type"], "Name": all_ous[parent["Id"]]})

I can make it so that instead of directly referencing the dictionary, I call a function that will attempt to find the OU and if it's not loaded yet, it will lazy load the OU name in. It would work it's just annoying.

^^ I think this would be more API efficient than recursive calls, since you will only fetch the details you know that you need vs. making a bunch of list calls a priori.

[mikegrima]

@ericwestfall Friendly ping. I am planning on starting to work on this tomorrow so this can be wrapped up by fwd:cloudsec 2023.

[mikegrima]

@ericwestfall do me a favor and run the code I have in that draft PR and let me know if that fixes the issue for you

[ericwestfall]

@ericwestfall do me a favor and run the code I have in that draft PR and let me know if that fixes the issue for you

Hey @mikegrima, while the change in #53 correctly resolves the name of the OU that is the direct parent of an account, even when the OU is missing from the top-level OU map, we are still not recursively resolving the path from the account all the way back to the root. This means the resulting account index will only contain the root and the direct parent which contains the account, but will be missing any intermediary ancestors between the root and the direct parent OU.

AWS Organizations supports nesting of OUs up to 5 levels deep within a root, which is to say the depth of any given account is bounded by the constraint $1 <= d <= 6$.

For a given account whose depth exceeds 2 (e.g., Root -> Workloads -> Prod -> Account), the generated account index entry will contain a disjoint path (e.g., Root -> Prod) instead of the fully connected path (e.g., Root -> Workloads -> Prod).

Additional Context

Provided below is the structure of the organization in my research environment:

├── Root (r-olqy)
│   ├── Workloads (ou-olqy-e604f59w)
│   │   ├── Non-Prod (ou-olqy-nndadfrq)
│   │   │   ├── aws-security-research-lab-workload-dev (712861421548)
│   │   ├── Prod (ou-olqy-q8z9vop9)
│   │   │   ├── aws-security-research-lab-workload-prod (228001810812)
│   ├── Core Infrastructure (ou-olqy-q97w37sn)
│   │   ├── aws-security-research-lab-devops (629225780623)
│   │   ├── aws-security-research-lab-network (896367252297)
│   │   ├── aws-security-research-lab-secops (150958791683)
│   ├── Sandbox (ou-olqy-7puk9u2d)
│   ├── Security (ou-olqy-u144jfki)
│   │   ├── Audit (950175702822)
│   │   ├── Log Archive (240561585551)
└── aws-security-research-lab-management (401643647498)

For a given AWS account 228001810812 ($d=3$), this is the resulting account index entry generated (code at revision 8405d19) which contains a disjoint path in the Parents attribute:

{
  "228001810812": {
    "Id": "228001810812",
    "Arn": "arn:aws:organizations::401643647498:account/o-qsujwb57n9/228001810812",
    "Email": "[email protected]",
    "Name": "aws-security-research-lab-workload-prod",
    "Status": "ACTIVE",
    "JoinedMethod": "CREATED",
    "JoinedTimestamp": "2023-03-10 08:52:08.254000-08:00",
    "Tags": {},
    "Parents": [
      {
        "Id": "ou-olqy-q8z9vop9",
        "Type": "ORGANIZATIONAL_UNIT",
        "Name": "Prod"
      },
      {
        "Id": "r-olqy",
        "Type": "ROOT",
        "Name": "ROOT"
      }
    ],
    "Regions": [
      "us-east-1",
      "us-east-2"
    ]
  }
}

Ideally, the account index object generated for the account should contain a fully connected path to the root, including all intermediary ancestors between the account's immediate parent and the root:

{
  "228001810812": {
    "Id": "228001810812",
    "Arn": "arn:aws:organizations::401643647498:account/o-qsujwb57n9/228001810812",
    "Email": "[email protected]",
    "Name": "aws-security-research-lab-workload-prod",
    "Status": "ACTIVE",
    "JoinedMethod": "CREATED",
    "JoinedTimestamp": "2023-03-10 08:52:08.254000-08:00",
    "Tags": {},
    "Parents": [
      {
        "Id": "ou-olqy-q8z9vop9",
        "Type": "ORGANIZATIONAL_UNIT",
        "Name": "Prod"
      },
      {
        "Id": "ou-olqy-e604f59w",
        "Type": "ORGANIZATIONAL_UNIT",
        "Name": "Workloads"
      },
      {
        "Id": "r-olqy",
        "Type": "ROOT",
        "Name": "ROOT"
      }
    ],
    "Regions": [
      "us-east-1",
      "us-east-2"
    ]
  }
}

Additional Considerations

This also has implications for account resolution. For example, consider a worker that implements an Account fan out strategy and needs to resolve the following IncludeAccounts payload stanza:

IncludeAccounts:
    ByOrgUnits:
        - Workloads
ExcludeAccounts:
    ByNames:
        - aws-security-research-lab-workload-prod

Considering the Workloads OU contains multiple child OUs and accounts at varying levels, what is the expected behavior and what is the users intent? Should this resolve to all accounts that are contained directly in the Workloads OU and any OUs nested below the Workloads OU?

This requires that we have the capability to resolve the complete subset of accounts for any given path, which is why I was pursuing a broader solution in parallel to this effort. Let me know your thoughts, happy to chat about this in more detail ...

[mikegrima]

Yup -- fixing that now

[mikegrima]

Question: Does list_parents only ever return 1 parent? Why call it list_parents and make it paginated for???? <-- this is a question for the orgs team

[mikegrima]

Just updated the PR. I think this should finally do it. What a PITA! I'll test internally on Monday just to triple check.

[mikegrima]

I have confirmed that the latest changes on that branch will work with nested OUs and everything. Though, please triple check for me if there are any outstanding issues.

[mikegrima]

@ericwestfall I just merged in my PR. I do believe that this has been fully addressed now based on my own internal testing with nesting accounts in various OUs within my organization.

However, please let me know if you continue to see any issues.