awslabs / awsprocesscreds

Process credential providers for AWS SDKs and Tools

AWS Process Credential Providers

A collection of process-based credential providers to be used with the AWS CLI and related tools.

This is an experimental package, breaking changes may occur on any minor version bump.

Installation

The easiest way to install is to use pip:

pip install awsprocesscreds

Requirements

This package requires a version of python to be installed. Currently supported python versions are:

• 2.7.9+
• 3.3.x
• 3.4.x
• 3.5.x
• 3.6.x

SAML Forms-Based Authentication

If you have a SAML identity provider, you can use awsprocesscreds-saml to configure programmatic access to your AWS resources. It has four required arguments:

• -e / --endpoint - Your SAML idp endpoint.
• -u / --username - Your SAML username.
• -p / --provider - The name of your SAML provider. Currently okta and adfs are supported.
• -a / --role-arn- The role arn you wish to assume. Your SAML provider must be configured to give you access to this arn.

This will cache your credentials by default, which will allow you to run multiple commands without having to enter your password each time. You can disable the cache by specifying --no-cache.

Additionally, you can show logs by specifying -v or --verbose.

To configure this provider, you need create a profile using the credential_process config variable. See the AWS CLI Config docs for more details on this config option.

Example okta configuration:

[profile okta]
region = us-west-2
credential_process = awsprocesscreds-saml -e https://example.okta.com/home/amazon_aws/blob/123 -u '[email protected]' -p okta -a arn:aws:iam::123456789012:role/okta-dev

Example adfs configuration:

[profile adfs]
region = us-west-2
credential_process = awsprocesscreds-saml -e 'https://corp.example.com/adfs/ls/IdpInitiatedSignOn.aspx?loginToRp=urn:amazon:webservices' -u Monty -p adfs -a arn:aws:iam::123456789012:role/ADFS-Dev

Custom Providers

The mechanism this package uses to provide credentials is generally available, and not specific to this package. It can be used to implement any custom credential provider that will work with the AWS CLI, boto3, and other SDKs as they implement support.

A detailed breakdown of this mechanism along with a live demo of implementing a credential provider that hooks into the macOS keychain can be seen on this recorded talk from re:Invent 2017: AWS CLI: 2107 and Beyond

The CLI will call the process provided as the value for credential_process. This process must return credentials on stdout in the following JSON form:

{
   "Version": 1,
   "AccessKeyId": "string",
   "SecretAccessKey": "string",
   "SessionToken": "string",
   "Expiration": "2019-01-31T21:45:41+00:00"
}

Where Expiration is an RFC 3339 compatible timestamp. As the expiration time nears, the process will be called again to get a new set of credentials. The Version denotes the version of this format, whose only current valid value is 1. The remaining keys are the AWS credentials you wish to use.

awsprocesscreds's Issues

HTML Parsing is not robust enough

I am pointing to a login page that has some oddities in it, exposing some assumptions about the pages being processed.

Two examples:

1. escape function does not like None being passed in
2. name tag is required, yet login_form_html_node.findall(".//input")) does not filter

Both of these are valid markup - the first being a boolean attribute being present, the second being an input tag without a name attribute.

Feature request: Ability to accept additional input from end-user

As noted in the discussion around adding MFA support for Okta, in order to accept additional input from an end-user you need to use a workaround: getpass.getpass or STDERR.

I would like to add a couple more features relative to Okta: dynamic role discovery and selection, as well as aws account selection. These would involve accepting input from the end-user, so, it would be great to have a different way to communicate back to the main AWS CLI tool other than STDOUT.

Is the Credential Provider have to use the Windows password?

Hello there, great product:
I am using CredentialProvider to develop my custom login method，After my research，the Credential Provider seems to require the user's original Windows password. my question is : Is the Credential Provider have to know the original Windows password, and pass it to the operating system for verification, Otherwise the user cannot log in successfully?

Archive project

This project is clearly abandoned. Please archive it.

awsprocesscreds.saml.SAMLError: Unable to choose role

If I use an --role-arn which is different than my "initial" role but is valid for me to switch to, I get the following error:

Error when retrieving credentials from custom-process: Traceback (most recent call last):
  File "/usr/local/bin/awsprocesscreds-saml", line 11, in <module>
    sys.exit(saml())
  File "/Library/Python/2.7/site-packages/awsprocesscreds/cli.py", line 81, in saml
    creds = fetcher.fetch_credentials()
  File "/Library/Python/2.7/site-packages/awsprocesscreds/saml.py", line 348, in fetch_credentials
    creds = super(SAMLCredentialFetcher, self).fetch_credentials()
  File "/Library/Python/2.7/site-packages/botocore/credentials.py", line 555, in fetch_credentials
    return self._get_cached_credentials()
  File "/Library/Python/2.7/site-packages/botocore/credentials.py", line 565, in _get_cached_credentials
    response = self._get_credentials()
  File "/Library/Python/2.7/site-packages/awsprocesscreds/saml.py", line 357, in _get_credentials
    kwargs = self._get_assume_role_kwargs()
  File "/Library/Python/2.7/site-packages/awsprocesscreds/saml.py", line 398, in _get_assume_role_kwargs
    arns = self._get_role_and_principal_arn(assertion)
  File "/Library/Python/2.7/site-packages/awsprocesscreds/saml.py", line 379, in _get_role_and_principal_arn
    self._config.get('role_arn'), role_arns
awsprocesscreds.saml.SAMLError: Unable to choose role "arn:aws:iam::1234567890:role/RoleInDifferentAccount" from ['arn:aws:iam::0987654321:role/InitialRole']

I have obscured the actual account numbers and role names above but the important thing is that they are different roles in different accounts (it is a sub accounts versus a master account in our organization) and I have no problem switching between them in the AWS Console.

I do not get the error if I use the "InitialRole" as the argument to --role-arn but obviously then I am not in the account I want to be, nor do I have the correct role.

This is when using the ADFS provider and I had to make manual changes to be able to log in as described in issue #11

ssl error - cant ignore

Jordan
I have configured everything to have code commit to use processcreds to use AD to SSO to AWS, get token and then call code commit.
I have tested individual pieces - AD to SSO is working
Code commit with AWS Access key/secret key is working
then ur code is having issue - so tried to run tests in your repository and getting create_assertion failing.
Actual error ssl_error, i tried to config .gitconfig http.sslverify=false with url, and not working.

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

While doing git clone using command line getting the following issue:

Traceback (most recent call last):
File "C:\Users\ykapare\AppData\Local\Programs\Python\Python36\Scripts\git-remote-codecommit-script.py", line 11, in
load_entry_point('git-remote-codecommit==0.5', 'console_scripts', 'git-remote-codecommit')()
File "c:\users\ykapare\appdata\local\programs\python\python36\lib\site-packages\git_remote_codecommit_init_.py", line 152, in main
context = Context.from_url(remote_url)
File "c:\users\ykapare\appdata\local\programs\python\python36\lib\site-packages\git_remote_codecommit_init_.py", line 125, in from_url
credentials = session.get_credentials()
File "C:\Users\ykapare\AppData\Roaming\Python\Python36\site-packages\botocore\session.py", line 425, in get_credentials
'credential_provider').load_credentials()
File "C:\Users\ykapare\AppData\Roaming\Python\Python36\site-packages\botocore\credentials.py", line 1693, in load_credentials
creds = provider.load()
File "C:\Users\ykapare\AppData\Roaming\Python\Python36\site-packages\botocore\credentials.py", line 789, in load
creds_dict = self._retrieve_credentials_using(credential_process)
File "C:\Users\ykapare\AppData\Roaming\Python\Python36\site-packages\botocore\credentials.py", line 815, in retrieve_credentials_using
parsed = botocore.compat.json.loads(stdout.decode('utf-8'))
File "c:\users\ykapare\appdata\local\programs\python\python36\lib\json_init.py", line 354, in loads
return _default_decoder.decode(s)
File "c:\users\ykapare\appdata\local\programs\python\python36\lib\json\decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "c:\users\ykapare\appdata\local\programs\python\python36\lib\json\decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Support for setting DurationSeconds(MaxSessionDuration) during AssumeRoleWithSaml

It would be great if we can specify the DurationSeconds parameter as MaxSessionDuration while assuming the role. Currently the code uses the default 1 hour which is also the IAM role default. However we can set the MaxSessionDuration for a IAM Role to be maxed at 12 hours

Creating an optional parameter which will be defaulted to 1 would be ideal. When providing a DurationSeconds bigger than MaxSessionDuration then we would overwrite the DurationSeconds parameter with MaxSessionDuration retrieved from role details.

Support for Shibboleth IdPv3 and shibcas authenticator

Currently ADFS and Okta are supported. In Higher Education the Shibboleth IdP is a very popular SAML2 IdP.

A "vanilla" deployment of shib uses a basic form-based credential, and would probably work with very little in the way of code changes. The Shib IdP can also be configured to delegate actual authentication to a separate system. At my institution, we delegate authentication to Apereo CAS. We use the shibcas authenticator by Unicon.

This means that the awsprocesscreds helper would need to be able to follow the initial request from the Shib IdP, to the CAS authentication service, and back to the Shib IdP to intercept the SAML2 response. I have some experience with this, and I could submit a PR if you are interested.

I also submitted a separate issue because we protect our CAS authentication with Duo Security MFA. This makes the authentication flow have an additional step. The 2 issues are separate, but they are related in a way, as they both require some mechanism where the basic authentication flow has expanded steps.

F5 SSO provider

F5's implementation of SAML seems pretty straight forward.

With minor tweaks to my locally installed saml.py I was able to get it working. I'll see about submitting a PR if I get a chance to set up a development environment.

Okta

Hi,
When using the cli I now have this error
```'utf-8' codec can't decode byte 0xe9 in position 487: invalid continuation byte````
The only change I can see is that we now use Okta Verify mobile but it's not supposed to be activated for AWS yet.
Also, the main page is in French.
Anybody got the same error and found the resolution?

Plug-in Architecture for Authenticators

I have a need for a custom form authenticator, specifically one that does not conform with the existing Generic, Okta or ADFS versions.

I am unable to influence the existing page design and for compatibility reasons, we are unlikely to change it. A plugin-in model would allow me to bring my own authenticators to the mix.

CLI option for specifying User Agent

In our SAML environment we use the User-Agent to determine if forms based auth is going to be allowed or if something else will be required. I would like to be able to pass the desired User-Agent as a command line option.

Can't use a default profile

I ran in to an issue recently where I kept getting prompted for a password. Entering the correct password resulted in another password prompt:

$ aws s3 ls
Password: 
Password: 
Password: 

It keeps doing this until I enter an incorrect password or ^C out of it.

After some serious debugging, I discovered that this is because I was using a default profile in my .aws/config:

[profile default]
credential_process = awsprocesscreds-saml ...

Changing that to anything other than default fixes the problem.

Does anyone know a way around this? For an interim fix, I've created a bash alias that invokes aws --profile my_profile, but a real fix would be nice to have.

Code is posting form back to url specified in configuration, not to the url you are on

We have a SAML flow that starts with a landing URL that sets some specific session metadata and then re-routes you to the real login form. This is causing issues, specifically with the code in the method: GenericFormsBasedAuthenticator._retrieve_login_form_from_endpoint(endpoint, verify=True)

        response = self._requests_session.get(endpoint, verify=True)
        ....
        form_action = urljoin(endpoint,
                              login_form_html_node.attrib.get('action', ''))

The above code assumes that the page you visited initially is the same URL you are currently on, which is not always true. Regardless, the HTML spec is pretty specific: https://www.w3.org/TR/html52/sec-forms.html#form-submission-algorithm

8. If action is the empty string, let action be the document’s URL of the form document.

Naively I just replaced the endpoint with response.url and it worked as expected. but I haven't pored through the rest of the code to see if this would affect another place (like, session management?)

It keeps asking for the password

I have a new issue now.

I configured Okta as provider and if I try to use it with credential_process it keeps asking for the password.

If I then run the command once directly in the command line and enter the password it works on subsequent calls as it uses the cached response.

What is the problem here?

--verbose parameter causes json parsing issue

When I use the --verbose (or -v) parameter, I can successfully authenticate and get a json object from the awsprocesscreds tool, but when this object is sent to the core cli tool I always get the following error:

Expecting value: line 1 column 1 (char 0)

I suspect that the --verbose flag is corrupting the STDOUT sent to the cli.

I am testing on a clean Ubuntu 18.04 image with Python 2.7.9. I've also tested with Python 3.6 and gotten the same result.

Doesn't seem to work if MFA is configured on Okta

If my Okta account has an MFA then authenticating fails with:

Traceback (most recent call last):
  File "/usr/local/bin/awsprocesscreds-saml", line 11, in <module>
    sys.exit(saml())
  File "/usr/local/lib/python2.7/dist-packages/awsprocesscreds/cli.py", line 81, in saml
    creds = fetcher.fetch_credentials()
  File "/usr/local/lib/python2.7/dist-packages/awsprocesscreds/saml.py", line 348, in fetch_credentials
    creds = super(SAMLCredentialFetcher, self).fetch_credentials()
  File "/home/philip/.local/lib/python2.7/site-packages/botocore/credentials.py", line 507, in fetch_credentials
    return self._get_cached_credentials()
  File "/home/philip/.local/lib/python2.7/site-packages/botocore/credentials.py", line 517, in _get_cached_credentials
    response = self._get_credentials()
  File "/usr/local/lib/python2.7/dist-packages/awsprocesscreds/saml.py", line 357, in _get_credentials
    kwargs = self._get_assume_role_kwargs()
  File "/usr/local/lib/python2.7/dist-packages/awsprocesscreds/saml.py", line 393, in _get_assume_role_kwargs
    assertion = self._authenticator.retrieve_saml_assertion(config)
  File "/usr/local/lib/python2.7/dist-packages/awsprocesscreds/saml.py", line 238, in retrieve_saml_assertion
    session_token = parsed['sessionToken']
KeyError: 'sessionToken'

If I unset the MFA in Okta, authentication succeeds.

Please add support for MFA as this is going to be a requirement if we adopt this authentication model.

Thanks.

Cannot parse form field

I am attempting to run the following line:

awsprocesscreds-saml --endpoint https://my.company.com/adfs/ls/IdpInitiatedSignOn.aspx?loginToRp=urn:amazon:webservices --username '[email protected] --provider adfs --role-arn arn:aws:iam::595513102389:role/IaaSOperations --verbose

I am getting the following error:

Traceback (most recent call last):
  File "c:\python27\lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\Scripts\awsprocesscreds-saml.exe\__main__.py", line 9, in <module>
  File "c:\python27\lib\site-packages\awsprocesscreds\cli.py", line 81, in saml
    creds = fetcher.fetch_credentials()
  File "c:\python27\lib\site-packages\awsprocesscreds\saml.py", line 348, in fetch_credentials
    creds = super(SAMLCredentialFetcher, self).fetch_credentials()
  File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\botocore\credentials.py", line 507, in fetch_credentials
    return self._get_cached_credentials()
  File "C:\Users\me\AppData\Roaming\Python\Python27\site-packages\botocore\credentials.py", line 517, in _get_cached_credentials
    response = self._get_credentials()
  File "c:\python27\lib\site-packages\awsprocesscreds\saml.py", line 357, in _get_credentials
    kwargs = self._get_assume_role_kwargs()
  File "c:\python27\lib\site-packages\awsprocesscreds\saml.py", line 393, in _get_assume_role_kwargs
    assertion = self._authenticator.retrieve_saml_assertion(config)
  File "c:\python27\lib\site-packages\awsprocesscreds\saml.py", line 138, in retrieve_saml_assertion
    self._fill_in_form_values(config, form_data)
  File "c:\python27\lib\site-packages\awsprocesscreds\saml.py", line 178, in _fill_in_form_values
    self._ERROR_MISSING_FORM_FIELD % self.USERNAME_FIELD)
awsprocesscreds.saml.SAMLError: Error parsing HTML form, could not find the form field: "ctl00$ContentPlaceHolder1$UsernameTextBox"

Support for AWS SSO?

When logging in through AWS SSO, it can retrieve temporary keys for users to copy/paste into their shell or a config file. This isn't ideal; it's still difficult to use on a routine basis from the command line as you need to get the credentials out of the browser. Seems like something this package could help with, yes?

It appears that AWS SSO works by adding a SAML identity provider within the configured accounts. I would think it ought to be possible then to use a package like this to interface with SSO to retrieve the credentials and make them available via credential_process. Does that already work and I'm just not seeing how? Or, any idea how to go about adding that support?

Feature Request: Support Google Login

Hi.

It would be awesome if we could add google login to this tool.

Here's an article showing how to get the SAML response from Google: https://wheniwork.engineering/finally-saml-command-line-access-for-aws-sts-b0e33bbba26e

Thanks.

process hangs when connecting to ADFS server

Traceback (most recent call last):
  File "/usr/local/bin/awsprocesscreds-saml", line 11, in <module>
    sys.exit(saml())
  File "/Library/Python/2.7/site-packages/awsprocesscreds/cli.py", line 81, in saml
    creds = fetcher.fetch_credentials()
  File "/Library/Python/2.7/site-packages/awsprocesscreds/saml.py", line 348, in fetch_credentials
    creds = super(SAMLCredentialFetcher, self).fetch_credentials()
  File "/Library/Python/2.7/site-packages/botocore/credentials.py", line 507, in fetch_credentials
    return self._get_cached_credentials()
  File "/Library/Python/2.7/site-packages/botocore/credentials.py", line 517, in _get_cached_credentials
    response = self._get_credentials()
  File "/Library/Python/2.7/site-packages/awsprocesscreds/saml.py", line 357, in _get_credentials
    kwargs = self._get_assume_role_kwargs()
  File "/Library/Python/2.7/site-packages/awsprocesscreds/saml.py", line 393, in _get_assume_role_kwargs
    assertion = self._authenticator.retrieve_saml_assertion(config)
  File "/Library/Python/2.7/site-packages/awsprocesscreds/saml.py", line 137, in retrieve_saml_assertion
    endpoint)
  File "/Library/Python/2.7/site-packages/awsprocesscreds/saml.py", line 148, in _retrieve_login_form_from_endpoint
    response = self._requests_session.get(endpoint, verify=True)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/Library/Python/2.7/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/Library/Python/2.7/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/Library/Python/2.7/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/Library/Python/2.7/site-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/Library/Python/2.7/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 352, in wrap_socket
    _context=self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 579, in __init__
    self.do_handshake()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 808, in do_handshake
    self._sslobj.do_handshake()
KeyboardInterrupt

It should print out what is going wrong or at least fail faster.

Okta broken

Hello,

It seems that okta login no longer works. I am putting error log (I replaced there sensitive information)

username@MAC ~ % awsprocesscreds-saml --verbose -e https://organization.okta.com/app/amazon_aws/randomstringg/sso/saml -u '[email protected]' -p okta -a arn:aws:iam::1234567890:role/my-role

Password: 
Sending HTTP POST with username ([email protected]) and password to Okta API endpoint: https://organization.okta.com/api/v1/authn
Traceback (most recent call last):
  File "/Users/username/.pyenv/versions/3.8.0/bin/awsprocesscreds-saml", line 8, in <module>
    sys.exit(saml())
  File "/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/awsprocesscreds/cli.py", line 81, in saml
    creds = fetcher.fetch_credentials()
  File "/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/awsprocesscreds/saml.py", line 353, in fetch_credentials
    creds = super(SAMLCredentialFetcher, self).fetch_credentials()
  File "/Users/username/.local/lib/python3.8/site-packages/botocore/credentials.py", line 643, in fetch_credentials
    return self._get_cached_credentials()
  File "/Users/username/.local/lib/python3.8/site-packages/botocore/credentials.py", line 653, in _get_cached_credentials
    response = self._get_credentials()
  File "/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/awsprocesscreds/saml.py", line 362, in _get_credentials
    kwargs = self._get_assume_role_kwargs()
  File "/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/awsprocesscreds/saml.py", line 403, in _get_assume_role_kwargs
    assertion = self._authenticator.retrieve_saml_assertion(config)
  File "/Users/username/.pyenv/versions/3.8.0/lib/python3.8/site-packages/awsprocesscreds/saml.py", line 240, in retrieve_saml_assertion
    session_token = parsed['sessionToken']
KeyError: 'sessionToken'

Support for disabling ssl verification

Since aws cli supports --no-verify-ssl but in case of awsprocesscreds there is no option for disabling the ssl verification.

Support for Duo Security MFA

I specifically have a use case for incorporating a Duo Security MFA flow into the credential processor. At my institution, we use Duo Security's Duo Prompt to provide a 2nd factor for authentication. After a user enters her institutional credentials (username and password), the Duo Prompt appears in an IFRAME element and allows the user to select a device and authentication method. We allow U2F devices, phone callback, one-use codes, or Duo Push. The current form scraping credential processor doesn't handle this additional flow.

I have some experience with the kind of web scraping that would be needed to get this to work, and I could submit a PR if you are interested.