dahlke / terrasnek

Terraform Cloud Python Client Library

I'm Neil.

Enterprise software veteran with a lot left to learn. Raised in Chicago, living in San Francisco. Feel free to approach me - I’m always up for a chat. If you don’t know where to start, ask me about: data, cloud infrastructure, skiing, music, or basketball. I suspect we'll find common ground.

Twitter / Instagram / Medium / LinkedIn / Strava / Résumé

terrasnek's Issues

Filter value in test should be passed as string

A minor issue, but the filter value used in the test for the policy_sets list method should be updated to use a string, rather than a boolean. Specifically, this:

https://github.com/dahlke/terrasnek/blob/master/test/policy_sets_test.py#L53

Rather than False, this value should be "false".

Unable to create notification configuration

How to reproduce:

from terrasnek.api import TFC
import os

TOKEN = os.getenv("TOKEN", None)
TFC_URL = "https://app.terraform.io"

if __name__ == "__main__":
    api = TFC(TOKEN, url=TFC_URL)
    api.set_org("mytestorg")

    create_workspace_payload = {
      "data": {
        "attributes": {
          "name": "testws"
        },
        "type": "workspaces"
      }
    }
    
    created_workspace = api.workspaces.create(create_workspace_payload)

    create_variable_payload = {
      "data": {
        "type":"vars",
        "attributes": {
          "key":"MY_ACCESS_KEYS",
          "value":"some_value",
          "description":"some description",
          "category":"terraform",
          "hcl":"false",
          "sensitive":"true"
        },
        "relationships": {
          "workspace": {
            "data": {
              "id":created_workspace['data']['id'],
              "type":"workspaces"
            }
          }
        }
      }
    }

    api.vars.create(create_variable_payload)

    create_notification_configuration = {
      "data": {
        "type": "notification-configuration",
        "attributes": {
          "destination-type": "generic",
          "name": "Webhook server test",
          "url": "https://httpstat.us/200",
          "triggers": [
            "run:created"
          ]
        }
      }
    }

    api.notification_configs.create(payload=create_notification_configuration, workspace_id=created_workspace['data']['id'])

This will error out with:

{'errors': [{'status': '404', 'title': 'not found'}]}

The Terraform API documentation is misleading, it says in the description that type must be notification-configuraton while samples are plural. I've tried both to no avail and different permutations of the json payload.

My environment:
MacOS 10.15.6
Python 3.8.5

My pip freeze

astroid==2.4.2
awscli==1.18.115
beautifulsoup4==4.9.1
boto3==1.14.38
botocore==1.17.38
certifi==2020.6.20
cffi==1.14.1
chardet==3.0.4
click==7.1.2
colorama==0.4.3
configparser==3.8.1
cryptography==3.0
dnspython==2.0.0
docutils==0.15.2
email-validator==1.1.1
fido2==0.8.1
gimme-aws-creds==2.3.4
idna==2.10
inflection==0.5.1
isort==5.5.3
jmespath==0.10.0
keyring==21.3.0
lazy-object-proxy==1.4.3
mccabe==0.6.1
okta==0.0.4
pep8==1.7.1
pyasn1==0.4.8
pycodestyle==2.6.0
pycparser==2.20
pydantic==1.6.1
pylint==2.6.0
pyterprise==0.0.15
python-dateutil==2.8.0
PyYAML==5.3.1
requests==2.24.0
rsa==4.5
s3transfer==0.3.3
six==1.15.0
soupsieve==2.0.1
terrasnek==0.0.8
tfc-client==0.7.2
toml==0.10.1
urllib3==1.25.10
wrapt==1.12.1

type checking incompatibilities

The way api.py is initialized via the eventual calls to the dict here is incompatible with the type annotations found here.

Possible approaches are to make the initialization more "static analysis" friendly (mypy is unable to determinethat the dict values are classes being instantiated) with less indirection or to annotate fully using structural typing.

Possible acceptance criteria would be to help type checkers to either not flag type errors due to the above problem (everything would pass as there would be no type hints) or to fix the type hints by adding full type hints. An example of the latter approach can be found in this branch

Registry Modules API does not expose List All or pagination options

Registry Modules API supports pagination but terrasnek only exposes a list option with no way to be able to pass page, size pagination options.

Add support for retries with exponential backoff when encountering certain responses (429s, etc)

We are running into some 429s while using teams.list_all, would you be open to adding retry with exponential backoff support to terrasnek?

I assume we are hitting 429s because _list_all (https://github.com/dahlke/terrasnek/blob/master/terrasnek/endpoint.py#L374-L383) code doesn't have a sleep between the self._list calls so it is sending the API requests as fast as it can. That would be fine as long as retries with exponential backoff were enabled.

Also thank you for creating terrasnek, it is great!

Exceptions are never thrown with context...

I appreciate that messages are displayed via logger, but the error itself has no message whatsoever. This means that catching these and transforming them with context is impossible.

        elif req.status_code == HTTP_NOT_FOUND:
            err = json.loads(req.content.decode("utf-8"))
            self._logger.error(err)
            raise TFCHTTPNotFound()

If you catch this exception there is no context.

I suggest at the very least to pass err as an argument to the exceptions.

terrasnek.api.InvalidTFCTokenException

Hi,

I was trying to call our TFE workspace state version and found Terrasnek to do but while I'm trying to do the first step. It keeps failing with following error. I tried with several python3 versions between 3.7 to 3.11. All of them fail.

`>>> import os
>>> from terrasnek.api import TFC
>>>
>>> TFC_TOKEN = os.getenv("I have Provided User API Token here", None)
>>> TFC_URL = os.getenv("My Paid TFE URL", None)  # ex: https://app.terraform.io
>>>
>>> api = TFC(TFC_TOKEN, url=TFC_URL)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/AL20827/Library/Python/3.9/lib/python/site-packages/terrasnek/api.py", line 151, in __init__
    raise InvalidTFCTokenException
terrasnek.api.InvalidTFCTokenException
>>> api.set_org("My Organization")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'api' is not defined
>>>`

Any thing to mitigate this issue? I'm sure I was not missing anything to atleast to get started with Terrasnek.

Linter and auto-completion with Terranesk

Hello and thanks for Terrasnek !

It seem's that the dynamic attributes adding into the TFC super class make things harder for linter and auto-completion.
I got a lot of 'Instance of 'TFC' has no 'XXX' member', and autocompletion into IDE doesn't work too.

Will you be OK to replace the dynamic attributes adding by a less smart but more user friendly static attributes adding ? If so I can make a PR in this way.

Have a nice day !

Support for search[tags]

Hi, I'm a little confused someone could probably help out but in reviewing the terraform provider it seems like search/list workspaces by tag is possible, but the TF API documentation https://www.terraform.io/cloud-docs/api-docs/workspaces#list-workspaces does not mention this. I would be more than happy to add this feature if you are all ok with it

Check version method break library in air-gaped environments

terrasnek/terrasnek/api.py

Line 258 in 0807ca8

def _check_version(self):

When using the library a lambda without connectivity to pypi the constructor of the api object cant be invoked and fails with the following error:

[ERROR] ConnectionError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/terrasnek/json (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) Traceback (most recent
call last):   File "/var/task/example_lambda.py", line 423, in lambda_handler     tfe_api = terrasnek.api.TFC(   File "/var/task/terrasnek/api.py", line 236, in init     self._check_version()   File "/var/task/terrasnek/api.py",
line 257, in _check_version     r = requests.get(f'https://pypi.org/pypi/{self.package_name}/json'))
  File "/var/task/requests/api.py", line 73, in get     return request("get", url, params=params, **kwargs)   File "/var/task/requests/api.py", line 59, in request     return session.request(method=method, url=url, **kwargs)   File "/var/task/requests/sessions.py",
line 587, in request     resp = self.send(prep, **send_kwargs)   File "/var/task/requests/sessions.py", line 701, in send     r = adapter.send(request, **kwargs)   File "/var/task/requests/adapters.py", line 565, in send     raise ConnectionError(e, request=request)

Would it be possible to make this check non blocking?

Regards,
Miguel

pagination

just discovered this, awesome!
writing these here so I don't forget,

• would like to implement pagination. for example when getting workspaces it's limited to the first 20. adding a ?page%5Bsize%5D=100 to the URL would increase it to 100 for example.
• also may be worth adding a "how to install" note on the readme, specifying python3. I initially tried running in python2.7 and that did not work.

Missing required filter option for the Team Membership List Method

First off, terrasnek is amazing! It really is by far the best method for interacting with the TFC/E API that exists. That said, I've found a minor issue, outlined below.

The team membership endpoint has a list function that requires a filter parameter of [workspace][id], as documented here:
https://www.terraform.io/docs/cloud/api/team-access.html#query-parameters

At present, this is missing from terrasnek here:
https://github.com/dahlke/terrasnek/blob/master/terrasnek/team_access.py#L30

Thank you!

Workspaces List of API endpoint support filters

Hi!

I checked the List endpoint of the Workspaces API, and it supports filter. The documentation didn't list the feature, but the following request shows the feature support of using filters.

https://app.terraform.io/api/v2/organizations/xxx/workspaces?fields[workspace][]=current_run&fields[workspace][]=latest_change_at&fields[workspace][]=name&fields[workspace][]=vcs_repo_identifier&fields[workspace][]=locked&fields[workspace][]=tag_names&fields[workspace][]=assessments_enabled&fields[workspace][]=current_assessment_result&fields[workspace][]=permissions&fields[workspace][]=execution_mode&fields[workspace][]=organization&fields[run][]=status&fields[assessment-result][]=drifted&fields[assessment-result][]=succeeded&fields[assessment-result][]=all_checks_succeeded&filter[current-run][status]=errored&include=current_run,current_assessment_result,organization&organization_name=xxx&page[number]=1&page[size]=20&search[name]=dns-&sort=name

I tried this locally, and it works fine with a simple patch, but I'm not familiar with python tests.

    def list(self, page=None, page_size=None, include=None, search=None, filters=None):
        """
        ``GET /organizations/:organization_name/workspaces``

        `Workspaces List API Doc Reference \
            <https://www.terraform.io/docs/cloud/api/workspaces.html#list-workspaces>`_

        `Query Parameter(s) Details \
            <https://www.terraform.io/docs/cloud/api/workspaces.html#query-parameters>`__
        """
        return self._list(self._org_api_v2_base_url, \
            page=page, page_size=page_size, include=include, search=search, filters=filters)

When will a new release come out with filters functionality for teams and workspaces?

Do you know when you plan to release a new version of terrasnek on PyPy? We have some python code that uses the teams.list_all which currently takes ~65 seconds to return and workspaces.list_all which currently takes ~90 seconds to return because we can't include a filter in the API call with the 0.1.10 release.

I would like to use filters that was added to teams.py in d6b8a9c and workspaces.py in 9d6bd20 but those changes have not been released yet.

Thanks!
Paul

How to catch error from _post method ?

Hello,

Some functions like "runs.apply" use the "_post" function of TFCEndpoint.

When an error occur during the post (like a bad HTTP response code), the error is logged, but nothing is returned, nor exception raised by the _post function.

So, is there a way for the main calling function to know that something goes wrong during the terrasnek function ? (in the two case the function only return 'None')

Example : If i do a terrasnek "apply.run" function call, how do i know if the call was a success or a fail ?

Thanks !

Thank you

I just wanted to say thank you for an excellent Python library. It's helping me a lot, and I'm definitely still a beginner so I appreciate the straightforward documentation and helpful code.

Thank you!

Handle non-json error responses

I'm trying to list terraform states, and I'm getting back a 504 response: gateway timeout.

The _get method in endpoint.py falls through to its final else clause, where it should raise a TFCHTTPUnclassified exception.

But it fails before that when it tries to parse the json in the response, because whatever front-end we're using returned a message in HTML.

So all I see in the logs is a json.decoder.JSONDecodeError "Expecting value: line 1 column 1 (char 0)" error.

It would be helpful to catch that exception and do something like log the raw req.content, and possibly the status code (I couldn't tell what was failing until I stepped through it with a debugger).

I can submit a merge request, if that would be helpful. Honestly, this seems like something that should apply to all the end-points.

str include param get exploded by join

It would be nice to be able to send a single string for includes

Trying tf.teams.list(include='users') will turn 'users' into [ 'u', 's', 'e', 'r', 's'] and join that into a path query.

https://github.com/dahlke/terrasnek/blob/master/terrasnek/endpoint.py

if include is not None:
joined_include = ",".join(include)
q_options.append(f"include={joined_include}"))

Support for 429 API rate limiting exception handling

Hey! Love the utility, been using it happily...

I have a very large TFE org in terms of workspaces (several hundred+) to maintain and we've been the hammering the API. While I can increase it on our private instance, it would be not ideal for us when using cloud. (https://www.terraform.io/docs/cloud/api/index.html#rate-limiting)

Right now, I am catching these parsing TFCHTTPUnclassified, but looking at the other known HTTP response type handling, having a specific exception for this this would allow me to more easily catch and sleep when hitting the limit.

Proposing something like this: paullschock@dbba341

Happy to make a PR with commit, but wanted to check with issue first.

Query Parameters coverage for Workspaces

Is it possible to extend the Workspaces List endpoint to support query parameters as per the API documentation.

It appears to be listed in the documentation/comments, but no obvious way to pass the query as the only supported inputs are: page, page_size & include?

I would like to be able to pass through a list like:

query = {"search[\"name\"]":"my-awesome-prefix"}
workspaces = tfc_api.workspaces.list_all(query=query)

I would imagine the implementation would be similar to that for filters?

runs not filtering based on the commit Id

List API doesn't support search based on the commit id because of the following code in endpoint.py file'

        if search is not None:
            if "name" in search:
                q_options.append(f"search[name]={search['name']}")

            if "tags" in search:
                q_options.append(f"search[tags]={search['tags']}")

It should be modified as

        if search is not None:
            if "name" in search:
                q_options.append(f"search[name]={search['name']}")

            if "tags" in search:
                q_options.append(f"search[tags]={search['tags']}")
            
            if "commit" in search:
                q_options.append(f"search[commit]={search['commit']}")

api.vars error |

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I'm reporting the issue to the correct repository (for multi-repository projects)

When running
api.vars
The following error is returned
AttributeError: 'TFC' object has no attribute 'vars'

After

PEP8 and related formatting issues

IDEs such as Jetbrains Pycharm and VS Code flag numerous formatting violations in various places. The usual gubbins like trailing white space, inconsistent use of line continuation markers, and lack of trailing white space on end of file.

A possible solution would be to reformat the code base in black providing a consistent, uncompromising style. Black has the distinct benefit of absolving projects of any current or future formatting discussions. But some of its choices are controversial.

Possible acceptance criteria would be: code base is reformatted / PEP8 errors are all resolved and a CI step is added to enforce formatting / prevent formatting regressions.

Payload to assign varsets to workspaces and projects is wrong

When I tried to use these endpoints I see that it returns success but the assignment is never made. After debugging I found out the payload that terrasnek uses is:
"<workspace-id>"
or
"<project-id>"

Expected payload, verified using Postman:

{
  "data": [
    {
      "type": "workspaces",
      "id": "<workspace-id>"
    }
  ]
}

or

{
  "data": [
    {
      "type": "projects",
      "id": "<project-id>"
    }
  ]
}

You can see this in the documentation: https://developer.hashicorp.com/terraform/cloud-docs/api-docs/variable-sets#apply-variable-set-to-workspaces
and
https://developer.hashicorp.com/terraform/cloud-docs/api-docs/variable-sets#apply-variable-set-to-projects

If you use the current payload it fails silently returning 204, but it is a no-op.

return objects and types?

Hello. I wasn't sure where to ask so please do close this issue if it does not belong here.

Is it a project goal to at some point offer more "client side" support for validating input, and/or providing type hinted return objects?

Search in workspace list_all function is not working for version >=0.1.11

Hello,

The search in workspace list_all function is not working for version >=0.1.11; it ignores the search filter and always returns all workspaces within the organization. Search is working as excepted for version 0.1.10
https://terrasnek.readthedocs.io/en/latest/workspaces.html#terrasnek.workspaces.TFCWorkspaces.list_all

python version: 3.7.5

from terrasnek.api import TFC

TFC_TOKEN="<Token>"
tfc_api = TFC(TFC_TOKEN, "https://app.terraform.io", verify=False)
tfc_api.set_org("<organization>")

all_ws = tfc_api.workspaces.list_all(search='test')["data"]

print(all_ws)

Thanks.

api.vars error | SOLVED with api.variables

• I am running the latest version
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I'm reporting the issue to the correct repository (for multi-repository projects)

When running
api.vars
The following error is returned
AttributeError: 'TFC' object has no attribute 'vars'

After a deep dive into the api.py file I found variables
The following code might be funky endpoint replace

The following is working fine for me, but this should be corrected or a new convention should be chosen
api.variables.create(create_variable_payload)

handle query strings via requests() parameters

In _get() the query string is constructed via string concatenation. Can we replace with requests params dicts instead? I believe this is proper prima facie but below I've shared my reasons for future benefits as well.

To replace the custom string concatenation for params with regular requests parameters would bring benefits:

• As an immediate benefit, the ordering of the string concatenation would no longer be significant, requests will process dict-like objects into parameters correctly with no additional processing needed, eliminating string processing and concatenation logic and reducing surface area for needed unit tests.
• Converting to use dict-like objects as requests params will give us the ability to in the future create possibly type hinted objects or classes which will be processed to_dict() such as include, page, sort, and relations, and perform client-side validation of API parameters before calling _get(). The benefits of bringing client-side validation to parameters are discussed in #34
• API-specific _get() options like found here can move to their own objects to take advantage of the mentioned type hinting / input parameter validation, in the future.

An example of how this would work can be found in this commit.

Request: Support `include` query parameter on state version GET requests

In the State Version API Docs, at the very bottom there is an include query parameter that allows retrieving state version(s) and parsed outputs in a single request, but I don't see that terrasnek supports this yet for State Versions.

For example, something like:

• api.state_versions.list(filters, include='outputs')
• api.state_versions.list_all(filters, include='outputs')
• api.state_versions.show(sv_id, include='outputs')

Let me know if I've missed something. Thanks!

logic bug in _list_all

I think there is a logic bug in the code at

terrasnek/terrasnek/endpoint.py

Lines 366 to 387 in e1c4fbe

	current_page_number = 1
	list_resp = \
	self._list(url, page=current_page_number, page_size=MAX_PAGE_SIZE, include=include, \
	search=search, filters=filters, query=query)
	if "meta" in list_resp:
	total_pages = list_resp["meta"]["pagination"]["total-pages"]
	else:
	total_pages = list_resp["pagination"]["total_pages"]
	included = []
	data = []
	while current_page_number <= total_pages:
	list_resp = \
	self._list(url, page=current_page_number, page_size=MAX_PAGE_SIZE, \
	include=include, search=search, filters=filters, query=query)
	data += list_resp["data"]
	if "included" in list_resp:
	included += list_resp["included"]
	current_page_number += 1

current_page = 1 continues to be true during the while loop and ends up being retrieved twice before being incremented to page 2.

Source: setting up mocks for the workspaces.list_all() and the mock was called 3 times for 2 pages.

RUNS class : list Method not have "Filters" enabled

TFCRuns.list_all method missing filters options .

not consider it an issue, but a missing part I needed to finalize my script

<runs.py>

    def list_all(self, workspace_id, include=None, filters=None):
        """
        This function does not correlate to an endpoint in the TFC API Docs specifically,
        but rather is a helper function to wrap the `list` endpoint, which enumerates out
        every page so users do not have to implement the paging logic every time they just
        want to list every run for a workspace.

        Returns an object with two arrays of objects.
        """
        url = f"{self._ws_api_v2_base_url}/{workspace_id}/runs"
        return self._list_all(url, include=include, filters=filters)