nrdg / cloudknot Goto Github PK

View Code? Open in Web Editor NEW

70.0 70.0 17.0 3.97 MB

A python library to run your existing code on AWS Batch

Home Page: https://nrdg.github.io/cloudknot/

License: Other

Makefile 0.38% Python 98.08% Dockerfile 0.71% Shell 0.84%

cloudknot's People

Contributors

Stargazers

Watchers

Forkers

arokem dtenenba ccurtis7 afcarl roncard tokozedg ltirrell dash2927 dav009 zirkonium88 strossi22 36000 richford sumants-dev maouw

cloudknot's Issues

Accept dockerImage object input in Knot.init()

In IamRole, allow user to specify name or arn

Log some logging.info() stuff on cloudknot import

Example notebook: no region error

When trying to run cell number 8 in that notebook, this line:

ecr = boto3.client('ecr')

Prints:

DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable config_file from defaults.
DEBUG:botocore.session:Loading variable credentials_file from defaults.
DEBUG:botocore.session:Loading variable data_path from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable region from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable ca_bundle from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable api_versions from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable credentials_file from defaults.
DEBUG:botocore.session:Loading variable config_file from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable metadata_service_timeout from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.session:Loading variable metadata_service_num_attempts from defaults.
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.credentials:Looking for credentials via: env
DEBUG:botocore.credentials:Looking for credentials via: assume-role
DEBUG:botocore.credentials:Looking for credentials via: shared-credentials-file
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
DEBUG:botocore.loaders:Loading JSON file: /Users/arokem/anaconda3/lib/python3.6/site-packages/botocore/data/endpoints.json
DEBUG:botocore.session:Loading variable profile from defaults.
DEBUG:botocore.loaders:Loading JSON file: /Users/arokem/anaconda3/lib/python3.6/site-packages/botocore/data/ecr/2015-09-21/service-2.json
DEBUG:botocore.hooks:Event creating-client-class.ecr: calling handler <function add_generate_presigned_url at 0x10bc15598>

And then raises:

---------------------------------------------------------------------------
NoRegionError                             Traceback (most recent call last)
<ipython-input-8-f2213056781e> in <module>()
----> 1 ecr = boto3.client('ecr')
      2 # Use a unique name so as not to conflict with any of your pre-existing repositories
      3 response = ecr.create_repository(repositoryName=str(uuid.uuid4()))
      4 # Get the new repositories name, URI, and registry ID
      5 repo_name = response['repository']['repositoryName']

~/anaconda3/lib/python3.6/site-packages/boto3/__init__.py in client(*args, **kwargs)
     81     See :py:meth:`boto3.session.Session.client`.
     82     """
---> 83     return _get_default_session().client(*args, **kwargs)
     84 
     85 

~/anaconda3/lib/python3.6/site-packages/boto3/session.py in client(self, service_name, region_name, api_version, use_ssl, verify, endpoint_url, aws_access_key_id, aws_secret_access_key, aws_session_token, config)
    261             aws_access_key_id=aws_access_key_id,
    262             aws_secret_access_key=aws_secret_access_key,
--> 263             aws_session_token=aws_session_token, config=config)
    264 
    265     def resource(self, service_name, region_name=None, api_version=None,

~/anaconda3/lib/python3.6/site-packages/botocore/session.py in create_client(self, service_name, region_name, api_version, use_ssl, verify, endpoint_url, aws_access_key_id, aws_secret_access_key, aws_session_token, config)
    859             is_secure=use_ssl, endpoint_url=endpoint_url, verify=verify,
    860             credentials=credentials, scoped_config=self.get_scoped_config(),
--> 861             client_config=config, api_version=api_version)
    862         return client
    863 

~/anaconda3/lib/python3.6/site-packages/botocore/client.py in create_client(self, service_name, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, api_version, client_config)
     69         client_args = self._get_client_args(
     70             service_model, region_name, is_secure, endpoint_url,
---> 71             verify, credentials, scoped_config, client_config, endpoint_bridge)
     72         service_client = cls(**client_args)
     73         self._register_retries(service_client)

~/anaconda3/lib/python3.6/site-packages/botocore/client.py in _get_client_args(self, service_model, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, client_config, endpoint_bridge)
    281         return args_creator.get_client_args(
    282             service_model, region_name, is_secure, endpoint_url,
--> 283             verify, credentials, scoped_config, client_config, endpoint_bridge)
    284 
    285     def _create_methods(self, service_model):

~/anaconda3/lib/python3.6/site-packages/botocore/args.py in get_client_args(self, service_model, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, client_config, endpoint_bridge)
     43         final_args = self.compute_client_args(
     44             service_model, client_config, endpoint_bridge, region_name,
---> 45             endpoint_url, is_secure, scoped_config)
     46 
     47         service_name = final_args['service_name']

~/anaconda3/lib/python3.6/site-packages/botocore/args.py in compute_client_args(self, service_model, client_config, endpoint_bridge, region_name, endpoint_url, is_secure, scoped_config)
    109 
    110         endpoint_config = endpoint_bridge.resolve(
--> 111             service_name, region_name, endpoint_url, is_secure)
    112 
    113         # Override the user agent if specified in the client config.

~/anaconda3/lib/python3.6/site-packages/botocore/client.py in resolve(self, service_name, region_name, endpoint_url, is_secure)
    354         region_name = self._check_default_region(service_name, region_name)
    355         resolved = self.endpoint_resolver.construct_endpoint(
--> 356             service_name, region_name)
    357         if resolved:
    358             return self._create_endpoint(

~/anaconda3/lib/python3.6/site-packages/botocore/regions.py in construct_endpoint(self, service_name, region_name)
    120         for partition in self._endpoint_data['partitions']:
    121             result = self._endpoint_for_partition(
--> 122                 partition, service_name, region_name)
    123             if result:
    124                 return result

~/anaconda3/lib/python3.6/site-packages/botocore/regions.py in _endpoint_for_partition(self, partition, service_name, region_name)
    133                 region_name = service_data['partitionEndpoint']
    134             else:
--> 135                 raise NoRegionError()
    136         # Attempt to resolve the exact region for this partition.
    137         if region_name in service_data['endpoints']:

NoRegionError: You must specify a region.

Add docstrings to everything that should have a docstring

Unit tests leave behind orphaned compute environments

I think this is because the compute environment's batch service role is being deleted before the compute environment is completely deleted. I've been able to fix this manually using the AWS console by associating the compute environment with a new batch service role and then deleting it.

What needs to be done:

Write a function that automates the above process to clean up any orphaned compute environments.
Fix the actual root cause. Maybe in IamRole.clobber(), if the role is a batch service role, find any associated compute environments first. If their status is DELETING, just wait, if not, throw an error and tell user that there's a dependency issue.

Write test_BatchJob in test_aws.py

Careful here. Don't submit a job or it will cost money.

In DockerImage(), allow user to specify remote docker image without building locally

Use **kwargs in init methods

Add module and submodule docstrings

Create pyAFQ HCP example

Integrate with cottoncandy for I/O

See: http://gallantlab.github.io/cottoncandy/

Make stuff async

Disable set_profile()

Until we resolve #59, just put a NotImplementedError in set_profile().

Testing takes too long

Make tests for dependent objects prereq fixtures for owner objects. e.g. test_Pars and test_DockerImage should just feed into test_Knot.

Add `clobbered` parameter to all resource classes

and prevent any methods from executing if self.clobbered is True.

Use moto for unit testing,

maybe?

Add `test_Pars()` to `test_cloudknot.py`

test_aws is leaving behind dependent objects

Make clobber methods idempotent

Right now, we can't clobber twice. That makes no sense. Fix it.

Add policies input to `Knot.init()`

Let the user specify policies on Knot.__init__() rather than making them build a custom PARS first.

Keep track of region in config file

Presently, the user could create a few VPCs, switch regions, create a few more, then be unable to retrieve the old region's VPCs.

Keep track of profile in config file

See issue #41 for comparison.

Add docker-containers section to config file

Allow user to set/get AWS account to use

Create Jars object in cloudknot.py, add to unit tests

Add ResourceExistsException and ResourceDoesNotExistException to ecr.py

Don't need to import duecredit, since we are not referring to papers

https://github.com/richford/cloudknot/blob/0672c3059eb1124c3f4de559cb62494019b33d03/cloudknot/cloudknot.py#L10

Also: https://github.com/richford/cloudknot/blob/master/cloudknot/cloudknot.py#L14-L20

Write test_DockerContainer() in test_aws.py

Add capability to write logs to file

using an environment variable or something like that.

Add example notebook

So that @arokem can play with this a bit on his machine.

In JobDefinition, if retrieving pre-existing inactive job def, activate it

Add to config.prune()

prune docker images
prune docker repos
prune pars
prune knots

Add docstrings to classes, not just init

Include "Raises" and "Methods" sections where applicable.

Test config.py in unit tests

Add clean_jobs() method to config.py

to remove config references to finished/non-running jobs.

Create KeyPair object

Create a key pair in ~/.ssh/cloudknot_keys
Import it to AWS (all regions)
Use paramiko to generate keypairs
Pass as input to ComputeEnvironment
Put something in logging.info() with info on how to ssh into running instances
Write some method that will output the ssh command for running jobs

Add `get_log_urls` method to `BatchJob`

Query 'logStreamName' from batch.describe_jobs

Return a dict of log stream urls for each attempt.

Publish to pypi, update version.py

Return function output to user

Put function output in a cloudpickle and send back to user (perhaps with intermediate storage in S3).

Write clobber method for DockerImage

Add more logging.info() statements

Testing errors

I am currently seeing the following errors when testing on my latpop:


D-173-250-189-251:cloudknot (elminiate-circular-import) $make test
py.test --pyargs cloudknot --cov-report term-missing --cov=cloudknot
Test session starts (platform: darwin, Python 3.5.2, pytest 3.0.6, pytest-sugar 0.8.0)
rootdir: /Users/arokem/source/cloudknot, inifile: 
plugins: sugar-0.8.0, django-3.1.2, cov-2.4.0, flaky-3.3.0


――――――――――――― ERROR at setup of test_wait_for_compute_environment ――――――――――――――

    @pytest.fixture(scope='module')
    def pars():
>       p = ck.Pars(name='unit-test')

cloudknot/tests/test_aws.py:38: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cloudknot/cloudknot.py:232: in __init__
    self._vpc = aws.ec2.Vpc(name=vpc_name)
cloudknot/aws/ec2.py:96: in __init__
    self._vpc_id = self._create()
cloudknot/aws/ec2.py:243: in _create
    'Value': self.name
../../anaconda3/lib/python3.5/site-packages/botocore/client.py:253: in _api_call
    return self._make_api_call(operation_name, kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <botocore.client.EC2 object at 0x106fbf588>
operation_name = 'CreateTags'
api_params = {'Resources': ['vpc-7a9f7102'], 'Tags': [{'Key': 'owner', 'Value': 'cloudknot'}, {'Key': 'Name', 'Value': 'unit-test-cloudknot-vpc'}]}

    def _make_api_call(self, operation_name, api_params):
        operation_model = self._service_model.operation_model(operation_name)
        request_context = {
            'client_region': self.meta.region_name,
            'client_config': self.meta.config,
            'has_streaming_input': operation_model.has_streaming_input,
            'auth_type': operation_model.auth_type,
        }
        request_dict = self._convert_to_request_dict(
            api_params, operation_model, context=request_context)
    
        handler, event_response = self.meta.events.emit_until_response(
            'before-call.{endpoint_prefix}.{operation_name}'.format(
                endpoint_prefix=self._service_model.endpoint_prefix,
                operation_name=operation_name),
            model=operation_model, params=request_dict,
            request_signer=self._request_signer, context=request_context)
    
        if event_response is not None:
            http, parsed_response = event_response
        else:
            http, parsed_response = self._endpoint.make_request(
                operation_model, request_dict)
    
        self.meta.events.emit(
            'after-call.{endpoint_prefix}.{operation_name}'.format(
                endpoint_prefix=self._service_model.endpoint_prefix,
                operation_name=operation_name),
            http_response=http, parsed=parsed_response,
            model=operation_model, context=request_context
        )
    
        if http.status_code >= 300:
            error_code = parsed_response.get("Error", {}).get("Code")
            error_class = self.exceptions.from_code(error_code)
>           raise error_class(parsed_response, operation_name)
E           botocore.exceptions.ClientError: An error occurred (InvalidVpcID.NotFound) when calling the CreateTags operation: The vpc ID 'vpc-7a9f7102' does not exist

../../anaconda3/lib/python3.5/site-packages/botocore/client.py:557: ClientError
                                                                   8% ▉         

―――――――――――――――――― ERROR at setup of test_wait_for_job_queue ―――――――――――――――――――

    @pytest.fixture(scope='module')
    def pars():
>       p = ck.Pars(name='unit-test')

cloudknot/tests/test_aws.py:38: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cloudknot/cloudknot.py:232: in __init__
    self._vpc = aws.ec2.Vpc(name=vpc_name)
cloudknot/aws/ec2.py:96: in __init__
    self._vpc_id = self._create()
cloudknot/aws/ec2.py:243: in _create
    'Value': self.name
../../anaconda3/lib/python3.5/site-packages/botocore/client.py:253: in _api_call
    return self._make_api_call(operation_name, kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <botocore.client.EC2 object at 0x106fbf588>
operation_name = 'CreateTags'
api_params = {'Resources': ['vpc-7a9f7102'], 'Tags': [{'Key': 'owner', 'Value': 'cloudknot'}, {'Key': 'Name', 'Value': 'unit-test-cloudknot-vpc'}]}

    def _make_api_call(self, operation_name, api_params):
        operation_model = self._service_model.operation_model(operation_name)
        request_context = {
            'client_region': self.meta.region_name,
            'client_config': self.meta.config,
            'has_streaming_input': operation_model.has_streaming_input,
            'auth_type': operation_model.auth_type,
        }
        request_dict = self._convert_to_request_dict(
            api_params, operation_model, context=request_context)
    
        handler, event_response = self.meta.events.emit_until_response(
            'before-call.{endpoint_prefix}.{operation_name}'.format(
                endpoint_prefix=self._service_model.endpoint_prefix,
                operation_name=operation_name),
            model=operation_model, params=request_dict,
            request_signer=self._request_signer, context=request_context)
    
        if event_response is not None:
            http, parsed_response = event_response
        else:
            http, parsed_response = self._endpoint.make_request(
                operation_model, request_dict)
    
        self.meta.events.emit(
            'after-call.{endpoint_prefix}.{operation_name}'.format(
                endpoint_prefix=self._service_model.endpoint_prefix,
                operation_name=operation_name),
            http_response=http, parsed=parsed_response,
            model=operation_model, context=request_context
        )
    
        if http.status_code >= 300:
            error_code = parsed_response.get("Error", {}).get("Code")
            error_class = self.exceptions.from_code(error_code)
>           raise error_class(parsed_response, operation_name)
E           botocore.exceptions.ClientError: An error occurred (InvalidVpcID.NotFound) when calling the CreateTags operation: The vpc ID 'vpc-7a9f7102' does not exist

../../anaconda3/lib/python3.5/site-packages/botocore/client.py:557: ClientError
                                                                  17% █▋        
 cloudknot/tests/test_aws.py ✓✓                                   33% ███▍      


 cloudknot/tests/test_aws.py ✓✓✓✓                                 50% █████     

――――――――――――――――――――― ERROR at setup of test_JobDefinition ―――――――――――――――――――――

    @pytest.fixture(scope='module')
    def pars():
>       p = ck.Pars(name='unit-test')

cloudknot/tests/test_aws.py:38: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cloudknot/cloudknot.py:232: in __init__
    self._vpc = aws.ec2.Vpc(name=vpc_name)
cloudknot/aws/ec2.py:96: in __init__
    self._vpc_id = self._create()
cloudknot/aws/ec2.py:243: in _create
    'Value': self.name
../../anaconda3/lib/python3.5/site-packages/botocore/client.py:253: in _api_call
    return self._make_api_call(operation_name, kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <botocore.client.EC2 object at 0x106fbf588>
operation_name = 'CreateTags'
api_params = {'Resources': ['vpc-7a9f7102'], 'Tags': [{'Key': 'owner', 'Value': 'cloudknot'}, {'Key': 'Name', 'Value': 'unit-test-cloudknot-vpc'}]}

    def _make_api_call(self, operation_name, api_params):
        operation_model = self._service_model.operation_model(operation_name)
        request_context = {
            'client_region': self.meta.region_name,
            'client_config': self.meta.config,
            'has_streaming_input': operation_model.has_streaming_input,
            'auth_type': operation_model.auth_type,
        }
        request_dict = self._convert_to_request_dict(
            api_params, operation_model, context=request_context)
    
        handler, event_response = self.meta.events.emit_until_response(
            'before-call.{endpoint_prefix}.{operation_name}'.format(
                endpoint_prefix=self._service_model.endpoint_prefix,
                operation_name=operation_name),
            model=operation_model, params=request_dict,
            request_signer=self._request_signer, context=request_context)
    
        if event_response is not None:
            http, parsed_response = event_response
        else:
            http, parsed_response = self._endpoint.make_request(
                operation_model, request_dict)
    
        self.meta.events.emit(
            'after-call.{endpoint_prefix}.{operation_name}'.format(
                endpoint_prefix=self._service_model.endpoint_prefix,
                operation_name=operation_name),
            http_response=http, parsed=parsed_response,
            model=operation_model, context=request_context
        )
    
        if http.status_code >= 300:
            error_code = parsed_response.get("Error", {}).get("Code")
            error_class = self.exceptions.from_code(error_code)
>           raise error_class(parsed_response, operation_name)
E           botocore.exceptions.ClientError: An error occurred (InvalidVpcID.NotFound) when calling the CreateTags operation: The vpc ID 'vpc-7a9f7102' does not exist

../../anaconda3/lib/python3.5/site-packages/botocore/client.py:557: ClientError
 cloudknot/tests/test_aws.py ✓✓✓✓                                 58% █████▉    

―――――――――――――――――― ERROR at setup of test_ComputeEnvironment ―――――――――――――――――――

    @pytest.fixture(scope='module')
    def pars():
>       p = ck.Pars(name='unit-test')

cloudknot/tests/test_aws.py:38: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cloudknot/cloudknot.py:232: in __init__
    self._vpc = aws.ec2.Vpc(name=vpc_name)
cloudknot/aws/ec2.py:96: in __init__
    self._vpc_id = self._create()
cloudknot/aws/ec2.py:243: in _create
    'Value': self.name
../../anaconda3/lib/python3.5/site-packages/botocore/client.py:253: in _api_call
    return self._make_api_call(operation_name, kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <botocore.client.EC2 object at 0x106fbf588>
operation_name = 'CreateTags'
api_params = {'Resources': ['vpc-7a9f7102'], 'Tags': [{'Key': 'owner', 'Value': 'cloudknot'}, {'Key': 'Name', 'Value': 'unit-test-cloudknot-vpc'}]}

    def _make_api_call(self, operation_name, api_params):
        operation_model = self._service_model.operation_model(operation_name)
        request_context = {
            'client_region': self.meta.region_name,
            'client_config': self.meta.config,
            'has_streaming_input': operation_model.has_streaming_input,
            'auth_type': operation_model.auth_type,
        }
        request_dict = self._convert_to_request_dict(
            api_params, operation_model, context=request_context)
    
        handler, event_response = self.meta.events.emit_until_response(
            'before-call.{endpoint_prefix}.{operation_name}'.format(
                endpoint_prefix=self._service_model.endpoint_prefix,
                operation_name=operation_name),
            model=operation_model, params=request_dict,
            request_signer=self._request_signer, context=request_context)
    
        if event_response is not None:
            http, parsed_response = event_response
        else:
            http, parsed_response = self._endpoint.make_request(
                operation_model, request_dict)
    
        self.meta.events.emit(
            'after-call.{endpoint_prefix}.{operation_name}'.format(
                endpoint_prefix=self._service_model.endpoint_prefix,
                operation_name=operation_name),
            http_response=http, parsed=parsed_response,
            model=operation_model, context=request_context
        )
    
        if http.status_code >= 300:
            error_code = parsed_response.get("Error", {}).get("Code")
            error_class = self.exceptions.from_code(error_code)
>           raise error_class(parsed_response, operation_name)
E           botocore.exceptions.ClientError: An error occurred (InvalidVpcID.NotFound) when calling the CreateTags operation: The vpc ID 'vpc-7a9f7102' does not exist

../../anaconda3/lib/python3.5/site-packages/botocore/client.py:557: ClientError
 cloudknot/tests/test_aws.py ✓✓✓✓                                 67% ██████▋   

――――――――――――――――――――――― ERROR at setup of test_JobQueue ――――――――――――――――――――――――

    @pytest.fixture(scope='module')
    def pars():
>       p = ck.Pars(name='unit-test')

cloudknot/tests/test_aws.py:38: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cloudknot/cloudknot.py:232: in __init__
    self._vpc = aws.ec2.Vpc(name=vpc_name)
cloudknot/aws/ec2.py:96: in __init__
    self._vpc_id = self._create()
cloudknot/aws/ec2.py:243: in _create
    'Value': self.name
../../anaconda3/lib/python3.5/site-packages/botocore/client.py:253: in _api_call
    return self._make_api_call(operation_name, kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <botocore.client.EC2 object at 0x106fbf588>
operation_name = 'CreateTags'
api_params = {'Resources': ['vpc-7a9f7102'], 'Tags': [{'Key': 'owner', 'Value': 'cloudknot'}, {'Key': 'Name', 'Value': 'unit-test-cloudknot-vpc'}]}

    def _make_api_call(self, operation_name, api_params):
        operation_model = self._service_model.operation_model(operation_name)
        request_context = {
            'client_region': self.meta.region_name,
            'client_config': self.meta.config,
            'has_streaming_input': operation_model.has_streaming_input,
            'auth_type': operation_model.auth_type,
        }
        request_dict = self._convert_to_request_dict(
            api_params, operation_model, context=request_context)
    
        handler, event_response = self.meta.events.emit_until_response(
            'before-call.{endpoint_prefix}.{operation_name}'.format(
                endpoint_prefix=self._service_model.endpoint_prefix,
                operation_name=operation_name),
            model=operation_model, params=request_dict,
            request_signer=self._request_signer, context=request_context)
    
        if event_response is not None:
            http, parsed_response = event_response
        else:
            http, parsed_response = self._endpoint.make_request(
                operation_model, request_dict)
    
        self.meta.events.emit(
            'after-call.{endpoint_prefix}.{operation_name}'.format(
                endpoint_prefix=self._service_model.endpoint_prefix,
                operation_name=operation_name),
            http_response=http, parsed=parsed_response,
            model=operation_model, context=request_context
        )
    
        if http.status_code >= 300:
            error_code = parsed_response.get("Error", {}).get("Code")
            error_class = self.exceptions.from_code(error_code)
>           raise error_class(parsed_response, operation_name)
E           botocore.exceptions.ClientError: An error occurred (InvalidVpcID.NotFound) when calling the CreateTags operation: The vpc ID 'vpc-7a9f7102' does not exist

../../anaconda3/lib/python3.5/site-packages/botocore/client.py:557: ClientError
 cloudknot/tests/test_aws.py ✓✓✓✓✓✓                               92% █████████▎
 cloudknot/tests/test_cloudknot.py ✓                             100% ██████████

---------- coverage: platform darwin, python 3.5.2-final-0 -----------
Name                                Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------------------------------
cloudknot/__init__.py                   5      0      0      0   100%
cloudknot/aws/__init__.py               6      0      0      0   100%
cloudknot/aws/base_classes.py          71     24     14      1    56%   108, 112, 121-141, 148-164, 105->108
cloudknot/aws/batch.py                467    398    170      0    11%   61-151, 173-210, 220-248, 257-264, 345-589, 635-717, 727-766, 776-833, 859-943, 969-1002, 1012-1029, 1033-1046, 1056-1080, 1120-1178, 1207-1234, 1244-1266, 1279-1285, 1299-1304
cloudknot/aws/ec2.py                  212     14     64      3    92%   18-20, 267-269, 583, 587, 592, 611, 616-622, 586->587, 591->592, 610->611
cloudknot/aws/ecr.py                   82     61     33      0    18%   43-72, 83-104, 108-145, 149, 155-162, 174-184, 193
cloudknot/aws/iam.py                  125      0     49      0   100%
cloudknot/cloudknot.py                216    153     66      8    25%   26-30, 36-42, 48-51, 60, 67-68, 73-74, 83-158, 162-163, 179-180, 185-186, 200-201, 206-207, 221-222, 227-229, 234-280, 286-305, 311-334, 355-373, 379-397, 401-411, 420-477, 59->60, 66->67, 72->73, 81->83, 161->162, 184->185, 205->206, 226->227
cloudknot/tests/__init__.py             0      0      0      0   100%
cloudknot/tests/test_aws.py           571    325     46      0    43%   39-40, 53-64, 76-97, 759-938, 989-1373, 1484-1667
cloudknot/tests/test_cloudknot.py       3      0      0      0   100%
-------------------------------------------------------------------------------
TOTAL                                1758    975    442     12    42%

===Flaky Test Report===


===End Flaky Test Report===

Results (42.59s):
       7 passed
       5 error
make: *** [test] Error 1

Should I do something to configure my environment to run without error?

Wrap boto3 calls in exponential backoff retries

I think a lot of the intermittent build errors are due to eventual consistency.

To solve this, wrap the boto3 methods in exponential backoff error retries.

Rather than write our own wrapper, use tenacity.

Logically separate DockerImage and DockerRepo classes

Rename DockerReqs class to DockerImage.
Rename DockerImage class to DockerRepo.

Move build, tag, push methods to new DockerImage class, leaving only create_repo and clobber methods in DockerRepo.

Thus, DockerImage will create all the necessary files on init and control all docker behavior. DockerImage should accept a DockerRepo instance in it's push method and be able to push to that specific docker repo.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble