stelligent / sagemaker-pipeline Goto Github PK

View Code? Open in Web Editor NEW

58.0 68.0 71.0 1.12 MB

Sagemaker pipeline for AWS Summit New York

License: MIT License

Python 96.65% Shell 3.35%

sagemaker codepipeline machine-learning devops

sagemaker-pipeline's People

Contributors

Stargazers

Watchers

sagemaker-pipeline's Issues

Create visualization of machine learning output

describe components (CodePipeline, S3, Sagemaker, CodeBuild, GitHub, CloudFormation, etc.), Launch Stack button, describe the model and how it all works, step-by-step usage (e.g. launch CFN infrastructure - describe params, launch CFN stack for CodePipeline - describe params.

Record demo of how Amazon Sagemaker works through the console

Decide if we want to launch a new endpoint with each pipeline run or to update existing endpoint

Change the Name of Pipeline Template

Fix CodeBuild Naming Conflicts

Minor - make sure to name the cloud-formation stack something other than sagemaker-pipeline

If you name your stack - sagemaker-pipeline, the stack creation will fail because of S3 bucket name clash with the original author's buckets.

Change Name of 'TemplateSource' Artificat to 'src'

Record demo of annotated CodePipeline workflow

Describe each steps in time elapsed format

Visualization of prediction data (e.g. QuickSight, etc.)

Update/Replace Endpoint with Stack Update

[Article request] Chalice with TF Serving payload format

Hi, I've followed along with your Medium article successfully. I've also done this sagemaker tutorial: tensorflow_bring_your_own/tensorflow_bring_your_own.ipynb. The format required to send an image to the TF Serving RESTful endpoint in this tutorial is:

import json
import numpy as np
from PIL import Image
img_path = './img.jpg'
image = np.asarray(Image.open(img_path)).astype(np.float32)
image = np.expand_dims(image, axis=0)
data = {'instances': image}
data = json.dumps({k: _ndarray_to_list(v) for k, v in six.iteritems(data)}) # or sagemaker.predictor.json_serializer
request_args = {}
request_args['Body'] = data
request_args['EndpointName'] = 'my-endpoint-name'
request_args['ContentType'] = 'application/json'
request_args['Accept'] = 'application/json'
response = sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)

I'm struggling to figure out how I can send either a jpg or image as numpy array to Chalice in order to preprocess the image into this format required for TF serving. Any chance you might be able to help?

Remove Quotes from PipelineRole, CFNRole, and CodeBuild Role Permissions

README updates

Update the formatting (particularly for the CFN parameters). See the https://github.com/stelligent/dromedary/blob/master/README.md I'd shared before
Add a step in the CFN template in which it publishes the pipeline.yaml to a public S3 bucket (in the Stelligent AWS "Prod" account). Here's an example of some code that will do that: https://github.com/stelligent/devops-essentials/blob/master/samples/static/pipeline.yml#LL143-LL169
Add a Launch Stack button (the link goes to the template hosted in S3 - see step 2)
Show the CLI command execution of the CFN template (the --template-url refers to the template hosted in S3 - see step 2)
Create a diagram that shows the overall architecture (see the attached). Should include CloudFormation, CodePipeline, CodeBuild, Git, S3, Sagemaker, etc. Might consider using http://cloudcraft.co/ for this

numpy installation failed in TestEndpoint

This error occurs in the TestEndpoint stage:

[Container] 2020/06/12 12:03:09 Running command pip3 install boto3
--
31 | Collecting boto3
32 | Downloading https://files.pythonhosted.org/packages/8f/47/fd52106b41769acac53dfe1923bb363d6e4cc583a3a951c54a85a415593f/boto3-1.14.1-py2.py3-none-any.whl (128kB)
33 | Collecting botocore<1.18.0,>=1.17.1 (from boto3)
34 | Downloading https://files.pythonhosted.org/packages/a5/72/3143262c07466d01a475237d2578e5b81879a2cc6830872fc7c09953ec59/botocore-1.17.1.tar.gz (6.5MB)
35 | Collecting jmespath<1.0.0,>=0.7.1 (from boto3)
36 | Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl
37 | Collecting s3transfer<0.4.0,>=0.3.0 (from boto3)
38 | Downloading https://files.pythonhosted.org/packages/69/79/e6afb3d8b0b4e96cefbdc690f741d7dd24547ff1f94240c997a26fa908d3/s3transfer-0.3.3-py2.py3-none-any.whl (69kB)
39 | Collecting docutils<0.16,>=0.10 (from botocore<1.18.0,>=1.17.1->boto3)
40 | Downloading https://files.pythonhosted.org/packages/22/cd/a6aa959dca619918ccb55023b4cb151949c64d4d5d55b3f4ffd7eee0c6e8/docutils-0.15.2-py3-none-any.whl (547kB)
41 | Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.18.0,>=1.17.1->boto3)
42 | Downloading https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227kB)
43 | Collecting urllib3<1.25.8,>=1.20 (from botocore<1.18.0,>=1.17.1->boto3)
44 | Downloading https://files.pythonhosted.org/packages/b4/40/a9837291310ee1ccc242ceb6ebfd9eb21539649f193a7c8c86ba15b98539/urllib3-1.25.7-py2.py3-none-any.whl (125kB)
45 | Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.18.0,>=1.17.1->boto3)
46 | Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
47 | Building wheels for collected packages: botocore
48 | Running setup.py bdist_wheel for botocore
49 | Stored in directory: /root/.cache/pip/wheels/8e/e6/8b/dfb196ecf69fe9e4af7e912923df94cad2e58f31ace096a617
50 | Successfully built botocore
51 | Installing collected packages: jmespath, docutils, six, python-dateutil, urllib3, botocore, s3transfer, boto3
52 | Successfully installed boto3-1.14.1 botocore-1.17.1 docutils-0.15.2 jmespath-0.10.0 python-dateutil-2.8.1 s3transfer-0.3.3 six-1.15.0 urllib3-1.25.7
53 | You are using pip version 7.1.2, however version 20.1.1 is available.
54 | You should consider upgrading via the 'pip install --upgrade pip' command.
55 |  
56 | [Container] 2020/06/12 12:03:16 Running command pip3 install numpy
57 | Collecting numpy
58 | Downloading https://files.pythonhosted.org/packages/01/1b/d3ddcabd5817be02df0e6ee20d64f77ff6d0d97f83b77f65e98c8a651981/numpy-1.18.5.zip (5.4MB)
59 | Complete output from command python setup.py egg_info:
60 | Traceback (most recent call last):
61 | File "<string>", line 20, in <module>
62 | File "/tmp/pip-build-2wct20o0/numpy/setup.py", line 32, in <module>
63 | raise RuntimeError("Python version >= 3.5 required.")
64 | RuntimeError: Python version >= 3.5 required.
65 |  
66 | ----------------------------------------
67 | Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-2wct20o0/numpy
68 | You are using pip version 7.1.2, however version 20.1.1 is available.
69 | You should consider upgrading via the 'pip install --upgrade pip' command.
70 |  
71 | [Container] 2020/06/12 12:03:18 Command did not exit successfully pip3 install numpy exit status 1
72 | [Container] 2020/06/12 12:03:18 Phase complete: INSTALL State: FAILED

Add Stack Assumptions to README

The configuration for this demo assumes that:

Your pipeline stack name is less than 20 characters long
You are launching the pipeline in US East (N. Virginia)

QA Launch endpoint error

Getting the below error:

Action execution failed
Parameter 'SageMakerRole' must contain at most 64 characters (Service: AmazonCloudFormation; Status Code: 400; Error Code: ValidationError; Request ID: c9d7f5dc-9950-42d9-b755-3585182c2332)

Please help

stelligent / sagemaker-pipeline Goto Github PK

sagemaker-pipeline's People

Contributors

Stargazers

Watchers

Forkers

sagemaker-pipeline's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs