stelligent / sagemaker-pipeline Goto Github PK
View Code? Open in Web Editor NEWSagemaker pipeline for AWS Summit New York
License: MIT License
Sagemaker pipeline for AWS Summit New York
License: MIT License
describe components (CodePipeline, S3, Sagemaker, CodeBuild, GitHub, CloudFormation, etc.), Launch Stack button, describe the model and how it all works, step-by-step usage (e.g. launch CFN infrastructure - describe params, launch CFN stack for CodePipeline - describe params.
If you name your stack - sagemaker-pipeline, the stack creation will fail because of S3 bucket name clash with the original author's buckets.
Describe each steps in time elapsed format
Hi, I've followed along with your Medium article successfully. I've also done this sagemaker tutorial: tensorflow_bring_your_own/tensorflow_bring_your_own.ipynb. The format required to send an image to the TF Serving RESTful endpoint in this tutorial is:
import json
import numpy as np
from PIL import Image
img_path = './img.jpg'
image = np.asarray(Image.open(img_path)).astype(np.float32)
image = np.expand_dims(image, axis=0)
data = {'instances': image}
data = json.dumps({k: _ndarray_to_list(v) for k, v in six.iteritems(data)}) # or sagemaker.predictor.json_serializer
request_args = {}
request_args['Body'] = data
request_args['EndpointName'] = 'my-endpoint-name'
request_args['ContentType'] = 'application/json'
request_args['Accept'] = 'application/json'
response = sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
I'm struggling to figure out how I can send either a jpg or image as numpy array to Chalice in order to preprocess the image into this format required for TF serving. Any chance you might be able to help?
pipeline.yaml
to a public S3 bucket (in the Stelligent AWS "Prod" account). Here's an example of some code that will do that: https://github.com/stelligent/devops-essentials/blob/master/samples/static/pipeline.yml#LL143-LL169--template-url
refers to the template hosted in S3 - see step 2)This error occurs in the TestEndpoint stage:
[Container] 2020/06/12 12:03:09 Running command pip3 install boto3
--
31 | Collecting boto3
32 | Downloading https://files.pythonhosted.org/packages/8f/47/fd52106b41769acac53dfe1923bb363d6e4cc583a3a951c54a85a415593f/boto3-1.14.1-py2.py3-none-any.whl (128kB)
33 | Collecting botocore<1.18.0,>=1.17.1 (from boto3)
34 | Downloading https://files.pythonhosted.org/packages/a5/72/3143262c07466d01a475237d2578e5b81879a2cc6830872fc7c09953ec59/botocore-1.17.1.tar.gz (6.5MB)
35 | Collecting jmespath<1.0.0,>=0.7.1 (from boto3)
36 | Downloading https://files.pythonhosted.org/packages/07/cb/5f001272b6faeb23c1c9e0acc04d48eaaf5c862c17709d20e3469c6e0139/jmespath-0.10.0-py2.py3-none-any.whl
37 | Collecting s3transfer<0.4.0,>=0.3.0 (from boto3)
38 | Downloading https://files.pythonhosted.org/packages/69/79/e6afb3d8b0b4e96cefbdc690f741d7dd24547ff1f94240c997a26fa908d3/s3transfer-0.3.3-py2.py3-none-any.whl (69kB)
39 | Collecting docutils<0.16,>=0.10 (from botocore<1.18.0,>=1.17.1->boto3)
40 | Downloading https://files.pythonhosted.org/packages/22/cd/a6aa959dca619918ccb55023b4cb151949c64d4d5d55b3f4ffd7eee0c6e8/docutils-0.15.2-py3-none-any.whl (547kB)
41 | Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.18.0,>=1.17.1->boto3)
42 | Downloading https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl (227kB)
43 | Collecting urllib3<1.25.8,>=1.20 (from botocore<1.18.0,>=1.17.1->boto3)
44 | Downloading https://files.pythonhosted.org/packages/b4/40/a9837291310ee1ccc242ceb6ebfd9eb21539649f193a7c8c86ba15b98539/urllib3-1.25.7-py2.py3-none-any.whl (125kB)
45 | Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.18.0,>=1.17.1->boto3)
46 | Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
47 | Building wheels for collected packages: botocore
48 | Running setup.py bdist_wheel for botocore
49 | Stored in directory: /root/.cache/pip/wheels/8e/e6/8b/dfb196ecf69fe9e4af7e912923df94cad2e58f31ace096a617
50 | Successfully built botocore
51 | Installing collected packages: jmespath, docutils, six, python-dateutil, urllib3, botocore, s3transfer, boto3
52 | Successfully installed boto3-1.14.1 botocore-1.17.1 docutils-0.15.2 jmespath-0.10.0 python-dateutil-2.8.1 s3transfer-0.3.3 six-1.15.0 urllib3-1.25.7
53 | You are using pip version 7.1.2, however version 20.1.1 is available.
54 | You should consider upgrading via the 'pip install --upgrade pip' command.
55 |
56 | [Container] 2020/06/12 12:03:16 Running command pip3 install numpy
57 | Collecting numpy
58 | Downloading https://files.pythonhosted.org/packages/01/1b/d3ddcabd5817be02df0e6ee20d64f77ff6d0d97f83b77f65e98c8a651981/numpy-1.18.5.zip (5.4MB)
59 | Complete output from command python setup.py egg_info:
60 | Traceback (most recent call last):
61 | File "<string>", line 20, in <module>
62 | File "/tmp/pip-build-2wct20o0/numpy/setup.py", line 32, in <module>
63 | raise RuntimeError("Python version >= 3.5 required.")
64 | RuntimeError: Python version >= 3.5 required.
65 |
66 | ----------------------------------------
67 | Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-2wct20o0/numpy
68 | You are using pip version 7.1.2, however version 20.1.1 is available.
69 | You should consider upgrading via the 'pip install --upgrade pip' command.
70 |
71 | [Container] 2020/06/12 12:03:18 Command did not exit successfully pip3 install numpy exit status 1
72 | [Container] 2020/06/12 12:03:18 Phase complete: INSTALL State: FAILED
The configuration for this demo assumes that:
Getting the below error:
Action execution failed
Parameter 'SageMakerRole' must contain at most 64 characters (Service: AmazonCloudFormation; Status Code: 400; Error Code: ValidationError; Request ID: c9d7f5dc-9950-42d9-b755-3585182c2332)
Please help
Add some configuration in pipeline.yaml
in which it publishes the pipeline.yaml
to a public S3 bucket (in the Stelligent AWS "Prod" account). This way we can do an embedded link to the Launch Stack button described in the README.
Here's an example of some code that will do that: https://github.com/stelligent/devops-essentials/blob/master/samples/static/pipeline.yml#LL143-LL169
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.