microsoft / agogosml Goto Github PK
View Code? Open in Web Editor NEWagogosml is a flexible data processing pipeline that addresses the common need for operationalizing ML models at scale
License: MIT License
agogosml is a flexible data processing pipeline that addresses the common need for operationalizing ML models at scale
License: MIT License
Does your issue follow our Code of Conduct?
Ensure you are up-to-date with master.
Search existing issues.
Part of the project with issues: (ie. Agogosml, Deployment, CLI)
Indicate OS, version of Python (Note: Only Python 3.7 is currently supported.)
Steps of reproduce issue. Provide a link to a Github Gist or paste here if short.
agogosml/agogosml/agogosml/common/flask_http_listener_client.py
Lines 37 to 39 in 5e60339
Sending unvalidated data to a web browser can result in the browser executing malicious code.
Refinement and improvement required testing of CI.
make dist
command as pipeline artifactsTAG |
---|
latest |
python3.5-1.0.1 |
python3.6-1.0.1 |
python3.7-1.0.1 |
during a run of make test-all
tests fails. see pyenv/pyenv-virtualenv#206 and tox-dev/tox-pyenv#4
flake8 runtests: commands[0] | flake8 cli
__________________________________________________________________________________________________________ summary __________________________________________________________________________________________________________
py35: commands succeeded
ERROR: py36: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError("Failed to get version_info for python3.6: pyenv: python3.6: command not found\n\nThe `python3.6' command exists in these Python versions:\n 3.6.8\n 3.6.8/envs/py36\n 3.6.8/envs/py368\n py36\n py368\n\n", None)
py37: commands succeeded
flake8: commands succeeded
(py35) cicorias@cicoria-msi:/c/g/cse/tiem/agogosml/agogosml_cli$
more verbose
GLOB finish: packaging after 0.24 seconds
copying new sdistfile to '/root/.tox/distshare/example-pkg-your-username-0.0.1.zip'
package .tmp/package/1/example-pkg-your-username-0.0.1.zip links to dist/example-pkg-your-username-0.0.1.zip (/app/.tox)
py35 start: getenv /app/.tox/py35
py35 cannot reuse: no previous config /app/.tox/py35/.tox-config1
py35 create: /app/.tox/py35
ERROR: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError("Failed to get version_info for python3.5: pyenv: python3.5: command not found\n\nThe `python3.5' command exists in these Python versions:\n 3.5.6\n\n", None)
py35 finish: getenv after 0.05 seconds
py36 start: getenv /app/.tox/py36
py36 cannot reuse: no previous config /app/.tox/py36/.tox-config1
py36 create: /app/.tox/py36
ERROR: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError("Failed to get version_info for python3.6: pyenv: python3.6: command not found\n\nThe `python3.6' command exists in these Python versions:\n 3.6.8\n\n", None)
py36 finish: getenv after 0.05 seconds
___________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________ERROR: py35: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError("Failed to get version_info for python3.5: pyenv: python3.5: command not found\n\nThe `python3.5' command exists in these Python versions:\n 3.5.6\n\n", None)
ERROR: py36: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError("Failed to get version_info for python3.6: pyenv: python3.6: command not found\n\nThe `python3.6' command exists in these Python versions:\n 3.6.8\n\n", None)
cleanup /app/.tox/.tmp/package/1/example-pkg-your-username-0.0.1.zip
Does your issue follow our Code of Conduct?
Ensure you are up-to-date with master.
Search existing issues.
Part of the project with issues: (ie. Agogosml, Deployment, CLI): Agogosml Library
Indicate OS, version of Python (Note: Only Python 3.7 is currently supported.) All.
Steps of reproduce issue. Provide a link to a Github Gist or paste here if short.
Does your issue follow our Code of Conduct?
Ensure you are up-to-date with master.
Search existing issues.
Part of the project with issues: (ie. Agogosml, Deployment, CLI)
Indicate OS, version of Python (Note: Only Python 3.7 is currently supported.)
Steps of reproduce issue. Provide a link to a Github Gist or paste here if short.
agogosml generate
is invoked the other files are created as mentioned in the document however .env file with initial set of keys does not get created.Code modules are being bypassed in coverage to address target of 70% - these lines are marked with # pragma: no cover
As these modules are changed, the goal is to remove pragma statements where code is changing and ensure coverage is in place when meaningful.
In the simple app main.py template, a very low level construct is used to serve http requests. The http server validates the incoming requests against a JSON schema, transforms the data and then creates a response.
This is a common pattern in the web world: we want to ensure that some request corresponds to our desired schema, call a method on the request body and respond with some data. A great abstraction for this workflow is connexion which is a framework that lets you create http servers based on OpenAPI/Swagger specifications. The framework takes care of validating that the request matches the Swagger schema, routing the request to a particular Python function and serializing the response. As such, it seems like a great match for our use-case.
For example, to translate the simple app into connexion, the following would be all that's required:
File api.spec.yaml
swagger: '2.0'
info:
title: My simple API
version: '0.1'
basePath: '/'
paths:
'/':
post:
summary: Transform the data.
operationId: datahelper.transform
consumes:
- application/json
parameters:
- $ref: '#/parameters/Data'
responses:
200:
description: The transformed data.
parameters:
Data:
name: data
description: The data to transform.
in: body
schema:
$ref: '#/definitions/Data'
required: true
definitions:
Data:
properties:
key:
description: The key of the input.
type: string
intValue:
description: The value of the input.
type: number
required:
- key
- intValue
File main.py
from connexion import App
app = App(__name__)
app.add_api('api.spec.yaml')
app.run(port=8080, host='0.0.0.0')
The business logic method datahelper.transform
doesn't even have to know that it's dealing with JSON inputs anymore! It simply gets a deserialized object passed into it that has the keys/values specified in the Swagger spec. Another advantage of connexion is that we can chose from a wide range of server backends like flask, tornado, aiohttp, etc. so it's trivial to get good performance out of the app with just a configuration change.
Last but not least, we can abstract the Swagger spec and use templating so that the user only has to provide the type definitions for the data and never edit the entire Swagger spec file, e.g.:
File input.spec.yaml
Data:
properties:
key:
description: The key of the input.
type: string
intValue:
description: The value of the input.
type: number
required:
- key
- intValue
There's an example of how to bind this fragment into the Swagger spec here.
Another benefit of this approach is that we'll get self-documenting APIs since a Swagger spec will be auto-loaded to ${basePath}/swagger.json
The version of pyyaml referenced differently in
./agogosml/Pipfile
./agogosml/Pipfile.lock
./agogosml/setup.py
./agogosml_cli/Pipfile.lock
[packages]
pyyaml = ">=4.2b1"
Does your issue follow our Code of Conduct?
Ensure you are up-to-date with master.
Search existing issues.
Part of the project with issues: (ie. Agogosml, Deployment, CLI)
Indicate OS, version of Python (Note: Only Python 3.7 is currently supported.)
The three containers add to overall runtime complexity and increase build times while making it harder to maintain the semantics of the queue/stream to which the containers are connected. Much can go wrong in the two extra http hops. Given how we create agogosml apps/containers from templates, we could easily create a single-container template.
agogosml is currently a more brittle (at runtime), less capable model hosting environment than Kafka Streams ( https://kafka.apache.org/21/documentation/streams/core-concepts ). We should simplify agogosml with the idea of moving over to Kafka Streams when the Kafka head for Azure Event Hub supports it.
Does your issue follow our Code of Conduct?
Ensure you are up-to-date with master.
Search existing issues.
Part of the project with issues: (ie. Agogosml, Deployment, CLI)
Indicate OS, version of Python (Note: Only Python 3.7 is currently supported.)
Steps of reproduce issue. Provide a link to a Github Gist or paste here if short.
To reproduce use the generate command to generate the template and then look at the input reader and out put writer docker files. both of them will have paths as follows :
FROM ${CONTAINER_REG}agogosml:${AGOGOSML_TAG} as builder
There needs to be a "/" after ${CONTAINER_REG} . because of the path being incorrect, the Azure Devops build fails with the following message
ull access denied for *********.azurecr.ioagogosml, repository does not exist or may require 'docker login'
After adding "/" build is successful
I will submit PR for this as well.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.