sensiml / piccolo Goto Github PK
View Code? Open in Web Editor NEWSensiML's open-source AutoML solution for Edge AI model development
Home Page: https://sensiml.org
License: Other
SensiML's open-source AutoML solution for Edge AI model development
Home Page: https://sensiml.org
License: Other
Pipeline logs should be easily accessible for developers to help debug issues they are having. We would like to show a list of previously executed pipelines, the summary of the pipeline that ran, and the logs generated by them.
Currently, all logs are written to log files. This makes it difficult to track issues when multiple jobs are running concurrently. We would also like to start surfacing more information during model training to users about the accuracy of the models being trained.
We already have an endpoint that returns the pipeline history on the server. We will need an additional endpoint to return the logs. The UI will then need to perform some formatting to display them.
Potentially adds a high-write database table, or additional logs. I don't expect any issues at the scale for a local server.
Adds a few extra API calls on the front end when loading the UI
No dependencies
No response
Trying to upload a video to the Piccolo AI server from the Data Studio is causing this error in the server logs.
[16/Jul/2024 05:59:00] ERROR [datamanager.exceptions:59] Traceback (most recent call last):
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
sensiml.cloud-1 | response = handler(request, *args, **kwargs)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/drf_spectacular/drainage.py", line 159, in wrapped_method
sensiml.cloud-1 | return method(self, request, *args, **kwargs)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/generics.py", line 242, in post
sensiml.cloud-1 | return self.create(request, *args, **kwargs)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/mixins.py", line 19, in create
sensiml.cloud-1 | self.perform_create(serializer)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/mixins.py", line 24, in perform_create
sensiml.cloud-1 | serializer.save()
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/serializers.py", line 212, in save
sensiml.cloud-1 | self.instance = self.create(validated_data)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/contextlib.py", line 79, in inner
sensiml.cloud-1 | return func(*args, **kwds)
sensiml.cloud-1 | File "/home/sml-app/install/server/datamanager/serializers/capture_video.py", line 140, in create
sensiml.cloud-1 | move(uploaded_file.file.name, os.path.join(folder, file_key_name))
sensiml.cloud-1 | AttributeError: 'NoneType' object has no attribute 'file'
sensiml.cloud-1 | Traceback (most recent call last):
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
sensiml.cloud-1 | response = handler(request, *args, **kwargs)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/drf_spectacular/drainage.py", line 159, in wrapped_method
sensiml.cloud-1 | return method(self, request, *args, **kwargs)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/generics.py", line 242, in post
sensiml.cloud-1 | return self.create(request, *args, **kwargs)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/mixins.py", line 19, in create
sensiml.cloud-1 | self.perform_create(serializer)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/mixins.py", line 24, in perform_create
sensiml.cloud-1 | serializer.save()
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/serializers.py", line 212, in save
sensiml.cloud-1 | self.instance = self.create(validated_data)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/contextlib.py", line 79, in inner
sensiml.cloud-1 | return func(*args, **kwds)
sensiml.cloud-1 | File "/home/sml-app/install/server/datamanager/serializers/capture_video.py", line 140, in create
sensiml.cloud-1 | move(uploaded_file.file.name, os.path.join(folder, file_key_name))
sensiml.cloud-1 | AttributeError: 'NoneType' object has no attribute 'file'
fails to upload video to the server
video is successfully synced to the remote server
No response
Model file is generated and downloaded to my local machine
There was an error downloading. See the attached build log: buildlog.txt
unable to deselect these in the feature generator pannel
The columns are unable to be deselected
Able to select/deselect columns in the dropdown
No response
Feature
The objective is to build a feature extraction functionality to enhance the feature development while creating a pipeline.
We propose to add a feature extractor into WebUI Pipeline Builder that gives
Building a feature extractor for the WebUI unlocks powerful benefits for model development:
Simplified experimentation: Users can easily extract and analyze specific features from data directly within the WebUI, without running a whole pipeline.
Improved model training: Feature extraction allows users to focus models on relevant data aspects, potentially leading to faster training times and better model performance.
Enhanced user experience: Integrating feature extraction into the WebUI empowers users with a much more flexible development, fostering a smoother workflow.
UI
We propose to build a feature extractor based on the pipeline builder. Also, we propose to use 3 different pipeline builder modes: Feature Extractor, Train Model, AutoML
And change the left nav menu to group these screens
They all have the same home screen with a pipeline table and templates.
Existing pipelines can be opened with any of those modes, which gives the flexibility to build models based on a properly selected set of transforms.
Feature Extractor detailed screen:
To run it calls action submitOptimizationRequest with post request project/${projectUuid}/sandbox-async/${pipelineUuid}/
with parameter
execution_type = pipeline
To get data for the Feature Visualisation and Feature Graph it calls an action with get request project/${projectUuid}/sandbox-async/${pipelineUuid}/
Train Model and AutoML use pipeline builder with different custom_training parameter and execution_type = automl
calls action submitOptimizationRequest with post request project/${projectUuid}/sandbox-async/${pipelineUuid}/
Back-end
The endpoint project/${projectUuid}/sandbox-async/${pipelineUuid}/
should also return
feature_summary
To implement this we need to implement feature_summary calculation base on feature_table similarly to model's feature_summary
The endpoint project/${projectUuid}/sandbox-async/${pipelineUuid}/
should also return result_type
field
There are some extra calculation of feature_summary during get request project/${projectUuid}/sandbox-async/${pipelineUuid}/
for pipelines with execution_type = pipeline
No 3rd party dependencies would be added
Benefit: This feature simplifies experimentation, improves model training, and enhances user experience by enabling feature extraction directly within the UI.
Faster workflows: Extract and analyze features without running entire pipelines.
Optimized models: Focus models on relevant data aspects for potentially faster training and better performance.
Smoother development: Integrates seamlessly into UI fostering a streamlined workflow.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.