sensiml / piccolo Goto Github PK

RFC Feature Enhancement

Description

Pipeline logs should be easily accessible for developers to help debug issues they are having. We would like to show a list of previously executed pipelines, the summary of the pipeline that ran, and the logs generated by them.

Motivation

Currently, all logs are written to log files. This makes it difficult to track issues when multiple jobs are running concurrently. We would also like to start surfacing more information during model training to users about the accuracy of the models being trained.

Design Proposal

Showing Pipeline History

Displaying Formatted Logs for Pipeline History

Storing Log Files

• Postprocess current logs files and store results into a single file after the pipeline finishes execution
• Add additional logging that appends to a log file for each pipeline in a pipeline execution folder that can be retrieved via REST API
• Add another table in the PostgreSQL database that we store pipeline logs and index against the task ID so we can quickly pull them up
• Add another type of database better suited for the logs like MongoDB and write there

Retrieving Log Files

We already have an endpoint that returns the pipeline history on the server. We will need an additional endpoint to return the logs. The UI will then need to perform some formatting to display them.

Performance Implications

Potentially adds a high-write database table, or additional logs. I don't expect any issues at the scale for a local server.
Adds a few extra API calls on the front end when loading the UI

Dependencies

No dependencies

User Impact

No response

Description

Trying to upload a video to the Piccolo AI server from the Data Studio is causing this error in the server logs.

[16/Jul/2024 05:59:00] ERROR [datamanager.exceptions:59] Traceback (most recent call last):
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
sensiml.cloud-1 | response = handler(request, *args, **kwargs)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/drf_spectacular/drainage.py", line 159, in wrapped_method
sensiml.cloud-1 | return method(self, request, *args, **kwargs)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/generics.py", line 242, in post
sensiml.cloud-1 | return self.create(request, *args, **kwargs)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/mixins.py", line 19, in create
sensiml.cloud-1 | self.perform_create(serializer)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/mixins.py", line 24, in perform_create
sensiml.cloud-1 | serializer.save()
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/serializers.py", line 212, in save
sensiml.cloud-1 | self.instance = self.create(validated_data)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/contextlib.py", line 79, in inner
sensiml.cloud-1 | return func(*args, **kwds)
sensiml.cloud-1 | File "/home/sml-app/install/server/datamanager/serializers/capture_video.py", line 140, in create
sensiml.cloud-1 | move(uploaded_file.file.name, os.path.join(folder, file_key_name))
sensiml.cloud-1 | AttributeError: 'NoneType' object has no attribute 'file'
sensiml.cloud-1 | Traceback (most recent call last):
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
sensiml.cloud-1 | response = handler(request, *args, **kwargs)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/drf_spectacular/drainage.py", line 159, in wrapped_method
sensiml.cloud-1 | return method(self, request, *args, **kwargs)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/generics.py", line 242, in post
sensiml.cloud-1 | return self.create(request, *args, **kwargs)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/mixins.py", line 19, in create
sensiml.cloud-1 | self.perform_create(serializer)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/mixins.py", line 24, in perform_create
sensiml.cloud-1 | serializer.save()
sensiml.cloud-1 | File "/usr/local/lib/python3.10/site-packages/rest_framework/serializers.py", line 212, in save
sensiml.cloud-1 | self.instance = self.create(validated_data)
sensiml.cloud-1 | File "/usr/local/lib/python3.10/contextlib.py", line 79, in inner
sensiml.cloud-1 | return func(*args, **kwds)
sensiml.cloud-1 | File "/home/sml-app/install/server/datamanager/serializers/capture_video.py", line 140, in create
sensiml.cloud-1 | move(uploaded_file.file.name, os.path.join(folder, file_key_name))
sensiml.cloud-1 | AttributeError: 'NoneType' object has no attribute 'file'

Reproduction steps

1. Open Google Chrome (v125.0.6422.142, Official Build, 64-bit)
2. Go to '...'
3. Click on '...'
4. Scroll down to '...'
5. See error '...'
   ...

Current behavior

fails to upload video to the server

Expected behavior

video is successfully synced to the remote server

Additional context and Screenshots

No response

Steps to reproduce

1. Open the Analytics Studio
2. Generate an AutoML model
3. Open the Download Model page
4. Select a platform (In my test I used Windows x86_64)
5. Click Download

Expected Behavior

Model file is generated and downloaded to my local machine

Actual Behavior

There was an error downloading. See the attached build log: buildlog.txt

Description

unable to deselect these in the feature generator pannel

Reproduction steps

1. Go to the main branch
2. Start the server
3. Open a pipeline
4. Open the feature generator
5. Click + more features
6. Try to change the sensors

Current behavior

The columns are unable to be deselected

Expected behavior

Able to select/deselect columns in the dropdown

Additional context and Screenshots

No response

Please identify RFC type

Feature

Description

The objective is to build a feature extraction functionality to enhance the feature development while creating a pipeline.
We propose to add a feature extractor into WebUI Pipeline Builder that gives

Motivation

Building a feature extractor for the WebUI unlocks powerful benefits for model development:

• Simplified experimentation: Users can easily extract and analyze specific features from data directly within the WebUI, without running a whole pipeline.

• Improved model training: Feature extraction allows users to focus models on relevant data aspects, potentially leading to faster training times and better model performance.

• Enhanced user experience: Integrating feature extraction into the WebUI empowers users with a much more flexible development, fostering a smoother workflow.

Design Proposal

UI

We propose to build a feature extractor based on the pipeline builder. Also, we propose to use 3 different pipeline builder modes: Feature Extractor​, Train Model​, AutoML
And change the left nav menu to group these screens

They all have the same home screen with a pipeline table and templates.

Existing pipelines can be opened with any of those modes, which gives the flexibility to build models based on a properly selected set of transforms.

Feature Extractor detailed screen:

To run it calls action submitOptimizationRequest with post request project/${projectUuid}/sandbox-async/${pipelineUuid}/ with parameter
execution_type = pipeline

To get data for the Feature Visualisation and Feature Graph it calls an action with get request project/${projectUuid}/sandbox-async/${pipelineUuid}/

Train Model and AutoML use pipeline builder with different custom_training​ parameter and execution_type = automl
calls action submitOptimizationRequest with post request project/${projectUuid}/sandbox-async/${pipelineUuid}/

Back-end
The endpoint project/${projectUuid}/sandbox-async/${pipelineUuid}/ should also return
feature_summary
To implement this we need to implement feature_summary calculation base on feature_table similarly to model's feature_summary

The endpoint project/${projectUuid}/sandbox-async/${pipelineUuid}/ should also return result_type field

Performance Implications

There are some extra calculation of feature_summary during get request project/${projectUuid}/sandbox-async/${pipelineUuid}/ for pipelines with execution_type = pipeline

Dependencies

No 3rd party dependencies would be added

User Impact

Benefit: This feature simplifies experimentation, improves model training, and enhances user experience by enabling feature extraction directly within the UI.

• Faster workflows: Extract and analyze features without running entire pipelines.

• Optimized models: Focus models on relevant data aspects for potentially faster training and better performance.

• Smoother development: Integrates seamlessly into UI fostering a streamlined workflow.