This repository provides sample code for Flask backend application that controls Console for AITRIOS by using Console REST API and Next.js application that retrieves inference results from backend and display visualization in web browser, as well as configuration files for building a development environment.
This application retrieves Keypoint Detection AI model metadata from Console for AITRIOS and display pose estimation.
For basic device and console operation, please refer to the Device Setup Guide and Console User Manual.
To import the AI model into Console, see tutorial in import_model_to_console directory.
To deploy the imported model to the device, see 3.4.1. Deploy registered AI model to device
in Console User Manual.
To get metadata for keypoint detection from the device, deploy the Vision and Sensing application obtained from download site to the device.
See 3.9.1. Manage Vision and Sensing Application
for instructions on importing application into Console, and 3.4.2. Deploy Vision and Sensing Application
for instructions on deploying application to the device in Console User Manual.
To control the device's inference operation, Command Parameter must be bound to the device.
See 3.9.2. Manage Command Parameter file
for instructions on importing Command Parameter file into Console, and 3.3.4. Bind Command Parameter file to device
for instructions on binding Command Parameter to the device in Console User Manual.
The following is an example of minimum operable parameters. Save it as a JSON file of any name and import it into Console for use.
Note
- To reduce device processing delays, recommended to set the "Mode" parameter to "2" (output inference results only).
- Set "ModelID" parameter to the model ID imported into Console above.
{
"commands": [
{
"command_name": "StartUploadInferenceData",
"parameters": {
"MaxDetectionsPerFrame": 5,
"Mode": 2,
"ModelId": "PoseNet_KeypointDet",
"NumberOfInferencesPerMessage": 1,
"PPLParameter": {
"header": {
"id": "00",
"version": "01.01.00"
},
"input_height": 353,
"input_width": 481
},
"UploadInterval": 30,
"UploadMethodIR": "MQTT"
}
}
]
}
-
Clone or fork this application in your GitHub environment.
-
Click the
+
button to create a codespace on the GitHub environment. See how to create a codespace. -
Create console access setting configuration file with real values in backend/src/config/console_access_settings.yaml.
backend/src/config/console_access_settings.yaml
console_access_settings: console_endpoint: "__console_endpoint__" portal_authorization_endpoint: "__portal_authorization_endpoint__" client_secret: "__client_secret__" client_id: "__client_id__"
Note
- Specify the base URL
- Specify the token URL
- Specify registered application secret for client_secret
- Specify registered application client ID for client_id
To obtain the endpoint URL, see REST API Authentication, or see the Portal user manual for more details.
Warning
The above config file is referenced by specifying the path in backend/src/app.py and backend/src/pipeline/data_loader/console_data_loader.py. If you change the path to the config file, you must also change the paths specified in these files.
backend/src/app.py
28 # Load Console access settings 29 with open(os.path.join(os.getcwd(), "src", "config", "console_access_settings.yaml")) as file:
backend/src/pipeline/data_loader/console_data_loader.py
26 with open( 27 os.path.join( 28 os.path.dirname(__file__), "..", "..", "config", "console_access_settings.yaml" 29 )
-
Run the sample application using the following command.
vscode ➜ /workspace/aitrios-application_keypoint-detection (main) $ docker compose up # or run apps in the background vscode ➜ /workspace/aitrios-application_keypoint-detection (main) $ docker compose up -d
Note
To make this app fully operational, set the Port Visibility setting of the backend app to public as follows.
- Click the
PORTS
tab in your Codespace. - Right-click the port of the backend app(
5000
by default), click thePort Visibility
. - Change value Private to Public.
See the Github Codespace documentation for more infomations.
- Click the
-
Access the host address from the pop-up in the lower right corner, and verify that the application is running.
-
Open the cloned repository from VSCode's "Open Folder" in "File" tab
-
Open Command Palette in VSCode(Ctrl + Shift + P)
-
Type and choose
"Dev Containers: Open Folder in Container..."
-
Select root directory of the project and wait for building DevContainer.
/your/local/workspace/aitrios-application_keypoint-detection
-
Create console access setting configuration file with real values in backend/src/config/console_access_settings.yaml.
backend/src/config/console_access_settings.yaml
console_access_settings: console_endpoint: "__console_endpoint__" portal_authorization_endpoint: "__portal_authorization_endpoint__" client_secret: "__client_secret__" client_id: "__client_id__"
Note
- Specify the base URL
- Specify the token URL
- Specify registered application secret for client_secret
- Specify registered application client ID for client_id
To obtain the endpoint URL, see REST API Authentication, or see the Portal user manual for more details.
Warning
The above config file is referenced by specifying the path in backend/src/app.py and backend/src/pipeline/data_loader/console_data_loader.py. If you change the path to the config file, you must also change the paths specified in these files.
backend/src/app.py
28 # Load Console access settings 29 with open(os.path.join(os.getcwd(), "src", "config", "console_access_settings.yaml")) as file:
backend/src/pipeline/data_loader/console_data_loader.py
26 with open( 27 os.path.join( 28 os.path.dirname(__file__), "..", "..", "config", "console_access_settings.yaml" 29 )
-
Run the sample application using the following command.
vscode ➜ /workspace/aitrios-application_keypoint-detection (main) $ docker compose up # or run apps in the background vscode ➜ /workspace/aitrios-application_keypoint-detection (main) $ docker compose up -d
-
Open
localhost:3000
in a browser and verify that the application is running.
Package | Version | Description |
---|---|---|
Node.js |
v18 |
Cross-platform JavaScript runtime environment and library for running web applications. |
TypeScript |
^4.9.5 |
Syntactic superset of JavaScript which adds static typing. |
Next.js |
13.1.6 |
React-based web application development framework with server-side rendering and static website generation. |
React |
18.2.0 |
Front-end JavaScript library for building user interfaces based on components. |
MUI |
^5.11.10 |
Library of UI design components that developers can easily import to build React applications. |
Python |
^3.9 |
Interpreted, object-oriented, high-level programming language with dynamic semantics. |
Flask |
^2.3.2 |
Micro web framework written in Python. |
Flatbuffers |
1.11.0 |
Cross platform serialization library for various languages. |
Console REST API | - | REST APIs that provides accesses and controls of the Console for AITRIOS. |
- Inference result viewer
- A list of edge AI devices connected to the Console is displayed on the screen.
- A list of AI models deployed on the specified device is displayed on the screen.
- A text filed that display Command Parameter binded on the specified device on the screen.
- A button that updates and binds Command Parameter to the specified device.
- A button that that controlls uploading Inference Results of the specified device.
- To run this app properly, deploy PoseNet AI model and Keypoint Detection PPL to the device.
- It can be run on Dev Containers/GitHub Codespaces.
- When the page is first opened, the GetDevices API is called to get the DeviceID list of edge AI devices registered in the Console.
- The gotten Device ID list of edge AI devices is set to the Device selector drop-down list.
- Choose Device ID of purpose from the Device selector drop-down list.
- When a Device ID is selected, the GetCommandParameters API is called to get the info of Command Parameters registerd in the Console.
- Search for Command Parameter bound to a specific device from the info of Command Parameters.
- Binded Command Parameter is displayed in JSON format in the Text Field on the sidebar of the screen.
- Parameters in Command Parameter can be changed to any value.
- When a Device ID is selected, the GetDevice API is called to get a list of AI models deployed to the certain device.
- The gotten AI model list of certain device is set to the AI model selector list on the sidebar.
- The default value of the selector is set to the AI model set in the ModelID key of the Command Parameter obtained in the Command Parameter field.
- Selecting an AI model from the selector will change the ModelID in the Command Parameter field.
- When the button is pressed, the UpdateCommandParameter API is called to update the Command Parameter currently bound to the device with the contents of the Command Parameter field. At this time, the contents of the registered Command Parameter File are overwritten.
- If the update is successful, a notification alert will be displayed.
- When the button is pressed, the StartUploadInferenceResult API is called and the device starts uploading the inference results.
- When the device starts inference, subthreads are started in the backend that loops to retrieve the latest inference result and deserialize them.
- If the start sequence is successful, a notification alert will be displayed. The button label changes to Stop Inference while inference is running.
- If you press the button again while inference is running, the StopUploadInferenceResult API will be called and upload from the device will stop.
- Stop subthreads that is retrieving and deserializing inference results.
- If the stop sequence is successful, a notification alert will be displayed.
-
While inference is running, a subthread loops to retrieve the latest inference result from the Console by calling GetInferenceResult API.
-
Decode the obtained inference result in base64 and deserialize it in Flatbuffers. You can obtain the keypoint coordinates of human pose estimation as follows.
{ "score": 0.916876, "keypoint_list": [ { "point": { "x": 175, "y": 91 }, "name": "Nose" }, { "point": { "x": 181, "y": 87 }, "name": "LeftEye" }, ] }
-
Based on the keypoint coordinates infomation, a pose estimation is drawn on the window of the Inference result viewer.
-
By pressing the icon button in the upper left corner of the window, you can view the inference results in JSON format.
Using the Docker in Docker
technique, start the frontend and backend apps in the DevContainer according to the description in docker-compose.yaml
.
Following commands can be executed in the root directory.
docker compose up
Starts the frontend and backend apps as containers. In the container, start the dev server according to the npm run dev
and flask run
commands, respectively.
docker compose up -d
- Starts the containers in the background and leaves them running.
docker compose down
Stops containers and removes containers, networks, volumes, and images created by docker compose up
.
Following commands can be executed in the /frontend
directory.
npm install
Installs locally the dependent packages listed in package.json
that exist in the current directory. When using DevContainer or Codespaces, this is done automatically at container startup.
npm run dev
Starts the development server for the Next.js
application. By default, the app is hosted on port 3000.
npm run lint
Runs the static analysis tool ESlint
to detect code with execution problems. See .eslintrc.json
for detection rule definitions.
npm run lint:fix
- Automatically fix any detected codes that can be done automatically.
npm run fmt
Runs the automatic formatting tool Prettier
to perform automatic formatting of codes based on coding rules. See .prettierrc.json
for formatting rule definitions.
Following commands can be executed in the /backend
directory.
flask run
Starts the development server for the Flask
application. By default, the app is hosted on port 5000.
python -m pflake8
{source_file_or_directory}
Runs the static analysis tool flake8
to detect code with execution problems. The pflake8
is PEP 8 compliant linter.
python -m black
{source_file_or_directory}
Runs the automatic formatting tool black
to perform automatic code formatting. The black
is PEP 8 compliant formatter.
Since some APIs are not provided in the ConsoleAccessLibrary
, an HTTP client defined in OpenAPI is used to access the Console for AITRIOS instead.
- Get Started
- Console REST API reference
- aitrios-sdk-console-access-lib-python
- aitrios-sdk-cloud-app-sample-python
Before using Codespaces, please read the Site Policy of GitHub and understand the usage conditions.
This library aims to adhere to Semantic Versioning 2.0.0.
See the Release Note from Releases for this repository.
Each release is generated in the main branch. Pre-releases are generated in the develop branch. Releases will not be provided by other branches.