ubcagrobot / appliedai

Central Codebase for Applied AI Team

AppliedAI

Central Codebase for Applied AI Team

appliedai's Issues

Preprocessing Node

• Node for resizing, stacking and padding images

determine which is better: resized image dimensions or original image dimensions (scaling the bounding boxes in husan's node)

MVP Priority

• Convert color filter code and extermination array to C++
• Dedicated preprocessing node to combine+pad images
• Verify CUDA/TensorRT performance
• Convert final array w/ transformer code to C++
• Play around with dummy models
• Metric for accuracy in display node (find sample data)

ROSberry Pi x 2 + Google Coral + Intel Realsense Wrapper Setup

With the Raspberry Pi now being used as our main source for ROS for the foreseeable term please complete all items in this issue.

Equipment needed:

• Raspberry Pi x 2
• Google Coral Board
• Intel Realsense D455

Checklist:

1. Prepare Raspberry Pi

• Flash Ubuntu 22.04 Desktop onto two raspberry Pi 4 devices
• Set up SSH for remote access
• Set up with UBC wi-fi (eduroam)

2. Install ROS2

• Install ROS2 Humble
• Set up the ROS2 environment variables
• Verify and Test ROS2 installation

3. Intel Realsense D455 Setup

• Install the necessary Intel Realsense D455 drivers and libraries
• Verify the camera is recognized and functional

4. ROS2 Workspace Configuration

• Create a ROS2 workspace on each Raspberry Pi
• Integrate with Github
• Install ROS2 dependencies for working with the Intel Realsense camera

5. Google Coral Setup

• Setup Google Coral Board

6. Sample Application Setup:

• Test out entire setup and workflow to ensure that the Intel Realsense D455 camera is working with ROS2 and that the Google Coral board is setup

7. Create Comprehensive Documentation:

• Create comprehensive documentation with clear, step-by-step instructions
• Create image of Raspberry Pi so that we can distribute and use in future

Jerome's RTOS Work

• ROS2 Real-time system testing
• Community RTOS Kernel

Hard real-time systems have strict, non-negotiable timing constraints where tasks must complete within a guaranteed timeframe.

• Use real-time operating systems (RTOS) to ensure deterministic scheduling and response times.
• Prioritize critical tasks and allocate fixed time slots for their execution.

Soft real-time systems have timing constraints that are important but not as strict as hard real-time systems. Missing a deadline is undesirable but not catastrophic.

• Implement priority scheduling to ensure critical tasks are given preference.
• Set priorities based on the criticality of the task.

Task Scheduling: Use appropriate task scheduling algorithms to ensure timely execution.

• Employ priority-based scheduling algorithms to give precedence to high-priority tasks.
• Consider rate monotonic scheduling for periodic tasks.

Real-Time Clocks and Timers:

• Utilize hardware-based real-time clocks for accurate timekeeping.
• Set timers to trigger periodic or one-shot tasks at precise intervals.

Deadline Monitoring:

• Monitor task execution to ensure that deadlines are not missed.
• Implement mechanisms to handle tasks that exceed their allotted time.

Deterministic Execution Paths: Minimize variations in execution times to achieve deterministic behavior.

• Profile and optimize code to reduce execution time variability.
• Minimize the use of non-deterministic constructs in the code.

Real-Time Guarantees: Provide guarantees for response times and completion times.

• Analyze worst-case execution times (WCET) to determine task execution bounds.
• Use scheduling and resource allocation techniques to meet guarantees.

Input and Output Constraints: Minimize the latency associated with input and output operations.

• Use asynchronous I/O operations to overlap computation and communication.
• Optimize data serialization and deserialization for efficient communication.

Task Offloading: Offload non-critical tasks to secondary processors or devices.

• Identify tasks that can be offloaded without affecting critical functions.
• Distribute computational load to improve overall system performance.

Interrupt Handling:

• Prioritize interrupts based on criticality.
• Minimize interrupt service routine (ISR) execution time.

Resource Reservation: Reserve resources such as CPU time, memory, and bandwidth for critical tasks.

• Use resource reservation mechanisms to guarantee availability for critical tasks.
• Monitor and enforce resource usage limits.

Real-Time Networking: Minimize communication latency in real-time networks.

• Use Real-Time Publish-Subscribe (RTPS) middleware for efficient and low-latency communication.
• Optimize network configurations for minimal jitter and latency.

Testing and Validation:

• Use stress testing and worst-case scenario testing to ensure the system meets real-time constraints.
• Simulate different operating conditions to verify performance under varying workloads.

Continuous Monitoring:

• Use real-time monitoring tools to track task execution times, resource usage, and system health.
• Implement alarms or notifications for deviations from expected performance.

Decide SOW/Goal 23-C-08_Weed

Weed object detection will be a crucial task involved in FRE 2025. Please read more about the competition here. We will be required to perform object detection (bounding boxes + class probabilities) and deliver the resultant bounding boxes and probabilities to the extermination subteam. Please keep in mind that this model that is developed will have to be run on Nvidia systems.

We are aiming to create a general weed classifier that can identify various types of weeds as the competition has not stated what weeds will be used. The first step to solve this problem effectively is to define the scope of work, methods, and approaches to develop a reliable object detection solution.

Action Items:

• Research problem + current approaches
• Find existing datasets if possible
• See if problem can be simplified
• Discuss problem with project group
• Fill out problem solver document (clone from google drive)
• Discuss problem solver document with Robbie + your ideas

Metric Logging and File Versioning

Research industry approaches to log metric data for robotics applications

• Use profiling tools for C++ and complete the PID approach that Valery has started (std::chrono)
• Verify the results match JTOP statistics
• Print FPS within cv2 display window
• Print preprocess, inference and postprocess latency values in terminal (rolling average over 1 second intervals)
• Add the ability to save recorded metric data to a logging file to later visualize
• Add model metadata and architecture/topic statistics to keep track of what logging files came from where
• Determine whether Python display node w/ cross-compilation or C++ Prometheus alternative is required

Logging Specific:
rclcpp::TimeSource and rclcpp::Clock:

• Utilize the ROS2 time-related classes to log timestamps and durations.
• Compare timestamps to measure latency between events.

Logging Statements (RCLCPP_INFO, RCLCPP_WARN, etc.):

• Add logging statements in your code to measure the time taken by specific operations.
• Analyze the log outputs to identify latency sources.

Lighting Condition Robustness

Histogram equalization can help improve the contrast of an image, which can be beneficial for object detection in varying lighting conditions.

import cv2

def preprocess_image(image):
    # Convert image to grayscale
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    # Apply histogram equalization
    equalized = cv2.equalizeHist(gray)
    return equalized`

Gamma correction adjusts the brightness of an image, which can help compensate for variations in lighting conditions.

import cv2
import numpy as np

def gamma_correction(image, gamma=1.0):
    # Build a lookup table mapping the pixel values [0, 255] to their adjusted gamma values
    inv_gamma = 1.0 / gamma
    table = np.array([((i / 255.0) ** inv_gamma) * 255 for i in np.arange(0, 256)]).astype("uint8")
    # Apply gamma correction using the lookup table
    corrected = cv2.LUT(image, table)
    return corrected

Adaptive thresholding can be useful for segmenting objects from the background in images with varying lighting conditions.

import cv2

def adaptive_thresholding(image):
    # Convert image to grayscale
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    # Apply adaptive thresholding
    thresholded = cv2.adaptiveThreshold(gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)
    return thresholded

Color normalization techniques can help reduce the impact of varying lighting conditions on color-based features in object detection.

import cv2

def color_normalization(image):
    # Convert image to LAB color space
    lab = cv2.cvtColor(image, cv2.COLOR_BGR2LAB)
    # Compute mean and standard deviation of the L channel
    l_mean, l_std = cv2.meanStdDev(lab[:, :, 0])
    # Normalize the L channel
    normalized_l = (lab[:, :, 0] - l_mean) / l_std
    # Merge normalized L channel with original AB channels
    normalized_lab = cv2.merge([normalized_l, lab[:, :, 1], lab[:, :, 2]])
    # Convert back to BGR color space
    normalized_image = cv2.cvtColor(normalized_lab, cv2.COLOR_LAB2BGR)
    return normalized_image

You can combine multiple preprocessing techniques to further enhance the robustness of your object detection system against varying lighting conditions.

def preprocess_image(image):
    # Apply a combination of preprocessing techniques
    equalized = cv2.equalizeHist(image)
    gamma_corrected = gamma_correction(equalized, gamma=1.2)
    thresholded = cv2.adaptiveThreshold(gamma_corrected, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)
    return thresholded

Decide SOW/Goal 23-M-11_GrapeQA

Grape quality assessment is a fundamental practice in viticulture (the cultivation of grapevines) that involves evaluating the attributes of grape clusters to determine their overall quality and readiness for harvest. This assessment is of paramount importance for vineyards and winemaking for several reasons:

1. Wine Quality: The quality of grapes directly influences the quality of the wine produced. Winemakers seek grapes with specific attributes, such as sugar content, acidity levels, and flavour compounds, to create wines with desired taste, aroma, and mouthfeel characteristics.
2. Harvest Timing: Determining the optimal time to harvest is crucial. Assessing grape quality helps vineyard managers and winemakers decide when to pick the grapes to achieve the desired balance of ripeness and flavour.
3. Consistency: Consistent grape quality assessment practices help maintain a consistent quality standard for wines produced by a vineyard, contributing to brand reputation and customer loyalty.
4. Resource Management: Efficiently allocating resources, such as labor and equipment, for harvesting grapes is essential. Accurate grape quality assessment ensures that resources are used effectively.
5. Sustainability: By assessing grape quality, vineyards can minimize waste, reduce the need for excessive chemical treatments, and promote sustainable viticulture practices.

Machine learning models can significantly enhance grape quality assessment by providing objective, data-driven insights and automation capabilities. Here's how machine learning contributes to grape quality assessment:

• Predictive Modeling: Machine learning algorithms can predict grape quality attributes, such as sugar content (Brix), acidity levels, and ripeness, based on historical data and current environmental factors. This enables more accurate timing for grape harvesting.
• Automated Image Analysis: Machine learning models, particularly computer vision algorithms, can analyze images of grape clusters to detect visual indicators of grape quality, such as color, size, and uniformity. This helps assess grape ripeness and overall condition.
• Quality Grading: Machine learning can be used to classify grape clusters into different quality grades, streamlining the sorting process during harvest.

Attack one of the above concepts for grape quality assessment. From my understanding, quality grading may be the most straightforward, but I think also automated image analysis would be interesting to tackle

Requirements:

• Accuracy: accurately detect grape quality, minimizing false positives
• Real time processing: the model should be designed to process images or video frames in real-time or near real time to support timely decision making in vineyard operations
• Adaptable: The model should be adaptable to different vineyard environments and camera setups
• Integration: Ensure that the model can be integrated with different types of cameras, drones, or sensors commonly used in vineyards

Action Items:

• Research problem + current approaches
• Find existing datasets if possible
• See if problem can be simplified
• Discuss problem with project group
• Fill out problem solver document (clone from google drive)
• Discuss problem solver document with Robbie + your ideas

ROS2 Launch Configuration and Parameters

Add ROS2 Launch configuration commands with parameters that incorporate the following:

• Ability to launch all 3 nodes at the same time
• Ability to launch just the camera and jetson nodes together
• Ability to alter parameters like image source used (static image publishing, Zed camera, Web Cam, video frames)
• Ability to alter the path of the model file being loaded
• Ability to alter the path of the image source folder being loaded (for .jpg and .mp4)
• Add default values for all the above parameters for fast Jetson debugging (Zed camera, all 3 nodes, repository folders, etc.)
• Make sure to document in detail what each parameter does
• Add logic for starting, stopping and reconfiguring nodes without system restart
• Example ROS2 Launch parameters arguments

More documentation and Examples:

• ROS2 Parameters
• RQT console
• Composable Nodes
• Launch File Formats
• Parameters Example
• ROS environment configuration
• ROS Domain ID

Verify CUDA/TensorRT performance

Convert filter node to cpp

• Exclusion factor for area of objects
• Implemenet union between RCNN and color filter output
• Include code for publishing to extermination output (if bounding box is within region of interest)
• (under experiments/Python POCs/Camera Subscriber, take publishing functinoality and put into C++)
• Write a unit test template for specific functions of your code with maize pictures

Benchmarking

Metric for accuracy in display node (find sample data)

• Create performance report for ONNX to TensorRT for both Pytorch and Tensorflow converted models respectively
• Create performance report for Pytortch to Torch-TRT
• Create performance report for Tensorflow to TF-TRT

Profile ROS2 Nodes:

• ros2 topic hz Command: measure the publication rate and latency of a topic
• ros2 latency Package: measuring round-trip latency between nodes
• ros2 tracing: Use rqt with the ros2_trace plugin for visualization
• rclcpp Profiler: ROS2 provides a built-in profiler for C++ nodes
• System Profilers (e.g., perf, gprof, valgrind): Example with perf: sudo perf record -e cycles -g -- ros2 run your_package your_node

Decide SOW/Goal 23-C-07_Maize

Maize object detection will be a crucial task involved in FRE 2025. Please read more about the competition here. We will be required to perform object detection (bounding boxes + class probabilities) and deliver the resultant bounding boxes and probabilities to the extermination subteam. Please keep in mind that this model that is developed will have to be run on Nvidia systems.

In general, it is useful for various agricultural applications, including yield estimation, pest detection, and crop monitoring. However, the first step to solve this problem effectively is to define the scope of work, methods, and approaches to develop a reliable object detection solution.

Action Items:

• Research problem + current approaches
• Find existing datasets if possible
• See if problem can be simplified
• Discuss problem with project group
• Fill out problem solver document (clone from google drive)
• Discuss problem solver document with Robbie + your ideas

investigation: handling different model types

Play around with dummy models
Research on whether the .h files for different model types will affect how the jetson node is configured

different models to test:

• RCNN
• SSD
• YOLO

YOLO and FRCNN in the Test models Folder

Following this, reach out to these people to test their models and ask for any necessary pre-processing steps or output:

• Husan for maize model
• Sanskar for his model
• AJ for his FR-CNN model

Jerome's Checklist

• Converting .onnx file to .trt in colab
• Implementing the TensorRT engine inference using NVIDIA examples and other tutorials in C++
• Optimizing the inference with batching, CUDA contexts, multithreading/asynchronous streams
• Include best practices: NVIDIA TensorRT optimizations (include creating interfaces or abstract classes that define the common functionality required for neural network processing)
• Comparison w/ industry examples (Nvidia, Zed, Github examples) to adapt our case to the Jetson hardware
• Metric logging for benchmarking using Nvidia tutorials
• Test different batch sizes to maximize the GPU utlilization
• Investigate any other potential areas for multiprocessing or parallel execution to speed up task processing

Links Checklist:

• TensorRT C++ API Repository
• Youtube Tutorial
• Nvidia Support Matrix
• Nvidia Archived Best Practices
• Nvidia Object Detection Examples
• Nvidia Guide for Benchmarking/Profiling
• Nvidia Guide for Optimization
• Nvidia Example ONNX Deployment
• Nvidia Basic Workflow (for making docs)

Links Checklist (some of these may overlap w/ Jerome's so maybe meet with him to discuss progress and what needs to be revisited or reviewed)

• YOLOV8
• ZedBox Custom Input
• ONNX TensorRT
• Python Tutorial Script
• Nvidia Tutorial Example
• Updated Nvidia Tutorial
• TRTexec Tool Script

Camera Node

• Webcam version of C++ node

• Zed Camera Node

• Include preprocessing to remove alpha channel

Conversion Tools

Interconverter for benchmarking between:

• TensorFlow to ONNX

• PyTorch to ONNX

• test more model architectures (found inside the dev model folder, test data and the model)

• verify it works using online compiler

• script for converting models to trt .engine file (can be python or C++, try links below)

Decide SOW/Goal 23-M-10_GrapeLD

Leaf diseases in vineyards are fungal, bacterial, or viral infections that affect the leaves of grapevines. These diseases can have a significant impact on vineyards and are of great importance to both vineyard owners and farmers for several reasons:

1. Reduced Grape Quality: Leaf diseases can lead to a reduction in grape quality, affecting factors such as taste, aroma, and overall wine quality. Infected leaves may not photosynthesize efficiently, resulting in less sugar accumulation in grapes and lower wine production quality.
2. Yield Loss: Severe leaf diseases can cause yield losses, reducing the quantity of grapes harvested. This can result in financial losses for vineyard owners and decreased profitability for farmers.
3. Increased Costs: Treating and managing leaf diseases can be expensive. Farmers may need to invest in fungicides, labor, and equipment to prevent or control outbreaks. These costs can significantly impact the overall operation of a vineyard.
4. Crop Management: Leaf diseases require careful crop management. Farmers must monitor vine health, identify diseases early, and implement appropriate treatments to minimize damage. This adds complexity to vineyard operations.
5. Sustainability: Sustainable farming practices are increasingly important in the wine industry. Leaf diseases can lead to increased chemical pesticide usage, which may be detrimental to the environment and long-term sustainability goals.
6. Impact on Grape Varieties: Different grape varieties have varying susceptibility to leaf diseases. Managing these diseases is essential for preserving the health and quality of specific grape varieties that are often associated with specific wine types.
7. Vineyard Reputation: The quality of wine produced by a vineyard contributes to its reputation. Leaf diseases can damage the reputation of both vineyards and the wines they produce.
8. Crop Insurance and Financing: For vineyard owners who rely on loans or crop insurance, the presence of leaf diseases can impact financial agreements and insurance coverage.
9. Research and Development: Managing leaf diseases requires ongoing research and development efforts. Farmers and vineyard owners may need access to the latest advancements in disease prevention and management.

To address these challenges, vineyard owners and farmers employ various strategies, including disease-resistant grape varieties, integrated pest management (IPM) practices, regular monitoring, and the use of fungicides when necessary. Early detection and prevention are crucial for minimizing the impact of leaf diseases on vineyard operations and the quality of wine produced.

Leaf diseases in vineyards can be caused by various fungi, bacteria, and viruses. Some of the common types of leaf diseases that affect grapevines include:

• Powdery Mildew (Erysiphe necator): Powdery mildew is one of the most widespread and destructive grapevine diseases. It appears as a white, powdery substance on the leaves, shoots, and fruit clusters. It can reduce photosynthesis, affect grape quality, and lead to yield losses.
• Downy Mildew (Plasmopara viticola): Downy mildew affects both leaves and fruit. It appears as yellow or greenish lesions on the upper side of leaves and a downy growth on the lower side. It can lead to defoliation, reduced grape quality, and yield loss.
• Esca (Various Fungi): Esca is a complex of grapevine trunk diseases caused by various fungi. Symptoms include yellowing and browning of leaves, dieback of shoots, and reduced vine vigor. It can lead to long-term damage and vine decline.
• Botrytis Bunch Rot (Botrytis cinerea): Botrytis can affect not only the fruit but also leaves and shoots. It tends to be more common in regions where conditions are conducive to its development, such as areas with high humidity.

Aim to only detect one leaf disease, but aim to do so very well and meet the requirements

Requirements:

• Accuracy: accurately detect leaf diseases, minimizing false negatives
• Real time processing: the model should be designed to process images or video frames in real-time or near real time to support timely decision making in vineyard operations
• Adaptable: The model should be adaptable to different vineyard environments and camera setups
• Integration: Ensure that the model can be integrated with different types of cameras, drones, or sensors commonly used in vineyards

Action Items:

• Research problem + current approaches
• Find existing datasets if possible
• See if problem can be simplified
• Discuss problem with project group
• Fill out problem solver document (clone from google drive)
• Discuss problem solver document with Robbie + your ideas

Use multithreading in nodes

Nvidia Jetson Nano Setup

The Nvidia Jetson Nano needs to be set up to run ROS 2 Foxy and support the Intel Realsense D455 camera.

Hardware

• Nvidia Jetson Nano
• Intel Realsense D455

Software

• Jetpack
• Ubuntu 20.04
• ROS2 Foxy
• Realsense Wrapper

For more information about setting up a Jetson Nano please see this link
For more information about installing ROS 2 Foxy on a Jetson please see this link
For more information about installing the Realsense Wrapper please see this link

Decide SOW/Goal 23-M-09_GrapeCD

Grape cluster detection is a computer vision and image processing task that involves identifying and locating grape clusters within vineyard images or videos. This technology is particularly important for vineyards and farmers for several reasons:

1. Harvest Planning: Grape cluster detection helps vineyard owners and farmers plan their harvest more effectively. By knowing the location and quantity of grape clusters in advance, they can schedule the picking process efficiently, ensuring that grapes are harvested at the optimal time for quality and ripeness.
2. Yield Estimation: Accurate grape cluster detection allows for better yield estimation. Farmers can estimate the expected crop yield early in the growing season, helping with production planning, resource allocation, and market forecasting.
3. Quality Control: Identifying grape clusters enables farmers to monitor the health and quality of the grapes. Detection systems can identify clusters with signs of disease, pests, or other issues, allowing for targeted intervention and treatment.
4. Resource Optimization: Vineyards can optimize resource usage, such as labor and machinery, by directing them to the areas where grape clusters are most abundant. This reduces unnecessary manual labor and minimizes damage to the vines.
5. Grape Maturity Assessment: Grape cluster detection can be used to assess the maturity of the grapes. Different grape clusters may ripen at different rates, and this information can help determine when specific clusters should be harvested for the desired wine characteristics.
6. Precision Agriculture: Grape cluster detection is part of the broader field of precision agriculture. By using technology to precisely manage vineyard operations, farmers can reduce costs, increase productivity, and minimize environmental impacts.
7. Research and Development: In the research and development of new grape varieties or cultivation techniques, grape cluster detection can provide valuable data for analysis and experimentation.
8. Wine Quality: The quality of wine is closely tied to the quality of the grapes used in production. Grape cluster detection contributes to the production of high-quality grapes, which, in turn, leads to better-quality wines.
9. Sustainability: Efficient grape cluster detection can help minimize waste and optimize resource usage, contributing to sustainable farming practices and reduced environmental impact.

Grape cluster detection is typically achieved using computer vision techniques and machine learning algorithms that analyze images or videos captured in the vineyard. These systems can be automated and integrated into the overall vineyard management process, making them valuable tools for modern viticulture and wine production.

Requirements:

• Accuracy: accurately detect grape bunches, minimizing false positives and false negatives
• Real time processing: the model should be designed to process images or video frames in real-time or near real time to support timely decision making in vineyard operations
• Adaptable: The model should be adaptable to different vineyard environments and camera setups
• Integration: Ensure that the model can be integrated with different types of cameras, drones, or sensors commonly used in vineyards

Action Items:

• Research problem + current approaches
• Find existing datasets if possible
• See if problem can be simplified
• Discuss problem with project group
• Fill out problem solver document (clone from google drive)
• Discuss problem solver document with Robbie + your ideas

ROS Development Experiments

Please complete all items in this issue

Prerequisites:

• Download Ubuntu 22.04 Desktop
• Download ROS2 Humble Hawksbill
• Download Realsense Wrapper
• Download Tensorflow

Background Knowledge Required:

• ROS2 Knowledge
• Tensorflow Background

What We Want to Find Out

1. Python vs C++ ROS Node Performance and Output Lagtime
2. Best Method of Building Nodes for Neural Networks
3. Best Method of Building Camera Nodes and Transferring Frames

Python vs C++ ROS Node Performance

Objective:
The main objective of this experiment is to measure and analyze the performance differences between ROS2 Python and ROS2 C++ nodes when executing different neural networks. This information will help the team make informed decisions when choosing the language for their ROS2 projects involving neural networks.

Tasks:

• Create YOLO, F-RCNN, and MobileNet Neural Networks
• Implement ROS2 nodes for Python and C++ to run the specified neural networks (YOLO, F-RCNN, and MobileNet)
• Create a test environment with representative hardware and software configurations to ensure realistic performance testing
• Collect performance metrics such as execution time, CPU and memory usage, latency, stability over time and network bandwidth
• Analyze gathered data and create performance comparison report
• Document and share the results, insights and recommendations with the team

Best Method of Building Nodes for Neural Networks

Objective:
The main objective of this issue is to foster a collaborative discussion on how to structure ROS2 nodes for neural networks in a way that achieves the following goals:

• Low-Latency: Minimize the time it takes for neural network inference to produce results, which is vital for real-time robotics applications.
• Scalability: Ensure that the ROS2 node can effectively scale with increased computational demands, such as larger or more complex neural networks.
• Adaptability: Make the node easily adaptable to different neural network architectures and configurations without extensive code modifications.

Tasks:

• Node Architecture: Figure out and test the overall structure of a ROS2 node for NN. Should it be monolithic, modular, or something else? What design patterns are most suitable for low-latency performance?
• Interface Design: How should the node interface with ROS2, taking into consideration message passing, services, and actions?
• Hardware Acceleration: Report on items we could purchase or use to improve performance and by how much expected
• Configuration Management: Explore and test strategies for managing neural network configurations and adapting to different networks with minimal code changes.
• Data Preprocessing: How should data preprocessing be handled to optimize NN input and output?
• Real-time Constraints: What real time constraints and conditions can be implemented to ensure better results? How can we ensure low-latency performance within these constraints?
• Benchmarking and Profiling: Benchmark and profile the ROS2 nodes to measure and optimize latency
• Analyze gathered data and create performance comparison report
• Document and share the results, insights and recommendations with the team

Best Method of Building Camera Nodes and Transferring Frames

Objective:
The primary objective of this issue is to encourage a collaborative discussion on how to structure ROS2 nodes for RGB-D cameras, with a focus on achieving the following goals:

• Low-Latency: Minimize the time it takes to capture, process, and publish RGB-D data, ensuring real-time responsiveness for robotics applications.
• Scalability: Design nodes that can efficiently handle varying resolutions, frame rates, and camera models, and adapt to the computational demands of different applications.
• Adaptability: Create a node structure that can easily accommodate different RGB-D camera models and configurations without extensive code modifications.

Tasks:

• Node Architecture: Discuss the overall structure of ROS2 nodes for RGB-D cameras. Should it be modular, configurable, or use a specific design pattern? What strategies can be employed to reduce latency in data processing?
• Camera Interface: Explore how the node should interface with RGB and RGB-D cameras, considering ROS2 message types, configuration options, and synchronization mechanisms for RGB and depth data.
• Real-time Constraints: Address the real-time constraints involved in robotics applications and how the node design can ensure low-latency performance within these constraints.
• Configuration Management: Figure out methods for managing camera configurations, such as resolution, frame rate, and synchronization, while allowing easy adaptation to different camera models.
• Data Preprocessing: Figure out best practices for preprocessing RGB-D data to optimize the information received from the camera and reduce latency.
• Synchronization Techniques: Figure out methods for synchronizing RGB and depth data efficiently, especially when dealing with cameras that may have slight timing discrepancies.
• Benchmarking and Profiling: Benchmark and profile the Camera ROS2 nodes to measure and optimize latency
• Analyze gathered data and create performance comparison report
• Document and share the results, insights and recommendations with the team

Architecture Documentation

• readme.md summary of node architecture
• UML diagram for different architecture versions
• Explanation of TensorRT optimization methods (including model architeecture best practices)
• Explanation of currently utilized optimizations

Extermination Node Testing

• Verify maximum latency from taking frame to outputting array value < 100 ms
• Verify object counting in sample video is consistent (or implement object identifier)
• Refine ROI to match sprayer area (using ROI calibration tool)
• Refine horizontal shift to match latency (using latency calibration tool)
• Test how Gaussian blurring impacts performance (do the boxes still rapidly spray on/off?)
• Test measuring velocity with onboard accelerometer of Zed Camera
• Implement plugin architecture to support different neural networks (allow easy swapping of models without modifying core node code)