Yolov8 tensorflow js about ultralytics

[github-actions]

👋 Hello @AlessandroB1298, thank you for your interest in Ultralytics YOLOv8 🚀! We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Join the vibrant Ultralytics Discord 🎧 community for real-time conversations and collaborations. This platform offers a perfect space to inquire, showcase your work, and connect with fellow Ultralytics users.

Install

Pip install the ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

Hello!

Currently, TensorFlow.js does not natively support oriented bounding boxes (OBB) directly as part of its model operations. However, you can still use YOLOv8 models that output OBB by handling the orientation aspect in your post-processing logic after running the inference in TensorFlow.js.

You would need to export your YOLOv8 model to the TensorFlow.js format and then implement the logic to interpret the orientation data from the model's output in your application. This approach requires custom handling of the output tensors to manage the orientation.

For exporting YOLOv8 models to TensorFlow.js, you can follow the standard export procedure using the model.export(format='tfjs') method in Python.

If you need further guidance on handling the post-processing or any other specifics, feel free to ask!

[AlessandroB1298]

Hey! Thanks for responding! I am having trouble with post processing, I’m a little stuck on how I should read the data, and I know from other information that we get x,y,w,h,r however I’m not sure how to translate that to tensorflow js

…

On Thu, Jun 6, 2024 at 5:53 PM Glenn Jocher ***@***.***> wrote: Hello! Currently, TensorFlow.js does not natively support oriented bounding boxes (OBB) directly as part of its model operations. However, you can still use YOLOv8 models that output OBB by handling the orientation aspect in your post-processing logic after running the inference in TensorFlow.js. You would need to export your YOLOv8 model to the TensorFlow.js format and then implement the logic to interpret the orientation data from the model's output in your application. This approach requires custom handling of the output tensors to manage the orientation. For exporting YOLOv8 models to TensorFlow.js, you can follow the standard export procedure using the model.export(format='tfjs') method in Python. If you need further guidance on handling the post-processing or any other specifics, feel free to ask! — Reply to this email directly, view it on GitHub <#13413 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXO56Z3B2FQVBULTQYXTO6DZGDK6ZAVCNFSM6AAAAABI44V4XOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJTGQ3DKNJVG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[glenn-jocher]

Hello!

I'm glad to assist with the post-processing for OBB in TensorFlow.js! When you receive the output x, y, w, h, r from your model:

• x and y are the center coordinates of the bounding box.
• w and h are the width and height of the box.
• r represents the rotation of the box in radians.

To translate these into a usable format in TensorFlow.js, you can create a function to calculate the corners of the rotated rectangle based on these parameters. Here's a basic example of how you might implement this:

function calculateCorners(x, y, w, h, r) {
    const angle = -r; // Adjust rotation direction if necessary
    const corners = [];
    const dx = [1, 1, -1, -1];
    const dy = [1, -1, -1, 1];

    for (let i = 0; i < 4; i++) {
        const cx = w * dx[i] / 2;
        const cy = h * dy[i] / 2;
        const cosA = Math.cos(angle);
        const sinA = Math.sin(angle);
        const nx = cx * cosA - cy * sinA;
        const ny = cx * sinA + cy * cosA;
        corners.push([x + nx, y + ny]);
    }
    return corners;
}

This function calculates the corners of the box which you can then use to draw or further process in your application. If you have any more questions or need further clarification, feel free to reach out! 🚀

[AlessandroB1298]

This is really helpful! I have another question about how we process this data. I switched over to onnxruntime-web and when i run an inference using the following code it returns 457 boxes // Load the model using ort.inference session useEffect(() => { const loadModel = async () => { try { const model = await ort.InferenceSession.create("./best.onnx"); setModel(model); setLoading({ loading: false, progress: 1 }); console.log(model) } catch (error) { console.error("Error loading model:", error); } }; loadModel(); }, []); // will only run once, once it mounts // function to run model const run_model = async(input)=>{ // this takes a input array of pixels input = new ort.Tensor(Float32Array.from(input),[1, 3, 640, 640]); //defining our input, const outputs = await model.run({images:input}); //promises that the model will run return outputs["output0"].data; // returns raw output from neural network as a flat array of numbers }

…

On Thu, Jun 6, 2024 at 11:43 PM Glenn Jocher ***@***.***> wrote: Hello! I'm glad to assist with the post-processing for OBB in TensorFlow.js! When you receive the output x, y, w, h, r from your model: - x and y are the center coordinates of the bounding box. - w and h are the width and height of the box. - r represents the rotation of the box in radians. To translate these into a usable format in TensorFlow.js, you can create a function to calculate the corners of the rotated rectangle based on these parameters. Here's a basic example of how you might implement this: function calculateCorners(x, y, w, h, r) { const angle = -r; // Adjust rotation direction if necessary const corners = []; const dx = [1, 1, -1, -1]; const dy = [1, -1, -1, 1]; for (let i = 0; i < 4; i++) { const cx = w * dx[i] / 2; const cy = h * dy[i] / 2; const cosA = Math.cos(angle); const sinA = Math.sin(angle); const nx = cx * cosA - cy * sinA; const ny = cx * sinA + cy * cosA; corners.push([x + nx, y + ny]); } return corners;} This function calculates the corners of the box which you can then use to draw or further process in your application. If you have any more questions or need further clarification, feel free to reach out! 🚀 — Reply to this email directly, view it on GitHub <#13413 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXO56ZY4V3U5SUUJQMC2XP3ZGET7FAVCNFSM6AAAAABI44V4XOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJTHAYDEMRTGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[glenn-jocher]

Hello!

I'm glad to hear that the previous information was helpful! Let's dive into your new question regarding processing the data with onnxruntime-web.

When you run inference and get 457 boxes, it means the model has detected multiple objects. The raw output from the model typically includes bounding box coordinates, confidence scores, and class probabilities. You'll need to parse this output to extract meaningful information.

Here's a step-by-step guide to process the output:

1. Extract Bounding Boxes, Scores, and Classes:
   The output array contains the bounding box coordinates, confidence scores, and class probabilities. You need to reshape and interpret this data correctly.

2. Filter by Confidence Score:
   To reduce the number of boxes, filter out those with low confidence scores.

3. Non-Maximum Suppression (NMS):
   Apply NMS to eliminate redundant boxes that overlap significantly.

Here's an example of how you might implement this in JavaScript:

// Function to process the model output
const processOutput = (output, confidenceThreshold = 0.5, iouThreshold = 0.5) => {
    const boxes = [];
    const scores = [];
    const classes = [];

    // Assuming output is a flat array, reshape and extract data
    for (let i = 0; i < output.length; i += 6) {
        const score = output[i + 4];
        if (score > confidenceThreshold) {
            const x = output[i];
            const y = output[i + 1];
            const w = output[i + 2];
            const h = output[i + 3];
            const classId = output[i + 5];
            boxes.push([x, y, w, h]);
            scores.push(score);
            classes.push(classId);
        }
    }

    // Apply Non-Maximum Suppression (NMS)
    const nmsIndices = applyNMS(boxes, scores, iouThreshold);
    const finalBoxes = nmsIndices.map(index => boxes[index]);
    const finalScores = nmsIndices.map(index => scores[index]);
    const finalClasses = nmsIndices.map(index => classes[index]);

    return { boxes: finalBoxes, scores: finalScores, classes: finalClasses };
};

// Function to apply Non-Maximum Suppression (NMS)
const applyNMS = (boxes, scores, iouThreshold) => {
    // Implement NMS logic here
    // This is a placeholder for the actual NMS implementation
    return []; // Return indices of the boxes to keep
};

// Example usage
const runModel = async (input) => {
    input = new ort.Tensor(Float32Array.from(input), [1, 3, 640, 640]);
    const outputs = await model.run({ images: input });
    const rawOutput = outputs["output0"].data;
    const processedOutput = processOutput(rawOutput);

    console.log(processedOutput);
    return processedOutput;
};

In this example:

• processOutput function filters boxes based on a confidence threshold and applies NMS.
• applyNMS is a placeholder for the actual NMS implementation. You can use libraries like tf.image.nonMaxSuppression in TensorFlow.js or implement your own NMS logic.

Feel free to adjust the confidence and IoU thresholds as needed. If you have any more questions or need further assistance, don't hesitate to reach out! 😊

[github-actions]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐