GithubHelp home page GithubHelp logo

camera_tutorial's People

Contributors

ugu11 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

camera_tutorial's Issues

Error when build

I have cloned your respository and try to run it. But i met an error when build on ios/android:

../../development/flutter/.pub-cache/hosted/pub.dartlang.org/ffi-0.1.3/lib/src/utf8.dart:63:33: Error: The getter 'addressOf' isn't defined for the class 'Utf8'.
 - 'Utf8' is from 'package:ffi/src/utf8.dart' ('../../development/flutter/.pub-cache/hosted/pub.dartlang.org/ffi-0.1.3/lib/src/utf8.dart').
Try correcting the name to the name of an existing getter, or defining a getter or field named 'addressOf'.
  String toString() => fromUtf8(addressOf);
                                ^^^^^^^^^
../../development/flutter/.pub-cache/hosted/pub.dartlang.org/ffi-0.1.3/lib/src/utf8.dart:23:7: Error: Struct 'Utf8' is empty. Empty structs and unions are undefined behavior.
class Utf8 extends Struct {
      ^
../../development/flutter/.pub-cache/hosted/pub.dartlang.org/ffi-0.1.3/lib/src/utf16.dart:16:7: Error: Struct 'Utf16' is empty. Empty structs and unions are undefined behavior.
class Utf16 extends Struct {
      ^
../../development/flutter/.pub-cache/hosted/pub.dartlang.org/ffi-0.1.3/lib/src/allocation.dart:47:33: Error: Expected type 'T' to be a valid and instantiated subtype of 'NativeType'.
  final int totalSize = count * sizeOf<T>();
                                ^

Command PhaseScriptExecution failed with a nonzero exit code

Big thanks for your solution.

Convert uint32_t array to openCV Mat c++

Hello.
How can i convert your uint32_t *image to openCV Mat in c++?
I had converted but not successed
Here is my code in c++

cv::Mat intMat_RGBA = cv::Mat(cv::Size(oriImgWidth, oriImgHeight), CV_8UC4, image);
std::vector<cv::Mat> RGBA_channels;
cv::split(intMat_RGBA, RGBA_channels);

// remove the alpha channel:
RGBA_channels.pop_back();
std::reverse(RGBA_channels.begin(), RGBA_channels.end());

// and merge back to image:
cv::Mat intMat_BGR;
cv::merge(RGBA_channels, intMat_BGR);

Invalid argument(s): Failed to lookup symbol (dlsym(RTLD_DEFAULT, convertImage): symbol not found)

I met a problem with this function

Convert conv = convertImageLib
      .lookup<NativeFunction<convert_func>>('convertImage')
      .asFunction<Convert>();
[VERBOSE-2:ui_dart_state.cc(199)] Unhandled Exception: Invalid argument(s): Failed to lookup symbol (dlsym(RTLD_DEFAULT, convertImage): symbol not found)
#0      DynamicLibrary.lookup (dart:ffi-patch/ffi_dynamic_library_patch.dart:31:29)
#0      DynamicLibrary.lookup (dart:ffi-patch/ffi_dynamic_library_patch.dart:31:29)
#1      convertCameraImage
package:flutter_app/utils/cameraImage.util.dart:20
#2      _CameraState.handleStream.<anonymous closure>
package:flutter_app/main.dart:74
#3      _CameraState.handleStream.<anonymous closure>
package:flutter_app/main.dart:66
#4      CameraController.startImageStream.<anonymous closure>
package:camera/src/camera_controller.dart:395

iOS error

device: iPhone6s plus
iOS version: 13.3.1

Launching lib/main.dart on 干志雄的 iPhone in debug mode...
Found saved certificate choice "iPhone Developer: Guangjian Wang (N8AC8H42N4)". To clear, use "flutter config".
Signing iOS app for device deployment using developer identity: "iPhone Developer: Guangjian Wang (N8AC8H42N4)"
Running Xcode build...
Xcode build done.                                           13.1s
Installing and launching...                                        18.4s
Debug service listening on ws://localhost:1024/ws
Syncing files to device 干志雄的 iPhone...
flutter: ══╡ EXCEPTION CAUGHT BY WIDGETS LIBRARY ╞═══════════════════════════════════════════════════════════


flutter: The following ArgumentError was thrown building Builder:


flutter: Invalid argument(s): Failed to lookup symbol (dlsym(RTLD_DEFAULT, convertImage): symbol not found)


flutter: 


flutter: The relevant error-causing widget was:


flutter:   MaterialApp


flutter:   file:///Users/ganzhixiong/Documents/yilisheng/TemperatureSensor/test/camera_tutorial-master.zip%20Folder/camera_tutorial-master/lib/main.dart:23:12


flutter: 


flutter: When the exception was thrown, this was the stack:


flutter: #0      DynamicLibrary.lookup (dart:ffi-patch/ffi_dynamic_library_patch.dart:33:29)


flutter: #1      _MyHomePageState.initState (package:camera_tutorial/main.dart:57:28)


flutter: #2      StatefulElement._firstBuild (package:flutter/src/widgets/framework.dart:4644:58)


flutter: #3      ComponentElement.mount (package:flutter/src/widgets/framework.dart:4480:5)


flutter: ...     Normal element mounting (24 frames)


flutter: #27     Element.inflateWidget (package:flutter/src/widgets/framework.dart:3450:14)


flutter: #28     MultiChildRenderObjectElement.mount (package:flutter/src/widgets/framework.dart:5951:32)


flutter: ...     Normal element mounting (119 frames)


flutter: #147    Element.inflateWidget (package:flutter/src/widgets/framework.dart:3450:14)


flutter: #148    MultiChildRenderObjectElement.mount (package:flutter/src/widgets/framework.dart:5951:32)


flutter: ...     Normal element mounting (261 frames)


flutter: #409    Element.inflateWidget (package:flutter/src/widgets/framework.dart:3450:14)


flutter: #410    Element.updateChild (package:flutter/src/widgets/framework.dart:3218:18)


flutter: #411    RenderObjectToWidgetElement._rebuild (package:flutter/src/widgets/binding.dart:1129:16)


flutter: #412    RenderObjectToWidgetElement.mount (package:flutter/src/widgets/binding.dart:1100:5)


flutter: #413    RenderObjectToWidgetAdapter.attachToRenderTree.<anonymous closure> (package:flutter/src/widgets/binding.dart:1042:17)


flutter: #414    BuildOwner.buildScope (package:flutter/src/widgets/framework.dart:2607:19)


flutter: #415    RenderObjectToWidgetAdapter.attachToRenderTree (package:flutter/src/widgets/binding.dart:1041:13)


flutter: #416    WidgetsBinding.attachRootWidget (package:flutter/src/widgets/binding.dart:922:7)


flutter: #417    WidgetsBinding.scheduleAttachRootWidget.<anonymous closure> (package:flutter/src/widgets/binding.dart:903:7)


flutter: (elided 11 frames from class _RawReceivePortImpl, class _Timer, dart:async, and dart:async-patch)


flutter: 


flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════


Lost connection to device.
 

Conversion to images using either C or Dart takes time to compare

Why is using Dart(convertYUV420toImageColor) slower under release than debug?

Test Environment

phone: huawei Mate7
Android version: 6.0
CPU: Hisilicon Kirin 925

Let's look at the results first

debug result

  • using C elapsed time 170-190ms
    image

  • using Dart elapsed time 230-250ms
    image

release result

flutter run --release

  • using C elapsed time 35-46ms

  • using Dart elapsed time 431-593ms
    why release slow?

my code

import 'dart:ffi';
import 'dart:io';
import 'dart:typed_data';

import 'package:ffi/ffi.dart';
import 'package:flutter/material.dart';
import 'package:camera/camera.dart';
import 'package:image/image.dart' as imglib;

import 'ImagePreview.dart';
import 'image_converter.dart';

typedef convert_func = Pointer<Uint32> Function(
    Pointer<Uint8>, Pointer<Uint8>, Pointer<Uint8>, Int32, Int32, Int32, Int32);
typedef Convert = Pointer<Uint32> Function(Pointer<Uint8>, Pointer<Uint8>, Pointer<Uint8>, int, int, int, int);

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Camera App',
      theme: ThemeData(
        primarySwatch: Colors.blue,
//        visualDensity: VisualDensity.adaptivePlatformDensity,
      ),
      home: MyHomePage(title: 'Camera App'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  MyHomePage({Key key, this.title}) : super(key: key);
  final String title;

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  CameraController _camera;
  bool _cameraInitialized = false;
  CameraImage _savedImage;

  final DynamicLibrary convertImageLib =
      Platform.isAndroid ? DynamicLibrary.open("libconvertImage.so") : DynamicLibrary.process();
  Convert conv;

  @override
  void initState() {
    super.initState();
    _initializeCamera();

    // Load the convertImage() function from the library
    conv = convertImageLib.lookup<NativeFunction<convert_func>>('convertImage').asFunction<Convert>();
  }

  void _initializeCamera() async {
    // Get list of cameras of the device
    List<CameraDescription> cameras = await availableCameras();

    // Create the CameraController
    _camera = new CameraController(cameras[0], ResolutionPreset.medium);
    _camera.initialize().then((_) async {
      // Start ImageStream
      await _camera.startImageStream((CameraImage image) => _processCameraImage(image));
      setState(() {
        _cameraInitialized = true;
      });
    });
  }

  void _processCameraImage(CameraImage image) async {
    setState(() {
      _savedImage = image;
    });
  }

  List<int> _durations = [];

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: Stack(
        children: [
          Center(
              child: (_cameraInitialized)
                  ? AspectRatio(
                      aspectRatio: _camera.value.aspectRatio,
                      child: CameraPreview(_camera),
                    )
                  : CircularProgressIndicator()),
          Center(
              child: CircularProgressIndicator(
            backgroundColor: Colors.red,
          ))
        ],
      ),
      floatingActionButton: Row(
        mainAxisAlignment: MainAxisAlignment.spaceEvenly,
        children: [
          RaisedButton(
            onPressed: () async {
//              print('''
//
//tap FloatingActionButton get img
//''');
              // The jump has animation, and all the pushing that you see is not going very fast
//          Navigator.push(context, MaterialPageRoute(builder:
//            (context) => new ImagePreview(img: img)));
              await _convertImageByC();
            },
            child: Text('ConvertImageByC'),
          ),
          RaisedButton(
            onPressed: () async {
              await _convertImageByDart();
            },
            child: Text('ConvertImageByDart'),
          ),
        ],
      ), // This trailing comma makes auto-formatting nicer for build methods.
      floatingActionButtonLocation: FloatingActionButtonLocation.centerFloat,
    );
  }

  _convertImageByC() {
    Stopwatch stopwatch = Stopwatch();
    stopwatch.start();

    // Allocate memory for the 3 planes of the image
    Pointer<Uint8> p = allocate(count: _savedImage.planes[0].bytes.length);
    Pointer<Uint8> p1 = allocate(count: _savedImage.planes[1].bytes.length);
    Pointer<Uint8> p2 = allocate(count: _savedImage.planes[2].bytes.length);

    // Assign the planes data to the pointers of the image
    Uint8List pointerList = p.asTypedList(_savedImage.planes[0].bytes.length);
    Uint8List pointerList1 = p1.asTypedList(_savedImage.planes[1].bytes.length);
    Uint8List pointerList2 = p2.asTypedList(_savedImage.planes[2].bytes.length);
    pointerList.setRange(0, _savedImage.planes[0].bytes.length, _savedImage.planes[0].bytes);
    pointerList1.setRange(0, _savedImage.planes[1].bytes.length, _savedImage.planes[1].bytes);
    pointerList2.setRange(0, _savedImage.planes[2].bytes.length, _savedImage.planes[2].bytes);

    // Call the convertImage function and convert the YUV to RGB
    Pointer<Uint32> imgP = conv(p, p1, p2, _savedImage.planes[1].bytesPerRow, _savedImage.planes[1].bytesPerPixel,
        _savedImage.width, _savedImage.height);

    // Get the pointer of the data returned from the function to a List
    List imgData = imgP.asTypedList((_savedImage.width * _savedImage.height));

    stopwatch.stop();
    int getImgDataTime = stopwatch.elapsedMilliseconds;

    stopwatch.reset();
    stopwatch.start();

    // Generate image from the converted data
    imglib.Image img = imglib.Image.fromBytes(_savedImage.height, _savedImage.width, imgData);

    // Free the memory space allocated
    // from the planes and the converted data
    free(p);
    free(p1);
    free(p2);
    free(imgP);

    stopwatch.stop();
    print(
        'get imgData: ${getImgDataTime}ms, imglib.Image.fromBytes: ${stopwatch.elapsedMilliseconds}ms, total: ${getImgDataTime + stopwatch.elapsedMilliseconds}ms');

    _durations.add(getImgDataTime + stopwatch.elapsedMilliseconds);
    if (_durations.length == 10) {
      int totalMilliseconds = 0;
      for (var value in _durations) {
        totalMilliseconds += value;
      }
      double averageMilliseconds = totalMilliseconds / 10;
      print('Ten takes an average time: ${averageMilliseconds}ms');
      _durations.clear();
    }
  }

  _convertImageByDart() async {
    Stopwatch stopwatch = Stopwatch();
    stopwatch.start();

    await convertYUV420toImageColor(_savedImage);

    stopwatch.stop();
    print('convertYUV420toImageColor time: ${stopwatch.elapsedMilliseconds}ms');

    _durations.add(stopwatch.elapsedMilliseconds);
    if (_durations.length == 10) {
      int totalMilliseconds = 0;
      for (var value in _durations) {
        totalMilliseconds += value;
      }
      double averageMilliseconds = totalMilliseconds / 10;
      print('Ten takes an average time: ${averageMilliseconds}ms');
      _durations.clear();
    }
  }

  static const shift = (0xFF << 24);

  Future<List<int>> convertYUV420toImageColor(CameraImage image) async {
    try {
      final int width = image.width;
      final int height = image.height;

      final int uvRowStride = image.planes[1].bytesPerRow;
      final int uvPixelStride = image.planes[1].bytesPerPixel;

//    print("uvRowStride: " + uvRowStride.toString());
//    print("uvPixelStride: " + uvPixelStride.toString());

      // imgLib -> Image package from https://pub.dartlang.org/packages/image
      var img = imglib.Image(width, height); // Create Image buffer

      // Fill image buffer with plane[0] from YUV420_888
      for (int x = 0; x < width; x++) {
        for (int y = 0; y < height; y++) {
          final int uvIndex = uvPixelStride * (x / 2).floor() + uvRowStride * (y / 2).floor();
          final int index = y * width + x;

          final yp = image.planes[0].bytes[index];
          final up = image.planes[1].bytes[uvIndex];
          final vp = image.planes[2].bytes[uvIndex];
          // Calculate pixel color
          int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
          int g = (yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91).round().clamp(0, 255);
          int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);
          // color: 0x FF  FF  FF  FF
          //           A   B   G   R
          img.data[index] = shift | (b << 16) | (g << 8) | r;
        }
      }
      return img.getBytes();
    } catch (e) {
      print(">>>>>>>>>>>> ERROR:" + e.toString());
    }
    return null;
  }
}

Live streaming

Hey Hugand. Love your work. So I've been trying to send live frames to server after converting to base64. But the conversion from yuv to normal image for every image captured takes a lot of time. Is it possible to make it real-time? I just want to send every frame to the server and I can't convert yuv or bgra directly to base64 . Please help.

why use c will be a lot faster, Is the dart vm the problem? Or something else?

I looked at the code for your converter. C file,It is almost the same as the code I found from this issue(https://github.com/flutter/flutter/issues/26348#issuecomment-462321428), why is the code of c fast
Your code:

uint32_t *convertImage(uint8_t *plane0, uint8_t *plane1, uint8_t *plane2, int bytesPerRow, int bytesPerPixel, int width, int height){
     int hexFF = 255;
    int x, y, uvIndex, index;
    int yp, up, vp;
    int r, g, b;
    int rt, gt, bt;

    uint32_t *image = malloc(sizeof(uint32_t) * (width * height));

    for(x = 0; x < width; x++){
        for(y = 0; y < height; y++){
            
            uvIndex = bytesPerPixel * ((int) floor(x/2)) + bytesPerRow * ((int) floor(y/2));
            index = y*width+x;

            yp = plane0[index];
            up = plane1[uvIndex];
            vp = plane2[uvIndex];
            rt = round(yp + vp * 1436 / 1024 - 179);
            gt = round(yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91);
            bt = round(yp + up * 1814 / 1024 - 227);
            r = clamp(0, 255, rt);
            g = clamp(0, 255, gt);
            b = clamp(0, 255, bt);
            image[getRotatedImageByteIndex(y, x, height)] = (hexFF << 24) | (b << 16) | (g << 8) | r;
        }
    }
    return image;
}

my code

Future<Image> convertYUV420toImageColor(CameraImage image) async {
      try {
        final int width = image.width;
        final int height = image.height;
        final int uvRowStride = image.planes[1].bytesPerRow;
        final int uvPixelStride = image.planes[1].bytesPerPixel;

        print("uvRowStride: " + uvRowStride.toString());
        print("uvPixelStride: " + uvPixelStride.toString());
        
        // imgLib -> Image package from https://pub.dartlang.org/packages/image
        var img = imglib.Image(width, height); // Create Image buffer

        // Fill image buffer with plane[0] from YUV420_888
        for(int x=0; x < width; x++) {
          for(int y=0; y < height; y++) {
            final int uvIndex = uvPixelStride * (x/2).floor() + uvRowStride*(y/2).floor();
            final int index = y * width + x;

            final yp = image.planes[0].bytes[index];
            final up = image.planes[1].bytes[uvIndex];
            final vp = image.planes[2].bytes[uvIndex];
            // Calculate pixel color
            int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
            int g = (yp - up * 46549 / 131072 + 44 -vp * 93604 / 131072 + 91).round().clamp(0, 255);
            int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);     
            // color: 0x FF  FF  FF  FF 
            //           A   B   G   R
            img.data[index] = (0xFF << 24) | (b << 16) | (g << 8) | r;
          }
        }

        imglib.PngEncoder pngEncoder = new imglib.PngEncoder(level: 0, filter: 0);
        List<int> png = pngEncoder.encodeImage(img);
        muteYUVProcessing = false;
        return Image.memory(png);  
      } catch (e) {
        print(">>>>>>>>>>>> ERROR:" + e.toString());
      }
      return null;
  }

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.