Runtime Version Detected Sim always same even changed the person about lite.ai.toolkit

[DefTruth]

you should also change the new image path to test_onnxruntime function in test_lite_glint_arcface.cpp. It's seems that Default Version Detected Sim and Default Version Detected Sim is ok. you can try the latest version of lite.ai.toolkit? some bug were fixed. and can you should me the src code you have tried? like:

static void test_default()
{
  std::string onnx_path = "../../../hub/onnx/cv/ms1mv3_arcface_r100.onnx";
  std::string test_img_path0 = "../../../examples/lite/resources/test_lite_faceid_0.png";
  std::string test_img_path1 = "../../../examples/lite/resources/test_lite_faceid_1.png";

  lite::cv::faceid::GlintArcFace *glint_arcface = new lite::cv::faceid::GlintArcFace(onnx_path);

  lite::types::FaceContent face_content0, face_content1;
  cv::Mat img_bgr0 = cv::imread(test_img_path0);
  cv::Mat img_bgr1 = cv::imread(test_img_path1);
  glint_arcface->detect(img_bgr0, face_content0);
  glint_arcface->detect(img_bgr1, face_content1);

  if (face_content0.flag && face_content1.flag)
  {
    float sim = lite::utils::math::cosine_similarity<float>(
        face_content0.embedding, face_content1.embedding);
    std::cout << "Default Version Detected Sim: " << sim << std::endl;
  }

  delete glint_arcface;
}

static void test_onnxruntime()
{
#ifdef ENABLE_ONNXRUNTIME
  std::string onnx_path = "../../../hub/onnx/cv/ms1mv3_arcface_r100.onnx";
  std::string test_img_path0 = "../../../examples/lite/resources/test_lite_faceid_0.png";
  std::string test_img_path1 = "../../../examples/lite/resources/test_lite_faceid_2.png";

  lite::onnxruntime::cv::faceid::GlintArcFace *glint_arcface =
      new lite::onnxruntime::cv::faceid::GlintArcFace(onnx_path);

  lite::types::FaceContent face_content0, face_content1;
  cv::Mat img_bgr0 = cv::imread(test_img_path0);
  cv::Mat img_bgr1 = cv::imread(test_img_path1);
  glint_arcface->detect(img_bgr0, face_content0);
  glint_arcface->detect(img_bgr1, face_content1);

  if (face_content0.flag && face_content1.flag)
  {
    float sim = lite::utils::math::cosine_similarity<float>(
        face_content0.embedding, face_content1.embedding);
    std::cout << "ONNXRuntime Version Detected Sim: " << sim << std::endl;
  }

  delete glint_arcface;
#endif
}

[MyraBaba]

@DefTruth thanks for the fix and your help.

I am trying to use also :

antelopev2 | RetinaFace-10GF | ResNet100@Glint360K | 2d106 & 3d68 | Gender&Age | 407MB
buffalo_l | RetinaFace-10GF | ResNet50@WebFace600K | 2d106 & 3d68 | Gender&Age | 326MB
buffalo_m | RetinaFace-2.5GF | ResNet50@WebFace600K | 2d106 & 3d68 | Gender&Age | 313MB

antelopev2 onnx gives error when I put above test_lite_glint_arcface.cpp

libc++abi.dylib: terminating with uncaught exception of type std::length_error: vector

which sample that we can use above Retina models.

Best

THX

[MyraBaba]

@DefTruth
By the way all face feature extractor assumes face is already aligned and 112X112 size and just right ?

[DefTruth]

@DefTruth By the way all face feature extractor assumes face is already aligned and 112X112 size and just right ?

the src code of GlintArcFace:

Ort::Value GlintArcFace::transform(const cv::Mat &mat)
{
  cv::Mat canva = mat.clone();
  cv::resize(canva, canva, cv::Size(input_node_dims.at(3),
                                    input_node_dims.at(2)));
  cv::cvtColor(canva, canva, cv::COLOR_BGR2RGB);
  // (1,3,112,112)
  ortcv::utils::transform::normalize_inplace(canva, mean_val, scale_val);
  return ortcv::utils::transform::create_tensor(
      canva, input_node_dims, memory_info_handler,
      input_values_handler, ortcv::utils::transform::CHW);
}

void GlintArcFace::detect(const cv::Mat &mat, types::FaceContent &face_content)
{
  if (mat.empty()) return;
  // 1. make input tensor
  Ort::Value input_tensor = this->transform(mat);
  // 2. inference
  auto output_tensors = ort_session->Run(
      Ort::RunOptions{nullptr}, input_node_names.data(),
      &input_tensor, 1, output_node_names.data(), num_outputs
  );
  Ort::Value &embedding = output_tensors.at(0);
  auto embedding_dims = output_node_dims.at(0); // (1,512)
  const unsigned int hidden_dim = embedding_dims.at(1); // 512
  const float *embedding_values = embedding.GetTensorMutableData<float>();
  std::vector<float> embedding_norm(embedding_values, embedding_values + hidden_dim);
  cv::normalize(embedding_norm, embedding_norm); // l2 normalize
  face_content.embedding.assign(embedding_norm.begin(), embedding_norm.end());
  face_content.dim = hidden_dim;
  face_content.flag = true;
}

the input size of the model will be auto detected by ONNXRuntime, see the src code in ort_handler.cpp:

  // 3. type info.
  Ort::TypeInfo type_info = ort_session->GetInputTypeInfo(0);
  auto tensor_info = type_info.GetTensorTypeAndShapeInfo();
  input_tensor_size = 1;
  input_node_dims = tensor_info.GetShape();  // auto detect input size
  for (unsigned int i = 0; i < input_node_dims.size(); ++i)
    input_tensor_size *= input_node_dims.at(i);
  input_values_handler.resize(input_tensor_size);

and, you can only load the specific onnx files provide by this repo into GlintArcFace class because of the post-process of each specific model is very different and have been "freeze" after the onnx file done exported, otherwise, you will be encounter unknow errors that i can't fix if you try to load an another model file(not provide by my repo) into lite::cv::faceid::GlintArcFace. the onnx files listed below can be load into GlintArcFace, all is well in my device. download these files from lite.ai.toolkit.hub.onnx.md , more examples can be found at examples/lite/cv

Class	Pretrained ONNX Files	Rename or Converted From (Repo)	Size
lite::cv::faceid::GlintArcFace	ms1mv3_arcface_r100.onnx	insightface	248Mb
lite::cv::faceid::GlintArcFace	ms1mv3_arcface_r50.onnx	insightface	166Mb
lite::cv::faceid::GlintArcFace	ms1mv3_arcface_r34.onnx	insightface	130Mb
lite::cv::faceid::GlintArcFace	ms1mv3_arcface_r18.onnx	insightface	91Mb

can you should me the code, i really don't know want the problem you have encounter.

[MyraBaba]

@DefTruth
code is test_lite_glint_arcface.cpp
model is :

std::string onnx_path = "/Users/tulpar/Projects/lite.ai/hub/onnx/insightmodels/buffalo_l/w600k_r50.onnx";

Gdrive for models:

models
you should reimplement the c++ inference process if you want to use the model converted by your self, You can refer to the c++ reimplementation of GlintArcFace in my repo, i hope it helps~

[MyraBaba]

@DefTruth
Hi I tried to load glintr100.onnx model from insight-face but GlintArcFace gives : libc++abi.dylib: terminating with uncaught exception of type std::length_error: vector

[DefTruth]

@DefTruth Hi I tried to load glintr100.onnx model from insight-face but GlintArcFace gives : libc++abi.dylib: terminating with uncaught exception of type std::length_error: vector

try to check the index of hidden dims in your onnx model is [1] or not ?

const unsigned int hidden_dim = embedding_dims.at(1); // 512 assume the `hidden_dim` is locate at index [1]
const float *embedding_values = embedding.GetTensorMutableData<float>();
std::vector<float> embedding_norm(embedding_values, embedding_values + hidden_dim); // std::length_error: vector might be encounter here if the hidden_dim is wrong

[MyraBaba]

Hi,

I am fairly new to onnx. But models seems different. I attached the screen shot.

Alsso attached gdrive for model : https://drive.google.com/file/d/1fENaqD2rol9a_nninDVnVlERIOL9X_To/view?usp=sharing

[DefTruth]

[None,3,112,112], the 'None' means that the input batch-size is dynamic, my code does't support dynamic shape now. may be you can export you onnx models with static input shape, let the batch-size=1 and don't use 'dynamic_axes' parameters when you are exporting onnx file with torch.onnx.export API.

[MyraBaba]

@DefTruth

I am cross checking the insightFace antelopev2 (https://github.com/deepinsight/insightface/tree/master/model_zoo) glintr100.onnx with the both glint360k_cosface_r100.onnx (from your repo) and mine glintr100.onnx converted to shape 1x3x112x122 and readable by your lite c++.

I am using same 112x112 aligned image :

python insightface antelope version gives different feature result than the c++ version of yours.

is that normal ? Or How I can get the same feature (512) both python insightface glintr100.onnx and yours c++ version ?

Which example and which model ?

Best

[MyraBaba]

@DefTruth

What is the best way to detect for sleeping driver ? Face landmark ? Which one is good if you have experience ?

We want to use for our NgO drivers

Best