ifm3d::buffer_id::XYZ to cv::Mat conversion

[520liangqiujuan]

Hi!
I'm wondering if there's an example on how to convert the ifm3d::buffer_id::XYZ to an cv::Mat?
I want to be able to do something like this:
cv::Mat xyz = frame.get()->GetBuffer(ifm3d::buffer_id::XYZ);

[lola-masson]

Hi @520liangqiujuan,
Assuming you are using an O3R, you can do something like this. If using another device, just replace the O3R() with the corresponding class.

/*
 * Copyright 2021-present ifm electronic, gmbh
 * SPDX-License-Identifier: Apache-2.0
 */
#include <chrono>
#include <iostream>
#include <thread>
#include <ifm3d/device/o3r.h>
#include <ifm3d/fg.h>
#include <opencv4/opencv2/core/core.hpp>

using namespace ifm3d::literals;
using namespace std::chrono_literals;

// LUT for image format conversion
static std::unordered_map<ifm3d::pixel_format, int> LUT_TYPE{
    {ifm3d::pixel_format::FORMAT_8U, CV_8U},
    {ifm3d::pixel_format::FORMAT_8S, CV_8S},
    {ifm3d::pixel_format::FORMAT_16U, CV_16U},
    {ifm3d::pixel_format::FORMAT_16S, CV_16S},
    {ifm3d::pixel_format::FORMAT_32S, CV_32S},
    {ifm3d::pixel_format::FORMAT_32F, CV_32F},
    {ifm3d::pixel_format::FORMAT_32F3, CV_32F},
    {ifm3d::pixel_format::FORMAT_64F, CV_64F}};

cv::Mat ConvertImageToMatCopy(ifm3d::Buffer &img) {
  auto mat = cv::Mat(img.height(), img.width(), CV_MAKETYPE(LUT_TYPE[img.dataFormat()], 3));
  std::memcpy(mat.data, img.ptr(0),
              img.width() * img.height() * LUT_SIZE[img.dataFormat()] * 3);
  return mat;
}

int main() {

  auto dev = std::make_shared<ifm3d::O3R>();

  const auto pcic_port =
      dev->Port("port2").pcic_port;
  auto fg = std::make_shared<ifm3d::FrameGrabber>(dev, pcic_port);

  // Set Schema and start the grabber
  fg->Start({ifm3d::buffer_id::XYZ});
  
  auto future = fg->WaitForFrame();
  if (future.wait_for(3s) != std::future_status::ready) {
    std::cerr << "Timeout waiting for camera!" << std::endl;
    return -1;
  }
  auto frame = future.get();

  auto xyz = frame->GetBuffer(ifm3d::buffer_id::XYZ);
  cv::Mat xyz_mat = ConvertImageToMatCopy(xyz);

  std::cout << "Number of channels: " << xyz_mat.channels() << std::endl;
  std::cout << "X value at pixel (100, 100): " << xyz_mat.at<cv::Vec3f>(100, 100)[0] << std::endl;
  std::cout << "Y value at pixel (100, 100): " << xyz_mat.at<cv::Vec3f>(100, 100)[1] << std::endl;
  std::cout << "Z value at pixel (100, 100): " << xyz_mat.at<cv::Vec3f>(100, 100)[2] << std::endl;

  fg->Stop();

  return 0;
}

[520liangqiujuan]

My camera is an O3D303 and I define LUT_SIZE as follows

static std::unordered_map<ifm3d::pixel_format, size_t> LUT_SIZE{
    {ifm3d::pixel_format::FORMAT_8U, 1},
    {ifm3d::pixel_format::FORMAT_8S, 1},
    {ifm3d::pixel_format::FORMAT_16U, 2},
    {ifm3d::pixel_format::FORMAT_16S, 2},
    {ifm3d::pixel_format::FORMAT_32S, 4},
    {ifm3d::pixel_format::FORMAT_32F, 4},
    {ifm3d::pixel_format::FORMAT_32F3, 12},
    {ifm3d::pixel_format::FORMAT_64F, 8}
};

The result of the run is as follows, is this normal?

Number of channels: 3
X value at pixel (100, 100): -nan
Y value at pixel (100, 100): 4.86859e-39
Z value at pixel (100, 100): 8.46348e-37
Number of channels: 3
X value at pixel (100, 100): -nan
Y value at pixel (100, 100): 4.96044e-39
Z value at pixel (100, 100): 9.46265e-37

[lola-masson]

Sorry about that I forgot to include the LUT_SIZE. It should be:

// LUT for image format size
static std::unordered_map<ifm3d::pixel_format, int> LUT_SIZE{
    {ifm3d::pixel_format::FORMAT_8U, 1},
    {ifm3d::pixel_format::FORMAT_8S, 1},
    {ifm3d::pixel_format::FORMAT_16U, 2},
    {ifm3d::pixel_format::FORMAT_16S, 2},
    {ifm3d::pixel_format::FORMAT_32S, 4},
    {ifm3d::pixel_format::FORMAT_32F, 4},
    {ifm3d::pixel_format::FORMAT_32F3, 4},
    {ifm3d::pixel_format::FORMAT_64F, 8}};

This is probably the cause of the weird results you get. Let me know if it works better with this LUT_SIZE.