問題描述

我在 OpenCV 的幫助下從網絡攝像機捕獲多個流.當我嘗試從 OpenCV 窗口(cv::namedWindow(...))顯示這些流時，它可以正常工作(到目前為止我已經嘗試了 4 個流).

I'm capturing multiple streams from ip cameras with the help of OpenCV. When i try to display these stream from an OpenCV window(cv::namedWindow(...)), it works without any problem (i have tried up to 4 streams so far).

當我嘗試在 Qt 小部件中顯示這些流時出現問題.由于捕獲是在另一個線程中完成的，我必須使用信號槽機制來更新 QWidget(在主線程中).

The problem arises when i try to show these streams inside a Qt widget. Since the capturing is done in another thread, i have to use the signal slot mechanism in order to update the QWidget(which is in main thread).

基本上，我從捕獲線程發出新捕獲的幀，GUI 線程中的一個插槽捕獲它.當我打開 4 個流時，我無法像以前那樣流暢地顯示視頻.

Basically, i emit the newly captured frame from the capture thread and a slot in the GUI thread catches it. When i open 4 streams, i can not display the videos smoothly like before.

這是發射器:

void capture::start_process() {
    m_enable = true;
    cv::Mat frame;

    while(m_enable) {
        if (!m_video_handle->read(frame)) {
            break;
        }
        cv::cvtColor(frame, frame,CV_BGR2RGB);

        qDebug() << "FRAME : " << frame.data;

        emit image_ready(QImage(frame.data, frame.cols, frame.rows, frame.step, QImage::Format_RGB888));
        cv::waitKey(30);
    }
}

這是我的插槽:

void widget::set_image(QImage image) {
    img = image;
    qDebug() << "PARAMETER IMAGE: " << image.scanLine(0);
    qDebug() << "MEMBER IMAGE: " << img.scanLine(0);
}

問題似乎是連續復制 QImages 的開銷.雖然 QImage 使用了隱式共享，但是當我通過 qDebug() 消息比較圖像的數據指針時，我看到了不同的地址.

The problem seems like the overhead of copying QImages continuously. Although QImage uses implicit sharing, when i compare the data pointers of images via qDebug() messages, i see different addresses.

1- 有沒有辦法將 OpenCV 窗口直接嵌入到 QWidget 中?

1- Is there any way to embed OpenCV window directly into QWidget ?

2- 處理顯示多個視頻的最有效方法是什么?例如，視頻管理系統如何同時顯示多達 32 個攝像頭?

2- What is the most efficient way to handle displaying multiple videos? For example, how video management systems show up to 32 cameras in the same time ?

3- 必須要走的路是什么?

3- What must be the way to go ?

推薦答案

使用 QImage::scanLine 強制深度復制，所以至少應該使用 constScanLine，或者更好的是，將插槽的簽名更改為:

Using QImage::scanLine forces a deep copy, so at the minimum, you should use constScanLine, or, better yet, change the slot's signature to:

void widget::set_image(const QImage & image);

當然，您的問題隨后變成了其他問題:QImage 實例指向位于另一個線程中的幀的數據，并且可以(并且將)隨時更改.

Of course, your problem then becomes something else: the QImage instance points to the data of a frame that lives in another thread, and can (and will) change at any moment.

對此有一個解決方案:需要使用堆上分配的新幀，并且需要在 QImage 中捕獲該幀.QScopedPointer 用于防止內存泄漏，直到 QImage 獲得幀的所有權.

There is a solution for that: one needs to use fresh frames allocated on the heap, and the frame needs to be captured within QImage. QScopedPointer is used to prevent memory leaks until the QImage takes ownership of the frame.

static void matDeleter(void* mat) { delete static_cast<cv::Mat*>(mat); }

class capture {
   Q_OBJECT
   bool m_enable;
   ...
public:
   Q_SIGNAL void image_ready(const QImage &);
   ...
};

void capture::start_process() {
  m_enable = true;
  while(m_enable) {
    QScopedPointer<cv::Mat> frame(new cv::Mat);
    if (!m_video_handle->read(*frame)) {
      break;
    }
    cv::cvtColor(*frame, *frame, CV_BGR2RGB);

    // Here the image instance takes ownership of the frame.
    const QImage image(frame->data, frame->cols, frame->rows, frame->step,
                       QImage::Format_RGB888, matDeleter, frame.take());       
    emit image_ready(image);
    cv::waitKey(30);
  }
}

當然，由于Qt默認在QThread中提供了本地消息分發和一個Qt事件循環，使用QObject很簡單> 用于捕獲過程.下面是一個完整的、經過測試的示例.

Of course, since Qt provides native message dispatch and a Qt event loop by default in a QThread, it's a simple matter to use QObject for the capture process. Below is a complete, tested example.

捕獲、轉換和查看器都在它們自己的線程中運行.由于 cv::Mat 是一個具有原子、線程安全訪問的隱式共享類，所以它是這樣使用的.

The capture, conversion and viewer all run in their own threads. Since cv::Mat is an implicitly shared class with atomic, thread-safe access, it's used as such.

轉換器具有不處理陳舊幀的選項 - 如果轉換僅用于顯示目的，則非常有用.

The converter has an option of not processing stale frames - useful if conversion is only done for display purposes.

查看器在 gui 線程中運行并正確丟棄過時的幀.觀眾永遠沒有理由處理陳舊的幀.

The viewer runs in the gui thread and correctly drops stale frames. There's never a reason for the viewer to deal with stale frames.

如果您要收集數據以保存到磁盤，則應以高優先級運行捕獲線程.您還應該檢查 OpenCV api，看看是否有辦法將本機相機數據轉儲到磁盤.

If you were to collect data to save to disk, you should run the capture thread at high priority. You should also inspect OpenCV apis to see if there's a way of dumping the native camera data to disk.

為了加快轉換速度，您可以使用 OpenCV 中的 GPU 加速類.

To speed up conversion, you could use the gpu-accelerated classes in OpenCV.

下面的示例確保除非復制需要，否則不會重新分配任何內存:Capture 類維護自己的幀緩沖區，該緩沖區可用于每個后續幀，Converter，ImageViewer 也是如此.

The example below makes sure that in none of the memory is reallocated unless necessary for a copy: the Capture class maintains its own frame buffer that is reused for each subsequent frame, so does the Converter, and so does the ImageViewer.

制作了兩個圖像數據的深層副本(除了 cv::VideoCatpure::read 內部發生的任何事情):

There are two deep copies of image data made (besides whatever happens internally in cv::VideoCatprure::read):

1. 復制到Converter的QImage.

復制到ImageViewer的QImage.

需要兩個副本來確保線程之間的解耦并防止由于需要分離具有更高引用計數的 cv::Mat 或 QImage 而導致的數據重新分配比 1. 在現代架構上，內存復制非常快.

Both copies are needed to assure decoupling between the threads and prevent data reallocation due to the need to detach a cv::Mat or QImage that has the reference count higher than 1. On modern architectures, memory copies are very fast.

由于所有圖像緩沖區都位于相同的內存位置，因此它們的性能是最佳的 - 它們保持分頁和緩存狀態.

Since all image buffers stay in the same memory locations, their performance is optimal - they stay paged in and cached.

AddressTracker 用于跟蹤內存重新分配以進行調試.

The AddressTracker is used to track memory reallocations for debugging purposes.

// https://github.com/KubaO/stackoverflown/tree/master/questions/opencv-21246766
#include <QtWidgets>
#include <algorithm>
#include <opencv2/opencv.hpp>

Q_DECLARE_METATYPE(cv::Mat)

struct AddressTracker {
   const void *address = {};
   int reallocs = 0;
   void track(const cv::Mat &m) { track(m.data); }
   void track(const QImage &img) { track(img.bits()); }
   void track(const void *data) {
      if (data && data != address) {
         address = data;
         reallocs ++;
      }
   }
};

Capture 類用捕獲的幀填充內部幀緩沖區.它通知幀更改.框架是類的用戶屬性.

The Capture class fills the internal frame buffer with the captured frame. It notifies of a frame change. The frame is the user property of the class.

class Capture : public QObject {
   Q_OBJECT
   Q_PROPERTY(cv::Mat frame READ frame NOTIFY frameReady USER true)
   cv::Mat m_frame;
   QBasicTimer m_timer;
   QScopedPointer<cv::VideoCapture> m_videoCapture;
   AddressTracker m_track;
public:
   Capture(QObject *parent = {}) : QObject(parent) {}
   ~Capture() { qDebug() << __FUNCTION__ << "reallocations" << m_track.reallocs; }
   Q_SIGNAL void started();
   Q_SLOT void start(int cam = {}) {
      if (!m_videoCapture)
         m_videoCapture.reset(new cv::VideoCapture(cam));
      if (m_videoCapture->isOpened()) {
         m_timer.start(0, this);
         emit started();
      }
   }
   Q_SLOT void stop() { m_timer.stop(); }
   Q_SIGNAL void frameReady(const cv::Mat &);
   cv::Mat frame() const { return m_frame; }
private:
   void timerEvent(QTimerEvent * ev) {
      if (ev->timerId() != m_timer.timerId()) return;
      if (!m_videoCapture->read(m_frame)) { // Blocks until a new frame is ready
         m_timer.stop();
         return;
      }
      m_track.track(m_frame);
      emit frameReady(m_frame);
   }
};

Converter 類將傳入的幀轉換為縮小的 QImage 用戶屬性.它通知圖像更新.保留圖像以防止內存重新分配.processAll 屬性選擇是轉換所有幀，還是只有最近的幀應該有多個排隊.

The Converter class converts the incoming frame to a scaled-down QImage user property. It notifies of the image update. The image is retained to prevent memory reallocations. The processAll property selects whether all frames will be converted, or only the most recent one should more than one get queued up.

class Converter : public QObject {
   Q_OBJECT
   Q_PROPERTY(QImage image READ image NOTIFY imageReady USER true)
   Q_PROPERTY(bool processAll READ processAll WRITE setProcessAll)
   QBasicTimer m_timer;
   cv::Mat m_frame;
   QImage m_image;
   bool m_processAll = true;
   AddressTracker m_track;
   void queue(const cv::Mat &frame) {
      if (!m_frame.empty()) qDebug() << "Converter dropped frame!";
      m_frame = frame;
      if (! m_timer.isActive()) m_timer.start(0, this);
   }
   void process(const cv::Mat &frame) {
      Q_ASSERT(frame.type() == CV_8UC3);
      int w = frame.cols / 3.0, h = frame.rows / 3.0;
      if (m_image.size() != QSize{w,h})
         m_image = QImage(w, h, QImage::Format_RGB888);
      cv::Mat mat(h, w, CV_8UC3, m_image.bits(), m_image.bytesPerLine());
      cv::resize(frame, mat, mat.size(), 0, 0, cv::INTER_AREA);
      cv::cvtColor(mat, mat, CV_BGR2RGB);
      emit imageReady(m_image);
   }
   void timerEvent(QTimerEvent *ev) {
      if (ev->timerId() != m_timer.timerId()) return;
      process(m_frame);
      m_frame.release();
      m_track.track(m_frame);
      m_timer.stop();
   }
public:
   explicit Converter(QObject * parent = nullptr) : QObject(parent) {}
   ~Converter() { qDebug() << __FUNCTION__ << "reallocations" << m_track.reallocs; }
   bool processAll() const { return m_processAll; }
   void setProcessAll(bool all) { m_processAll = all; }
   Q_SIGNAL void imageReady(const QImage &);
   QImage image() const { return m_image; }
   Q_SLOT void processFrame(const cv::Mat &frame) {
      if (m_processAll) process(frame); else queue(frame);
   }
};

ImageViewer 小部件相當于存儲像素圖的 QLabel.圖像是查看器的用戶屬性.傳入的圖像被深度復制到用戶屬性中，以防止內存重新分配.

The ImageViewer widget is the equivalent of a QLabel storing a pixmap. The image is the user property of the viewer. The incoming image is deep-copied into the user property, to prevent memory reallocations.

class ImageViewer : public QWidget {
   Q_OBJECT
   Q_PROPERTY(QImage image READ image WRITE setImage USER true)
   bool painted = true;
   QImage m_img;
   AddressTracker m_track;
   void paintEvent(QPaintEvent *) {
      QPainter p(this);
      if (!m_img.isNull()) {
         setAttribute(Qt::WA_OpaquePaintEvent);
         p.drawImage(0, 0, m_img);
         painted = true;
      }
   }
public:
   ImageViewer(QWidget * parent = nullptr) : QWidget(parent) {}
   ~ImageViewer() { qDebug() << __FUNCTION__ << "reallocations" << m_track.reallocs; }
   Q_SLOT void setImage(const QImage &img) {
      if (!painted) qDebug() << "Viewer dropped frame!";
      if (m_img.size() == img.size() && m_img.format() == img.format()
          && m_img.bytesPerLine() == img.bytesPerLine())
         std::copy_n(img.bits(), img.sizeInBytes(), m_img.bits());
      else
         m_img = img.copy();
      painted = false;
      if (m_img.size() != size()) setFixedSize(m_img.size());
      m_track.track(m_img);
      update();
   }
   QImage image() const { return m_img; }
};

該演示實例化上述類并在專用線程中運行捕獲和轉換.

The demonstration instantiates the classes described above and runs the capture and conversion in dedicated threads.

class Thread final : public QThread { public: ~Thread() { quit(); wait(); } };

int main(int argc, char *argv[])
{
   qRegisterMetaType<cv::Mat>();
   QApplication app(argc, argv);
   ImageViewer view;
   Capture capture;
   Converter converter;
   Thread captureThread, converterThread;
   // Everything runs at the same priority as the gui, so it won't supply useless frames.
   converter.setProcessAll(false);
   captureThread.start();
   converterThread.start();
   capture.moveToThread(&captureThread);
   converter.moveToThread(&converterThread);
   QObject::connect(&capture, &Capture::frameReady, &converter, &Converter::processFrame);
   QObject::connect(&converter, &Converter::imageReady, &view, &ImageViewer::setImage);
   view.show();
   QObject::connect(&capture, &Capture::started, [](){ qDebug() << "Capture started."; });
   QMetaObject::invokeMethod(&capture, "start");
   return app.exec();
}

#include "main.moc"

完整示例到此結束.注意:此答案的先前修訂版不必要地重新分配了圖像緩沖區.

This concludes the complete example. Note: The previous revision of this answer unnecessarily reallocated the image buffers.

這篇關于如何在 Qt 中有效地顯示 OpenCV 視頻?的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！