問題描述
您好，我的目標是開發用于飛機(模擬器)駕駛艙的頭部跟蹤功能，以提供 AR 以支持民用飛行員在視覺條件不佳的情況下著陸和飛行.
我的方法是檢測我知道其 3D 坐標的特征點(在黑暗的模擬器 LED 中)，然后計算估計的(頭戴相機的)姿勢 [R|t](旋轉與平移連接).
我遇到的問題是估計的姿勢似乎總是錯誤的，并且我的 3D 點的投影(我也用來估計姿勢)與2D 圖像點(或不可見).
我的問題是:
如何使用一組給定的 2D 到 3D 點對應關系估計相機位姿.
為什么我嘗試它時它不起作用，哪里可能出現錯誤?
測量(3D 和 2D 點以及相機矩陣)必須有多精確才能使理論解決方案在現實生活環境中發揮作用?
理論上該方法是否適用于共面點(x,y 軸發生變化)?
我使用的硬件是 Epson BT-200.
在飛機中，我定義了一個固定的縱坐標，我希望我的程序會產生相對平移和旋轉.該程序檢測(唯一)LED 的圖像坐標，并將它們與相應的 3D 坐標相匹配.使用我使用 open-cv 示例 android 代碼 (:0.png
1.png
編輯 22.03.2015:我終于能夠找到我犯的錯誤了.
- 我在 for 循環中修改了一個 Mat 對象，因為 OpenCV 工作量很大通過引用調用，我在這里不夠小心.所以重新投影的 tvec 和 rvec 不正確.
- 我在測試環境中的一個觀點有(在圖片中坐標)，由于軸方向混淆而被標記為錯誤.
所以我的方法總體上是正確的.我的測試數據集中至少(通常)沒有收到有效的重投影.
不幸的是，OpenCV PnP 算法:迭代、P3P、EPNP"返回各種結果，即使使用非常不準確但接近的內在猜測，結果也只是有時是正確的.P3P算法應該提供3種解決方案，但是OpenCV 只提供了一個.EPNP 應該返回良好的結果，但使用 EPNP OpenCV 返回最差結果，根據我的人類觀察評估.
現在的問題是，如何過濾不準確的值或確保 OpenCV 函數返回有效值.(也許我應該修改本機代碼以接收 3 個 PnP 解決方案).
此處的壓縮圖像 (37MB)，請顯示我當前的結果(使用迭代 PnP 求解器)，內在猜測為零旋轉和向上 75 厘米.打印輸出有一個向前的 x 軸、向左的 y 軸和向下的 z 軸，以及對應的滾動角、俯仰角和偏航角.
我在嘗試實現我的頭部跟蹤系統時學到的一件事是，你應該從簡單的問題開始，而不是轉向更復雜的問題.你的問題很長，不幸的是我沒有時間分析它并在你的代碼中搜索錯誤或邏輯錯誤，所以至少我會嘗試給你一些提示和工作示例.
這里是OpenCV查找對象平移和旋轉的教程.它是用 Python 編寫的，如果有問題 這里 部分我的舊 c++ 項目.
我的項目使用 solvePnP 或 solvePnPRansac 函數執行相同的任務(您可以更改模式).請注意，我的代碼是一些舊的游樂場"的一部分.項目，所以即使在我進行了清理之后，它也很混亂.當你運行它時，向相機顯示打印的棋盤，按'p'開始位置和旋轉估計，'m'改變模式(0-ransac，1-pnp，2-posit，這似乎不起作用......)或d"使用色散系數打開/關閉.
這兩個項目都依賴于尋找棋盤圖案，但應該很容易修改它們以使用其他對象.
相機校準 - 雖然我一直在研究我的頭部跟蹤系統，但我從來沒有成功地校準過兩次相同結果的相機......所以我決定使用我在 github 上找到的一些校準文件效果很好 - 這里你可以找到更多有關該文件的鏈接的信息.
嘗試從在某些(甚至簡單的)情況下產生良好結果的盡可能簡單的解決方案開始.在我看來，一個好的開始點是用教程中的打印棋盤替換測試環境中的一張紙(this one) 并使其工作.從這個轉向你的問題比從你的問題開始要容易得多.嘗試使用任何編程語言制作任何可行的解決方案 - 考慮使用 Python 或 C++ 版本的 OpenCV - 教程/示例比 Java 版本多得多，并且將代碼的結果與某些工作代碼的結果進行比較會更容易.當您有一些可行的解決方案時，請嘗試修改它以適應您的測試環境.有很多事情可能導致它現在無法正常工作 - 點數不足、代碼中的錯誤甚至 OpenCV Java 包裝器中的錯誤、對結果的錯誤解釋等等......
edit2:
使用您的代碼中的點，我設法得到以下結果:
<塊引用>rvec = [[-158.56293283]，[1.46777938]，[-17.32569125]]
tvec = [[ -36.23910413]，[-82.83704819]，[266.03157578]]
不幸的是，對我來說，很難說結果是否好...唯一可能錯誤對我來說是 2 個角度不同于 0(或 180).但是，如果您將 points2d 的最后一行從 (355,37), (353,72), (353,101) 更改為
(355,37), (355,72), (355,101)
(我猜這是你的錯誤，不是正確的結果)你會得到:
<塊引用>rvec = [[-159.34101842]，[1.04951033]，[-11.43731376]]
tvec = [[ -25.74308282]，[-82.58461674]，[268.12321097]]
這可能更接近正確的結果.更改相機矩陣會大大改變結果，因此請考慮測試 這篇文章中的值.
請注意，所有 rvec 值都乘以 180.0/3.14 - 在 c++ 中，solvePnPRansac 返回的 python rvec 向量包含以弧度表示的角度.
Hello my goal is to develop head-tracking functionality to be used in an aircraft (simulator) cockpit, in order to provide AR to suport civilian pilots to land and fly with bad visual conditions.
My approach is to detect characteristic points (in the dark simulator LEDs) of which I know the 3D coordinates and than compute the estimated (head worn camera's) pose [R|t] (rotation concatinated with translation).
The problem I do have is that the estimated pose seems to be always wrong and a projection of my 3D points (which I also used to estimate the pose) does not overlap with the 2D image points (or is not visible).
My questions are:
How can I estimate the camera pose with a given set of 2D-to-3D point correspondences.
Why does it not work how I try it and where might be sources of error?
How accurate must be the measurements (of 3D and 2D points and the camera matrix) to get the theoretical solution working in a real life environment?
Will the approach work for coplanar points (x,y axis changed) in theory?
The hardware I use is the Epson BT-200.
In the aircraft I defined a fixed ordinate to which I expect relative translations and rotations as result of my program. The program detects the image coordinates of (unique) LEDs and matches them to their corresponding 3D coordinate. With a camera matrix I obtained using the open-cv sample android code (https://github.com/Itseez/opencv/tree/master/samples/android/camera-calibration) I try to estimate the pose using solvePnP.
My camera matrix and distortion varries slightly. Here are some values I received from the procedure. I made sure that the circle-distance of my printed out circle pattern is the same as written down in the source-code (measured in Meters).
Here are some examples and how I create the OpenCV Mat of it.
// protected final double[] DISTORTION_MATRIX_VALUES = new double[]{
// /*This matrix should have 5 values*/
// 0.04569467373955304,
// 0.1402980385369059,
// 0,
// 0,
// -0.2982135315849994
// };
// protected final double[] DISTORTION_MATRIX_VALUES = new double[]{
// /*This matrix should have 5 values*/
// 0.08245931646421553,
// -0.9893762277047577,
// 0,
// 0,
// 3.23553287438898
// };
// protected final double[] DISTORTION_MATRIX_VALUES = new double[]{
// /*This matrix should have 5 values*/
// 0.07444480392067945,
// -0.7817175834131075,
// 0,
// 0,
// 2.65433773093283
// };
protected final double[] DISTORTION_MATRIX_VALUES = new double[]{
/*This matrix should have 5 values*/
0.08909941096327206,
-0.9537960457721699,
0,
0,
3.449728790843752
};
protected final double[][] CAMERA_MATRIX_VALUES = new double[][]{
/*This matrix should have 3x3 values*/
// {748.6595405553738, 0, 319.5},
// {0, 748.6595405553738, 239.5},
// {0, 0, 1}
// {698.1744297982436, 0, 320},
// {0, 698.1744297982436, 240},
// {0, 0, 1}
// {707.1226937511951, 0, 319.5},
// {0, 707.1226937511951, 239.5},
// {0, 0, 1}
{702.1458656346429, 0, 319.5},
{0, 702.1458656346429, 239.5},
{0, 0, 1}
};
private void initDestortionMatrix(){
distortionMatrix = new MatOfDouble();
distortionMatrix.fromArray(DISTORTION_MATRIX_VALUES);
}
private void initCameraMatrix(){
cameraMatrix = new Mat(new Size(3,3), CvType.CV_64F);
for(int i=0;i<CAMERA_MATRIX_VALUES.length; i++){
cameraMatrix.put(i, 0, CAMERA_MATRIX_VALUES[i]);
}
}
To estimate the camera pose I do use solvePnP (and solvePnPRansac) as described in several locations (1,2,3,4). The result of solvePnP I use as input for the Projection (Calib3d.projectPoints). The inverse of the concatinated result [R|t] I do use as estimated pose.
Because my results in the productive environment were too bad I created a testing environment. In that environment I place the camera (which is because of it's 3D-shape (it's a glass) slightly rotated downwards at a table's edge. This edge I do use as ordinate of the world-coordinate system. I searched how the open-cv coordinate system might be oriented and found different answers (one on stackoverflow and one in an official youtube-talk about opencv). Anyways I tested if I got the coordinate system right by projection 3D points (described in that coordinate system) on an image and checked if the given world shape stays constant.
So I came up wiht z pointing foreward, y downward and x to the right.
To get closer to my solution I estimated the pose in my testing environment. The translation vector-output and euler angel output refers to the inverse of [R|t]. The euler angels might not be displayed correct (they might be swaped or wrong, if we take order into account) because I compute it with the convetional (I assume refering to the airplane coordinate system) equations, using an open-cv coordinate system. (The computation happens in the class Pose which I will attach). But anyways even the translation vector (of the inverse) appeard to be wrong (in my simple test).
In one test with that Image I had a roll (which might be pitch in airplane coordinates) of 30° and a translation upwards of 50cm. That appeard to be more reasonable. So I assumed because my points are coplanar, I might get ambiguous results. So I realized an other test with a point which changed in the Z-Axis. But with this test even the projection failed.
For solvePnP I tried all different solving-algorithm-flags and different parameters for the ransac algorithm.
Maybe you can somehow help me to find my mistake, or showing me a good path to solve my initial problem. I am going to attach also my debugging source-code with many println statements and the debugging images. This code contains my point measurements.
Thanks for your help in advance.
Class Main.java: Class Pose.java: 0.png
1.png
EDIT 22.03.2015: Finally I have been able to find mistakes I made.
- I modified a Mat object in a for-loop, because OpenCV works a lot with call by reference, and I was not careful enough here. So the tvec and rvec for the reprojection were not right.
- One of my points in the testing environment had (in the image coordinates), was tagged wrong due to an axis-direction confusion.
So my approach in general was right. I am not receiving at least (often) valid reprojections in my test-dataset.
Unfortunately the OpenCV PnP algorithms: "ITERATIVE, P3P, EPNP" return various results, and even with using a very unaccurate but close intrinsic guess, the results are only sometimes correct. The P3P algorithm is supposed to provide 3 solutions, but OpenCV only provides one. EPNP is supposed to return good results, but with EPNP OpenCV returns the worst results, evaluated from my human obersation.
The problem now is, how to filter the inaccurate values or ensure the OpenCV function returns valid ones. (Maybe I shuold modify the native code to receive 3 solutions for PnP).
The compressed images here (37MB), do show my current results (with the ITERATIVE PnP-Solver) , with an intrinsic guess of zero rotation and 75 cm upwards. The print-out has an x-axis foreward, y-axis to the left and z-down, and corrosponding roll, pitch, and yaw angles.
One thing that i've learned during trying to implement my head tracking system is that you should start from simple problem and than move to more complicated one. Your question is quite ong and unfortunetely i don't have time to analyze it and search for a bug or logical mistake in your code, so at least i will try to give you some hints and working examples.
Here is OpenCV tutorial for finding object translation and rotation. It's written in Python, if it is a problem here part of my old c++ project.
My project performs the same task using solvePnP or solvePnPRansac function (you can change mode). Note that my code it's a part of some old "playground" project, so even after cleaining which i performed it's quite messy. When you run it, show printed chessboard to the camera, press 'p' to start position and rotation estimation, 'm' to change mode (0-ransac, 1-pnp, 2-posit which seems to not work...) or 'd' to turn on/off using dissortion coefficients.
Both projects relies on finding chessboard pattern, but it shoud be easy to modify them to use some other objects.
Camera calibration - while i've been working on my head tracking system i've never managed to calibrate camera twice with the same results... So i decided to use some calibartion file which i've found on github and it worked well - here you can found a litte more information about that an link to this file.
edit:
Try to start with as simple as possible solution that gives good results in some (even simple) situation. A good point to start in my opinion is to replace a sheet of paper from your testing environment with printed chessboard from tutorial (this one) and make it working. Moving from this to your problem will be much easier than beginning with you problem. Try to make any working solution in any programming language - consider using Python or C++ version of OpenCV - there is much more tutorials/examples than to Java version and comparing results from your code with results from some working code will make it much easier. When you will have some working solution try to modify it to work with your testing environment. There is a lot of things which may cause it not working right now - not enough points, bug in your code or even in OpenCV Java wrapper, bad interpretation of results, etc...
edit2:
Using points from your code i've managed to get following results:
rvec = [[-158.56293283], [ 1.46777938], [ -17.32569125]]
tvec = [[ -36.23910413], [ -82.83704819], [ 266.03157578]]
Unfortunetely, for me it's hard to say whether results are good or not... The only thing that might be wrong to me is that 2 angles are different from 0 (or 180). But if you change last row of points2d from (355,37), (353,72), (353,101) to
(355,37), (355,72), (355,101)
(i guess it's your mistake, not a correct result) you will get:
rvec = [[-159.34101842], [ 1.04951033], [ -11.43731376]]
tvec = [[ -25.74308282], [ -82.58461674], [ 268.12321097]]
which might be much closer to the correct result. Changing camera matrix changes results much, so consider testing values from this post.
Note that all rvec values are multiplied by 180.0/3.14 - in c++ and python rvec vector returned by solvePnPRansac contains angles in radians.
這篇關于如何估計具有 3d 到 2d 點對應的相機姿勢(使用 opencv)的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！