Page MenuHome

Tracker keyframe selection code mistake
Open, NormalPublic

Description

System Information
All, dosent matter.

Blender Version
Broken: all

Short description of error
keyframe selection can't work as expected.

Exact steps for others to reproduce the error
according the paper KEY FRAME EXTRACTION AND BROWSER-BASED VISUALIZATION FOR 3D RECONSTRUCTION FROM VIDEO STREAMS page 8

SymetricEpipolarDistance (libmv/multiview/fundamental.cc) function does return PELC (PointToEpipolarLineCost) page 8, equation 2.5 instead of what it should, page 8, equation 2.4
so keyframe selection cant work as expected.

The positive point is that we allready have PELC criterion computed witch shoud fix TODO(sergey): STEP 4: PELC criterion arround line 265

should be something like that :

double SymmetricEpipolarDistance(const Mat &F, const Vec2 &x1, const Vec2 &x2) {
  Vec3 x(x1(0), x1(1), 1.0);
  Vec3 y(x2(0), x2(1), 1.0);

  Vec3 F_x = F * x;
  Vec3 Ft_y = F.transpose() * y;
  // d(y,F_x)+d(x,F_y)
  return Square(F_x(0)-y(0)) + Square(F_x(1)-y(1)) + 
	     Square(Ft_y(0)-x(0)) + Square(F_y(1)-x(1));
}

double PointToEpipolarLineCost(const Mat &F, const Vec2 &x1, const Vec2 &x2) {
  Vec3 x(x1(0), x1(1), 1.0);
  Vec3 y(x2(0), x2(1), 1.0);

  Vec3 F_x = F * x;
  Vec3 Ft_y = F.transpose() * y;
  double y_F_x = y.dot(F_x);

  return Square(y_F_x) * (  1 / F_x.head<2>().squaredNorm()
                          + 1 / Ft_y.head<2>().squaredNorm());
}

Details

Type
To Do

Event Timeline

Equation 2.5 is derived from 2.4 in the paper you've referenced to by substituting d(x, Fx) with a proper distance to the epipolar line. The distance itself is used as a cost in your paper.

Now, if i'm not missing something, your suggested implementation of SymmetricEpipolarDistance() uses euclidean distance for d(x, Fx), which makes it effectively the same as SymmetricGeometricDistance.

There might be some confusion about naming of PELC here, but mr. Zisserman calls it (see [1], [2]). The book [2] is what is referenced as HZ in the multiview/.

It is also interesting to see an exact case which demonstrates that your suggested line distance gives better keyframes.

The issue with lack of PELC criteria in the current code is not caused by the lack of PELC calculation, but because it requires some experimentally acquired weights. See Eq. 5 from [3].

P.S. While it is really interesting to investigate this, i don't see how this is a bug.

[1] http://users.rsise.anu.edu.au/hartley/public_html/Papers/CVPR99-tutorial/tut_4up.pdf
[2] http://cvrs.whu.edu.cn/downloads/ebooks/Multiple%20View%20Geometry%20in%20Computer%20Vision%20(Second%20Edition).pdf
[3] https://www.cs.ait.ac.th/~mdailey/papers/Tahir-KeyFrame.pdf

Sergey Sharybin (sergey) changed Type from Bug to To Do.May 31 2017, 12:30 PM

Different names used in papers for same function lead to confusion.
Facing issues with reconstruction process quality under automatic selection make me think there is a bug somewhere. Feel free to label this as "enhancement".

SymmetricEpipolarDistance use euclidian distance to epipolar d(x, Fx)
where SymmetricGeometricDistance use euclidian distance between mesured and apparent position d(x, x^)

Anyway, as far as i can understand, seem current implementation only use GRIC as criterion to select frames, where weighted mix of GRIC and PELC may significantely enhance result [1] page 27 3.6 comparison page 45 4.2

[1] https://upcommons.upc.edu/bitstream/handle/2099.1/7737/60425.pdf