Tracker keyframe selection code mistake #50919

New Issue

stephen leger · 2017-03-11T07:59:11+01:00

stephen leger commented

2017-03-11 07:59:11 +01:00

System Information
All, dosent matter.

Blender Version
Broken: all

Short description of error
keyframe selection can't work as expected.

Exact steps for others to reproduce the error
according the paper KEY FRAME EXTRACTION AND BROWSER-BASED VISUALIZATION FOR 3D RECONSTRUCTION FROM VIDEO STREAMS page 8

SymetricEpipolarDistance (libmv/multiview/fundamental.cc) function does return PELC (PointToEpipolarLineCost) page 8, equation 2.5 instead of what it should, page 8, equation 2.4
so keyframe selection cant work as expected.

The positive point is that we allready have PELC criterion computed witch shoud fix TODO(sergey): STEP 4: PELC criterion arround line 265

should be something like that :

double SymmetricEpipolarDistance(const Mat &F, const Vec2 &x1, const Vec2 &x2) {
  Vec3 x(x1(0), x1(1), 1.0);
  Vec3 y(x2(0), x2(1), 1.0);

  Vec3 F_x = F * x;
  Vec3 Ft_y = F.transpose() * y;
  // d(y,F_x)+d(x,F_y)
  return Square(F_x(0)-y(0)) + Square(F_x(1)-y(1)) + 
	     Square(Ft_y(0)-x(0)) + Square(F_y(1)-x(1));
}

double PointToEpipolarLineCost(const Mat &F, const Vec2 &x1, const Vec2 &x2) {
  Vec3 x(x1(0), x1(1), 1.0);
  Vec3 y(x2(0), x2(1), 1.0);

  Vec3 F_x = F * x;
  Vec3 Ft_y = F.transpose() * y;
  double y_F_x = y.dot(F_x);

  return Square(y_F_x) * (  1 / F_x.head<2>().squaredNorm()
                          + 1 / Ft_y.head<2>().squaredNorm());
}

**System Information** All, dosent matter. **Blender Version** Broken: all **Short description of error** keyframe selection can't work as expected. **Exact steps for others to reproduce the error** according the paper [KEY FRAME EXTRACTION AND BROWSER-BASED VISUALIZATION FOR 3D RECONSTRUCTION FROM VIDEO STREAMS ](https://upcommons.upc.edu/bitstream/handle/2099.1/7737/60425.pdf) page 8 SymetricEpipolarDistance (libmv/multiview/fundamental.cc) function does return PELC (PointToEpipolarLineCost) page 8, equation 2.5 instead of what it should, page 8, equation 2.4 so keyframe selection cant work as expected. The positive point is that we allready have PELC criterion computed witch shoud fix TODO(sergey): STEP 4: PELC criterion arround line 265 should be something like that : ``` double SymmetricEpipolarDistance(const Mat &F, const Vec2 &x1, const Vec2 &x2) { Vec3 x(x1(0), x1(1), 1.0); Vec3 y(x2(0), x2(1), 1.0); Vec3 F_x = F * x; Vec3 Ft_y = F.transpose() * y; // d(y,F_x)+d(x,F_y) return Square(F_x(0)-y(0)) + Square(F_x(1)-y(1)) + Square(Ft_y(0)-x(0)) + Square(F_y(1)-x(1)); } double PointToEpipolarLineCost(const Mat &F, const Vec2 &x1, const Vec2 &x2) { Vec3 x(x1(0), x1(1), 1.0); Vec3 y(x2(0), x2(1), 1.0); Vec3 F_x = F * x; Vec3 Ft_y = F.transpose() * y; double y_F_x = y.dot(F_x); return Square(y_F_x) * ( 1 / F_x.head<2>().squaredNorm() + 1 / Ft_y.head<2>().squaredNorm()); } ```

stephen leger commented

2017-03-11 07:59:11 +01:00

Changed status to: 'Open'

stephen leger commented

2017-03-11 07:59:11 +01:00

Added subscriber: @StephenLeger

Bastien Montagne commented

2017-03-14 15:32:12 +01:00

Added subscribers: @Sergey, @mont29

Sergey Sharybin was assigned by Bastien Montagne

2017-03-14 15:32:12 +01:00

Bastien Montagne commented

2017-03-14 15:32:12 +01:00

@Sergey might understand this? ;)

Sergey Sharybin commented

2017-05-31 12:30:41 +02:00

Equation 2.5 is derived from 2.4 in the paper you've referenced to by substituting d(x, Fx) with a proper distance to the epipolar line. The distance itself is used as a cost in your paper.

Now, if i'm not missing something, your suggested implementation of SymmetricEpipolarDistance() uses euclidean distance for d(x, Fx), which makes it effectively the same as SymmetricGeometricDistance.

There might be some confusion about naming of PELC here, but mr. Zisserman calls it (see - [x], - [x]). The book - [x] is what is referenced as HZ in the multiview/.

It is also interesting to see an exact case which demonstrates that your suggested line distance gives better keyframes.

The issue with lack of PELC criteria in the current code is not caused by the lack of PELC calculation, but because it requires some experimentally acquired weights. See Eq. 5 from [3].

P.S. While it is really interesting to investigate this, i don't see how this is a bug.

Equation 2.5 is derived from 2.4 in the paper you've referenced to by substituting `d(x, Fx)` with a proper distance to the epipolar line. The distance itself is used as a cost in your paper. Now, if i'm not missing something, your suggested implementation of `SymmetricEpipolarDistance()` uses euclidean distance for `d(x, Fx)`, which makes it effectively the same as `SymmetricGeometricDistance`. There might be some confusion about naming of PELC here, but mr. Zisserman calls it (see - [x], - [x]). The book - [x] is what is referenced as HZ in the `multiview/`. It is also interesting to see an exact case which demonstrates that your suggested line distance gives better keyframes. The issue with lack of PELC criteria in the current code is not caused by the lack of PELC calculation, but because it requires some experimentally acquired weights. See Eq. 5 from [3]. P.S. While it is really interesting to investigate this, i don't see how this is a bug. - [x] http://users.rsise.anu.edu.au/hartley/public_html/Papers/CVPR99-tutorial/tut_4up.pdf - [x] http://cvrs.whu.edu.cn/downloads/ebooks/Multiple%20View%20Geometry%20in%20Computer%20Vision%20(Second%20Edition).pdf - [x] https://www.cs.ait.ac.th/~mdailey/papers/Tahir-KeyFrame.pdf

stephen leger commented

2017-05-31 20:06:33 +02:00

Different names used in papers for same function lead to confusion.
Facing issues with reconstruction process quality under automatic selection make me think there is a bug somewhere. Feel free to label this as "enhancement".

SymmetricEpipolarDistance use euclidian distance to epipolar d(x, Fx)
where SymmetricGeometricDistance use euclidian distance between mesured and apparent position d(x, x^)

Anyway, as far as i can understand, seem current implementation only use GRIC as criterion to select frames, where weighted mix of GRIC and PELC may significantely enhance result - [x] page 27 3.6 comparison page 45 4.2

https://upcommons.upc.edu/bitstream/handle/2099.1/7737/60425.pdf

Different names used in papers for same function lead to confusion. Facing issues with reconstruction process quality under automatic selection make me think there is a bug somewhere. Feel free to label this as "enhancement". SymmetricEpipolarDistance use euclidian distance to epipolar d(x, Fx) where SymmetricGeometricDistance use euclidian distance between mesured and apparent position d(x, x^) Anyway, as far as i can understand, seem current implementation only use GRIC as criterion to select frames, where weighted mix of GRIC and PELC may significantely enhance result - [x] page 27 3.6 comparison page 45 4.2 - [x] https://upcommons.upc.edu/bitstream/handle/2099.1/7737/60425.pdf

Sergey Sharybin commented

2021-10-06 12:34:21 +02:00

Finally having some time to check on this task!

@StephenLeger, Thanks for the paper, it even provided weights! Did you try implementing KFWS and see if it improves quality for your scene?
Also, would be really helpful to have the scene which we can use to validate quality of changes.

Finally having some time to check on this task! @StephenLeger, Thanks for the paper, it even provided weights! Did you try implementing KFWS and see if it improves quality for your scene? Also, would be really helpful to have the scene which we can use to validate quality of changes.

stephen leger commented

2021-10-06 21:07:34 +02:00

Hi Sergey,
Looks like the paper is no more the same, was originally something from bf itself if i recall correctly, with a blender's implementation sample.
Will try to find original paper with source code on older laptop ;)
While i do rougthly understand the math behind the code, i'm by far not good enough on code side to manage required deps compilation.

Hi Sergey, Looks like the paper is no more the same, was originally something from bf itself if i recall correctly, with a blender's implementation sample. Will try to find original paper with source code on older laptop ;) While i do rougthly understand the math behind the code, i'm by far not good enough on code side to manage required deps compilation.

Sign in to join this conversation.