shaurya950 / int-3dhuman-i1

This is a personal fun implementation of I1 (heatmap + integral loss) in Integral Regression. Trained only on H36M w/ 2-stage coarse to fine Hourglass. JUST FOR REFERENCE.

int-3dhuman-I1

Hey there, this is the implementation of I1 (heatmap loss + integral loss) in this paper. Training is on H36M exclusively with 2-stage coarse to fine Hourglass. Kindly jump to Training procedure below if you are not using C++.

---

Sketch

${POSE_ROOT}
+-- caffe_code
+-- data
+-- figs
+-- models
+-- training
+-- testing
+-- README.md

---

---

External links (closely related projects)

H1

I1 with group normalization, batch size = 1

---

Source code & Installation

Please refer to the "caffe_code/" folder and "Installation" section in readme of this repo

---

Configuration

This is for Caffe Ubuntu with 12 GB Titan card.

---

Data

Same as "data/". Note this is different from commonly used data like effective baseline. Major difference is my training(testing) set has 1559571(548999) images.

---

MPJPE Performance

d2	MPJPE(mm) of this repo (I1)	Caffe Model	(For reference) MPJPE(mm) of corresponding H1 repo
16	71.0	net_iter_526322.caffemodel	73.6
32	55.9	net_iter_567946.caffemodel	68.6
64	65.9	net_iter_671782.caffemodel	67.1

Note that in order to restrain one from tuning loss weight of 2D/3D HM/integral to the best of one's ability, adaptive euclidean loss weight balancing technique (as detailed below) is initialized insofar as it does not degrade performance.

I have a deep skepticsm concerning the bearing of different training technique on final number. And so the point I would like to make here is that what matters most is the algorithm itself rather than the superiority or inferiority of number.

As to depth dimension d2, the original paper uses 64. This paper uses 16. This repo prefers 32.

Piece of advice here, insofar as d2 is at least 16, it's okay.

---

Training procedure

First off, THERE IS NO GUARANTEE THAT OTHER TRAINING STRATEGY WOULD NOT YIELD SAME/SIMILAR OR BETTER RESULTS. THIS IS JUST FOR REFERENCE.

Having said that, I WOULD BE MORE THAN HAPPY TO KNOW IF ANYONE HAS A MORE ELEGANT WAY TO TRAIN THIS WACKY STUFF. MUCH APPRECIATED. THANKS!

Back to our topic,

The starting point is MPII pretrained model. See head of training section .

Note 1 [heatmap2 init std]: The init of layer "heatmap2" (which reduces dimension to generate final 3d heatmap output) is gaussian with standard deviation of a hyper param.

Note 2 [heatmap2_flat_scale]: Before softmax normalization, hereinafter a scale 30.0 is multiplied by 3d heatmap output for numerical reason. The semantic of heatmap no longer preserves any more, as alluded in this issue. Otherwise I have no idea how to train I1. CAN ANYONE ENLIGHTEN ME?

Note 3 [loss]: Adaptive weight balancing is employed. Details here. Found this paper several weeks after writing the AdaptiveWeightEucLossLayer

1. Adaptive H1 adaptively computes gradient magnitude of 2D/3D heatmap w.r.t neuron, and tries to balance gradients flowing from 2D and 3D heatmap. Search AdaptiveWeightEucLoss in this pdf for expatiation.

2. Adaptive I1 adaptively balances the gradient of 2D/3D heatmap/integral loss w.r.t neuron, whereupon restraining one from tuning weight ratio between heatmap euclidean loss and integral regression loss.

3. Manual sets loss ratio of heatmap:integral manually after Adaptive I1 warm-up.

The following is sorted by d2 (depth dimension of 3d heatmap) in increasing order.

• d2 = 4

  cd training/d2=4
  $CAFFE_ROOT/build/tools/caffe train --solver=solver_d4_ada.prototxt --weights=improved-hourglass_iter_640000.caffemodel
  

  d2	lr	"heatmap2" init std	loss	Caffe Model	Solver State
  4	2.5e-5	0.01	Adaptive H1	net_iter_160533.caffemodel	net_iter_160533.solverstate

• d2 = 8

  • Adaptive I1

  cd ../../training/d2=8
  $CAFFE_ROOT/build/tools/caffe train --solver=solver_d8_ada.prototxt --snapshot=net_iter_160533.solverstate
  

  d2	lr	"heatmap2" init std	loss	Caffe Model	Solver State
  8	2.5e-5	0.01	Adaptive I1	net_iter_265747.caffemodel	net_iter_265747.solverstate

  • Manual

  $CAFFE_ROOT/build/tools/caffe train --solver=solver_d8_ada_manual.prototxt --snapshot=net_iter_265747.solverstate
  

  d2	lr	loss ratio of 2D HM:3D HM:integral	loss	Caffe Model	Solver State
  8	5e-6	1:0.1:1	Manual	net_iter_350433.caffemodel	net_iter_350433.solverstate

  net_iter_350433.caffemodel train around 71 mm, test around 83 mm.

• d2 = 16

  • Adaptive I1

  cd ../../training/d2=16
  $CAFFE_ROOT/build/tools/caffe train --solver=solver_16_ada.prototxt --snapshot=net_iter_350433.solverstate
  

  d2	lr	"heatmap2" init std	loss	Caffe Model	Solver State
  16	2.5e-5	0.002	Adaptive I1	net_iter_434657.caffemodel	net_iter_434657.solverstate

  net_iter_434657.caffemodel train around 50 mm, test around 73 mm.

  • Manual

  $CAFFE_ROOT/build/tools/caffe train --solver=solver_16_ada_manual.prototxt --snapshot=net_iter_434657.solverstate
  

  d2	lr	loss ratio of 2D HM:3D HM:integral	loss	Caffe Model	Solver State
  16	5e-6	1:0.3:1	Manual	net_iter_526322.caffemodel	net_iter_526322.solverstate

  net_iter_526322.caffemodel train around 47 mm, test around 71 mm (click this for screenshot).

• d2 = 32

  • Adaptive I1

  cd ../../training/d2=32
  $CAFFE_ROOT/build/tools/caffe train --solver=solver_32_ada.prototxt --snapshot=net_iter_526322.solverstate
  

  d2	lr	"heatmap2" init std	loss	Caffe Model	Solver State
  32	2.5e-5	0.002	Adaptive I1	net_iter_567946.caffemodel	net_iter_567946.solverstate

  net_iter_567946.caffemodel train around 49 mm test around 56 mm (click this for screenshot).

• d2 = 64

  • Adaptive I1

  cd ../../training/d2=64
  $CAFFE_ROOT/build/tools/caffe train --solver=solver_64_ada.prototxt --snapshot=net_iter_567946.solverstate
  

  d2	lr	"heatmap2" init std	loss	Caffe Model	Solver State
  64	2.5e-5	0.002	Adaptive I1	net_iter_600890.caffemodel	net_iter_600890.solverstate

  net_iter_600890.caffemodel train around 43 mm, test around 68 mm.

  • Manual

  $CAFFE_ROOT/build/tools/caffe train --solver=solver_64_ada_manual.prototxt --snapshot=net_iter_600890.solverstate
  

  d2	lr	loss ratio of 2D HM:3D HM:integral	loss	Caffe Model	Solver State
  64	5e-6	1:0.08:1	Manual	net_iter_671782.caffemodel	net_iter_671782.solverstate

  net_iter_671782.caffemodel train around 40 mm, test around 66 mm (click this for screenshot).

---

Testing steps

Put corresponding trained models under folder models. (Simply scroll up to MPJPE Performance for names of models)

d2 = 32

cd testing/d2=32
$CAFFE_ROOT/build/tools/caffe test -model test_d32_statsfalse.prototxt -weights models/net_iter_567946.caffemodel	-iterations 274500

Denouement: 55.9 mm

d2 = 64

cd testing/d2=64
$CAFFE_ROOT/build/tools/caffe test -model test_d64_statsfalse.prototxt -weights models/net_iter_671782.caffemodel	-iterations 274500

Denouement: 65.9 mm

d2 = 16

cd testing/d2=16
$CAFFE_ROOT/build/tools/caffe test -model test_d16_statsfalse.prototxt -weights models/net_iter_526322.caffemodel	-iterations 274500

Denouement: 71.0 mm

FAQ

Glad to answer any concern. Just email [email protected]. Thanks!