jeff-sjtu / res-loglikelihood-regression

Code for "Human Pose Regression with Residual Log-likelihood Estimation", ICCV 2021 Oral

Human Pose Regression with Residual Log-likelihood Estimation

[Paper] [arXiv] [Project Page]

Human Pose Regression with Residual Log-likelihood Estimation
Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, Cewu Lu
ICCV 2021 Oral

Regression with Residual Log-likelihood Estimation

TODO

• Provide minimal implementation of RLE loss.
• Add model zoo.
• Provide implementation on Human3.6M dataset.
• Provide implementation on COCO dataset.

Installation

1. Install pytorch >= 1.1.0 following official instruction.
2. Install rlepose:

pip install cython
python setup.py develop

3. Install COCOAPI.

pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'

4. Init data directory:

mkdir data

5. Download COCO data, MPII (annotations) and Human3.6M data (from PoseNet or ours):

|-- data
`-- |-- coco
    |   |-- annotations
    |   |   |-- person_keypoints_train2017.json
    |   |   `-- person_keypoints_val2017.json
    |   `-- images
    |       |-- train2017
    |       |   |-- 000000000009.jpg
    |       |   |-- 000000000025.jpg
    |       |   |-- 000000000030.jpg
    |       |   |-- ... 
    |       `-- val2017
    |           |-- 000000000139.jpg
    |           |-- 000000000285.jpg
    |           |-- 000000000632.jpg
    |           |-- ... 
    |-- mpii
    |   |-- annotations
    |   |   `-- annot_mpii.json
    |   `-- images
            |-- 000001163.jpg
            |-- 000003072.jpg
            |-- 000004812.jpg
            |--- ...
    |-- h36m
    `-- |-- annotations
        |   |-- Sample_trainmin_train_Human36M_protocol_2.json
        |   `-- Sample_64_test_Human36M_protocol_2.json
        `-- images
            |-- s_01_act_02_subact_01_ca_01
            |   |-- ...
            |-- s_01_act_02_subact_01_ca_02
            |   |-- ...
            `-- ... 

Training

Train on MSCOCO

./scripts/train.sh ./configs/256x192_res50_regress-flow.yaml train_rle_coco

Train on Human3.6M

./scripts/train.sh ./configs/256x192_res50_3d_h36mmpii-flow.yaml train_rle_h36m

Evaluation

Validate on MSCOCO

Download the pretrained model from Google Drive.

./scripts/validate.sh ./configs/256x192_res50_regress-flow.yaml ./coco-laplace-rle.pth

Validate on Human3.6M

Download the pretrained model from Google Drive.

./scripts/validate.sh ./configs/256x192_res50_3d_h36mmpii-flow.yaml ./h36m-laplace-rle.pth

Citing

If our code helps your research, please consider citing the following paper:

@inproceedings{li2021human,
    title={Human Pose Regression with Residual Log-likelihood Estimation},
    author={Li, Jiefeng and Bian, Siyuan and Zeng, Ailing and Wang, Can and Pang, Bo and Liu, Wentao and Lu, Cewu},
    booktitle={ICCV},
    year={2021}
}

res-loglikelihood-regression's Issues

validate.py报错

我用了您提供的与训练模型，在coco val2017上测试模型的性能，但是报出如下错误（python3.6 torch1.8）：

Timed out initializing process group in store based barrier on rank: 2

Hi, Thank you for sharing your great work!

I tried to run your training scripts. But my machine only has 4 GPUs. So I changed its WORLD SIZE to 4 from 8 in original yaml file.

Then it says "Timed out initializing process group in store based barrier on rank: 2" or sometimes it suddenly crashes during the epoch and my docker container shutdowns (indicating memory leak?).

Any advise on successfully running your training code?

Thank you for your cooperation.

out_coord.shape[2] == 2

Hello, could you tell me why the third dimension must ensure is 2?
" out_coord = self.fc_coord(x).reshape(BATCH_SIZE, self.num_joints, 2)
assert out_coord.shape[2] == 2"

A question about Regression-based method

Hi Jeff,
Sorry to bother you.

I notice that you classify the Integral Pose method as Heatmap-based method and you use FC to regress the joint coords directly.
I understand that maybe you think the soft-argmax based method is not purely regression-based, but according to my experience, the soft-argmax based method(eg. DSNT, Integral pose) can achieve better performance than fc-based one.
So I wonder if you have tried ablation experiment like using RLE on Integral Pose Regression?
Looking forward to your reply~
Best,
Tau

LogQ in criterion.py

def logQ(self, gt_uv, pred_jts, sigma):
return torch.log(sigma / self.amp) + torch.abs(gt_uv - pred_jts) / (math.sqrt(2) * sigma + 1e-9)

from the function definition, it looks like the Q is laplacian distribution but Q = exp( - abs(x - mu) / b) / 2b, after taking log it is log(1/2b) - abs(x - mu) / b

when seeing loss = nf_loss + Q_logprob, nf_loss = log(sigma) - log_phi, compare with the paper saying loss = -log(Q(bar_mu)) +log(sigma)-log_phi, I guess the Q_logprob already has the negative sign inside but then, why it is not -log(1/2b) + abs(x-mu)/b? i.e. the first term torch.log(sigma / self.amp) should have the minus sign.

On the other hand, how do you choose the value of b? in the distribution sense, b = sqrt(variance / 2)

when will you release the 3d version and the minimal implementation?

Hello, thank you for sharing your awesome work.
Could you tell me when will you guys release the 3d version and the minimal implementation?

3d output vis has problem

I visually checked the output results of 3D model and found that the key points of 2D image are quite accurate, but the display effect in 3D is not very ideal. Does the author have a demo for 3D visualization?

Can this method used for multi-person directly regression.

Great jobs! Can I ask a question? Can this method used for multi-person directly regression. I mean inputting an image I, it directly generates candidate keypoints of all persons. (In your paper, you have mentioned the experiments for one-stage hpe, but your experiment is based on point-anchors.)

trian error

one gpu training，run trian.py meet this error

``
Traceback (most recent call last):
File "D:/rhnet-daima/res-loglikelihood-regression-master/res-loglikelihood-regression-master/scripts/train.py", line 172, in
main()
File "D:/rhnet-daima/res-loglikelihood-regression-master/res-loglikelihood-regression-master/scripts/train.py", line 45, in main
mp.spawn(main_worker, nprocs=ngpus_per_node, args=(opt, cfg))
File "D:\anaconda3\envs\PyTorch_YOLOv4-master\lib\site-packages\torch\multiprocessing\spawn.py", line 199, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "D:\anaconda3\envs\PyTorch_YOLOv4-master\lib\site-packages\torch\multiprocessing\spawn.py", line 157, in start_processes
while not context.join():
File "D:\anaconda3\envs\PyTorch_YOLOv4-master\lib\site-packages\torch\multiprocessing\spawn.py", line 118, in join
raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "D:\anaconda3\envs\PyTorch_YOLOv4-master\lib\site-packages\torch\multiprocessing\spawn.py", line 19, in _wrap
fn(i, *args)
File "D:\rhnet-daima\res-loglikelihood-regression-master\res-loglikelihood-regression-master\scripts\train.py", line 55, in main_worker
init_dist(opt)
File "D:\rhnet-daima\res-loglikelihood-regression-master\res-loglikelihood-regression-master\rlepose\utils\env.py", line 24, in init_dist
world_size=opt.world_size, rank=opt.rank)
File "D:\anaconda3\envs\PyTorch_YOLOv4-master\lib\site-packages\torch\distributed\distributed_c10d.py", line 434, in init_process_group
init_method, rank, world_size, timeout=timeout
File "D:\anaconda3\envs\PyTorch_YOLOv4-master\lib\site-packages\torch\distributed\rendezvous.py", line 82, in rendezvous
raise RuntimeError("No rendezvous handler for {}://".format(result.scheme))
RuntimeError: No rendezvous handler for tcp://

about the comparison between hrnet and rle-hrnet

hi, i am prepare to cite you paper, but i am confuse about the table2 in your paper. there are 2 issue.
first, the simple baseline in your paper is 71.0, but in the simple baseline in its official paper is 70.4.
second, the hrnet in your paper is 74.1, but the result in official paper is 74.4.
can you provide more detail of those experiment? thank you!

There is something wrong with the COCO folder structure in the readme.md

The "images" folder actually shouldn't appear. Otherwise, custom_dataset will not able to access data and will raise error like this:

cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) cv2.error: OpenCV(3.4.4) C:\projects\opencv python\opencv\modules\imgproc\src\color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cv::cvtColor'

the structure below would be the right structure. Anyway, thank you for your really really great work. Hope this tip will help :-)

data{coco{annotations/train2017/val2017}}

instead of

data{coco{annotations/images{train2017/val2017}}}

for example, the train2017 folder should be like
<repo_root>/data/coco/train2017/xxx.jpg
instead of
<repo_root>/data/coco/images/train2017/xxx.jpg

loss gets nan problem

Hi Jeff,
I still have trouble training with RLE in my project.
the loss can decrease correctly at beginning, but after some iters, it increases immediately and finally becomes to nan.
I'm using Adam and a cosine scheduler with warm-up strategy.

I implement the regression module as follow:
(this is a handpose project, I add two fc heads to predict hand validness and handtype(left/right hand))

class RegressFlow3D(nn.Module):
    def __init__(self, cfg, in_dim):
        super(RegressFlow3D, self).__init__()
        self.num_joints = cfg.joint_num
        self.root_idx = cfg.wrist_joint_idx

        self.avg_pool = nn.AdaptiveAvgPool2d(1)

        self.hand_type_fc = make_linear_layers([in_dim, 128, 1], relu_final=False)
        self.hand_valid_fc = make_linear_layers([in_dim, 64, 1], relu_final=False)

        self.fc_coord = Linear(in_dim, self.num_joints * 3)
        self.fc_sigma = Linear(in_dim, self.num_joints * 3)

        # self.fc_layers = [self.fc_coord, self.fc_sigma]

        prior = distributions.MultivariateNormal(torch.zeros(2), torch.eye(2), validate_args=False)
        masks = torch.from_numpy(np.array([[0, 1], [1, 0]] * 3).astype(np.float32))
        prior3d = distributions.MultivariateNormal(torch.zeros(3), torch.eye(3), validate_args=False)
        masks3d = torch.from_numpy(np.array([[0, 0, 1], [1, 1, 0]] * 3).astype(np.float32))

        self.flow2d = RealNVP(nets, nett, masks, prior)
        self.flow3d = RealNVP(nets3d, nett3d, masks3d, prior3d)

    # def _initialize(self):
    #     for m in self.fc_layers:
    #         if isinstance(m, nn.Linear):
    #             nn.init.xavier_uniform_(m.weight, gain=0.01)

    def forward(self, feat, labels=None):
        BATCH_SIZE = feat.shape[0]

        feat = self.avg_pool(feat).reshape(BATCH_SIZE, -1)

        hand_type = self.hand_type_fc(feat)
        hand_valid = self.hand_valid_fc(feat)

        out_coord = self.fc_coord(feat).reshape(BATCH_SIZE, self.num_joints, 3)
        # (B, N, 3)
        pred_jts = out_coord.reshape(BATCH_SIZE, self.num_joints, 3)
        pred_jts[:, :, 2] = pred_jts[:, :, 2] - pred_jts[:, self.root_idx:self.root_idx + 1, 2]

        if labels is not None:
            gt_uvd = labels['target_coord'].reshape(pred_jts.shape)
            gt_3d_mask = labels['mask3d']

            out_sigma = self.fc_sigma(feat).reshape(BATCH_SIZE, self.num_joints, -1)
            sigma = out_sigma.reshape(BATCH_SIZE, self.num_joints, -1).sigmoid() + 1e-9
            scores = 1 - sigma
            scores = torch.mean(scores, dim=2, keepdim=True)
            bar_mu = (pred_jts - gt_uvd) / sigma
            bar_mu = bar_mu.reshape(-1, 3)

            bar_mu_3d = bar_mu[gt_3d_mask > 0]
            bar_mu_2d = bar_mu[gt_3d_mask < 1][:, :2]

            log_phi = torch.zeros_like(bar_mu[:, 0])
            # (B, K, 3)
            num_3d = bar_mu_3d.shape[0]
            num_2d = bar_mu_2d.shape[0]
            if num_3d:
                log_phi_3d = self.flow3d.log_prob(bar_mu_3d)
                log_phi[gt_3d_mask > 0] = log_phi_3d
            if num_2d:
                log_phi_2d = self.flow2d.log_prob(bar_mu_2d)
                log_phi[gt_3d_mask < 1] = log_phi_2d

            log_phi = log_phi.reshape(BATCH_SIZE, self.num_joints, 1)
            nf_loss = torch.log(sigma) - log_phi

            return pred_jts, scores, nf_loss, sigma, hand_type, hand_valid
        else:
            return pred_jts, hand_type, hand_valid

and loss as follow:

class RLELoss3D(nn.Module):
    ''' RLE Regression Loss 3D
    '''

    def __init__(self, OUTPUT_3D=False, size_average=True):
        super(RLELoss3D, self).__init__()
        self.size_average = size_average
        self.amp = 1 / math.sqrt(2 * math.pi)

    def logQ(self, gt_uv, pred_jts, sigma):
        return torch.log(sigma / self.amp) + torch.abs(gt_uv - pred_jts) / (math.sqrt(2) * sigma + 1e-9)

    def forward(self, pred_jts, nf_loss, sigma, target):
        gt_uv = target.reshape(pred_jts.shape)
        Q_logprob = self.logQ(gt_uv, pred_jts, sigma)
        loss = nf_loss + Q_logprob

        if self.size_average:
            return loss.sum() / len(loss)
        else:
            return loss.sum()

Could you provide any suggestions about debugging?

训练第一个epoch的loss变成负数

loss: -8.48068483 | acc: 0.2207: 71%|█████████████████████▎ | 417/586 [47:52<27:15, 9.68s/it]

Two questions

Dear author，

I have 2 questions for you,

1. What's the purpose of following calculation:
   img[0].add_(-0.406)
   img[1].add_(-0.457)
   img[2].add_(-0.480)

2. Have you ever succeeded in converting pth to onnx format?

Thanks for your time.

Question Related the Norm Term in Linear Layer

Hi there, really thanks for your amazing work. When reading your source code, I found one place that I don't really understand its meaning. In your regression_nf.py (model), you re-construct the Linear layer and add a norm term. Later, when you regress the out_coord, you use the norm term; and when you regress the sigma, you didn't use that.

Thus, I would like to know what is that norm term for? Why do you use it for the coordinate regression?

Hope to get some feedback from you, thanks :))

How do multiple point detection works?

I am confused how multiple key point detection works. One-point is easy, just compare the result. Two-point is not, because the model could output A, B or B, A, while the ground truth is a, b. So, multiple-point can have different permutation to compare.

Another thing is, it is possible the model output shape is (B, N, 2) but the number of points needed to be detected is smaller than N. Then, how do you know the exact number of points dynamically.

How about model output repeated points? NMS?

Implementation of logQ

https://github.com/Jeff-sjtu/res-loglikelihood-regression/blob/203dc3195ee5a11ed6f47c066ffdb83247511359/rlepose/models/criterion.py#L37
Sorry for bothering you, I wonder if I'm right thinking the above line misimplemented the loss from $logQ(\bar\mu)$ in the paper to $logQ(\mu)$？

Mpii precision

Simplepose's accuracy on mPII validation set is 33.911 in his paper, why is it 25.4 in yours? The MPII annotations used in your code also appear to be different from other networks, such as SimpleBaseline.

A question about self.fcs

Hello, I just notice that you create a self.fcs in both 2d and 3d models, but it seems that it isn't used in anywhere

from regression_nf_3d.py line 92:
self.fcs, out_channel = self._make_fc_layer()

Could you tell me why does it exist?

Question about the norm in linear layer

Hi, I am curious about the normaliztion applied for fc_coord. What's the meaning of this line? Why should the output divide the norm of the input?

res-loglikelihood-regression/rlepose/models/regression_nf.py

Line 33 in 203dc31

y = y / x_norm

Per joint occlusion sensitivity analysis

How to calculate per joint occlusion sensitivity?

about conflict between the paper and this implement

thanks for your great work. i prepare to site your paper. but i meat the mask rcnn's setting issue. in your paper, you said mask rcnn is trained without heatmap pretrain and the trainin schedule is 3X (27000iter). but this is different from the setting you mentioned on another issue #12 , in which you said you trained the model using 2X and heatmap pretrain. so can you please provide the excat setting of the mask rcnn you used in you paper? thanks a lot!

Training error

valueerror occurs at self.flow.log_prob() during training.
NAN appears in the values.

the meaning of gt box and the det box？

Can not find 'Sample_64_test_Human36M_protocol_2.json'

Hi, I want to try to evalute on human3.6m, but I can't find the annotation file 'Sample_64_test_Human36M_protocol_2.json'. Have you provided this file?

about the implement of RLE on HRNet

Hi, i'm interesting in RLE. it is a nice work.
i see you use soft_argmax on HRNet, but i didn't see it in this repo. can you introduce the implement detail about this? eg. are you using the softmax or sigmoid heatmap? is it the same with intergral pose? for the sigma branch, are you using the p2 feature as the input? are there any else needs to notice? thank you very mcuh!

performance gap between 2d and 3d data

Hi,
I'm training by mixing 2d and 3d data, but the 3d-prediction performance is much worse than 2d.
the training log is like this:

the pink line is an Integral Pose model, and the green line is RLE.
As you can see, the 2d performance is competitive(RLE is better in fact), but the 3d performance is worse
Due to I'm doing mix-data training, I calculate my RLE loss as follow:

        is_2d = ~is_3d  # masks of 2d and 3d
        num_3d = is_3d.sum()
        num_2d = is_2d.sum()

        t_cnt = 0
        ### joint loss ###
        jc_loss = 0.
        if num_3d > 0:
            jc_loss += self.joint_loss(joint_coord3d[is_3d], nf_loss[is_3d], sigma[is_3d], target_coord[is_3d])
            t_cnt += 1.
        if num_2d > 0:
            jc_loss += self.joint_loss(joint_coord3d[is_2d, :, :2], nf_loss[is_2d, :, :2], sigma[is_2d, :, :2], target_coord[is_2d, :, :2])
            t_cnt += 1.
        jc_loss /= t_cnt

Do you have any suggestion?

can release the high AP model, e.g. HRNet + RLE (Ours) HRNet-W48 (75.7)?

The order of height and width of the heatmap in the metrics is different.

In this implementation, PCK@50 is used to evaluate the inference results.

res-loglikelihood-regression/rlepose/utils/metrics.py

Lines 115 to 150 in 71a7e8f

	def calc_accuracy(output, target):
	"""Calculate heatmap accuracy."""
	preds = output.heatmap
	labels = target['target_hm']
	labels_mask = target['target_hm_weight']
	preds = preds * labels_mask
	labels = labels * labels_mask
	preds = preds.cpu().data.numpy()
	labels = labels.cpu().data.numpy()
	num_joints = preds.shape[1]
	norm = 1.0
	hm_h = preds.shape[2]
	hm_w = preds.shape[3]
	preds, _ = get_max_pred_batch(preds)
	labels, _ = get_max_pred_batch(labels)
	norm = np.ones((preds.shape[0], 2)) * np.array([hm_w, hm_h]) / 10
	dists = calc_dist(preds, labels, norm)
	acc = 0
	sum_acc = 0
	cnt = 0
	for i in range(num_joints):
	acc = dist_acc(dists[i])
	if acc >= 0:
	sum_acc += acc
	cnt += 1
	if cnt > 0:
	return sum_acc / cnt
	else:
	return 0

However, I am wondering if the order of the height and width of the heatmap used for normalizing is reversed.

norm = np.ones((preds.shape[0], 2)) * np.array([hm_w, hm_h]) / 10

Other implementations of the awesome method (SimpleBaseline, HRNet, DarkPose)are as follows.

norm = np.ones((pred.shape[0], 2)) * np.array([h, w]) / 10

How to make \bar\mu always mean 0, variance 1?

From the paper \bar\mu seems should be 0 mean, variance 1 random variable.
Is there any mechanism to guarantee that? Thank you.

about the implement of RLE on two-stage 3D HPE mehtods

Hi,
I'm interested in RLE. it is a nice job.
I notice that you embed RLE into two-stage 3D HPE methods, such as SRnet, but I didn't see it in this repo. can you introduce the implementation detail about this? Is there any else that needs to notice?

Thanks very much!

Some related or extended questions

I implemented a tensorflow keras version for 2D single point regression with some success. The mu is more accurate (sometimes better than ground truth due to human label error) and sigma do have a meaningful representation.

But I have some questions,

I would like to know if sigma bounded by (0, 1) is solely for the ease of classifying missing point or there is another reason. I tried using softplus in sigma so that it is > 0, the result is also very good and the sigma no longer squeezed between (0, 1), a blurry/noisy image can have very high sigma > 4.

As it is possible that the key point is missing in the image, I used a simple classification output previously, i.e. output one more sigmoid value to represent if the point exists. Now I have sigma, but few normal cases' sigma do overlap with missing point cases' sigma around ~0.5 (using softplus). Previously, I can feed missing point image to train, but now I can't? Just asking for some advice, how do I improve missing point detection?

cannot reproduce the results on human3.6m

Hi,
I have tried to train the model on human3.6m dataset, but cannot get the same performance as the provided pretrained model 'h36m-laplace-rle.pth'.
I was using the annotation files and codes from this repository, and the environment is based on pytorch1.8.

The training behaives normally at the beginning, but after some epoch, the loss value was increased and MPJPE reached 200.

Train-9 epoch | loss:-179.81700464 | acc:0.5870
##### Epoch 9 | gt results: 78.48804327541252/78.48804327541252 #####
############# Starting Epoch 10 | LR: 0.001 #############
Train-10 epoch | loss:-178.24089413 | acc:0.5794
############# Starting Epoch 11 | LR: 0.001 #############
Train-11 epoch | loss:-105.90890308 | acc:0.2606
##### Epoch 11 | gt results: 206.2949851515231/78.48804327541252 #####
############# Starting Epoch 12 | LR: 0.001 #############
Train-12 epoch | loss:-99.68143688 | acc:0.2141
############# Starting Epoch 13 | LR: 0.001 #############
Train-13 epoch | loss:-134.96754921 | acc:0.3854

Before lr step, the model would collapse.

############# Starting Epoch 60 | LR: 0.001 #############
Train-60 epoch | loss:-99.61992546 | acc:0.2145
##### Epoch 60 | gt results: 183.75314692309206/71.80444962169011 #####
############# Starting Epoch 61 | LR: 0.001 #############
Train-61 epoch | loss:16064.89554501 | acc:0.0533
##### Epoch 61 | gt results: 973.8534021229914/71.80444962169011 #####
############# Starting Epoch 62 | LR: 0.001 #############
Train-62 epoch | loss:-18.63251462 | acc:0.0071

I doubted if it was from the very large lr, and then experimented with initial lr 10e-4. The loss value was lower but also increased after some epoch.

##### Epoch 64 | gt results: 83.14857669377162/64.38682879287299 #####
############# Starting Epoch 65 | LR: 0.0001 #############
Train-65 epoch | loss:-208.61684037 | acc:0.6639
##### Epoch 65 | gt results: 74.92741776420186/64.38682879287299 #####
############# Starting Epoch 66 | LR: 0.0001 #############
Train-66 epoch | loss:-207.51670638 | acc:0.6620
##### Epoch 66 | gt results: 77.90103486437523/64.38682879287299 #####
############# Starting Epoch 67 | LR: 0.0001 #############
Train-67 epoch | loss:-176.32472946 | acc:0.5414
##### Epoch 67 | gt results: 204.0743792529304/64.38682879287299 #####
############# Starting Epoch 68 | LR: 0.0001 #############
Train-68 epoch | loss:-126.63842224 | acc:0.3175
##### Epoch 68 | gt results: 124.20229629684249/64.38682879287299 #####

If the lr is dropped to 1e-5 before the , I got a final result of 66 MPJPE, compared to the 38 MPJPE by pretrained model.

Additionally, I have tried to downgrade the pytorch to 1.5, but got similar phenomenon.

Do you have any suggestions? Thanks a lot for your help.

Just curious about what 'SPPE' represents for?

How to replace normal distribution with laplace distribution

Great jobs！
but，How to replace normal distribution with laplace distribution

请问推理代码在哪呢？测试单个图片或视频，可视化结果？

感谢

mask rcnn + rle hyper parameters

Hi, I want to train mask rcnn with rle loss！

1. Could you give me some details about the optimizer and learning rate scheduler used in your experiments? I found rle uses the adam while the original mask rcnn uses the sgd?
2. Do you use the exactly same flow model architecture as released in this codebase?
3. Does the rle loss directly regress absolute coordinates of the image or relative coordinates of the pred boxes? (e.g. normalize the target keypoint to range [-0.5, 0.5])
   Thanks!

Comparing with uncertainty learning

Hi, thanks for your amazing work!
I am curious about whether you have compared your method with uncertainty learning qunatitatively?

Laplace RealNVP

I only find the implementation of MultivariateNormal of RealNVP. Could you open source your Laplace of RealNVP？I do not find MultivariateLaplace in pytorch.

l1Loss

Quesion about NUM_FC_FILTERS set by -1

@Jeff-sjtu
Hi, thank you for your great work! Recently, I study your code, and I found that NUM_FC_FILTERS is set to -1 in YAML file. It will cause the function is the identity mapping. Is it correct?

Error When Installing

Hi, I was just installing this repo, but when I run python setup.py develop, I get some errors in Post-processing (stage 2), and I don't know what do they mean. Could you please help me? Thanks a lot!

Post-processing (stage 2)...
Building modules...
        Building module "mvn"...
                Constructing wrapper function "mvnun"...
                  value,inform = mvnun(lower,upper,means,covar,[maxpts,abseps,releps])
                Constructing wrapper function "mvndst"...
                  error,value,inform = mvndst(lower,upper,infin,correl,[maxpts,abseps,releps])
                Constructing COMMON block support for "dkblck"...
                  ivls
        Wrote C/API module "mvn" to file "build/src.linux-x86_64-3.8/scipy/stats/mvnmodule.c"
        Fortran 77 wrappers are saved to "build/src.linux-x86_64-3.8/scipy/stats/mvn-f2pywrappers.f"
no previously-included directories found matching 'benchmarks/env'
no previously-included directories found matching 'benchmarks/results'
no previously-included directories found matching 'benchmarks/html'
no previously-included directories found matching 'benchmarks/scipy'
no previously-included directories found matching 'scipy/special/tests/data/boost'
no previously-included directories found matching 'scipy/special/tests/data/gsl'
no previously-included directories found matching 'scipy/special/tests/data/local'
no previously-included directories found matching 'doc/build'
no previously-included directories found matching 'doc/source/generated'
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.bak' found anywhere in distribution
warning: no previously-included files matching '*.swp' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
CCompilerOpt.generate_dispatch_header[2281] : dispatch header dir build/src.linux-x86_64-3.8/numpy/distutils/include does not exist, creating it

about h36m dataset

Hi Jeff，

I downloaded h36m dataset from the official website. Based on your dataset structure, there are supposed to be some sub directories for h36m data, e.g, "s_01_act_02_subact_01_ca_03". But no such sub directories are generated after I uncompress the images archives. My question is that if I have to download h36m from the website mentioned in this project, i.e:

5. Download COCO data, MPII (annotations) and Human3.6M data (from PoseNet or ours):

Thanks a lot.

rlepose

Loss becomes very big suddenly when after epoch 10

Dear author,

I encountered an issue when training on coco keypoints dataset. I just adjusted num_layers from 50 to 18, i.e, res18, and started to train based on coco keypoints dataset. The loss went down quickly and became negative soon. But after epoch 10, the loss became much bigger(over 30000) suddenly and then dropped again in next epoch. How do I understand such phenomenon? Any tips for that? Thanks!!

about training on COCOdataset

negative loss val problem

Thanks for the great work! I run the code and find sometimes the loss value becomes negative. Is it normal?

about heatmap_to_coord

Hi author，

For the function 'heatmap_to_coord' defined in rlepose/utils/transforms.py:

coords[:, :, 0] = (coords[:, :, 0] + 0.5) * hm_width
coords[:, :, 1] = (coords[:, :, 1] + 0.5) * hm_height

what's the aim of "0.5" ? Any reason for that? Wait for your feedback, thanks.

can't reproduce the result present in the paper

hi @Jeff-sjtu
i use the cmd ./scripts/train.sh ./configs/256x192_res50_regress-flow.yaml train_rle_coco to reproduce the result present in the paper, but only get ##### Epoch 255 | gt mAP: 0.7123198166036238 | det mAP: 0.6962231312113063 #####, lower than the released model you provide where gt box: 0.7218652898214926 mAP | det box: 0.7127219006071578 mAP. can u give the complete training pipeline to reproduce the result?
thanks

masks in the RealNVP model

Thanks for your excellent work. Do the masks in the RealNVP model correspond to the shuffle operation?