zhjohnchan / r2gen

[EMNLP-2020] The official implementation of Generating Radiology Reports via Memory-driven Transformer.

Hi there. I'm Zhihong 👋

r2gen's Issues

Annotation of Mimic-cxr

Could you please provide the annotation of mimic_cxr? Without this annotation, I cannot directly use your checkpoint for testing.
Looking forward to your reply.

The number of images and reports are not correct

Hi, thank you for the paper.

However, when I download the IU X-Ray dataset you provided, the number of images is not correct as you reported in the paper.
In detail, there are only 6091 images, and 2069, 296, 590 reports for training, validation and testing sets, respectively.

Can you check again?

Thank you very much.

about MIMIC dataset

Is it possible that the MIMIC data downloaded from this page contains MiMIC images processed differently from the MiMIC images available on the physionet site? Or is it just the MIMIC annotation files?

Couldn't visualize the captions generated on IU_Xray dataset by revised the plot_mimic_cxr.sh.

Hi, good work and thank you!

I followed the bash plot_mimic_cxr.sh step you have mentioned. I want to visualize the captions generated on IU_Xray dataset by using model_iu_xray.pth, since i am not yet able to download the MIMIC_CXR dataset.

But it has the following errors：

RuntimeError: Error(s) in loading state_dict for R2GenModel:
size mismatch for encoder_decoder.model.tgt_embed.0.lut.weight: copying a param with shape torch.Size([761, 512]) from checkpoint, the shape in current model is torch.Size([400, 512]).
size mismatch for encoder_decoder.logit.weight: copying a param with shape torch.Size([761, 512]) from checkpoint, the shape in current model is torch.Size([400, 512]).
size mismatch for encoder_decoder.logit.bias: copying a param with shape torch.Size([761]) from checkpoint, the shape in current model is torch.Size([400]).

Kindly any guidance， thanks!

Impression or Finding section

Hi, thanks for your work and for publishing your code!
One question, for the MIMIC-CXR dataset, do you predict the impression section, the finding section or concatenate both?
Thanks a lot in advance,
Chantal

Visualization of Captions

Hey, I followed the steps you have mentioned but I couldn't visualize the captions generated. I took IU_Xray dataset. Kindly guide me. Thank you.

Clinical efficacy score

Hi,

I used your trained mimic-cxr weight to infer the test set. I get almost the same result as the NLP score. But, I also followed your instructions to use Chexpert and Chexbert to label the report and calculate the CE score using your script. I tried the two label methods you gave me but I can't reproduce your result.

Here is my result.
{'F1_MACRO': 0.21054676943711254,
'F1_MICRO': 0.37574679039023773,
'RECALL_MACRO': 0.3051327263640022,
'RECALL_MICRO': 0.4828487422410977,
'PRECISION_MACRO': 0.19370866272550938,
'PRECISION_MICRO': 0.3075322513524761}

A beam search error occurs during the referencing process.

Hello, could you please tell me why I am getting beam search error in the "assert (beam_logprobs_sum == ys).all()" line of the caption_model.py file during the reproduction of the mimic dataset?

Running the checkpoint for 'mimic_cxr' results in error

When running the checkpoint, an error occurs. Why is it that running the checkpoint for 'iu_xray' works fine, but running the checkpoint for 'mimic_cxr' results in the following error?

RuntimeError: Error(s) in loading state_dict for R2GenModel:
size mismatch for encoder_decoder.model.tgt_embed.0.lut.weight: copying a param with shape torch.Size([4336, 512]) from checkpoint, the shape in current model is torch.Size([3595, 512]).
size mismatch for encoder_decoder.logit.weight: copying a param with shape torch.Size([4336, 512]) from checkpoint, the shape in current model is torch.Size([3595, 512]).
size mismatch for encoder_decoder.logit.bias: copying a param with shape torch.Size([4336]) from checkpoint, the shape in current model is torch.Size([3595]).