some problem about train.py on linux with tensorboardX 1.1 about sfmlearner-pytorch

[ClementPinard]

First guess is that your tensorboardX versio is out of date.
Now tensorboardX expects a CHW output to make it an image, regardless of the fact that's torch or numpy, which was not the case before.

so i suggest you try to update tensorboardX with pip

[zenithfang]

zfang@biwirender17:/scratch_net/biwidl204/zfang/SfmLearner-Pytorch-master$ CUDA_VISIBLE_DEVICES=$SGE_GPU python3 train.py /scratch_net/biwidl204_second/zfang/kitti_sfm/ -b4 -m0 -s2.0 --epoch-size 1000 --sequence-length 5 --log-output --with-gt

=> will save everything to checkpoints/kitti_sfm,epoch_size1000,seq5,s2.0/11-19-14:33
=> fetching scenes in '/scratch_net/biwidl204_second/zfang/kitti_sfm/'
38564 samples found in 54 train scenes
5164 samples found in 8 valid scenes
=> creating model
=> no mask loss, PoseExpnet will only output pose
=> setting adam solver


N/A% (0 of 200) |                        | Elapsed Time: 0:00:00 ETA:  --:--:--

 * Avg Loss : 2.038
100% (1000 of 1000) |####################| Elapsed Time: 0:02:38 ETA:  00:00:00


N/A% (0 of 1291) |                       | Elapsed Time: 0:00:00 ETA:  --:--:--

Traceback (most recent call last):
  File "/scratch_net/biwidl204/zfang/anaconda3/lib/python3.7/site-packages/PIL/Image.py", line 2460, in fromarray
    mode, rawmode = _fromarray_typemap[typekey]
KeyError: ((1, 1, 128), '|u1')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 515, in <module>
    main()
  File "train.py", line 219, in main
    errors, error_names = validate_with_gt(args, val_loader, disp_net, epoch, logger, output_writers)
  File "/scratch_net/biwidl204/zfang/anaconda3/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 46, in decorate_no_grad
    return func(*args, **kwargs)
  File "train.py", line 488, in validate_with_gt
    epoch)
  File "/scratch_net/biwidl204/zfang/anaconda3/lib/python3.7/site-packages/tensorboardX/writer.py", line 412, in add_image
    self.file_writer.add_summary(image(tag, img_tensor), global_step, walltime)
  File "/scratch_net/biwidl204/zfang/anaconda3/lib/python3.7/site-packages/tensorboardX/summary.py", line 205, in image
    image = make_image(tensor, rescale=rescale)
  File "/scratch_net/biwidl204/zfang/anaconda3/lib/python3.7/site-packages/tensorboardX/summary.py", line 243, in make_image
    image = Image.fromarray(tensor)
  File "/scratch_net/biwidl204/zfang/anaconda3/lib/python3.7/site-packages/PIL/Image.py", line 2463, in fromarray
    raise TypeError("Cannot handle this data type")
TypeError: Cannot handle this data type

It still arise problem.

[ClementPinard]

Last pypi package is 1.4, and it should work with it. Unfortunately, there is a breakage between version 1.4 and anterior versions, and it cannot work for both versions unless we put a switch between tqdm versions, which i don't find ideal for readability reasons.
https://pypi.org/project/tensorboardX/1.4/
you can easily install with the -U version
pip install -U tensorboardX

If you do not wish to update tensorbaordX to 1.4, you can pass a HWC tensor to the add_image function instead of CHW (the way it was used before)

[ClementPinard]

After some research, turns out the function tensor2array is actually broken, a patch is coming. However, the patch still won't make it work with version 1.1.

Once the function is patched, I advise you to also update this repo inaddition to tensorboardX to version 1.4

[ClementPinard]

I just pushed changes to the function tensor2array you can now choose if you want channel to be first or last. With an outdated version of tensorboardX, just set function calls of tensor2array with argument channel_first to False