How to use the PNG format in my own dataset? about deeplab2

[ClaireXie]

Hi ilaripih,

You can refer to how we created the panoptic map for KITTI-STEP dataset:

deeplab2/data/utils/create_step_panoptic_maps.py

Lines 106 to 112 in e038967

	def encode_panoptic_map(panoptic_map: np.ndarray) -> np.ndarray:
	"""Encodes the panoptic map in three channel image format."""
	# Encoding format: R: semantic | G: instance // 256 | B: instance % 256
	semantic_id = panoptic_map // INSTANCE_LABEL_DIVISOR
	instance_id = panoptic_map % INSTANCE_LABEL_DIVISOR
	return np.dstack(
	(semantic_id, instance_id // 256, instance_id % 256)).astype(np.uint8)

Here we encode the panoptic map with the following PNG format:

R: semantic
G: instance // 256
B: instance % 256

And we decode the png here:

deeplab2/data/build_step_data.py

Lines 181 to 186 in e038967

	semantic_map = panoptic_map[:, :, 0]
	instance_map = (
	panoptic_map[:, :, 1] * _ENCODED_INSTANCE_LABEL_DIVISOR +
	panoptic_map[:, :, 2])
	panoptic_map = semantic_map * _INSTANCE_LABEL_DIVISOR + instance_map
	return panoptic_map.tobytes()

[ilaripih]

Thanks for the quick response. To me it looks like the encode_panoptic_map code you're referring to is meant for creating intermediate PNG files before creating the final TFRecord dataset which uses the "raw" (int32) format. I'm creating the TFRecord dataset directly from my own dataset, skipping the intermediate steps.

I successfully encoded the panoptic labels in PNG format in "image/segmentation/class/encoded" and set the "image/segmentation/class/format" field to "png" but it looks like the Deeplab2 code still wants to use the raw format and is complaining about the PNG data:

Traceback (most recent call last):
  File "deeplab2/trainer/train.py", line 76, in <module>
    app.run(main)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "deeplab2/trainer/train.py", line 72, in main
    FLAGS.num_gpus)
  File "/workspace/deeplab2/trainer/train_lib.py", line 191, in run_experiment
    steps=config.trainer_options.solver_options.training_number_of_steps)
  File "/workspace/models/orbit/controller.py", line 240, in train
    self._train_n_steps(num_steps)
  File "/workspace/models/orbit/controller.py", line 439, in _train_n_steps
    train_output = self.trainer.train(num_steps_tensor)
  File "/workspace/models/orbit/standard_runner.py", line 146, in train
    self._train_loop_fn(self._train_iter, num_steps)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 885, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 950, in _call
    return self._stateless_fn(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 3040, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1964, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  Input to DecodeRaw has length 18802 that is not a multiple of 4, the size of int32
	 [[{{node DecodeRaw}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
	 [[RemoteCall]]
	 [[while/body/_1/IteratorGetNext]]
	 [[while/body/_1/DeepLabFamilyLoss/MaXDeepLabLoss/maximum_bipartite_matching/while/body/_8657/DeepLabFamilyLoss/MaXDeepLabLoss/maximum_bipartite_matching/while/find_augmenting_path/while/body/_8938/DeepLabFamilyLoss/MaXDeepLabLoss/maximum_bipartite_matching/while/find_augmenting_path/while/mul/_724]]
  (1) Invalid argument:  Input to DecodeRaw has length 18802 that is not a multiple of 4, the size of int32
	 [[{{node DecodeRaw}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
	 [[RemoteCall]]
	 [[while/body/_1/IteratorGetNext]]

I think I'm hitting this line that forces the use of the "raw" format in panoptic datasets: https://github.com/google-research/deeplab2/blob/main/data/data_utils.py#L353
I'll try the raw/int32 format.

[ilaripih]

With the "raw" format I can run the training code.

[ClaireXie]

Yes, current we only support "raw" format creating tfrecord dataset.
You might need to either try with raw format or modify the _decode_label in data_utils to decode png data.

[ilaripih]

Thanks. I can manage with the raw format, it's working.