Save images or videos in headless mode about igibson

[cgokmen]

Hi, definitely - if you just want to get them from the robot's perspective as you run simulation using the Environment APIs, you can simply get the observation returned by env.step(), pick out the camera image, and save that as an image using PIL.Image APIs.

If you want to dive deeper and be able to manipulate the camera outside the typical robotics environment context (e.g. no robot, no physics, just photos with a particular purpose) you can take a look at this example:

iGibson/igibson/utils/derivative_dataset/pipeline.py

Lines 157 to 189 in 1f52c34

	def save_images(self, objs_of_interest):
	img_id = self.total_image_count
	rgb, segmask, threed = self.env.simulator.renderer.render(("rgb", "ins_seg", "3d"))
	rgb_arr = np.uint8(rgb[:, :, :3] * 255)
	rgb_img = Image.fromarray(rgb_arr)
	depth = np.clip(-threed[:, :, 2:3], 0, self.max_depth) / self.max_depth
	depth_arr = np.uint8(depth[..., 0] * 255)
	depth_img = Image.fromarray(depth_arr)
	seg = np.round(segmask[:, :, 0] * MAX_INSTANCE_COUNT).astype(int)
	body_ids = self.env.simulator.renderer.get_pb_ids_for_instance_ids(seg)
	unique_body_ids, lowered_body_ids = np.unique(body_ids, return_inverse=True)
	seg_arr = np.uint8(lowered_body_ids.reshape(body_ids.shape))
	seg_img = Image.fromarray(seg_arr)
	out_dir = self.output_path
	filename = f"{self.prefix}{img_id}.png"
	if self.export_rgb:
	rgb_dir = os.path.join(out_dir, "rgb")
	os.makedirs(rgb_dir, exist_ok=True)
	rgb_img.save(os.path.join(rgb_dir, filename))
	if self.export_depth:
	depth_dir = os.path.join(out_dir, "depth")
	os.makedirs(depth_dir, exist_ok=True)
	depth_img.save(os.path.join(depth_dir, filename))
	if self.export_seg:
	seg_dir = os.path.join(out_dir, "seg")
	os.makedirs(seg_dir, exist_ok=True)
	seg_img.save(os.path.join(seg_dir, filename))

(this is unmerged development code, but you can do this in your own code - it's just to give you an idea)

Essentially the simulator.renderer.render() function gives you the image as a numpy array. You can use PIL or cv2 to save that in your desired format.

For videos, you can either just save all your frames in a directory and run ffmpeg offline later to combine them into a video, or you can use cv2's VideoWriter to write a video on the fly. I'd recommend ffmpeg.

I'm closing this now but feel free to reopen if you have more questions.