ralphmao / waymo-dataset-tool

A tool to download waymo open dataset (https://github.com/waymo-research/waymo-open-dataset) and convert it from TFrecord to general format

Waymo Open Dataset Download Tool

This tool downloads Waymo Open Dataset and converts it into KITTI-like format that consists of individual images and txt labels.

Prerequisite

• python 2 or 3
• gsutil (command line tool to download files from google cloud. Follow this Guide to install. After installing the Cloud SDK, simply run gcloud init, then you will be asked to log into your google account. Make sure it is the same account that has Waymo Dataset access.)
• waymo_open_dataset (follow the Quick Start)

How to use

Before you run the following code, make sure you have set up gsutil and waymo_open_dataset! Currently only the original training and validation set are supported. Feel free to modify the code to download other data splits.

python batch_download.py training --out-dir /ssd/dataset/waymo --resize 1.0

waymo-dataset-tool's Issues

You are not converting the calibration files?

Hello. Can you please also convert the calibration files. I tried to do it myself but, due to waymo using different reference axis as compared to kitti, I was unable to store the extrinsic and intrinsic parameter matrices (P2 and Tr_velo_to_cam) correctly.

Difference

What is the difference between this repo and https://github.com/Yao-Shao/Waymo_Kitti_Adapter?

Bugs occur when running gsutil cp

Hello, thanks for your kindness sharing! When I run your code, a bug occur when it goes gsutil cp gs://waymo_open_dataset_v_1_0_0/validation /validation_0000.tar ./ , which outputs ServiceException: 401 Anonymous caller does not have storage.objects.get access to waymo_open_dataset_v_1_0_0/validation/validation_0000.tar. It seems like I don't have permission to get data. Could you give me a hand? Thank you.

Dataset size

I am assuming that this is a 2d object detection dataset that would be downloaded? I am wondering about the size of the dataset? How big would the dataset be? I have 500 GB in my system? Will it fit? As I read that the original dataset in 2 TB, however I believe that it has everything including pointclound and lidar, however I only need 2D object detection.

gsutil batch downloading

Hey, never used gsutil before. How is the batch downloading working?

If looking here i need some project id. Whats the project id for the waymo dataset?
https://colab.research.google.com/notebooks/io.ipynb#scrollTo=S7c8WYyQdh5i
Or do I have to setup a project my own and somehow link the waymo data? Is it free?
Just want to batch download the waymo dataset somehow....

Thanks in advance!

len(labels) == 0

In convert_tfrecord.py, labels = frame.camera_labels where the labels is always [].