armandpl / quick_calib

Hi there 👋

My name is Armand, I can make things. Among those things are RL Robots 🤖, autonomous 1/10th racecars 🏎️, Minecraft hacks 🥷 and ✨jewels✨. I interned at Weights&Biases🪄🐝 then at Loyal For Dogs 🐶 and studied at ECE Paris.

Reach out at [email protected], DM me @armand_dpl :)

You'll find an index of my recent blog posts, videos and projects at armandpl.com.

quick_calib's Issues

investigate possibility of generating a synth dataset

lightly annoyed that the vpgnet dataset is/seems to be imbalanced. annoyed i don't have easy access to the DeepVP-1M dataset. Can I make my own dataset?

• generate new image using controlnet and and canny edges
• generate lane lines canny using blender then gen using controlnet
• try sdxl turbo to make it faster
• make simple blender scene with 4 lane lines, render edges, gen with sdxl and see
• add more stuff in the blender scene
  • cars/trucks (get very simple models)
  • randomly double/delete lane lines, make them dotted
  • automate the camera to get a bunch of different angles

establish manual baseline by finding the vps by hand

find vps by hand in the 5 labeled videos and compute the score we would get

plan

If we assume the car is going parallel to the lanes lines on flat ground, the direction of travel is the same as the extrinsics parameters which we can find using the vanishing point. Looking at openpilot code it seems that's how its done.
We could find the vanishing point for each image, remove outliers and average the predictions to get a better estimate of the extrinsics parameters. (also are we gaining that much by averaging over all images? can't we just select one by hand and find the vp by hand? if we're not gaining much there is no point in training a neural net. that said if we want to go further and actually find the direction of travel as opposed to extrinsics that might make sense)

To estimate if this would be enough to get <25% I averaged the 0-5 labels and got ~6%. However, the test videos look a bit more difficult than the labeled ones so maybe the assumption of flat ground + straight path won't hold.
I found two road vanishing point datasets:

• VPGNet Dataset which i easily downloaded
• DeepVP-1M which was scraped from Google StreetView, couldn't find a download link, emailed the researchers haven't received an answer yet

todo:

• #2
• #4
• #5
• #8
• remove bad images by looking at loss outliers
• save more metadata when generating images
  • nb lines
  • camera pose
  • which line is dotted or not?
  • other generation parameters e.g lane line width
• optimize sdxl generation speed? go back to sd 1.5? smaller images?

eda on vpgnet dataset

• convert from .mat to .png + labels
• count number of images
• look at the distribution of the vanishing point, maybe convert to pitch yaw and look at that distrib too

train nn to predict vp on synth dataset

• refactor training script
• random horizontal flip?
• log part of the validation set to wandb + display vp pred on it
• write script to manually inspect the dataset and mask images. make it plot the loss distrib before showing images.
  • have the banishing script load banished.npy if it exists
  • update the dataset class to not load banished images, make sure it then still works with the banishing script
  • add a reverse mode to the script where we only see banished images and can unbanish them?
  • add preds along label?
• add option to load multiple datasets at once then concat them
  • maybe a good time to save more metadata about datasets
  • and to better parametrize the gen script, maybe just save the config to the directory of the dataset? also add more info per image?
• viz on calib_challenge too, again maybe just one batch. consider direction of travel == vanishing point
• write testing func: inference on calib_challenge, average preds w/o outliers and eval
• split hydra config file into configs fileS