camenduru / panohead-colab

🐣 Please follow me for new updates https://twitter.com/camenduru
🔥 Please join our discord server https://discord.gg/k5BwmmvJJU
🥳 Please join my patreon community https://patreon.com/camenduru

🦒 Colab

🚦 WIP 🚦

Colab	Info
	PanoHead_colab (generates 3D models from generated images)
	PanoHead_custom_colab (generates 3D models from custom images)

Tutorial

https://www.youtube.com/watch?v=QwPAo-f5UgM

For .ply output please add --shapes=True

After running the first cell, add your custom image named test.jpg to the in folder.

Maybe for better 3D we can play with
max_batch = 3000000 voxel_resolution = 512 --num-steps=300 --num-steps-pti=300

--num-steps=100 --num-steps-pti=100 max_batch = 100000

Free Colab Limit

--num-steps=200 --num-steps-pti=200 max_batch = 100000

Main Repo

https://github.com/SizheAn/PanoHead

Page

https://sizhean.github.io/panohead

Paper

https://arxiv.org/abs/2303.13071

Output

--num-steps=300 --num-steps-pti=300

2023-06-29.15-38-28.mp4

panohead-colab's Issues

No longer works 2024

First step seems to work but the second block of code doesn't work

I think this is the problem:

SizheAn/PanoHead#49

I added --shapes=True to your hints, but in the end I didn't find the ply file, what can I do to get the 3d model?

I added --shapes=True to your hints, but in the end I didn't find the ply file, what can I do to get the 3d model?

Custom image colab endless loop

I started the custom image colab on a free plan.
Uploaded 512*512 jpg face
First it went into at 402 steps.
Then i decreased both steps down to 400.
After doing the 400 steps the colab says 'Saving optimization progress video "/content/output/easy-khair-180-gpc0.8-trans10-025000.pkl/0/proj.mp4"' and looks like stucking in some loop, the cell processing icon is running but nothing is happening for an hour or so already

Is it possible to save textures along with geometry?

Not an issue, rather a question.
I see in your demo video the textured head rotating which makes me ask this question.

dataset.json extension mismatch

The dataset.json leaves the original extension, but the file is converted into a jpeg which results in a keyerror

Loading networks from "/content/PanoHead/models/easy-khair-180-gpc0.8-trans10-025000.pkl"... Traceback (most recent call last): File "/content/PanoHead/projector_withseg.py", line 439, in <module> run_projection() # pylint: disable=no-value-for-parameter File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/content/PanoHead/projector_withseg.py", line 325, in run_projection c = torch.from_numpy(dataset._get_raw_labels()[idx:idx+1]).to(device) File "/content/PanoHead/training/dataset.py", line 59, in _get_raw_labels self._raw_labels = self._load_raw_labels() if self._use_labels else None File "/content/PanoHead/training/dataset.py", line 244, in _load_raw_labels labels = [labels[fname.replace('\\', '/')] for fname in self._image_fnames] File "/content/PanoHead/training/dataset.py", line 244, in <listcomp> labels = [labels[fname.replace('\\', '/')] for fname in self._image_fnames] KeyError: '[YOURFILENAME].jpg'

Readme Asset

i am getting an error as a result of recent updates can you help me?

/////////////////////////////////////////////////////
%cd /content/PanoHead
!python gen_videos_proj_withseg.py --output=/content/output/easy-khair-180-gpc0.8-trans10-025000.pkl/0/PTI_render/post.mp4 --latent=/content/output/easy-khair-180-gpc0.8-trans10-025000.pkl/0/projected_w.npz --trunc 0.7 --w-frames 120 --network /content/output/easy-khair-180-gpc0.8-trans10-025000.pkl/0/fintuned_generator.pkl --shapes 1 --cfg Head ##--nrr 256

//////////////////////////////////////////////////////in colab/////////////////
/content/PanoHead
Loading networks from "/content/output/easy-khair-180-gpc0.8-trans10-025000.pkl/0/fintuned_generator.pkl"...
Setting up PyTorch plugin "bias_act_plugin"... Done.
Setting up PyTorch plugin "upfirdn2d_plugin"... Done.
0% 0/120 [00:00<?, ?it/s]Generating shape for frame 0 / 120 ...

7% 10000000/134217728 [00:00<00:09, 13256492.97it/s]
0% 0/120 [00:11<?, ?it/s]
Traceback (most recent call last):
File "/content/PanoHead/gen_videos_proj_withseg.py", line 301, in
generate_images() # pylint: disable=no-value-for-parameter
File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/content/PanoHead/gen_videos_proj_withseg.py", line 296, in generate_images
gen_interp_video(G=G, mp4=output, ws=ws, bitrate='100M', grid_dims=grid, num_keyframes=num_keyframes, w_frames=w_frames, psi=truncation_psi, truncation_cutoff=truncation_cutoff, cfg=cfg, image_mode=image_mode, gen_shapes=shapes, device=device)
File "/content/PanoHead/gen_videos_proj_withseg.py", line 172, in gen_interp_video
sigma = G.sample_mixed(samples[:, head:head+max_batch], transformed_ray_directions_expanded[:, :samples.shape[1]-head], w.unsqueeze(0), truncation_psi=psi, noise_mode='const')['sigma']
File "", line 146, in sample_mixed
File "", line 148, in run_model
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "", line 179, in forward
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/activation.py", line 843, in forward
return F.softplus(input, self.beta, self.threshold)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.38 GiB (GPU 0; 14.75 GiB total capacity; 10.78 GiB already allocated; 1.69 GiB free; 12.04 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

My Own Image

How can I get some different angle image by my on in-the-wild image like the official readme: gen_samples

Generating Shapes on Free Colab

I couldn't generate .ply in colab, even with the provided settings for free colab limit.
I removed the save_video part in projector_withseg.py, ran again, and it did generate .ply.
Why is that?

Saving Textures

So I wanted to know the following: How do we save the textures for the 3d ply file?

Extra tips: In case you're wondering, I was playing around with the free version of collab and couldn't get it to run 180 samples, just 160ish, if you pay for collab plus be sure to change your enviroment to an a100 with ample ram