avik-pal / raytracer.jl

Differentiable RayTracing in Julia

RayTracer.jl

This package was written in the early days of Flux / Zygote. Both these packages have significantly improved over time. Unfortunately the current state of this package of has not been updated to reflect those improvements. It also seems that it might be better to gradually transition to defining the adjoints directly using ChainRules

A Ray Tracer written completely in Julia. This allows us to leverage the AD capablities provided by Zygote to differentiate through the Ray Tracer.

INSTALLATION

This package is registered. So open up a Julia 1.3+ repl and enter the pkg mode.

julia> ]
(v1.3) pkg> add RayTracer

To use the master branch (not recommended) do.

julia> ]
(v1.3) pkg> add RayTracer#master

TUTORIALS

Introductory Rendering Tutorial	Inverse Lighting Tutorial	Inverse Rendering with Optim.jl Tutorial

USAGE EXAMPLES

Follow the instructions below to run individual examples or use examples/script.sh to run all of them together.

First we need to get the versions of the packages used when these examples were written.

$ cd examples
$ julia --color=yes -e "using Pkg; Pkg.instantiate()"

Now we can run any of the file we need by julia --project=. --color=yes "/path/to/file"

Running Individual Examples

• teapot_rendering.jl -- We need to download the teapot.obj file.

wget https://raw.githubusercontent.com/McNopper/OpenGL/master/Binaries/teapot.obj

• performance_benchmarks.jl -- We need the mesh and texture for the yield sign board.

$ mkdir meshes
$ cd meshes
$ wget https://raw.githubusercontent.com/avik-pal/RayTracer.jl/master/test/meshes/sign_yield.obj
$ wget https://raw.githubusercontent.com/avik-pal/RayTracer.jl/master/test/meshes/sign_yield.mtl
$ cd ..

$ mkdir textures
$ cd textures
$ wget https://raw.githubusercontent.com/avik-pal/RayTracer.jl/master/test/textures/wood_osb.jpg
$ wget https://raw.githubusercontent.com/avik-pal/RayTracer.jl/master/test/textures/sign_yield.png
$ cd ..

This example requires a few arguments to be passes from command line. Chack them using

julia --project=. --color=yes "performance_benchmarks.jl" --help

• inverse_lighting.jl & optim_compatibility.jl -- We need to the tree.mtl and tree.obj files.

$ wget https://raw.githubusercontent.com/tejank10/Duckietown.jl/master/src/meshes/tree.obj 
$ wget https://raw.githubusercontent.com/tejank10/Duckietown.jl/master/src/meshes/tree.mtl

Additional Examples

Duckietown.jl uses RayTracer.jl for generating renders of a self-driving car environment. For more complex examples of RayTracer, checkout that project.

SUPPORTING AND CITING:

This software was developed as part of academic research. If you would like to help support it, please star the repository. If you use this software as part of your research, teaching, or other activities, we would be grateful if you could cite the following:

@article{Pal2020,
  doi = {10.21105/jcon.00037},
  url = {https://doi.org/10.21105/jcon.00037},
  year = {2020},
  publisher = {The Open Journal},
  volume = {1},
  number = {1},
  pages = {37},
  author = {Avik Pal},
  title = {RayTracer.jl: A Differentiable Renderer that supports Parameter Optimization for Scene Reconstruction},
  journal = {Proceedings of the JuliaCon Conferences}
}

CURRENT ROADMAP

These are not listed in any particular order

• Add more types of common objects (use mesh rendering for this)
• Add support for rendering arbitrary mesh
• Inverse Rendering Examples
• Texture Rendering
• Application in Machine Learning Models through Flux (work in progress)
• Major Overhaul using Flux3D.jl
• Exploit the latest improvements to Flux and Zygote

raytracer.jl's Issues

Possible to fix lighting and optimize polygon position?

Really nice work. Reading https://avik-pal.github.io/RayTracer.jl/dev/getting_started/inverse_lighting/ I was wondering if the opposite problem of fixing the lighting position and direction, and instead estimate how a fixed number of polygons would be positioned to create the seen image?

The reason I ask is because I'd like to estimate the spatial distribution of cloud water from a number of observations. I would split the scene I'm observing into a fixed number voxels. Based on the amount of water in each voxel a ray would get attenuated and reflected by a known amount. And so the question is to estimate the amount of water in each voxel. Does this make sense?

Zygote.forward Usage in RayTracer.jl/docs/src/inverse_lighting.md

Hello, I going through some of your posted tutorials and I ran into a bug when using the Zygote.forward function (see below). I looked through some 0.4 and 0.5 versions of Zygote and couldn't find that function, so I am curious what that function is being used for. My current guess is that it's being used like the forwarddiff function in Zygote.

for i in 1:401
    loss, back_fn = Zygote.forward(light_guess) do L
        sum((render(L, scene) .- target_img) .^ 2)
    end
    @show loss
    gs = back_fn(1.0f0)
    update!(opt, light_guess.intensity, gs[1].intensity)
    update!(opt, light_guess.position, gs[1].position)
    if i % 5 == 1
        save("./results_inv_lighting/iteration_$i.png",
             showimg(zeroonenorm(render(light_guess, scene))))
    end
end

Question about DTM rendering

Hi,

Could this package be used to create images like the one in this GMT post

JuliaCon submission comments

Here are some comments on your JuliaCon submission in pdf.

The main things are:

1. Title and abstract mislead me about the paper contents, so I have suggestions. Feel free to use your own rewordings, since I may have misplaced the emphasis.
2. Experiments need enough detail that they can be replicated, esp for BVH and light positioning.
3. Plots in fig 1 & 2 should factor out the expected trend and be single column.
4. Little, but important details: drop the brackets around figure & algorithm numbers (makes them look like paper citations.), and keep the aspect ratio from Fig 4 a,b in c,d.
5. Comments on writing, particularly in consistent use of terms and context first. See Gopen & Swan's https://cseweb.ucsd.edu/~swanson/papers/science-of-writing.pdf
6. The issue of ray tracing being non-differentiable is mentioned in the first paragraph of the intro, but you never say how that affects what you can attempt. Good science points out the limitations of a technique.

See attached pdfs
Comments by page on 10.21105.jcon.00031 (1).pdf
Comments on 10.21105.jcon.00031 (1).pdf

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Perturbing the Camera results in incorrect Type Promotion in Backward Pass

I was trying to differentiate with respect to the Camera position and therefore included the get_primary_rays(c::Camera) function in the loss function.
Using Zygote 0.4.15 (and Julia 1.3), this results in a MethodError in the reverse pass, because at some point a Camera is instanciated with a vfov of type Float64 instead of Float32.
I have attached a minimal working example and the Stacktrace that results when executing it.

I am fairly new to Julia, so my investigation of the Zygote internals was a fruitless toil.
However there is a dirty fix to this.
In src/gradients/zygote.jl change lines 306-308 from

    getproperty(c, :vfov), Δ -> (Camera(zero(c.lookfrom), zero(c.lookat), Δ, zero(c.focus),
                                        zero(c.fixedparams)), nothing)

to

    getproperty(c, :vfov), Δ -> (Camera(zero(c.lookfrom), zero(c.lookat), Float32.(Δ), zero(c.focus),
                                        zero(c.fixedparams)), nothing)

Attachments:
mwe_and_stacktrace.zip

Mesh rendering

Following would be required to render a mesh. It is a .obj file. It contains vertex list and .mtl file to use for material. .mtl file contains path to the texture image to be used and some attributes. We can make an object like pyglet.graphics.vertex_list to store the vertices and its attributes. This object can later be used while rendering.

• Getting texture from an image
• Rendering arbitrary meshes (ie vertices) with given texture

@nograd not defined when trying to run teapot_rendering.jl

$ julia --project=. --color=yes teapot_rendering.jl 
┌ Warning: Package Colors does not have Printf in its dependencies:
│ - If you have Colors checked out for development and have
│   added Printf as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with Colors
└ Loading Printf into Colors from project dependency, future warnings for Colors are suppressed.
┌ Warning: The call to compilecache failed to create a usable precompiled cache file for Flux [587475ba-b771-5e3f-ad9e-33799f191a9c]
│   exception = Required dependency Colors [5ae59095-9a9b-59fe-a467-6f913c188581] failed to load from a cache file.
└ @ Base loading.jl:1041
┌ Warning: Package Colors does not have Printf in its dependencies:
│ - If you have Colors checked out for development and have
│   added Printf as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with Colors
└ Loading Printf into Colors from project dependency, future warnings for Colors are suppressed.
WARNING: ImageQualityIndexes.psnr is deprecated, use assess_psnr instead.
  likely near /home/phil/.julia/packages/Images/0SDGT/src/Images.jl:60
┌ Warning: Package RayTracer does not have CUDAnative in its dependencies:
│ - If you have RayTracer checked out for development and have
│   added CUDAnative as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with RayTracer
└ Loading CUDAnative into RayTracer from project dependency, future warnings for RayTracer are suppressed.
ERROR: LoadError: LoadError: LoadError: UndefVarError: **@nograd not defined**
Stacktrace:

[1] top-level scope
 [2] include(::Module, ::String) at ./Base.jl:377
 [3] include(::String) at /home/phil/.julia/packages/RayTracer/rEd4h/src/RayTracer.jl:1
 [4] top-level scope at /home/phil/.julia/packages/RayTracer/rEd4h/src/RayTracer.jl:24
 [5] include(::Module, ::String) at ./Base.jl:377
 [6] top-level scope at none:2
 [7] eval at ./boot.jl:331 [inlined]
 [8] eval(::Expr) at ./client.jl:449
 [9] top-level scope at ./none:3
in expression starting at /home/phil/.julia/packages/RayTracer/rEd4h/src/gradients/zygote.jl:364
in expression starting at /home/phil/.julia/packages/RayTracer/rEd4h/src/gradients/zygote.jl:364
in expression starting at /home/phil/.julia/packages/RayTracer/rEd4h/src/RayTracer.jl:24
ERROR: LoadError: Failed to precompile RayTracer [60dacb86-48ff-11e9-0f01-03ab8794bbc9] to /home/phil/.julia/compiled/v1.4/RayTracer/sUryZ_rlO6L.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1272
 [3] _require(::Base.PkgId) at ./loading.jl:1029
 [4] require(::Base.PkgId) at ./loading.jl:927
 [5] require(::Module, ::Symbol) at ./loading.jl:922
 [6] include(::Module, ::String) at ./Base.jl:377
 [7] exec_options(::Base.JLOptions) at ./client.jl:288
 [8] _start() at ./client.jl:484
in expression starting at /home/phil/build/RayTracer.jl/examples/teapot_rendering.jl:11

Julia v1.4.0
(@v1.4) pkg> status
Status ~/.julia/environments/v1.4/Project.toml
[4c555306] ArrayLayouts v0.2.4
[08437348] AutoMLPipeline v0.1.11
[336ed68f] CSV v0.6.1
[3895d2a7] CUDAapi v4.0.0
[c5f51814] CUDAdrv v6.2.2
[be33ccc6] CUDAnative v3.0.4
[7057c7e9] Cassette v0.3.1
[3a865a2d] CuArrays v2.0.1
[a93c6f00] DataFrames v0.20.2
[31c24e10] Distributions v0.23.2
[ced4e74d] DistributionsAD v0.5.0
[587475ba] Flux v0.10.4
[7073ff75] IJulia v1.21.2
[6218d12a] ImageMagick v1.1.4
[86fae568] ImageView v0.10.8
[916415d5] Images v0.22.1
[e5e0dc1b] Juno v0.8.1
[1902f260] Knet v1.3.5
[1914dd2f] MacroTools v0.5.5
[91a5bcdd] Plots v1.0.10
[60dacb86] RayTracer v0.1.0 #ap/gpu (https://github.com/avik-pal/RayTracer.jl.git)
[37e2e3b7] ReverseDiff v1.2.0
[5e47fb64] TestImages v1.0.2
[e88e6eb3] Zygote v0.4.15
[10745b16] Statistics

inverse lighting example missing definition of `opt`

The code uses the variable opt but it is never defined.