Add support for floating point textures to OpenGL about processing4

[cacheflowe]

@Beervangeer - I never had the time/brainpower to get this submitted to the Processing core as a real solution, but I refined it within my own codebase in a way that's relatively easy to use and doesn't break the ways you'd normally use PGraphics and PShader objects. I wrapped up the code above into a class that mimics the Processing API as best as I could. You can add the class below that will create a PGraphics instance with an underlying 32-bit float texture. Then just call PGraphics32.newDataPG(this, 512, 512); to create your data texture with all of the appropriate settings to remove antialiasing, etc. Then you can use normal PShaders to manipulate individual pixel values and use them as high-resolution data storage and free yourself of the 8-bit limitations in the context of pixel data. One caveat: with this use case, 32-bit textures will be downsampled to 8-bit if you draw them to screen.

import java.nio.Buffer;

import com.jogamp.opengl.GL;

import processing.core.PApplet;
import processing.core.PConstants;
import processing.core.PGraphics;
import processing.opengl.PGL;
import processing.opengl.PGraphics3D;
import processing.opengl.PGraphicsOpenGL;
import processing.opengl.PJOGL;

/**
 * OpenGL renderer with 32-bit float texture.
 */
public class PGraphics32 
extends PGraphics3D {
	
	// proposed renderer name
	static final String P32 = "processing.opengl.PGraphics32";

	protected PGL createPGL(PGraphicsOpenGL pg) {
		return new PJOGL32(pg);
	}
	
	// override minimal code necessary to return 32-bit float texture 
	
	public class PJOGL32
	extends PJOGL {

		public PJOGL32(PGraphicsOpenGL pg) {
			super(pg);
		}

		@Override
		public void texImage2D(int target, int level, int internalFormat, int width, int height, int border, int format, int type, Buffer data) {
			gl.glTexImage2D(target, level, GL.GL_RGBA32F, width, height, border, format, type, data); 
		}
	}

	////////////////////////////////////////////////////////////////////
	// static factory method

	public static PGraphics createGraphics(PApplet p, int w, int h) {
		if (!p.g.isGL()) {
			throw new RuntimeException("createGraphics() with P32 requires size() to use P2D or P3D");
		}

		PGraphics pg = (PGraphics) (new PGraphics32());
		pg.setParent(p);
		pg.setPrimary(false);
		pg.setSize(w, h);
		return pg;
	}
	
	// static initializer with proper settings for texture data use
	
	public static PGraphics newDataPG(PApplet p, int w, int h) {
		PGraphics newPG = PGraphics32.createGraphics(p, w, h);
		newPG.noSmooth();
		((PGraphicsOpenGL)newPG).textureSampling(2);
		newPG.beginDraw();
		newPG.hint(PConstants.DISABLE_DEPTH_SORT);
		newPG.hint(PConstants.DISABLE_DEPTH_TEST);
		newPG.hint(PConstants.DISABLE_DEPTH_MASK);
		newPG.background(0, 0);
		newPG.noStroke();
		newPG.endDraw();
		return newPG;
	}
	
}

[cacheflowe]

I've revisited this after some conceptual direction from @benfry and arrived at a much simpler distillation that has little-to-no impact on the core. By overriding just a couple of things in PGraphics3D, we get a 32-bit buffer. The full example is seen below, and can be initialized with the normal PApplet.createGraphics() call if we add the new renderer class to PConstants:

PConstants.java addition

static final String P32 = "processing.opengl.PGraphics32";

PGraphics32.java (new class)

package processing.opengl;

import java.nio.Buffer;
import com.jogamp.opengl.GL;
import processing.core.PApplet;
import processing.core.PGraphics;
import processing.opengl.PGL;
import processing.opengl.PGraphics3D;
import processing.opengl.PGraphicsOpenGL;
import processing.opengl.PJOGL;

public class PGraphics32 extends PGraphics3D {
	// override minimal PJOGL code necessary to return 32-bit float texture
	@Override
	protected PGL createPGL(PGraphicsOpenGL pg) {
		return new PJOGL32(pg);
	}

	public class PJOGL32 extends PJOGL {
		public PJOGL32(PGraphicsOpenGL pg) {
			super(pg);
		}

		@Override
		public void texImage2D(int target, int level, int internalFormat, int width, int height, int border, int format, int type, Buffer data) {
			gl.glTexImage2D(target, level, GL.GL_RGBA32F, width, height, border, format, type, data);
		}
	}
}

It was suggested that this could be built as a library, but this is a much more minimal addition than I expected. I'd love to hear opinions about how this might get integrated as an option for creating PGraphics instances - I'm happy to go down either core or library route, depending on informed opinions. This code above would be a really easy pull request.

Some more info on what I'm doing (and I'll try to put together some complete examples), when you're creating a 32-bit texture to use as data, and manipulating each pixel in a shader to move a particle, you want to prep the PGraphics with the following configuration to have it behave properly. Otherwise, you'll get some funky artifacts:

// Create data texture for vertex shader
PGraphics dataPG = createGraphics(w, h, PConstants.P32);
dataPG.noSmooth();
((PGraphicsOpenGL) dataPG).textureSampling(2);
dataPG.beginDraw();
dataPG.hint(PConstants.DISABLE_DEPTH_SORT);
dataPG.hint(PConstants.DISABLE_DEPTH_TEST);
dataPG.hint(PConstants.DISABLE_DEPTH_MASK);
dataPG.background(0, 0);
dataPG.noStroke();
dataPG.endDraw();

Someone who's familiar with even more advanced shader techniques might have further additions or tweaks to OpenGL formats & specific texture/data configuration, but this example has been really helpful for my use-cases.

[cacheflowe]

I did some research and just got 32-bit textures working as a proof-of-concept with very minimal changes. In PJOGL, where textures are created, I overrode the internalFormat and type properties (with GL.GL_RGBA32F & GL.GL_FLOAT) in 3 functions:

@Override
public void texImage2D(int target, int level, int internalFormat, int width, int height, int border, int format, int type, Buffer data) {
  gl.glTexImage2D(target, level, GL.GL_RGBA32F, width, height, border, format, GL.GL_FLOAT, data);
}

@Override
public void copyTexImage2D(int target, int level, int internalFormat, int x, int y, int width, int height, int border) {
  gl.glCopyTexImage2D(target, level, GL.GL_RGBA32F, x, y, width, height, border);
}

@Override
public void compressedTexImage2D(int target, int level, int internalFormat, int width, int height, int border, int imageSize, Buffer data) {
  gl.glCompressedTexImage2D(target, level, GL.GL_RGBA32F, width, height, border, imageSize, data);
}

That's it! This change allows shaders to work as they do in WebGL and other 32-bit GLSL contexts (like Shadertoy), where textures can be used as high-precision vec4 float data! This allows for proper GPU particles and feedback effects, and would probably benefit most shaders' fidelity.

I've obviously just broken the intent of the PJOGL code, but I think there's the potential of adding a 32-bit texture option without much extra work. I'll try to think of a way to make this a reality, but I feel that someone like @codeanticode would have a good opinion on how to make this possible. Perhaps a toggle in settings() to switch everything over to 32-bit if the developer wants to?

Speaking to the previous question about PixelFlow's 32-bit support, you have to switch wholesale into the PixelFlow rendering techniques, and lose the option of using much of Processing's core functionality (this is my experience - I could be wrong). I think adding the 32-bit option into Processing's core would be a huge step forward.

[lindemeier]

@cacheflow, thank you very much for creating this class!

I tried it out, but I got a error on the static functions createGraphics and newDataPG. It says these methods cannot be declared static, because they can only be declared in a static or top level type. When I turn the new PGraphics32 calss into a static class, the error goes away.

It seems I can still load a shader and draw it to screen with image().
But when I try to pass the PGraphics32 object to another shader, it doesnt come through normally it seems.

Also when I use the PGRaphics32 I get this error: OpenGL error 1282 at bot endDraw():
It still draws though.

I attached the project if you are interested. Its simple fluid simulation with two shaders, one for calculations the other for displaying.

WaterShader.zip

For others with the same issue (static class):
Place the class in a java file besides the pde files to avoid nested import.

[timseverien]

Eventually I did manage to use PixelFlow to achieve what I couldn't with processing using the following code:

import com.jogamp.opengl.GL;
import com.thomasdiewald.pixelflow.java.DwPixelFlow;
import com.thomasdiewald.pixelflow.java.dwgl.DwGLSLProgram;
import com.thomasdiewald.pixelflow.java.dwgl.DwGLTexture;

DwPixelFlow context;
PGraphics2D canvas; // A PGraphics object to render the data to

// Two shaders: one to update data, one to visualise data to screen
DwGLSLProgram shader;
DwGLSLProgram shaderScreen;

// An object containing two frame buffer objects (textures) I can swap
DwGLTexture.TexturePingPong buffer = new DwGLTexture.TexturePingPong();

void setup() {
    context = new DwPixelFlow(this);
    canvas = (PGraphics2D) createGraphics(512, 512, P2D);

    shader = context.createShader("my-shader.glsl");
    shaderScreen = context.createShader("my-shader-screen.glsl");

    buffer.resize(context, GL.GL_RGBA32F, 512, 512, GL.GL_RGBA, GL.GL_FLOAT, GL.GL_NEAREST, GL.GL_REPEAT, 4, 1);
}

// Draw shader output to buffer.dest, where buffer.src acts as input data
void update() {
    context.begin();
    context.beginDraw(buffer.dest);
  
    shader.begin();
    shader.uniformTexture("uTexture", buffer.src);
    shader.drawFullScreenQuad();
    shader.end();
  
    context.endDraw();
    context.end();

    buffer.swap();
}

void draw() {
    update();

    // Render data to PGraphics object with a shader to visualise data
    context.begin();
    context.beginDraw(canvas);
    
    shaderScreen.begin();
    shaderScreen,uniformTexture("uTexture", buffer.src);
    shaderScreen.drawFullScreenQuad();
    shaderScreen.end();
    
    context.endDraw();
    context.end();

    // Render PGraphics object to screen
    image(canvas, 0, 0);
}

Resources:

• http://thomasdiewald.com/processing/libraries/pixelflow/reference/com/thomasdiewald/pixelflow/java/DwPixelFlow.html
• http://thomasdiewald.com/processing/libraries/pixelflow/reference/com/thomasdiewald/pixelflow/java/dwgl/DwGLTexture.html
• http://thomasdiewald.com/processing/libraries/pixelflow/reference/com/thomasdiewald/pixelflow/java/dwgl/DwGLTexture.TexturePingPong.html

[cacheflowe]

I'm in way over my head on this, but after seeing another performance hit while using a different camera library, I'm wondering if the realtime conversion between OpenGL texture formats could be causing the slowdown. But perhaps it's something else. I'll keep investigating towards a real implementation of this feature. Maybe it's a lot more involved than I'd hoped 😬 Any advice on OpenGL texture formats and how they interplay in PGL would be very welcome.

https://www.khronos.org/opengl/wiki/Pixel_Transfer
https://stackoverflow.com/a/34497547/352456

[sampottinger]

Hello! It may make sense to move this over to the Processing 4 repo. Things are pretty quiet on the 3.x side. @codeanticode / @SableRaf - should I move this over?

[SableRaf]

@sampottinger Done! Let me know if you need any more support from me on this.

[REAS]

@codeanticode will know the answer

[codeanticode]

Hi @AmnonOwed, it is not currently possible to define a floating point texture through Processing's API, the internal format can only be normal RGB or RGBA (i.e.: 1 byte per channel).

Quickly inspecting the code, seems like it shouldn't be too difficult to add GL.GL_RGBA32F or GL.GL_RGB32F as fp formats, but this would probably require an additional hint, or some other method to signal to the renderer that the user wishes to use fp textures.

[AmnonOwed]

Hey guys,

Thanks for the quick response. Would be great to add this functionality. I think the two options to let the user set this would be either a) per renderer or b) per texture. With regard to these two options:

a) per renderer
This option could be something like textureFormat(int format) similar to the way textureMode(int mode), textureWrap(int wrap) and textureSampling(int sampling) currently work.

b) per texture
This option could be something like GLGraphics's textureParameters solution, but specifically for each parameter. In this case setTextureFormat(int format).

I think option a (per renderer) is most in line with the current Processsing API. So it would make sense to implement this option.

The only question I would have then: can you use the global renderer setting, to set different formats for different textures in one sketch? So for example set / keep most textures to 1 byte per channel, but set one texture to 32FLOAT. I guess this could be possible by switching the global format setting back and forth at strategic points in the sketch? Not sure how this currently works for textureWrap() etc.

[benfry]

@codeanticode Can this be done inside begin/endPGL? If so, we should probably close. Seems akin to adding CMYK support instead of just RGB since it touches so many parts of the API.

[timseverien]

Any update on this? Floating point textures are great for GPGPU.

[codeanticode]

Hi, as mentioned earlier, this would require adding a bunch of new API to the core. I'm happy to have a more specific discussion, but this kind of functionality seems more suitable to incorporate through a contributed library... have you looked at PixelFlow, it goes pretty far in terms of GPGPU, and I believe it uses FP textures.

[timseverien]

Thanks for the link! PixelFlow seems to provide a lot of built-in GPGPU shaders, but I was unable to find utilities to use a custom shader. I'll have a more thorough look when I have more time.

but this kind of functionality seems more suitable to incorporate through a contributed library

Because Processing supports shaders, I think Processing should also provide the ability to choose how data is transferred to the GPU. The latter is essential for many projects. I was stoked finding out Processing supports shaders, and it feels counter-intuitive to download a library to work around this limitation. Nonetheless, I enjoy Processing a lot and am grateful of all your efforts.

Anyway, I like @AmnonOwed per-renderer approach. How about textureType(int type)? Additional constants can be introduced for various types. GL_BYTE, GL_FLOAT, etc.

[cacheflowe]

The more I've been using shaders for advanced feedback & particle techniques, the more I've run into this need. I didn't quite realize why my calculations were suffering until an OpenGL expert suggested that I might not be using a 32-bit texture for storage & computation. That explained my limitation of 1./255. as the smallest number that worked inside a shader. I've browsed the source, but think that I'm woefully underskilled to try to add this feature. I'll look some more though. In the meantime, I give this enhancement a 👍

[codeanticode]

@timseverien thanks for posting your solution, it sounds like the floating texture support in PixelFlow can be nicely integrated with the core API. Do you think this makes adding FP texture support in the code superfluous?

Perhaps we should write a tutorial about these advanced techniques... I wanted to write a follow up to the intro PShader tutorial for a long time, but never get the chance to do it :-)

[cacheflowe]

After further testing, I found some new info:

• This slows down the current Movie object in the processing-video library. There are potential performance impacts elsewhere, but Movie was noticeable.
• It seems to break the processing-video beta v2.0.4 version, which I've been using lately
• It has the potential of slowing things down, so we probably would want something like createGraphics(w, h, P32), to specifically enable this type of texture. This would also ensure backwards compatibility.

If anyone wants to try my recompiled core.jar, you can find it here.

[benfry]

Much nicer solution! That looks simple enough that adding a hint() for it might be appropriate. That way it can have the appropriate caveats, since it's really helpful in some circumstances, but can't be made the default due to the performance issues you mentioned.

[Beervangeer]

Hi @cacheflowe , i would love to try this. I want to pass float texture data like @timseverien between shaders through processing. Do I have to rebuild from source to use your solution? Or can I add this constant and class extension in a different way?

[jeremydouglass]

@benfry @sampottinger would a hint() for this be a incorporate-able in Processing3? or would it instead be a potential Processing4 feature?

[Beervangeer]

@cacheflow, thank you very much for creating this class!

I tried it out, but I got a error on the static functions createGraphics and newDataPG. It says these methods cannot be declared static, because they can only be declared in a static or top level type. When I turn the new PGraphics32 calss into a static class, the error goes away.

It seems I can still load a shader and draw it to screen with image().
But when I try to pass the PGraphics32 object to another shader, it doesnt come through normally it seems.

Also when I use the PGRaphics32 I get this error: OpenGL error 1282 at bot endDraw():
It still draws though.

I attached the project if you are interested. Its simple fluid simulation with two shaders, one for calculations the other for displaying.

WaterShader.zip

[cacheflowe]

@Beervangeer A couple of lines in the newDataPG() function are specifically there for using individual pixels as data for particle systems, and has adverse effects if you're using texture lookup for feedback effects. I probably should've noted this :) You might remove the following lines for your purpose. It seems to help, but I'm not sure what the desired result of your shader is:

// newPG.noSmooth();
// ((PGraphicsOpenGL)newPG).textureSampling(2);

[stmaccarelli]

Hello everyone.
I was guessing if any progress has been made here.
Supporting FP textures would be great for both GPGPU and any other high precision color handling.
Any news?

[aphid91]

The PGraphics32 class has been a life saver! Be very careful with newPG.noSmooth() as it can have some strange effects on texture lookups, even when using something like texelFetch() that should only care about individual pixels. See my comment here: processing/processing#5363
The problem I describe in that thread is present when using PGraphics32 as well. Hope this saves someone the headache of tracking down some truly baffling image artifacts.