Possible to trim CompositeSprite to make it only fit the rendered object? about pixi3d

[johot]

Thank you for all your pointers I will investigate! This sounds exactly like the kind of information I was looking for :) Guess I have some reading up to do about projection matrices then. Thank you for your time and for this awesome project @jnsmalm

[jnsmalm]

Hey and thanks @johot

Can't you just give it different width/height when creating the CompositeSprite?

  let compositeSprite = app.stage.addChild(new PIXI3D.CompositeSprite(app.renderer, {
    objectToRender: model,
    width: 512,
    height: 512
  }))

Just note that if the aspect ratio of the stage is different from those settings the object will be rendered weird. To fix this you need to change the aspect ratio of the camera you are using to render your Sprite3D. Either you change the main camera aspect or you change the aspect of the specific camera used by the Sprite3D.

PIXI3D.Camera.main.aspect = 1

// Or like this
sprite3d.camera.aspect = 1

[johot]

Thank you for your response!

The problem is not really what you describe, I saw the height and width properties and tried them but that was more a way of creating a final more low resolution or high resolution texture.

What I want to do is this:

Say I have a 1920x1080 stage and on screen I render a 3D model that will be displayed at say 300x200 pixels in screen size. If I now make a composite sprite from this I will end up with a 1920x1080 texture with a lot of white space / transparent pixels around that 300x200 sized 3D model. Basically a snapshot of the entire screen and not only the 3D object right?

What I ideally would like to happen is that the final composite sprite would end up being only 300x200 pixels wide. I looked at the source code and could see that the composite sprite will always be renderer.screen.width so I guess that explains why it takes a snapshot of the entire screen and not only the rendered pixels of the 3D object.

So my questions:

1. Would it be possible to make composite sprite take a snapshot of only the object and not the entire screen given the current camera size etc.
2. Or alternatively can I get the bounds of the 3D object in screen coordinates and clip the composite sprite myself?

In the best of worlds option 1 would ofc be the easiest to work with :) I hope I explain it good enough ;)

[jnsmalm]

The renderer.screen.width and renderer.screen.height is only used if you don't provide your own size. You do want to create a more low resolution texture (300x200 vs 1920x1080) it seems according to your description. But you may have to scale your 3d object when rendering it using the CompositeSprite so it fits the size of your CompositeSprite. If you render the object with a smaller scale, the object will come out as low res in that texture.

If this doesn't explain it, maybe you could provide a small app/code so I can take a look?

[johot]

Ah thanks! I think I understand then, so what I need to do then is to basically fill the renderers viewport with my 3D object then?

The only problem left then is to calculate the bounds of the 3D object so I know that I scaled it correctly right?

I couldn't really find any methods for getting the 2D bounds of a rendered 3D object does such utility functions exist in pixi3d? The one I actually need it the most for is the Sprite3D as I use pixi2d to create 3D effects of images for a video editor I am building :)

[jnsmalm]

Sorry for the long delay. Please have look at the following discussions:

#34
#74

Use model.getBoundingBox() to get the size of the model, then you can calculate (using the code in discussion) how far back the camera should be to fit the entire model in the camera view. This way the model will be as large as it can in the texture.

[johot]

No worries just happy to get any help 😊

I actually found those discussions earlier but my problem is that I can't find a getBoundingBox method for a Sprite3D, does it really exist? Couldn't really find an obvious way of getting the model / mesh for a Sprite3D either?

[jnsmalm]

Ah I forgot that you were rendering 3D sprites. Correct, getBoundingBox is not available there. But do you really need it? Width and height is already available on the 2D sprite (the one you are rendering to a texture), can't you just use that to get the size of the object?

[johot]

Not of it is rotated in 3D space, say I want to draw a bounding selection box around a Sprite3D rotated 45 degrees on the X or Y axis or fill a compound sprite then I will need that method 😅 If you have any code sample on how to get the xyz of the vertices and apply your previous methods I might be able to figure it out 😊

[jnsmalm]

Did you try to instead render Sprite3D with orthographic camera (and skip CompositeSprite)? That way you can render in screen coordinates.

If you still need to use CompositeSprite, do you really need the size to match exactly the rendered object if it's rotated?

[johot]

The reason I need all of this is that I am building an animation where images get rotated in 3D space, so I need a perspective camera. The reason I need a composite sprite is that I want to add filters such as motion blur to this 3D sprite. What I do now is that I create a composite sprite of the entire object and stage, this means I get a texture that is say 1920x1080 even though the sprite might only be 500x200 when rotated for example (this size will ofc change each frame as the animation is happening).

The current solution works but I am worried about things like overdraw since I get a lot of transparent pixels around the object. So I would need to crop my texture to only fit the rendered sprite to avoid the overdraw. Since this is also an image editor of sorts I would also like the user to be able to click the 3D sprite to show resize handles etc and in that case I need to calculate the bounds of the sprite given the current camera in screen coordinates, that is a regular Rectangle (2D).

Hope I make sense 😅

[jnsmalm]

The only way I can think of is to first render the Sprite3D to a very small texture, let's say 128x128. Then read the pixels of that texture and that way figure out the actual size of the rendered object. Then setting the size of the CompositeSprite using that result and render again.

Not sure how performance of this would compare against just guessing the size depending on the rotation of the object. Guess it also depends on how many of these objects you have an how often they get rotated (and need the CompositeSprite size recalculated).

[johot]

So its not possible to calculate the 2D bounds of a 3D object using the camera just like when using Camera.worldToScreen to translate 3D positions to 2D positions? A function like that would be very useful in pixi3d I think.

Maybe I could easily do it myself if I can get hold of the global xyz positions of the vertices and then use worldToScreen. Can I access the internal mesh of a Sprite3D somehow?

This unity forum thread touches on a similar solution: https://forum.unity.com/threads/finding-a-3d-objects-2d-bounds.231304/

If I can find the screen coordinates of the sprite vertices I could easily crop the composite sprite or scale the object to fit the viewport before creating the composite sprite.

[jnsmalm]

Maybe I could easily do it myself if I can get hold of the global xyz positions of the vertices and then use worldToScreen. Can I access the internal mesh of a Sprite3D somehow?

Sprite3D was created to be rendered in batches (just like the regular PixiJS sprite) so the data is a bit spread out.

ProjectionSprite is an object which holds the actual vertex data of the sprite (it inherits from regular PixiJS sprite). The vertices are calculated at https://github.com/jnsmalm/pixi3d/blob/develop/src/sprite/projection-sprite.ts#L51 almost identical to regular PixiJS.

ProjectSprite also holds an modelViewProjection matrix. That matrix is then applied to the vertex data inside the shader at https://github.com/jnsmalm/pixi3d/blob/develop/src/sprite/shader/sprite.vert. The modelViewProjection gets calculated inside the Sprite3D.render function at https://github.com/jnsmalm/pixi3d/blob/develop/src/sprite/sprite.ts#L128

If you want the rendered vertex positions you need to do the above calculation (modelViewProjection together with the vertex data ) in JavaScript instead, you can use the math functions for that. https://github.com/jnsmalm/pixi3d/tree/develop/src/math

Probably not so easy but it can be done, maybe you can even create your own Sprite3D class with the above code and it can function the way you want.