API and Error about texpresso

[jansol]

I haven't been entirely happy with the API myself either, but haven't had the time to properly think about what I would prefer it to look like. One major issue is that it requires expanding everything to be RGBA and will always decompress to RGBA, which adds unnecessary overhead as noted in the integration discussion over at the image crate. I was hoping to get some feedback from them about what kind of API they would prefer.

[localthomas]

I looked into the current implementation of the DXT module in the image crate and propose to add the following function for decoding:

#[non_exhaustive]
pub enum TodoError {
    /// The input for decoding or encoding did not meet the requirements for the variant.
    InvalidInputSize(usize),
    /// The output buffer for encoding or decoding did not meet the requirements for the variant.
    InvalidOutputSize(usize),
    // ...
}

impl Format {
    fn decompress(
        self,
        data: &[u8],
        width: usize,
        height: usize,
        output: &mut [u8],
    ) -> Result<(), TodoError>;
}

I am still unsure about the Error type, but this is probably the easiest solution.

For integration into the image crate, the current read_scanline could be modified to use the aforementioned function instead (with a height of 4). And maybe squish::Format could be used as a replacement for image::DxtVariant.
I would also suggest adding #[non_exhaustive] to squish::Format if a future version adds more formats.

I did not look into the encoding details yet, but it should be very similar.

---

One major issue is that it requires expanding everything to be RGBA and will always decompress to RGBA

I think this should just be changed to properly output with the correct byte stride for the format. The function decompressed_bytes_per_pixel(self) -> usize (or something similar; e.g. 3 for BC1) should be added to squish::Format.
This is the requirement to make the method of integration of above possible. This is possibly the most work, as it requires changing decompress_block and compress_block_masked. It would propose to make these two functions private for any new versions. External users can always use the decompress and compress functions with sizes of 4×4.

---

With your blessing, my next steps would be to first

1. Implement the Error type and add it to the decompress function
2. Add checks for the most obvious invalid parameter combinations
3. Create a simple fuzzing setup with cargo fuzz for decompression

Then I would try to switch from RGBA only modes to their proper ones (RGB, RGBA, RG, R).

[jansol]

Sure, go ahead. It is probably easiest to evaluate the options by seeing what they would look like in practice anyway.

I also plan to add a Bc1Alpha variant since OpenGL has two separate constants for DXT1 with and without transparency and are probably some gains to be had from being able to use the implicit black value in the palette. I just haven't got to actually doing that optimization in the encoder yet. If you are reworking the channel modes from RGBA-only it might make sense to add that format variant and just stub it out with unimplemented!() for now.