Retriving colors used in QR code about jsqr

[cozmo]

Currently jsQR discards colors as the first step of the process, specifically the binarize step. As such I think the best way to approach this would be to take the final result (with the location of the QR code identified) and use that to sample colors from the original image. extract gives back a mappingFunction which takes in x,y values for the QR code (0,0 is the top left, and if the QR is 21x21 then 21,21 would be the bottom right, etc) and returns the pixel coordinates in the original image.

Using that mapping function you could iterate through all the modules (squares) of the QR code and look up the color in the original image data. Then you could average out the values associated with "white" squares and the values associated with "black" squares to get the original colors.

I think this would be pretty effective. One thing to keep in mind however is that this isn't the primary use case of the library, and does add additional computational complexity to the scan (something which for standard uses needs to be efficient. As such if we're adding it to the library we need to make it something users can opt in to so if you don't need color data you don't incur the computational overhead. That said that's a bridge we can cross when it comes up.

Please let me know if this was helpful/if there is something I can clarify further.

Thanks!

[david-boles]

Thanks, all of that's really helpful! For the API, what do you think of adding a default false parameter to the scan function and optional property(ies) to the QRCode interface? If that sounds good, any naming preferences? Maybe something along the lines of retrieveColors, backgroundColor, and qrColor? Also, it seems like there can be some variation to QR codes (e.g. rounded finder patterns and logos in the center), any thoughts about how to sample colors or restricting sampling to certain areas?

[david-boles]

Also, any preferences on how to average together colors? For now I'll just do a arithmetic mean on the components.

[david-boles]

I've gotten it working here, let me know if there's anything you'd like done differently. How do you think tests should be implemented? I assume you'd want to test both with and without retrieving color, it seems like either having separate output JSON files for color or modifying the format of the existing output files to include both are the most reasonable two options... Alternatively, I could set up a separate set of test images for color retrieval so that testing time isn't basically doubled.

[cozmo]

For the API, what do you think of adding a default false parameter to the scan function and optional property(ies) to the QRCode interface

If we're adding more params to the function I think it should be an options object so we can extend it in the future with other options. That said lets get an implementation we're happy with first before thinking about how we want to integrate it. For example some other options could be keeping it on a fork of the library, or as a separate scan method we export. I think which we choose depends on the solidity of the results (I don't think we'd want to mainline the change if the quality we were seeing was low for example), and also how common of a need we foresee this becoming (if it's a literal one off it might not be worth adding a ton of complexity to the API, so a separate function etc could be ideal).

Also, any preferences on how to average together colors? For now I'll just do a arithmetic mean on the components.

Averaging colors, and generally doing arithmetic on RGP values is fraught with hidden complexity. For example to greyscale RGB you can't just average the three values. Rather you have to do this because of how humans perceive colors. You can learn more about why and also how those magic constants were derived here.

All that is to say - We should probably do better than an basic average. It seems like the simplest is to square the values and then sqrt the result. However this has some flaws (#ff0000 (red) and #0000ff (blue) creates a pink, instead of the potentially expected darker purple). You can also go way down the rabbit hole but that is probably unneeded (depending on your needs/expectations).

---

One more thing you may want to think about is how we want to handle images like this (or this or this). Specifically QR codes with graphics inserted. These will throw off the average, but unclear if that matters for your use case. If it does you could probably only average modules that weren't error corrected, but that would require some information being pulled out of the decode step.

---

Hope I didn't overwhelm with this info dump, just wanted to make sure you have enough context as you need, since you understand your needs more than I do. As for testing I'd say the first step should be just dumping out some easy to read information (maybe write a 10x10png of each color?) so that for the test suite images you can check to make sure you're getting the kinda results you expect?

[david-boles]

Info dump is great, makes it easier to reference back to :)

If we're adding more params to the function I think it should be an options object so we can extend it in the future with other options.

Will do. Assuming typescript doesn't give me any problems, I'll make an interface with optional retrieveColors property and Object.assign whatever's passed in to a constant set of defaults.

Averaging colors, and generally doing arithmetic on RGB values is fraught with hidden complexity.

Yeah... I'll poke around to see if there's anything simple and little bit fancier than a geometric mean. Otherwise I'll swap to that.

One more thing you may want to think about is how we want to handle images like this (or this or this).

Yeah, it doesn't affect me but I thought I'd bring it up. Maybe leave it simple for now and see if anyone wants it later? Also, this one you liked to doesn't have an image, were you talking about the shadow? That's an interesting additional layer of variation on what you might want the library to return...

As for testing I'd say the first step should be just dumping out some easy to read information (maybe write a 10x10png of each color?)

Seems fine to me to just print out the RGB values into JSON; it makes the testing code that much simpler and is pretty easy to a sanity check on with basic knowledge of RGB or an image editor... Any thoughts on whether/ how to run separate tests for color vs. normal scanning?

[david-boles]

So Object.assign doesn't seem to be a thing in jsQR's environment... It's not a problem right now but making default scan options work better may be something needed in the future. I was able to implement averaging in the CIELab color space (still untested) relatively easily, it might not be worth it but whatever.

[david-boles]

I just went ahead and implemented tests for color retrieval by changing the format of the output.jsons to:

{
  default: <output of a run with defaults>,
  color: <output of a run with color>
}

I can't seem to get jest to actually run the tests without getting 'heap out of memory' (I don't think this is related to my changes) errors but running generate-test-data didn't take too long and, poking around in the output, color mixing seems to work quite well. In case you're happy with it as is, I'm going to make a pull request. If not, just let me know and I'll change stuff!