[Feature request] Add support for Decimal type about explorer

[billylanchantin]

Rustler now supports 128 bit integers (thanks for initiating that, @Munksgaard!):

• rusterlium/rustler#600

I think we can consider this issue blocked until rustler does another release. Otherwise we'd have to go off their master branch, which probably isn't worth the headache since they seem to release regularly.

[LostKobrakai]

Seems like this has already been released with rustler 0.32.1

[philss]

The update of Rustler to v0.32.1 has landed on main :)

[billylanchantin]

(Carrying over some of my comment as well)

Polars does support a decimal datatype:

• https://github.com/pola-rs/polars/blob/849953b097e0d15fe107f3683f003228103b425d/py-polars/src/datatypes.rs#L34

If we want to support it as well, the tricky part will be representing things in Elixir. I see two options:

• Add Decimal as a dependency
• Custom datatype like Duration

I haven't played with the Decimal datatype in Polars or Arrow, so I don't know if it maps nicely to the Decimal library's representation. But I have no objections to exploring either option (or some other thing I didn't think of).

[josevalim]

There is a reference here: https://arrow.apache.org/docs/r/reference/data-type.html

The implementation is straight-forward, it is stored as a large integer, which we can easily convert to Decimal using Decimal.new/3. If a list of Decimals is given as input, we will need to compute the scale, using Decimal.scale. And for precision, we can use the maximum one.

[billylanchantin]

👍 I think I'm comfortable adding Decimal as a dep.

@Munksgaard PRs welcome! If you're not able to, then I can take a crack when I have some time.

[Munksgaard]

One complication that I can see is that Decimal allows both NaN and Infinity as values, which we don't have representations for in Polars as far as I can tell.

[josevalim]

Yes, we should raise when encoding those. We will already have to do a prepass converting all decimals into int::128 or int::256, so we can raise on NaN and Infinity.

[Munksgaard]

@Munksgaard PRs welcome! If you're not able to, then I can take a crack when I have some time.

I would love to help out, but realistically I won't be able to contribute anything significant in the next couple of weeks :-(

[cigrainger]

I'm comfortable with Decimal as a dep as well. It doesn't carry its own deps, pretty lightweight.

[alexpearce]

I had a crack at adding Decimal support and managed to add reading without too much trouble: alexpearce@709aa67

I found adding write support more challenging. It seems that Decimal support in polars-core isn't quite as mature as for other dtypes so write support requires a bit more work.

[billylanchantin]

@alexpearce Feel free to open a PR. If you're still having trouble, we can discuss there why write support is more difficult.