Integer value of h3Index about h3-js

[dfellis]

I found a blogpost on the subject

If your JS environment has BigInt defined, you can simply do BigInt('0x' + h3Index) or h3Indexes.map(h => BigInt('0x' + h)) to get the value you want, and then simply bigIntH3.toString(16) later to get back the string representation.

[dfellis]

Lots of reasons:

1. BigInt didn't exist anywhere when we started this project, so you can't fault us for not using it. ;)
2. There are still many large holes in compatibility with BigInt. According to this table Safari (desktop and mobile), Typescript, Babel, the JSON format (probably forever for this one), and Node 8.x and earlier lack BigInt support.
3. Emscripten, the tool used to transpile our C H3 library to Javascript represents 64-bit integers as 2 32-bit integers because BigInt did not exist when it was created, either, and conversion costs to and from BigInt will be required. (This is also true with the string representation, but that representation is compatible with everything and is probably cheaper than the BigInt representation in both memory and compute speed (just two simple itoa calls and then a string concat (which could simply be two memcpys) versus a special object for manipulating arbitrary precision integers).
4. Minor, but the beginning the H3Index is decently aligned with 4-bit hexadecimal chunks such that you can "read" it and recognize some properties of the H3Index visually. If it starts with an 8 it's a regular H3Index while if it starts with 1X it's a unidirectional edge (where the X indicates the edge direction 1 - 6). The next hex digit after that indicates its resolution, with 0 - F representing resolutions 0 to 15, so 89... is a regular H3Index at res 9. That makes the hex representation more useful for humans than the BigInt representation, so if we have to pay a performance price, either way, it seems more useful.

Finally, we do try to include many utility functions for manipulating the indexes without needing to worry about the low-level representation details. Would you mind sharing what you believe you need to do that the H3 library cannot do for you?

[dfellis]

So you want to use an h3index as a sort of index value, huh? ;)

I will admit that we never considered that case for Javascript when building the library because code written in Python, Java, or Go on the backend would need those sorts of tools while code written in Javascript was usually on the front-end and only needed hash-based lookups for rendering purposes.

I hope my earliest comment solved your immediate problems with sorting and searching the data (btw, a sorted array can be used in place of an actual tree with the metadata that would normally be attached to the tree stored in a hash table, as you can do a binary search for records, and then loop through those records to get the metadata you need with the hash table / object lookup).

As far as adding BigInt support to this library I would defer to @nrabinowitz . It doesn't seem too hard to do, but there would need to be some capability detection and then a wrapper around every binding function to do it, so there would be little-to-no performance improvement to do it in this library, but it might make it a slightly cleaner API.

[dfellis]

@dfellis Thanks for your comment,
Yes I want to use it as an index to access objects, However I am trying to make it's prototype in js and see if that actually works for large datasets of data or not,

I hope my earliest comment solved your immediate problems with sorting and searching the data (btw, a sorted array can be used in place of an actual tree with the metadata that would normally be attached to the tree stored in a hash table, as you can do a binary search for records, and then loop through those records to get the metadata you need with the hash table / object lookup).

I was thinking about this as well, but I don't know how that would work with window query or for knn query when we have a large array of items. lets say millions of cells.

So, that's a LOT of data to crunch in Javascript. I would recommend using Java instead if you can. There is a huge performance bottleneck in V8 with arrays around 100k and up in size.

I wrote a library to work around this year's ago but it is focused only on queue performance.

You could take our the array of arrays concept and mutate it into a lookup replacement, but it will be a lot of work.

If you try to do this with a more naive tree structure, the GC pauses will get crazy as it traces through your tree every time to see if anything can be discarded. When I did something similar, I got it working in Node after significant effort and debugging GC and V8 engine behavior. In Java you could just use TreeMap with H3-Java and be done a lot faster.

[dfellis]

I'm not as big of a Python guy as some others. I know there's nothing like that in the standard library, but there might be something in the ecosystem. @ajfriend might be able to answer that.

It's weird, but you could use Node's child process functionality to spawn a Java sidecar process to do these lookups for you, using something like JSON-RPC over stdin/stdout to handle the communication back and forth for you.

[ajfriend]

If you're going the Python route, would a normal python dict or set solve your problem?
You might also look at the bisect module, or numpy.searchsorted.

Getting a little more sophisticated, you might look at cykhash, which should be quite fast, and can be called from pure Python directly, or from compiled Cython code.

Or, I might be completely missing the point, and there's a good reason to use a tree over a hash table. 😄

[ajfriend]

numpy should be able to handle things up to a few million pretty easily.

Here's an example with numpy and a Python set for 100 million 64-bit integers. Both seem fast.

The set approach looks a bit faster for testing membership, and would definitely be more flexible and efficient around adding/removing members than the numpy sorted array.

[am2222]

@dfellis thanks so much,
So why this library is not using BigInt objects for h3index? is there anything related to possible issues? except some issues related to general support of BigInt by different environments.

[am2222]

@dfellis Thanks so much for your response,
Well I am want to us a kind of trees to search between hex indexes to find a specific cell in an array of cells and so I need to be able to convert them to numbers, suppose we have 1M cells and we want to query them by a range between [a,b] so I wanted to use a kind of tree structure to query hexes.

[am2222]

@dfellis Thanks for your comment,
Yes I want to use it as an index to access objects, However I am trying to make it's prototype in js and see if that actually works for large datasets of data or not,

I hope my earliest comment solved your immediate problems with sorting and searching the data (btw, a sorted array can be used in place of an actual tree with the metadata that would normally be attached to the tree stored in a hash table, as you can do a binary search for records, and then loop through those records to get the metadata you need with the hash table / object lookup).

I was thinking about this as well, but I don't know how that would work with window query or for knn query when we have a large array of items. lets say millions of cells.

[am2222]

@dfellis
I think I can switch to java at some point, but is there anything similar to TreeMap in python? It is easier to me to switch to python than Java since I want to load those cells from IPFS into the Tree structure and the current implementation of IPFS is not in complete in Java.

[am2222]

@dfellis Thanks very much for your help, If I think in that case if python is also not supporting that I do in java and handle the communications between nodejs and java since the objects that are needed to be transmitted are not that complex.

@ajfriend Thanks for your reply, Does those packages and libraries are able to handle large arrays? for example a few millions?

[am2222]

@ajfriend wow, thanks so much, You really saved my day.