Query nested field not working about elastiknn

[ReneHollander]

Looking into it further, it seems there is a error in the documentation: https://github.com/alexklibisz/elastiknn/blame/d86eecc7bb319cd8a1d6b9aa6279f299cec2e556/docs/pages/api.md#L590
This can be easily overlooked if you copy paste from the docs, instead of reading it properly...

My query now looks like this, but still finishes with an error:

{
    "query": {
        "elastiknn_nearest_neighbors": {
            "field": "features.color", 
            "vec": {
                "values": [0.00784313725490196, 0.30196078431372547, 0.34901960784313724, 0.3411764705882353, 0.23137254901960785, 0.0392156862745098, 0.01568627450980392, 0.00392156862745098, 0.00392156862745098, 0.011764705882352941, 0.011764705882352941, 0.027450980392156862, 0.03137254901960784, 0.2549019607843137, 0.2549019607843137, 0.34901960784313724, 0.28627450980392155, 0.3058823529411765, 0.011764705882352941, 0.3411764705882353, 0.25098039215686274, 0.24313725490196078, 0.28627450980392155, 0.011764705882352941, 0.011764705882352941, 0.0196078431372549, 0.011764705882352941, 0.011764705882352941, 0.01568627450980392, 0.0196078431372549, 0.20392156862745098, 0.33725490196078434, 0.2823529411764706, 0.00784313725490196, 0.23921568627450981, 0.24313725490196078, 0.19215686274509805, 0.058823529411764705, 0.11764705882352941, 0.1411764705882353, 0.1843137254901961, 0.03529411764705882, 0.14901960784313725, 0.10588235294117647, 0.20392156862745098, 0.37254901960784315, 0.26666666666666666, 0.10980392156862745, 0.18823529411764706, 0.10980392156862745, 0.2784313725490196, 0.27058823529411763, 0.12549019607843137, 0.0392156862745098, 0.27058823529411763, 0.21176470588235294, 0.36470588235294116, 0.30196078431372547, 0.17647058823529413, 0.24313725490196078, 0.30196078431372547, 0.27058823529411763, 0.20392156862745098, 0.17647058823529413, 0.16470588235294117, 0.18823529411764706, 0.0784313725490196, 0.0196078431372549, 0.023529411764705882, 0.13333333333333333, 0.054901960784313725, 0.03137254901960784, 0.43137254901960786, 0.5450980392156862, 0.43529411764705883, 0.5294117647058824, 0.36470588235294116, 0.803921568627451, 0.39215686274509803, 0.403921568627451, 0.2901960784313726, 0.6627450980392157, 0.3803921568627451, 0.7568627450980392, 0.3137254901960784, 0.2823529411764706, 0.16862745098039217, 0.3607843137254902, 0.2196078431372549, 0.5019607843137255, 0.3058823529411765, 0.3137254901960784, 0.12941176470588237, 0.16862745098039217, 0.1568627450980392, 0.5803921568627451, 0.25882352941176473, 0.6705882352941176, 0.6862745098039216, 0.6705882352941176, 0.7372549019607844, 0.7450980392156863, 0.6627450980392157, 0.796078431372549, 0.9058823529411765, 0.7137254901960784, 0.8627450980392157, 0.9137254901960784]
            },
            "model": "lsh",
            "similarity": "l2", 
            "candidates": 50
        }
    }
}
{"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: { }","index_uuid":"yalL9pznRLKIkjjgwVBtiQ","index":"posts"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"posts","node":"VJH4aR0HRp6dWr3e8pReAA","reason":{"type":"query_shard_exception","reason":"failed to create query: { }","index_uuid":"yalL9pznRLKIkjjgwVBtiQ","index":"posts","caused_by":{"type":"unchecked_execution_exception","reason":"java.lang.RuntimeException: Failed to retrieve mapping at index [posts] field [features.color]","caused_by":{"type":"runtime_exception","reason":"Failed to retrieve mapping at index [posts] field [features.color]","caused_by":{"type":"","reason":"Attempt to decode value on failed cursor: DownField(type)"}}}}}]},"status":400}

[alexklibisz]

@ReneHollander Hi, thanks for finding this. I had it working at one point but I actually don't have any tests for nested fields so I probably broke it at some point. I'll plan to look into this and expand the test suite soon.

[ReneHollander]

Thanks, for now I moved it out and successfully ran queries on my dataset.

Are you also on planning to write some docs on how to select values for rows, bands and width? I know that the docs mention in which way query performance will go for increasing/decreasing values. Especially how they relate to the size of your dataset. I basically tried it with the values in the API Example, and inserted around 3M documents. Queries took around 5 to 10 seconds on one node (-Xmx2048M for elasticsearch). Not sure if I can exepct more on one node, or if I can achieve better with more tuning. It may make sense to open another issue for that to help others get a quick start with selecting appropriate values :)

[alexklibisz]

Those parameters depend a lot on your dataset and your desired performance (i.e. recall vs latency trade-off). I'm actually working on some more comprehensive benchmarking with several datasets this weekend and hope to publish the results this coming week. I think those results will probably be a good starting point for picking parameters.

…

On Sat, Apr 25, 2020, 11:24 Rene Hollander ***@***.***> wrote: Thanks, for now I moved it out and successfully ran queries on my dataset. Are you also on planning to write some docs on how to select values for rows, bands and width? I know that the docs mention in which way query performance will go for increasing/decreasing values. Especially how they relate to the size of your dataset. I basically tried it with the values in the API Example, and inserted around 3M documents. Queries took around 5 to 10 seconds on one node (-Xmx2048M for elasticsearch). Not sure if I can exepct more on one node, or if I can achieve better with more tuning. It may make sense to open another issue for that to help others get a quick start with selecting appropriate values :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5E27DEQU3KJCZON4AWQBTROL6DXANCNFSM4MMWTGZQ> .

[alexklibisz]

@ReneHollander Also quick note on performance: the general unit of parallelism in Elasticsearch is the shard. Latency should be roughly proportional to the number of shards in your index (i.e. 5 shards is ~5x faster than 1 shard, assuming you have >= 5 threads). Eventually you will want to spread shards across nodes for replication and further parallelism.

[ReneHollander]

Thanks for the reminder with the sharding. After setting it properly things got much better.

After that I put together a python script over two days I let it run over different combinations of bands, rows and width and got a good result that achieves around 100ms latency with almost perfect recall and precision.

[alexklibisz]

Great! Are you using the published python client? I haven't published much documentation yet but its published to pypi: https://pypi.org/project/elastiknn-client/ so you can pip install it.

…

On Tue, Apr 28, 2020, 15:07 Rene Hollander ***@***.***> wrote: Thanks for the reminder with the sharding. After setting it properly things got much better. After that I put together a python script over two days I let it run over different combinations of bands, rows and width and got a good result that achieves around 100ms latency with almost perfect recall and precision. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5E27BVAZR2Y7XCKGVCQULRO4SODANCNFSM4MMWTGZQ> .

[ReneHollander]

For now I have just used called the REST API directly, as I did the testing manually at first, and then built the script from there.
The application I intend to use this in is in Python, so I will likely use it when I get o that stage.

[alexklibisz]

This is fixed in #106 .
@ReneHollander If you're still using the plugin, please have a look at the latest release.
Thanks!