Bug: Setting mongodb field value to null does not index it in Elasticsearch about monstache

[rwynn]

I tried both insert and update using $set. But in both cases the null was delivered for me.

I'll take a look at the repo though when you get a chance.

[rwynn]

Hi @meabed ,

I wasn't able to replicate this issue with the code in the rel6 branch. What version of monstache are you using? Are you able to give minimal steps to replicate?

This is what I am seeing in the logs when I set things to null in MongoDB...

INFO 2023/12/22 14:12:29 Started monstache version 6.7.17
INFO 2023/12/22 14:12:29 Go version go1.20.3
INFO 2023/12/22 14:12:29 MongoDB go driver v1.13.1
INFO 2023/12/22 14:12:29 Elasticsearch go driver 7.0.31
INFO 2023/12/22 14:12:29 Successfully connected to MongoDB version 6.0.12
INFO 2023/12/22 14:12:29 Successfully connected to Elasticsearch version 8.11.3
INFO 2023/12/22 14:12:29 Listening for events
INFO 2023/12/22 14:12:29 Sending systemd READY=1
WARN 2023/12/22 14:12:29 Systemd notification not supported (i.e. NOTIFY_SOCKET is unset)
INFO 2023/12/22 14:12:29 Watching changes on the deployment
TRACE 2023/12/22 14:12:42 POST /_bulk HTTP/1.1
Host: localhost:9200
User-Agent: elastic/7.0.31 (linux-amd64)
Content-Length: 145
Accept: application/json
Authorization: Basic ZZZ
Content-Type: application/x-ndjson
Accept-Encoding: gzip

{"update":{"_index":"test.test","_id":"6585cbfa8952c6532bf0db6c"}}
{"doc":{"a":{"zed":null},"bar":null,"foo":1,"zed":null},"doc_as_upsert":true}

TRACE 2023/12/22 14:12:42 HTTP/1.1 200 OK
Content-Length: 244
Content-Type: application/json
X-Elastic-Product: Elasticsearch

{"errors":false,"took":37,"items":[{"update":{"_index":"test.test","_id":"6585cbfa8952c6532bf0db6c","_version":7315497415933427715,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":6,"_primary_term":3,"status":200}}]}

[meabed]

Thank you @rwynn im using the latest docker build 6.7.14

I will make small repo to reproduce.

It happens on updating existing document in elastic search not on new document.

[meabed]

Hi @rwynn

Thank you so much, I have tested and its working as expected.

The issue I had it was from the javascript transformation it was removing the null and unset fields :)

Thank you for your support 🙏

[meabed]

Debugging the issue - this was actually from lodash _.omit method it was removing this fields - FYI For anyone having the same issue.