Sparknlp returning different embedding for manual spark dataframe vs reading from file spark dataframe about spark-nlp

[maziyarpanahi]

You are welcome. As I mentioned those discrepancies are minor and unavoidable. The moment you batch any inputs you must either pad or truncate. This results in small precision changes as you noticed.
I suggest tuning the batchSize for performance and scale specially on GPUs and not too worried about that small changes. (We have trained high-accurate models without any issues)

[maziyarpanahi]

Hi @rishabhindoria

I think it's because in one of those it batches the inputs. Could you please add .setBatchSize(1) to the MPNet annotator and try again?

[rishabhindoria]

Thank you @maziyarpanahi but then it won't be able to scale right if the batch size itself is 1 record at a time? Need to apply this to millions of names
What is the default batch size?

[maziyarpanahi]

I think it’s 4 or 8 but it’s just a default number.
It’s the same with any other library, if you batch a different group of sequences for inference, the output will change due to padding. (Doesn’t mean the quality degrades, it’s just different)

[rishabhindoria]

thank you @maziyarpanahi it works with setting batch size=1 but again how to scale it to hundreds of millions of names? do we need to set 1 cpu core per executor and lots of such executors like this so as to parallelize?
--conf spark.executor.instances=2000 --conf spark.executor.cores=1
any guidance from other projects/companies using that in production or youtube video on that would really help