bartosz25 / beam-learning

Apache Beam learning tests

This repository contains the code used to illustrate the posts on www.waitingforcode.com blog about Apache Beam.

beam-learning's Issues

How can Apache Beam determine in code that the data in two PCollection are equal?

I want to use Apache Beam to implement K-means, but I can't tell whether the new and old PCollection are equal or not in the code, do you have a better answer or idea?
Please check my StackOverflow question for details.
https://stackoverflow.com/questions/71686576/how-to-compare-the-equality-of-elements-in-two-pcollection-in-apache-beam

In Spark, I can get the data by using .colletion, and I can share it globally by using broadcast variables, but in Apache Beam, PCollectionView can implement similar broadcast variables, but there is no way to get the data in PCollection or determine whether two PCollections are equal.

I tried to use Object.eqauls, but it failed.
Also aggregating data into Sum.doublesGlobally() and converting it to PCollectionView, I found that I could only view it through the DoFn side input, then I modified the global variables through DoFn, and debugging found that the program would loop first and then enter the specific content of DoFn, so there was no way to determine convergence and then terminate the loop in the main program.