This repository contains the code used to illustrate the posts on www.waitingforcode.com blog about Apache Beam.
beam-learning's Introduction
beam-learning's People
beam-learning's Issues
How can Apache Beam determine in code that the data in two PCollection are equal?
I want to use Apache Beam to implement K-means, but I can't tell whether the new and old PCollection are equal or not in the code, do you have a better answer or idea?
Please check my StackOverflow question for details.
https://stackoverflow.com/questions/71686576/how-to-compare-the-equality-of-elements-in-two-pcollection-in-apache-beam
In Spark, I can get the data by using .colletion, and I can share it globally by using broadcast variables, but in Apache Beam, PCollectionView can implement similar broadcast variables, but there is no way to get the data in PCollection or determine whether two PCollections are equal.
I tried to use Object.eqauls, but it failed.
Also aggregating data into Sum.doublesGlobally() and converting it to PCollectionView, I found that I could only view it through the DoFn side input, then I modified the global variables through DoFn, and debugging found that the program would loop first and then enter the specific content of DoFn, so there was no way to determine convergence and then terminate the loop in the main program.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.