Comments (7)
I had thought that Netherite's implementation of Event Hub in combination with FASTER would make it unlikely that such multiple execution of Activity would occur, but it seems that it is occurring more than I had expected.
That is not surprising. Duplicate activity executions are considered a normal occurrence. The most common reason for duplicate activity executions is when partitions are moved for load balancing reasons. The partition manager is not being careful about this, i.e. it will not hesitate to move a partition to a different host even if it means that such movement creates duplicate activity executions.
from durabletask-netherite.
Hi @shibayan, thanks for reaching out.
It's a bit difficult to diagnose this issue without having more context. Would you be able to provide us with an example instanceID that experienced this issue, the activity name that got doubly executed, the app name on Azure, and the timerange in UTC where this issue can be observed? From there, we should be able to assist better. Thanks!
from durabletask-netherite.
@davidmrdavid Thanks for the reply. Since this issue occurred in a production environment, I would like to exchange App Service information and other information via email. Can you provide me with an email address?
from durabletask-netherite.
Thanks @shibayan.
Agreed, this isn't entirely surprising and duplicate activity executions are allowed by the framework (that's why activities need to be idempotent), they're just not supposed to happen very often.
If you still would like me to take a peek, you can DM me on twitter (https://twitter.com/davidjustodavid) and I'll send you my work email there (just trying to avoid making it public on GitHub). But yeah unless this is happening extremely frequently, this may very well be part of normal operation for the framework.
from durabletask-netherite.
@sebastianburckhardt @davidmrdavid Thank you for your reply, I understand that the multiple execution of Activities is an acceptable behavior by design.
This design behavior may not be well known by Durable Functions users. The documentation says that non-deterministic operations should be performed in Activity, so it seems that there are many cases of misunderstanding like mine.
Depending on which language you use, a built-in API for generating deterministic GUIDs or UUIDs may be available. Otherwise, use an activity function to return a randomly generated GUID or UUID.
from durabletask-netherite.
Thanks @shibayan. I'm trying to understand how to make this guidance clearer. You're correct that Activities are where non-deterministic operations should be performed. At the same time, activities may execute more than once. What the Durable Functions framework does in this case is that, as soon as one activity has its output durably stored, the activity should no longer execute. Therefore, even though the activity may execute more than once, only one of it's results will be acknowledged by the orchestrator and therefore it will be as-if it had only executed once from the perspective of the workflow itself.
Does that makes sense? Do you have any suggestions on how to rephrase our guidance so that this can be made clearer? Thanks!
from durabletask-netherite.
@davidmrdavid It makes sense. My customer really likes Durable Functions/Netherite and uses them extensively, so I wanted to make the behavior clear.
The question was successfully resolved. I would like to provide feedback on the documentation separately. Thanks!
from durabletask-netherite.
Related Issues (20)
- Crash using medium test case HOT 14
- Partition Table Document Issue HOT 1
- Netherite with medium to large payloads performance dive HOT 9
- 409 The specified container already exists after switching to Netherite (JavaScript)
- Event Hub Client Timeouts when exceeding Event Hub limits HOT 5
- Performance drop when using external events HOT 5
- Question regarding "Cannot access a disposed object. Object name: 'FaultTolerantAmqpObject`1'." and "Client object already closed." HOT 3
- ScaleMonitor is constantly logging "An item with the same key has already been added." errors HOT 4
- `BlobNotFound` error when running the DTFx sample
- Question: Event Hub traffic when no orchestrations are executing HOT 5
- Orchestrator start timed out when `PartitionManagement` is set to `ClientOnly` HOT 2
- Part11 discarded ActivityWorkItem 11A19122 because partition was terminated HOT 2
- Startnew Async behaves strange HOT 1
- .Net function app error once deployed HOT 2
- Task Hub Client Timeout Issue when directed to Specific Partition via Orchestration Id HOT 3
- Include child instance IDs in execution history HOT 7
- Non-Deterministic workflow detected ... current replay execution hasn't (yet?) scheduled this task HOT 1
- State Corruption in FASTER storage HOT 2
- Sudden Netherite "Client request timed out" exception HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from durabletask-netherite.