Recommendations on storing large payloads about durabletask-mssql

[cgillum]

If the payload will make its way into the orchestration history, then the biggest problem you'll run into is memory usage because each orchestration will need to pull that document into its memory (perhaps multiple times), which often takes up more memory than the actual size of the document content on disk. Database disk and network I/O will also be issues for the reasons you mentioned. If you decide to go this route, you'll want to be sure to carefully throttle your concurrency to avoid problems with resource exhaustion.

All that said, its best practice is to pass around references to data in your orchestration and have activities use those references to access the data directly from external storage.

[gauravvgat]

Thanks @cgillum . We are going ahead with storing payload in storage and passing reference. Also when and why would a payload get into the orchestration history?

[cgillum]

Payloads get stored in history when they are used as activity inputs or outputs, or as external event payloads. This is so that the values can be replayed to reconstruct the previous state of an orchestration.

[gauravvgat]

Thanks, closing this now.