Function Execution Timeouts about azure-functions-host

[brettsam]

This is now committed with #604 but not yet in prod. The behavior is as follows:

• By default, every function has a timeout of 5 minutes, which is also the maximum timeout. The minimum value is 1 second.
• When a function timeout is hit, the function application is shut down to prevent runaway functions from executing infinitely. When a new trigger fires, the site will come back up, which may happen immediately. This means that functions that are operating normally may also be shut down if another function in the function application has timed out.
• For C# functions, you can use a CancellationToken parameter in your function. That token will be canceled whenever a timeout occurs (or any host shutdown is imminent). You can then exit your function gracefully.
• Function logs will indicate that a function has exited due to a timeout.
• The timeout can be adjusted in the host.json file by specifying a TimeSpan value in the timeout property. This will apply to all functions. For example, this sets the timeout to 1 minute:

{
    "functionTimeout" : "00:01:00"
}

[mathewc]

The maximum of 5 minutes will only apply in our Dynamic SKU. If you run in one the classic paid SKUs (Basic/Standard) with AlwaysOn enabled there will be no limit. So for potentially long running "traditional" WebJobs scenarios, one of the paid SKUs would be the way to go.

@nicolocodev No, WEBJOBS_IDLE_TIMEOUT does not apply to Functions.

[mattscully]

I am in the same boat as @cgravill. We use PowerShell Azure Functions to upgrade/downgrade the service level for SQL Azure databases on a daily schedule. This single operation usually takes around 15 minutes and the timeout now prevents the function from publishing the results when the operation completes. Is Azure Functions no longer a viable solution for simple uni-task operations like this? We don't need the job to be always on to perform these simple tasks two times a day. It seems that if Functions doesn't even have a timeout large enough to perform autonomous ARM operations that perhaps the timeout is too small? @brettsam am I thinking about this wrong? I have not worked with traditional WebJobs yet, so perhaps that is a better alternative. But so far everything I've read about Functions matched up nicely with the problem I was trying to solve.

[mathewc]

Update on this - the default timeout value is being increased from 5 minutes to 20 minutes.

[ADringer]

When will the increased timeout be available? When I try to increase the timeout in my host.json file to anything over 5 minutes I get the error:

We are not able to retrieve the runtime master key. Please try again later.

I can only generate a new key if the timeout is set to 5 mins or below.
thanks

Hi folks, I have a couple of questions on this issue:

I have a Timer Trigger - PowerShell Azure Function that runs each week day at the same hour, my run.ps1 execute a .exe that makes a "heavy work", running by ~20 minutes on each execution. The issue is that when the function is triggered it never completes the process, I think it is a "timeout" on the Azure Function "pipeline". After a while the Log Console prints out: No new trace in the past {x} min(s). but none of my logs (Write-Output)

But when I execute my .exe from the Kudu Console the work run and finish as expected.

My two questions are: Is there a "timeout" on the Azure Function "pipeline"? How to "skip/configure" this timeout to run my "heavy work" .exe?

Thanks

[brettsam]

This is a bigger work item and won't be done by Iteration 20.

[cgravill]

@brettsam what's the thinking behind having a maximum timeout of 5 minutes? Would there be a way of renewing the timeout?

We're using functions that may take longer to execute and would be difficult to split over multiple calls. As default that's fine, but it's a shame to exclude using this for potentially long running computation. The Azure Functions has been really promising for deploying bits of scientific modelling code and easily adding an HTTP endpoint.

[Goz3rr]

That seems reasonable, however we have a bunch of queries copying rows from one Azure SQL database to another, then deleting those rows from the first database. We could split each insert/delete up into separate function calls but even then some of them will run for multiple hours maybe once or twice per year.

I don't believe it's possible to run a query, close the connection and return later to check on the progress either.

Moving this to an app service would mean we'd have to juggle around with scaling between free tier/paid tier and deployments, because it also doesn't seem to be possible to stop and deallocate them like you can with a regular VM and paying year round for something that runs for a couple of days seems like a huge waste. This is one of the reasons we're moving away from the split-merge service.

Would love to hear suggestions on a better approach for this!

[DarkNoir]

I am not hitting this time out in my implementation. I created my function to run a task that takes longer than 5 mins buy using a Task for my logic. The function returns right away obviously since the logic is now async. My async task runs to completion outside the 5 min timeout. Try this and see for yourself.

`DateTime d = DateTime.Now;
Task t = new Task(() =>
{
log.Info("Doing long task");
for (int cnt = 0; cnt < 100; cnt++)
{
log.Info(DateTime.Now.Subtract(d).ToString());
System.Threading.Thread.Sleep(10000);
}
});
t.Start();'

I am currently running them on a consumption plan for more than 5 mins, as per my post. There is a timeout, but it’s not on execution time obviously since I am running them longer than 5 mins. The time out is in the calling mechanism that executes the function. If you make it async, the caller returns and the function continues to execute. It is a common practice to put long running tasks on a separate thread/process than the calling thread/process. Keeping the orchestration layer of the function platform from blocking for long periods of time just makes sense. You should not block the calling layer for long running functions as it will seriously degrade the platform.

I wasn't trying to bypass the 5 min timeout, but rather trying to develop with good disciplines. I wanted an async function. I learned about the 'timeout' later. No one should be running long running functions synchronously anyway. Also, people should be aware of how to make async functions regardless of timeouts.

The interwebs can really be a silly place sometimes. I think people are getting so used to cut and pasting code, they are forgetting the basics.

PS. Please add your constraints to this Azure doc:
https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#app-service-limits

William Wallace

EDIT: If you attempt this async code, it will not execute to much beyond 5 minutes. I was experiencing longer runtimes because the Azure Portal was keeping the VM alive. If you return from your function while still executing a background task, and nothing else is keeping your App Service alive (Other function executions or the portal) , the VM will be shut down. There is also a timeout that will be hit at 5 minutes when executing normally.

There is no way to reliably execute an Azure function asynchronously. You can execute async and background tasks within the function but you must synchronize before returning to the caller.

[mathewc]

Correct - if you spin off an async task outside of the function execution (which we're monitoring), if your Function App stays alive due to other traffic, it won't be limited. The limit we have in place is all about making things predictable for users in our consumption plan. The underlying infrastructure will currently shut down the VM instance (as described elsewhere above) if no new triggers/invocations have happened in 5 minutes. That being the case, you can't really run reliably over 5 minutes.

However above we also link to Azure/Azure-Functions#75, where we discuss making lower level changes allowing us to relax this limit.

[mathewc]

I'll summarize all this by saying that any work started out of band from a normal function invocation (e.g. you spin up a background task that runs after the function returns) is not guaranteed to run reliably. Our underlying infrastructure can choose to release the VM at any time in such scenarios.

[fabiocav]

@DarkNoir , async functions will have the same guarantees, as long as you're following proper patterns and returning the Task representing your async work. As @mathewc pointed out, the pattern you've shared above is not going to be reliable and is definitely not something we recommend.

[christopheranderson]

RC justification: Needed to help control billing costs

[mathewc]

We might be able to leverage the SDK TimeoutAttribute for this. See example here. We could just expose a new function metadata property that maps to this.

[mathewc]

As part of this, for Node, we should also bring in the the tripwire module to catch while(true) errors (just like we did in Zumo).

Any update on whether timeouts on azure functions are configurable now? thanks!

[fabiocav]

This work is in progress with an open PR (#604).

great thanks! Any ETA on having it on prod?

@brettsam Do you know if the WEBJOBS_IDLE_TIMEOUT have some effect on Azure Functions timeouts?

[davidebbo]

Reopening so we can discuss options.

[Goz3rr]

Same here, we need to run several long (hours) database related tasks a few times per year, having to manually manage a VM for this would be a waste.

The Azure Functions Overview even mentions this specific scenario!

Consider Functions for tasks like image or order processing, file maintenance, long-running tasks that you want to run in a background thread, or for any tasks that you want to run on a schedule.

[davidebbo]

If your goal is to run a PowerShell script that runs on a schedule and manages your Azure resources, you should look at Azure Automation. It's exactly what it's for, and it likely a better fit than Azure Functions for such task.

[mattscully]

One of our long running tasks is scheduled but others may be triggered by an HTTP request or something else. Is Azure Functions no longer intended to support long running tasks? It seems that if the purpose of this change was to prevent tasks from getting stuck in a loop, perhaps a good approach would be:

• Default timeout of 5 minutes.
• Default timeout can still be overridden in host.json. This max can still be 5 minutes.
• Allow individual functions to be configured to override the default (perhaps with a max on the order of hours or one day?).

This would allow function authors to set a reasonable timeout for that particular task without raising the limit for all functions.

Of course, if the intent of Functions is now scoped to tasks <= 5 minutes, then that is good to know and we will just need to look into migrating our long running tasks elsewhere (Azure Automation, etc.).

[brettsam]

Thanks for the feedback. A little more background:

The reason we timeout at 5 minutes with the Dynamic plan is because under the Dynamic plan, the VM that is hosting your function will shut down (roughly) 5 minutes after the last trigger fired. Enforcing the timeout like we're doing today is a way to provide a consistent experience with logging that explains why the function stopped running. Before this, you would see functions simply disappear mid-invocation without any indication of why. We're evaluating improvements in this area and will update when we have more concrete details.

Long-running functions that cannot be decomposed down into smaller chunks will run into issues running in the Dynamic plan currently. However, there are often solutions for breaking down single long-running functions into smaller, quicker functions. For example, if you have a function that kicks off a long operation elsewhere, then polls for completion, could you get an 'operationId' from that operation, then put it in a Service Bus scheduled message (or even in a Queue message with an invisibility timeout), and have a second 'CheckStatus' function that reads those messages and polls for completion?

It may be interesting to capture some of the long-running scenarios here and see if we can establish some patterns. Feel free to chime in here with your long-running scenarios if you have them. That'll help drive future improvements in this area.

[mathewc]

Moving back to triage since this closed issue was reopened to discuss options.

[lorenzoaiello]

We have a similar scenario in which we run one or two long-running jobs (couple hours) once a week which we were hoping to use Azure Functions for on the Dynamic plan but the 5-minute cap is a blocker.

[lindydonna]

Please add any scenarios you might have to the top level issue here (Azure/Azure-Functions#75).

[paulbatum]

If platform work is done to enable functions to run for longer than 5 minutes (tracked by Azure/Azure-Functions#75) then we'll need updates here to relax our validation. Otherwise there is no further work here.

[DarkNoir]

@mathewc If I am running async functions, can you guarantee at least 5 mins? I am more than a little surprised that I have not seen any postings or talk on async implementations. What is your confidence level in async execution of a function? Will it possibly mess with your orchestration logic negatively if you cannot detect workloads outside synchronized calls?

EDIT:
Ok, when calling the function async, the 'other traffic' that @mathewc refers to can be traffic from the Azure portal. If the Azure portal is connected to your function app, it will override the timeout of the VM shutdown. This skewed much of my perception. When the portal is not connected and the function executes async, the VM will get released just after 5 min. I haven't checked to see if CLI connections to the log stream, or debugger connections also keep it alive, but I imagine they might since there are a number of other things that can be touching your Service App instance that host the functions.

[DarkNoir]

Apart from my questions and experiences with creating async functions and their interactions with this limit, for long running processes, I have maintained the state for the work being done outside the function and recursively called the function with a reference to the state when its duration approaches the 5 min limit. This allowed me to optimize execution count and time. This was worth the effort since this function was used intermittently. Using parallelism within the function can also decrease execution time. This also allowed me to have some control over throttling how many and how quickly the workitems were being processed. With this type of tech, since it scales so high and quickly, you can easily create your own DDOS attack on your internal systems if you are not careful.

[DarkNoir]

Yeah, unfortunately, the async pattern I was shooting for was the ‘fire and forget’ async that does not require synchronization, ever. (A proper pattern, just possibly not supported in your implementation at this time and thus cannot be recommended.). Having to synchronize Atomic and usually idempotent units of work chaffs a bit.

Or, I am missing something and it can be done and I haven’t figured it out yet and no one has pointed out the way to achieve async azure function calls is just not how I was attempting to do it.

2 facets where in my discussions today.

1. The 5 min execution time limit. I was seeing runtimes outside that because there were other things keeping my vm up outside my executing code. – RESOLVED, my perception was incorrect.

2. I was wanting to execute functions asynchronously (fire and forget, not wait for completion) I believe this is resolved and the answer is NOT SUPPORTED.

Is my understanding correct now? I really appreciate how much help you guys are giving me to make sense of what I am seeing. I am doing this so I can share this understanding with others also.

[mikezks]

@brettsam, related to your pattern suggestion for long running operations above:
Do I understand this szenario correctly that your described long running operation runs somewhere outside of Azure Functions and Azure Functions (AF) is used to get the status of this operation periodically via for example a message queue trigger?

1. AF1 starts long running operation outside of Azure Functions infrastructure
2. Long running operation return operationID immediately
3. AF1 sends message to queue which will be processed by AF2 e.g. 3 min later
4. AF2 processes the message after 3 min
5. AF2 checks long running operation status and logs the status (running, error, successful)
6. AF2 starts repeating process step (3) and following
7. AF2 notifies AF1 on success, error or status after a reached timelimit via a webhook call

Is this the way you described or did I get something wrong?

[mikezks]

Another question:
Related to long running Azure SQL operations, are there any plans to support Azure Functions inbound trigger for Azure SQL events (e.g. new row in table ABC was added)?

[niemyjski]

What about applying a timeout on a per function level?

[lindydonna]

@mikezks and @niemyjski Please open new issues or forum posts for new questions. We don't monitor closed issues.

[kchauhan-muhimbi]

Hi,

I have a continuous web job created using console app. I am calling external web service from the web jobs. Web service takes about 7-8 minutes to respond. Web job looses the web service connection after ~5 minutes but still keep on waiting for response until it times out after the "ReceiveTimeout" value. I am suspecting that it is the idle timeout out kicks in and drops the web service connection. Please note that the external web service is hosted on azure VM and has idle timeouts set to 30 mins.

Any inputs on this?

Regards,
Kiran Chauhan

[paulbatum]

@kchauhan-muhimbi With the current limits in place, you won't be able to work with such as web service from the consumption plan. You'll need to switch to a dedicated app service plan.

[kchauhan-muhimbi]

Hi Paul, We are on Basic tier of web app, would it still not work? I think, Basic falls under dedicated app service plan. Please advice. Thanks, Kiran Chauhan SharePoint Consultant

…

Sent from my iPhone On 02-May-2017, at 10:07 PM, Paul Batum <[email protected]<mailto:[email protected]>> wrote: @kchauhan-muhimbi<https://github.com/kchauhan-muhimbi> With the current limits in place, you won't be able to work with such as web service from the consumption plan. You'll need to switch to a dedicated app service plan. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#18 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AYAeSddjZ5fMtmC0x5ldQuHQueTwN1vSks5r11uwgaJpZM4HYgFo>.

[paulbatum]

@kchauhan-muhimbi Thanks for the clarification. I re-read your original comment. The issue you are experiencing is unrelated to this thread. This thread is about Azure Functions, and specifically the five minute execution limit when in the consumption plan.

[ttrace1995]

For anyone reading this, I am finding out that Azure Functions are incredibly fragile and unpredictable. For instance I just spent an hour researching how to debug the yellow exclamation mark response on the metrics console, and during that time the function magically started to work. Then magically fell back over 3 hours later after a simple code reformat. Most of the problems i'm having don't have to do with heavy workloads or anything. It almost seems like you need to give azure functions a couple hours to "warm up" before it starts interpreting the code correctly. Is anyone else having a similar experience?

[fabiocav]

@ttrace1995 sorry to hear you're running into issues. Would you mind opening a separate issue with more details about the issue you're seeing and the information requested on the issue template so we can identify and take a closer look at your app?

[alohaninja]

For those here looking to apply functionTimeout to Azure Functions - I was able to get this working with Timer Triggers.

After adding the timeout in host.json bindings you will see this if the process exceeds the threshold...

Timeout value of 00:30:00 exceeded by function 'Functions.YourFunctionName' (Id: '35190e05-1a8f-49d6-843e-da9bfd57bb70'). Initiating cancellation.

host.json

{
  "functionTimeout": "00:30:00" // 30 min timeout
}

[punitganshani]

@alohaninja FunctionTimeout must be between 00:00:01 and 00:10:00

[manoharreddyporeddy]

See: https://docs.microsoft.com/en-us/azure/azure-functions/functions-scale#consumption-plan

[JohnGalt1717]

This needs to be way more than 10 minutes. Doing things like encoding a video file in Azure Media Services takes way longer than that often and that's a basic use case especially for a QueueTrigger.

2 hours would start to be a reasonable number.