raisedapp / hangfire.storage.sqlite Goto Github PK
View Code? Open in Web Editor NEWAn Alternative SQLite Storage for Hangfire
Home Page: https://www.nuget.org/packages/Hangfire.Storage.SQLite
License: MIT License
An Alternative SQLite Storage for Hangfire
Home Page: https://www.nuget.org/packages/Hangfire.Storage.SQLite
License: MIT License
Hi and thanks for a great library! :-)
This is a feature request to implement SlidingInvisibilityTimeout.
Every now and then we run into issues with the InvisibilityTimeout for "long-running" jobs, which gets cancelled when the timeout is reached and then requeued. If the job does not implement the CancellationToken properly, it keeps running in the background while the new job is enqueued and duplicate processing occurs - there are some remedies for that as well, but none without some caveats.
Increasing the timeout also has its caveats as it might result in jobs not being executed for many hours in case of hard-failure on server/process.
The PostgreSQL provider has the same issue, which also explains the issue better/in more detail, and how the SQLServer provider has solved it:
And there is a pull-request for implementing SlidingInvisibilityTimeout for inspiration if someone wants to implement it here:
Issue
The internal functions GetTimelineStats
and GetHourlyTimelineStats
in SQLiteMonitoringApi.cs
return a dictionary of DateTime
to counts, but they are not matched correctly. This affects the following public API functions:
SucceededByDatesCount()
FailedByDatesCount()
HourlySucceededJobs()
HourlyFailedJobs()
This in turn causes the graphs in the Hangfire dashboard to be incorrect.
Cause
The statement on lines 195 and 228 is as follows:
var value = valuesAggregatorMap[valuesAggregatorMap.Keys.ElementAt(i)];
This is incorrect, as it disregards the keys of valuesAggregatorMap
and instead just indexes into the Keys collection, which is inherently unordered and so the result is not well defined. This means that the wrong counts are assigned to the wrong DateTime
keys in the final result.
Fix
By changing line 195 to this:
var value = valuesAggregatorMap[$"stats:{type}:{stringDates[i]}"];
and line 228 to this:
var value = valuesAggregatorMap[$"stats:{type}:{dates[i]:yyyy-MM-dd-HH}"];
the issue is fixed, as the map is correctly looked-up using the counter key.
Hello,
Global settings are set by this plugin here
There are two problems:
.UseSQLiteStorage()
and .UseRecommendedSerializerSettings()
set this global option, the end value of the serializer settings will be different depending on the order the two methods are called.DateFormatString
doesn't contain the timezone format (supposed to be K
at the end. See https://www.newtonsoft.com/json/help/html/P_Newtonsoft_Json_JsonSerializer_DateFormatString.htm)DateTime
and DateTimeOffset
.In my case, I'm using a job with a DateTimeOffset
argument.
I enqueue it with an offset of 0 (UTC).
When the job is called by Hangfire, the hours and minutes values didn't change, but the offset has been set to my local offset (+1)!
I spent a few hours on this to finally find that .UseRecommendedSerializerSettings()
after .UseSQLiteStorage()
instead of before fixes my issue.
I'm using SQLite as the database for my app. So I installed Microsoft.EntityFrameworkCore.Sqlite
version 3.1.7 nuget & ran the project (Blazor Server App) which caused following exception:
System.MissingMethodException: Method not found: 'System.String SQLitePCL.raw.sqlite3_column_name(SQLitePCL.sqlite3_stmt, Int32)'.
Without
Microsoft.EntityFrameworkCore.Sqlite
, Hangfire works perfectly.
Category: Hangfire.Storage.SQLite.SQLiteDistributedLock
EventId: 0
Unable to update heartbeat on the resource 'HangFire:job:1201:state-lock'. The resource is not locked or is locked by another owner.
Category: Hangfire.Storage.SQLite.SQLiteDistributedLock
EventId: 0
Unable to update heartbeat on the resource 'HangFire:job:1199:state-lock'. The resource is not locked or is locked by another owner.
Category: Hangfire.Storage.SQLite.SQLiteDistributedLock
EventId: 0
Unable to update heartbeat on the resource 'HangFire:job:1199:state-lock'. The resource is not locked or is locked by another owner.
How fix this error?
Result for .net 8 results in a warning
warning NETSDK1206: Found version-specific or distribution-specific runtime identifier(s): alpine-x64.
Affected libraries: SQLitePCLRaw.lib.e_sqlite3.
In .NET 8.0 and higher, assets for version-specific and distribution-specific runtime identifiers will not be found by default. See https://aka.ms/dotnet/rid-usage for details.
System.TypeInitializationException
HResult=0x80131534
Mensaje = The type initializer for 'SQLite.SQLiteConnection' threw an exception.
Origen = SQLite-net
Seguimiento de la pila:
en SQLite.SQLiteConnection..ctor(String databasePath, SQLiteOpenFlags openFlags, Boolean storeDateTimeAsTicks)
en Hangfire.Storage.SQLite.HangfireDbContext..ctor(String databasePath, String prefix)
en Hangfire.Storage.SQLite.HangfireDbContext.Instance(String databasePath, String prefix)
en Hangfire.Storage.SQLite.SQLiteStorage..ctor(String databasePath, SQLiteStorageOptions storageOptions)
en Hangfire.Storage.SQLite.SQLiteStorageExtensions.UseSQLiteStorage(IGlobalConfiguration configuration)
en Program.<>c.<
2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument) en Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context) en Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor
2.VisitCallSite(ServiceCallSite callSite, TArgument argument)2.GetOrAdd(TKey key, Func
2 valueFactory)Esta excepción se generó originalmente en esta pila de llamadas:
[Código externo]
Excepción interna 1:
TypeLoadException: Method 'sqlite3_soft_heap_limit64' in type 'SQLitePCL.SQLite3Provider_dynamic_cdecl' from assembly 'SQLitePCLRaw.provider.dynamic_cdecl, Version=2.0.3.851, Culture=neutral, PublicKeyToken=b68184102cba0b3b' does not have an implementation.
If a job is created while another thread is inserting a row elsewhere into the DB, it is sometimes assigned the wrong ID due to this issue in the underlying SQLite library.
Because of this, jobs aren't properly initialised, so they are left with a StateName
of NULL
, which means they silently fail, and their JobParameter
s end up being linked to the wrong jobs. If there are lots of jobs being created and run, this ends up happening quite often.
An example of the incorrect IDs being assigned in SQLiteConnectionFacts
:
[Fact, CleanDatabase]
public void CreateExpiredJob_CorrectlyPopulatesId_WhenManyThreadsAreInserting()
{
const int InsertCount = 2000;
var job = Job.FromExpression(() => SampleMethod("Hello"));
var parameters = new Dictionary<string, string>();
var createdAt = new DateTime(2012, 12, 12, 0, 0, 0, 0, DateTimeKind.Utc);
var expireIn = TimeSpan.FromDays(1);
UseConnection((database, connection) =>
{
var lastJobId = int.Parse(connection.CreateExpiredJob(job, parameters, createdAt, expireIn));
var finished = false;
var ok = true;
var expectedId = 0;
string jobId = null;
Parallel.Invoke(
// Create jobs and break if they are unexpected
() =>
{
for (var insertIndex = 1; insertIndex <= InsertCount; insertIndex++)
{
expectedId = lastJobId + insertIndex;
// Inserts the job with the expected ID, but returns the incorrect ID
jobId = connection.CreateExpiredJob(job, parameters, createdAt, expireIn);
ok = expectedId.ToString() == jobId;
if (!ok)
{
break;
}
}
finished = true;
},
// Another thread inserting rows to the db, e.g. other jobs completing
() =>
{
while (ok && !finished)
{
// Arbitrary database insert
database.Database.Insert(new Set());
}
}
);
if (!ok)
{
Assert.Equal(expectedId.ToString(), jobId);
}
});
}
Note that commenting out the inserting in the second thread fixes the issue
I noticed that the Recurring Jobs would execute immediately when they were first added to the queue.
Can we solve this problem? Thank you very much.
Add Unit Testing to the Project
Hello,
I'm trying to create several jobs, but after some jobs created, my application thrown this error.
Can help me?
Thanks,
mstiago
Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details.
---> System.ArgumentException: An item with the same key has already been added. Key: CurrentCulture
at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
at Hangfire.Storage.SQLite.SQLiteMonitoringApi.<>c__DisplayClass25_0.<JobDetails>b__0(HangfireDbContext _)
at Hangfire.Console.States.ConsoleApplyStateFilter.OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
at Hangfire.States.StateMachine.InvokeOnStateApplied(Tuple`2 x)
at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1 tuple)
at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfiler profiler, TInstance instance, Action`1 action, String message)
at Hangfire.States.StateMachine.ApplyState(ApplyStateContext initialContext)
at Hangfire.Client.CoreBackgroundJobFactory.<>c__DisplayClass14_0.<Create>b__3(Int32 attempt)
at Hangfire.Client.CoreBackgroundJobFactory.<>c__DisplayClass15_0.<RetryOnException>b__0(Int32 attempt)
at Hangfire.Client.CoreBackgroundJobFactory.RetryOnException[T](Int32& attemptsLeft, Func`2 action)
--- End of stack trace from previous location ---
at Hangfire.Client.CoreBackgroundJobFactory.RetryOnException[T](Int32& attemptsLeft, Func`2 action)
at Hangfire.Client.CoreBackgroundJobFactory.RetryOnException(Int32& attemptsLeft, Action`1 action)
at Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context)
at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass12_0.<CreateWithFilters>b__0()
at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
at Hangfire.BackgroundJobClient.Create(Job job, IState state)
--- End of inner exception stack trace ---
at Hangfire.BackgroundJobClient.Create(Job job, IState state)
at Infrastructure.BackgroundJobs.HangfireService.Enqueue(Expression`1 methodCall) in C:\Users\6110183\source\repos\ThomsonReuters.HighQIntegration\Infrastructure\BackgroundJobs\HangfireService.cs:line 16
at Infrastructure.Strategies.Implementations.TimesheetStrategy.RunAsync() in C:\Users\6110183\source\repos\ThomsonReuters.HighQIntegration\Infrastructure\Strategies\Implementations\TimesheetStrategy.cs:line 95
Are you planning to make your project compatible with Hangfire.SQLite? I.e. will the db structure be compatible? In either case, please consider being explicit about the compatibility in the README.
Hi there
is there any permissions needed in order to create the sqlilte db instance?
GlobalConfiguration.Configuration.UseSQLiteStorage();
Thanks
Alex
My ASP.NET Core 6 app shows this error very often:
Unable to update heartbeat on the resource 'HangFire:xxx'. The resource is not locked or is locked by another owner.
I believe this has to do with #68, and may be gone if that error is fixed.
But regardless of #68, if SQLiteDistributedLock cannot update the heartbeat because of that message, what's the point of keep retrying? I think the timer should be stopped in that case - or at the minimum, mute the error log so that it doesn't show up indefinitely in the logs.
Extension method UseSQLiteStorage
has a argument named nameOrConnectionString
but the sqlite-net
library's SQLiteConnection
takes databasePath
as argument.
I add another SQLite provider, like EF Core SQLite to project will throw exception from run project .UseSQLiteStorage()
.
Exception thrown: 'System.MissingMethodException' in SQLite-net.dll
An exception of type 'System.MissingMethodException' occurred in SQLite-net.dll but was not handled in user code
Method not found: 'System.String SQLitePCL.raw.sqlite3_column_name(SQLitePCL.sqlite3_stmt, Int32)'.
In ASP.NET Core 3.1 API PoC project,
.csproj
<ItemGroup>
<PackageReference Include="Hangfire.Core" Version="1.7.*" />
<PackageReference Include="Hangfire.AspNetCore" Version="1.7.*" />
<PackageReference Include="Hangfire.Storage.SQLite" Version="0.2.4" />
<PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="3.1.2" />
</ItemGroup>
startup.cs
public void ConfigureServices(IServiceCollection services)
{
services.AddHangfire(configuration => configuration
.UseSimpleAssemblyNameTypeSerializer()
.UseRecommendedSerializerSettings()
.UseSQLiteStorage());
services.AddHangfireServer();
services.AddControllers();
}
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
app.UseRouting();
app.UseAuthorization();
app.UseHangfireDashboard();
BackgroundJob.Enqueue(() => Console.WriteLine("Hello backgroundjobs."));
RecurringJob.AddOrUpdate("Hello", () => Console.WriteLine("Hello, recurringJob."), Cron.Minutely);
app.UseEndpoints(endpoints =>
{
endpoints.MapControllerRoute(
name: "default",
pattern: "{controller=Home}/{action=Index}/{id?}");
endpoints.MapControllers();
});
}
Consider the following test:
[Fact, CleanDatabase]
public void OnlySingleLockCanBeAcquired()
{
var connection = ConnectionUtils.CreateConnection();
var numThreads = 10;
long concurrencyCounter = 0;
var manualResetEvent = new ManualResetEventSlim();
var success = new bool[numThreads];
// Spawn multiple threads to race each other.
var threads = Enumerable.Range(0, numThreads).Select(i => new Thread(() =>
{
// Wait for the start signal.
manualResetEvent.Wait();
// Attempt to acquire the distributed lock.
using (new SQLiteDistributedLock("resource1", TimeSpan.FromSeconds(5), connection, new SQLiteStorageOptions()))
{
// Find out if any other threads managed to acquire the lock.
var oldConcurrencyCounter = Interlocked.CompareExchange(ref concurrencyCounter, 1, 0);
// The old concurrency counter should be 0 as only one thread should be allowed to acquire the lock.
success[i] = oldConcurrencyCounter == 0;
Interlocked.MemoryBarrier();
// Hold the lock for some time.
Thread.Sleep(100);
Interlocked.Decrement(ref concurrencyCounter);
}
})).ToList();
threads.ForEach(t => t.Start());
manualResetEvent.Set();
threads.ForEach(t => Assert.True(t.Join(TimeSpan.FromMinutes(1)), "Thread is hanging unexpected"));
// All the threads should report success.
Interlocked.MemoryBarrier();
Assert.DoesNotContain(false, success);
}
This test races threads trying to acquire a SQLiteDistributedLock
and confirms whether the thread was the only one executing at between acquire and release.
The SQLiteDistributedLock::Acquire()
method is unsafe. Consider the following execution through the function:
Now there are two different locks in the DB for the same resource because there is no uniqueness constraint over the Resource
column.
Suggested fixes:
Resource
column which would prevent the second insert from succeedingAside: I don't know the transactional capabilities nor the uniqueness constraints of LiteDB, but suspect the code this was taken from is similarly flawed.
I discovered this as the DisableConcurrentExecution
was failing to prevent concurrent execution of multiple instances of the same job.
Hello,
I noticed that there is a prefix
parameter in the HangfireDbContext, but it seems that it is not being utilized, resulting in the inability to customize the table prefix name. I attempted to set a custom prefix, but it doesn't reflect in the generated table names.
Is there a specific way to use the prefix
parameter, or is there any other workaround to achieve custom table prefix functionality in HangfireDbContext?
Thank you for your assistance.
/// <summary>
/// Starts SQLite database using a connection string for file system database
/// </summary>
/// <param name="databasePath">the database path</param>
/// <param name="prefix">Table prefix</param>
private HangfireDbContext(string databasePath, string prefix = "hangfire")
{
//UTC - Internal JSON
GlobalConfiguration.Configuration
.UseSerializerSettings(new JsonSerializerSettings()
{
DateTimeZoneHandling = DateTimeZoneHandling.Utc,
DateFormatHandling = DateFormatHandling.IsoDateFormat,
DateFormatString = "yyyy-MM-dd HH:mm:ss.fff"
});
Database = new SQLiteConnection(databasePath, SQLiteOpenFlags.ReadWrite |
SQLiteOpenFlags.Create | SQLiteOpenFlags.FullMutex, storeDateTimeAsTicks: true);
ConnectionId = Guid.NewGuid().ToString();
}
/// <summary>
/// Initializes initial tables schema for Hangfire
/// </summary>
public void Init(SQLiteStorageOptions storageOptions)
{
StorageOptions = storageOptions;
AutoClean(storageOptions);
Database.CreateTable<AggregatedCounter>();
Database.CreateTable<Counter>();
Database.CreateTable<HangfireJob>();
Database.CreateTable<HangfireList>();
Database.CreateTable<Hash>();
Database.CreateTable<JobParameter>();
Database.CreateTable<JobQueue>();
Database.CreateTable<HangfireServer>();
Database.CreateTable<Set>();
Database.CreateTable<State>();
Database.CreateTable<DistributedLock>();
AggregatedCounterRepository = Database.Table<AggregatedCounter>();
CounterRepository = Database.Table<Counter>();
HangfireJobRepository = Database.Table<HangfireJob>();
HangfireListRepository = Database.Table<HangfireList>();
HashRepository = Database.Table<Hash>();
JobParameterRepository = Database.Table<JobParameter>();
JobQueueRepository = Database.Table<JobQueue>();
HangfireServerRepository = Database.Table<HangfireServer>();
SetRepository = Database.Table<Set>();
StateRepository = Database.Table<State>();
DistributedLockRepository = Database.Table<DistributedLock>();
}
Implementing the following features doesn't take much effort
I already got those implemented and will provide a pull request
Hi Felix,
I noticed that the issue still exist for the functions "ScheduledJobs", "ProcessingJobs" and "FailedJobs" on the "SQLiteMonitoringApi"class, It seems that you missed adding validations on "ExceptionDetails", "ExceptionMessage" and "ExceptionType" in the "FailedJobs" function, "StartedAt" in the "ProcessingJobs" function and "ScheduledAt" in the "ScheduledJobs" function.
Can you please include this validations?
I'm sorry I don't know how to reliably reproduce this exception. All I can tell you is that MyJobClass.MyJobMethod
has this attribute: [DisableConcurrentExecution(timeoutInSeconds: 10 * 60)]
and that I'm using a TransactionScope to create two dependent jobs.
Hangfire.AutomaticRetryAttribute|Failed to process the job '123': an exception occurred. Retry attempt 1 of 10 will be performed in 00:00:39.
Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'Could not place a lock on the resource 'HangFire:MyJobClass.MyJobMethod': The lock request timed out.' resource.
at void Hangfire.Storage.SQLite.SQLiteDistributedLock.Acquire(TimeSpan timeout)
at new Hangfire.Storage.SQLite.SQLiteDistributedLock(string resource, TimeSpan timeout, HangfireDbContext database, SQLiteStorageOptions storageOptions)
at IDisposable Hangfire.Storage.SQLite.HangfireSQLiteConnection.AcquireDistributedLock(string resource, TimeSpan timeout)
at void Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContext filterContext)
at bool Hangfire.Profiling.ProfilerExtensions.InvokeAction<TInstance>(InstanceAction<TInstance> tuple)
at TResult Hangfire.Profiling.SlowLogProfiler.InvokeMeasured<TInstance, TResult>(TInstance instance, Func<TInstance, TResult> action, string message)
at void Hangfire.Profiling.ProfilerExtensions.InvokeMeasured<TInstance>(IProfiler profiler, TInstance instance, Action<TInstance> action, string message)
at PerformedContext Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func<PerformedContext> continuation)
Anyway I switched to HangFire.SqlServer and this problem did not occur anymore.
Still randomly getting distributed lock exception.
Hangfire.Storage.SQLite version: 0.3.0
Hangfire.Core version: 1.7.22
Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'Could not place a lock on the resource 'HangFire:extension:job-mutex:lock:StatusUpdateJob': The lock request timed out.' resource. at Hangfire.Storage.SQLite.SQLiteDistributedLock.Acquire(TimeSpan timeout) at Hangfire.Storage.SQLite.SQLiteDistributedLock..ctor(String resource, TimeSpan timeout, HangfireDbContext database, SQLiteStorageOptions storageOptions) at Hangfire.Storage.SQLite.HangfireSQLiteConnection.AcquireDistributedLock(String resource, TimeSpan timeout)
The following snippet shows a (valid?) usage of SQLiteDistributedLock which leaves/can leave the lock alive forever:
public async Task Method(PerformContext context)
{
using (context.Connection.AcquireDistributedLock(..., TimeSpan.FromMinutes(1)))
{
// await calls here
}
}
Expected: since the lock is in a using block, it should be released when the using loses scope.
Actual: the lock is held indefinitely.
I believe the problem resides in the use of ThreadLocal in SQLiteDistributedLock.
When using async, there's no guarantee that you resume in the same thread you were before - even less in ASP.NET Core where there's no sync context.
I'll try to add a test case and a fix for this issue.
I am getting this error while looking at the dashboad for the jobs that have been completed.
In the code I am doing something simple:
BackgroundJob.Enqueue(() => System.Console.WriteLine($"Hello world"));
{
"error": {
"code": "ExceptionErrorCode",
"message": "The given key 'SucceededAt' was not present in the dictionary.",
"target": "System.Collections.Generic.KeyNotFoundException",
"details": [
{
"code": "ExceptionStacktraceErrorCode",
"message": " at System.Collections.Generic.Dictionary2.get_Item(TKey key)\r\n at Hangfire.Storage.SQLite.SQLiteMonitoringApi.<>c.<SucceededJobs>b__33_1(JobDetailedDto sqlJob, Job job, Dictionary
2 stateData)\r\n at Hangfire.Storage.SQLite.SQLiteMonitoringApi.DeserializeJobs[TDto](ICollection1 jobs, Func
4 selector)\r\n at Hangfire.Storage.SQLite.SQLiteMonitoringApi.<>c__DisplayClass33_0.b__0(HangfireDbContext connection)\r\n at Hangfire.Dashboard.Pages.SucceededJobs.Execute()\r\n at Hangfire.Dashboard.RazorPage.TransformText(String body)\r\n at Hangfire.Dashboard.RazorPageDispatcher.Dispatch(DashboardContext context)\r\n at Hangfire.Dashboard.AspNetCoreDashboardMiddleware.Invoke(HttpContext httpContext)\r\n at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\r\n at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)\r\n at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)\r\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIIndexMiddleware.Invoke(HttpContext httpContext)\r\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext)\r\n at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\r\n at Enstoa.Application.Core.AspNetCore.Errors.ErrorHandlerMiddleware.Invoke(HttpContext context, ILogger`1 logger, IHostingEnvironment env)",
"details": []
}
]
}
}
Would anyone or the author consider this project to be ready for production environments yet, or are there significant limitations that haven't been worked out?
HangFire sqlitedb file get bigger every day
I want any configuration way to delete records from all tables that created from one week ago
Due to the use of an old version of sqlite-net-pcl, this package does not work on Apple M1 machines. Updating sqlite-net-pcl package to 1.9.172 fixes the issue.
Error message (partly):
"Unable to load shared library 'e_sqlite3' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: \ndlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.17/e_sqlite3.dylib, 0x0001): tried: '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.17/e_sqlite3.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.17/e_sqlite3.dylib' (no such file), '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.17/e_sqlite3.dylib' (no such file)
Ever since upgrading to 0.4.0 with Hangfire at 1.8.7, we have been seeing more errors from the system and some intermittent AccessViolation exceptions on Windows systems stemming from Hangfire. To be clear, on Linux, we don't see the AccessViolation exceptions but we do see new errors that weren't there before. Using Hangfire 1.8.7 with SQLite 0.3.4 seems to have no obvious issues.
I am not specifying any special properties on the SQLiteStorageOptions
during setup, so all settings are based on the default in both 0.3.4 and 0.4.0 i.e. using WAL and pool size of 20.
From my perspective, I think it's possible that this could be related to sqlite-net 1003. In 0.4.0, there was a major change to use a pooled DB connection strategy. I think it's possible that, if someone were to not dispose after fetching the IStorageConnection
, they could unintentionally lead to a situation where the IStorageConnection
(i.e. HangfireSQLiteConnection
) is actually garbage collected instead of disposed. That may trigger similar behavior as in the issue seen in the library.
Since this is a library to be consumed by Hangfire itself, it may not always be possible to guarantee correctness on "their side of the fence." From my brief review, I couldn't find anything else that really struck me as a possibility for having the finalizer be called without disposing beforehand.
From my decidedly narrow perspective, I think there are two easy paths:
EnqueueOrPhaseOut
, this would prove beyond a doubt that something did not call Dispose when it should have on the IStorageConnection
.HangfireSQLiteConnection
. Instead, pass either a function/lambda to fetch a DbConnection in the implementations and have a proper using
in place each time. This eliminates the need for worrying about other people correctly disposing of things leading to such serious situations.Things I'm not as confident about:
sqlite-net-pcl
and using Microsoft.Data.Sqlite
be an alternate path? Maybe it doesn't solve correctness issues, but maybe it avoids having AccessViolation exceptions during collection.sqlite-net-pcl
when there's an alternate explanation?I am happy to produce a PR for one of paths, but I'd like to hear which path would be desired.
2024-01-14 01:24:39.914 +00:00 [ERR] Execution RecurringJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:09
SQLite.SQLiteException: database is locked
at SQLite.PreparedSqlLiteInsertCommand.ExecuteNonQuery(Object[] source)
at SQLite.SQLiteConnection.Insert(Object obj, String extra, Type objType)
at Hangfire.Storage.SQLite.SQLiteWriteOnlyTransaction.<>c__DisplayClass21_0.<SetJobState>b__0(HangfireDbContext _)
at Hangfire.Storage.SQLite.SQLiteWriteOnlyTransaction.Commit()
at Hangfire.Server.RecurringJobScheduler.ScheduleRecurringJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, RecurringJobEntity recurringJob, DateTime now)
at Hangfire.Server.RecurringJobScheduler.TryEnqueueBackgroundJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, DateTime now)
at Hangfire.Server.RecurringJobScheduler.<>c__DisplayClass18_0.<EnqueueNextRecurringJobs>b__0(IStorageConnection connection)
at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)
at Hangfire.Server.RecurringJobScheduler.EnqueueNextRecurringJobs(BackgroundProcessContext context)
at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context)
at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)
at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)
2024-01-14 01:24:43.899 +00:00 [ERR] Unable to update heartbeat on the resource 'HangFire:job:12790378:state-lock'. The resource is not locked or is locked by another owner.
2024-01-14 01:24:46.916 +00:00 [ERR] Unable to update heartbeat on the resource 'HangFire:job:12790386:state-lock'. SQLite.SQLiteException: database is locked
2024-01-15 00:01:33.303 +00:00 [ERR] 10 state change attempt(s) failed due to an exception, moving job to the FailedState
SQLite.SQLiteException: database disk image is malformed
at SQLite.PreparedSqlLiteInsertCommand.ExecuteNonQuery(Object[] source)
at SQLite.SQLiteConnection.Insert(Object obj, String extra, Type objType)
at Hangfire.Storage.SQLite.SQLiteWriteOnlyTransaction.<>c__DisplayClass21_0.<SetJobState>b__0(HangfireDbContext _)
at Hangfire.Storage.SQLite.SQLiteWriteOnlyTransaction.Commit()
at Hangfire.States.BackgroundJobStateChanger.ChangeState(StateChangeContext context)
at Hangfire.Server.Worker.TryChangeState(BackgroundProcessContext context, IStorageConnection connection, String jobId, IState state, IReadOnlyDictionary`2 customData, String[] expectedStates, IFetchedJob completeJob,
All of these stacks ended with an AccessViolation leading to a total application failure.
Stack:
at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_finalize(IntPtr)
at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_finalize(IntPtr)
at SQLite.PreparedSqlLiteInsertCommand.Finalize()
Stack:
at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_finalize(IntPtr)
at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_finalize(IntPtr)
at SQLite.SQLite3.Finalize(SQLitePCL.sqlite3_stmt)
at SQLite.PreparedSqlLiteInsertCommand.Dispose(Boolean)
at SQLite.PreparedSqlLiteInsertCommand.Finalize()
Stack:
at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_open_v2(Byte*, IntPtr ByRef, Int32, Byte*)
at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_open_v2(Byte*, IntPtr ByRef, Int32, Byte*)
at SQLitePCL.SQLite3Provider_e_sqlite3.SQLitePCL.ISQLite3Provider.sqlite3_open_v2(SQLitePCL.utf8z, IntPtr ByRef, Int32, SQLitePCL.utf8z)
at SQLitePCL.raw.sqlite3_open_v2(SQLitePCL.utf8z, SQLitePCL.sqlite3 ByRef, Int32, SQLitePCL.utf8z)
at SQLitePCL.raw.sqlite3_open_v2(System.String, SQLitePCL.sqlite3 ByRef, Int32, System.String)
at SQLite.SQLite3.Open(System.String, SQLitePCL.sqlite3 ByRef, Int32, System.String)
at SQLite.SQLiteConnection..ctor(SQLite.SQLiteConnectionString)
at SQLite.SQLiteConnection..ctor(System.String, SQLite.SQLiteOpenFlags, Boolean)
at Hangfire.Storage.SQLite.SQLiteStorage+<>c__DisplayClass8_0.<.ctor>b__0()
at Hangfire.Storage.SQLite.SQLiteStorage.CreateAndOpenConnection()
at Hangfire.Server.ServerJobCancellationWatcher.Execute(Hangfire.Server.BackgroundProcessContext)
at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object)
at Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object)
at Hangfire.Processing.BackgroundDispatcher.DispatchLoop()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
NotSupportedException: Current storage doesn't support specifying queues directly for a specific job
Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details.
---> System.NotSupportedException: Current storage doesn't support specifying queues directly for a specific job. Please use the QueueAttribute instead.
at Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context)
at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass12_0.<CreateWithFilters>b__0()
at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass12_1.<CreateWithFilters>b__2()
at Hangfire.Client.BackgroundJobFactory.CreateWithFilters(CreateContext context, IEnumerable`1 filters)
at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
at Hangfire.BackgroundJobClient.Create(Job job, IState state, IDictionary`2 parameters)
--- End of inner exception stack trace ---
at Hangfire.BackgroundJobClient.Create(Job job, IState state, IDictionary`2 parameters)
at Hangfire.BackgroundJobClient.Create(Job job, IState state)
at Hangfire.BackgroundJobClientExtensions.Create(IBackgroundJobClient client, String queue, Expression`1 methodCall, IState state)
at Hangfire.BackgroundJobClientExtensions.Enqueue(IBackgroundJobClient client, String queue, Expression`1 methodCall)
at HangfirePlugin.Controllers.TestController.Test1()
at lambda_method15(Closure , Object )
[ApiController]
public class TestController : ControllerBase
{
private readonly IBackgroundJobClient _backgroundJobClient;
public TestController(IBackgroundJobClient backgroundJobClient)
{
_backgroundJobClient = backgroundJobClient;
}
[HttpGet]
public async Task<ActionResult> Test1()
{
{
_backgroundJobClient.Enqueue(queue: "test_queue_1", methodCall: () => Test1_NonAction(5));
_backgroundJobClient.Enqueue(queue: "test_queue_1", methodCall: () => Test1_NonAction(1));
_backgroundJobClient.Enqueue(queue: "test_queue_1", methodCall: () => Test1_NonAction(3));
}
return Content("Ok");
}
[NonAction]
public async Task Test1_NonAction(int num)
{
Thread.Sleep(1000 * num);
Console.WriteLine("--------------------------------------------------------------------------------");
Console.WriteLine($"{num} from Hangfire! - {DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss")}");
Console.WriteLine("--------------------------------------------------------------------------------");
}
}
<PackageReference Include="Hangfire.Core" Version="1.8.9" />
<PackageReference Include="Hangfire.AspNetCore" Version="1.8.9" />
<!-- https://github.com/raisedapp/Hangfire.Storage.SQLite -->
<PackageReference Include="Hangfire.Storage.SQLite" Version="0.4.0" />
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.