raisedapp / hangfire.storage.sqlite Goto Github PK

Hi and thanks for a great library! :-)

This is a feature request to implement SlidingInvisibilityTimeout.

Every now and then we run into issues with the InvisibilityTimeout for "long-running" jobs, which gets cancelled when the timeout is reached and then requeued. If the job does not implement the CancellationToken properly, it keeps running in the background while the new job is enqueued and duplicate processing occurs - there are some remedies for that as well, but none without some caveats.
Increasing the timeout also has its caveats as it might result in jobs not being executed for many hours in case of hard-failure on server/process.

The PostgreSQL provider has the same issue, which also explains the issue better/in more detail, and how the SQLServer provider has solved it:

• hangfire-postgres/Hangfire.PostgreSql#209

And there is a pull-request for implementing SlidingInvisibilityTimeout for inspiration if someone wants to implement it here:

• hangfire-postgres/Hangfire.PostgreSql#353

Issue
The internal functions GetTimelineStats and GetHourlyTimelineStats in SQLiteMonitoringApi.cs return a dictionary of DateTime to counts, but they are not matched correctly. This affects the following public API functions:

• SucceededByDatesCount()
• FailedByDatesCount()
• HourlySucceededJobs()
• HourlyFailedJobs()

This in turn causes the graphs in the Hangfire dashboard to be incorrect.

Cause
The statement on lines 195 and 228 is as follows:

var value = valuesAggregatorMap[valuesAggregatorMap.Keys.ElementAt(i)];

This is incorrect, as it disregards the keys of valuesAggregatorMap and instead just indexes into the Keys collection, which is inherently unordered and so the result is not well defined. This means that the wrong counts are assigned to the wrong DateTime keys in the final result.

Fix
By changing line 195 to this:

var value = valuesAggregatorMap[$"stats:{type}:{stringDates[i]}"];

and line 228 to this:

var value = valuesAggregatorMap[$"stats:{type}:{dates[i]:yyyy-MM-dd-HH}"];

the issue is fixed, as the map is correctly looked-up using the counter key.

Hello,

Global settings are set by this plugin here

There are two problems:

1. Overwriting default or user-provided settings inside a plugin is not very nice, and surprising to say the least.
   Since both .UseSQLiteStorage() and .UseRecommendedSerializerSettings() set this global option, the end value of the serializer settings will be different depending on the order the two methods are called.
2. The settings used by this plugin are wrong: the DateFormatString doesn't contain the timezone format (supposed to be K at the end. See https://www.newtonsoft.com/json/help/html/P_Newtonsoft_Json_JsonSerializer_DateFormatString.htm)
   This causes issues when using UTC DateTime and DateTimeOffset.

In my case, I'm using a job with a DateTimeOffset argument.
I enqueue it with an offset of 0 (UTC).
When the job is called by Hangfire, the hours and minutes values didn't change, but the offset has been set to my local offset (+1)!

I spent a few hours on this to finally find that .UseRecommendedSerializerSettings() after .UseSQLiteStorage() instead of before fixes my issue.

I'm using SQLite as the database for my app. So I installed Microsoft.EntityFrameworkCore.Sqlite version 3.1.7 nuget & ran the project (Blazor Server App) which caused following exception:

System.MissingMethodException: Method not found: 'System.String SQLitePCL.raw.sqlite3_column_name(SQLitePCL.sqlite3_stmt, Int32)'.

Without Microsoft.EntityFrameworkCore.Sqlite, Hangfire works perfectly.

Category: Hangfire.Storage.SQLite.SQLiteDistributedLock
EventId: 0

Unable to update heartbeat on the resource 'HangFire:job:1201:state-lock'. The resource is not locked or is locked by another owner.

Category: Hangfire.Storage.SQLite.SQLiteDistributedLock
EventId: 0

Unable to update heartbeat on the resource 'HangFire:job:1199:state-lock'. The resource is not locked or is locked by another owner.

Category: Hangfire.Storage.SQLite.SQLiteDistributedLock
EventId: 0

Unable to update heartbeat on the resource 'HangFire:job:1199:state-lock'. The resource is not locked or is locked by another owner.

How fix this error?

Result for .net 8 results in a warning

warning NETSDK1206: Found version-specific or distribution-specific runtime identifier(s): alpine-x64.
Affected libraries: SQLitePCLRaw.lib.e_sqlite3.

In .NET 8.0 and higher, assets for version-specific and distribution-specific runtime identifiers will not be found by default. See https://aka.ms/dotnet/rid-usage for details.

System.TypeInitializationException
HResult=0x80131534
Mensaje = The type initializer for 'SQLite.SQLiteConnection' threw an exception.
Origen = SQLite-net
Seguimiento de la pila:
en SQLite.SQLiteConnection..ctor(String databasePath, SQLiteOpenFlags openFlags, Boolean storeDateTimeAsTicks)
en Hangfire.Storage.SQLite.HangfireDbContext..ctor(String databasePath, String prefix)
en Hangfire.Storage.SQLite.HangfireDbContext.Instance(String databasePath, String prefix)
en Hangfire.Storage.SQLite.SQLiteStorage..ctor(String databasePath, SQLiteStorageOptions storageOptions)
en Hangfire.Storage.SQLite.SQLiteStorageExtensions.UseSQLiteStorage(IGlobalConfiguration configuration)
en Program.<>c.<

Esta excepción se generó originalmente en esta pila de llamadas:
[Código externo]

Excepción interna 1:
TypeLoadException: Method 'sqlite3_soft_heap_limit64' in type 'SQLitePCL.SQLite3Provider_dynamic_cdecl' from assembly 'SQLitePCLRaw.provider.dynamic_cdecl, Version=2.0.3.851, Culture=neutral, PublicKeyToken=b68184102cba0b3b' does not have an implementation.

If a job is created while another thread is inserting a row elsewhere into the DB, it is sometimes assigned the wrong ID due to this issue in the underlying SQLite library.

Because of this, jobs aren't properly initialised, so they are left with a StateName of NULL, which means they silently fail, and their JobParameters end up being linked to the wrong jobs. If there are lots of jobs being created and run, this ends up happening quite often.

An example of the incorrect IDs being assigned in SQLiteConnectionFacts:

[Fact, CleanDatabase]
public void CreateExpiredJob_CorrectlyPopulatesId_WhenManyThreadsAreInserting()
{
    const int InsertCount = 2000;
    var job = Job.FromExpression(() => SampleMethod("Hello"));
    var parameters = new Dictionary<string, string>();
    var createdAt = new DateTime(2012, 12, 12, 0, 0, 0, 0, DateTimeKind.Utc);
    var expireIn = TimeSpan.FromDays(1);
    
    UseConnection((database, connection) =>
    {
        var lastJobId = int.Parse(connection.CreateExpiredJob(job, parameters, createdAt, expireIn));
        
        var finished = false;
        var ok = true;
        var expectedId = 0;
        string jobId = null;
        
        Parallel.Invoke(
            // Create jobs and break if they are unexpected
            () =>
            {
                for (var insertIndex = 1; insertIndex <= InsertCount; insertIndex++)
                {
                    expectedId = lastJobId + insertIndex;

                    // Inserts the job with the expected ID, but returns the incorrect ID
                    jobId = connection.CreateExpiredJob(job, parameters, createdAt, expireIn);

                    ok = expectedId.ToString() == jobId;
                    if (!ok)
                    {
                        break;
                    }
                }

                finished = true;
            },
            // Another thread inserting rows to the db, e.g. other jobs completing
            () =>
            {
                while (ok && !finished)
                {
                    // Arbitrary database insert
                    database.Database.Insert(new Set());
                }
            }
        );

        if (!ok)
        {
            Assert.Equal(expectedId.ToString(), jobId);
        }
    });
}

Note that commenting out the inserting in the second thread fixes the issue

I noticed that the Recurring Jobs would execute immediately when they were first added to the queue.
Can we solve this problem? Thank you very much.

Add Unit Testing to the Project

Hello,

I'm trying to create several jobs, but after some jobs created, my application thrown this error.

Can help me?

Thanks,
mstiago

Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details.
---> System.ArgumentException: An item with the same key has already been added. Key: CurrentCulture
 at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
 at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
 at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
 at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
 at Hangfire.Storage.SQLite.SQLiteMonitoringApi.<>c__DisplayClass25_0.<JobDetails>b__0(HangfireDbContext _)
 at Hangfire.Console.States.ConsoleApplyStateFilter.OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
 at Hangfire.States.StateMachine.InvokeOnStateApplied(Tuple`2 x)
 at Hangfire.Profiling.ProfilerExtensions.InvokeAction[TInstance](InstanceAction`1 tuple)
 at Hangfire.Profiling.ProfilerExtensions.InvokeMeasured[TInstance](IProfiler profiler, TInstance instance, Action`1 action, String message)
 at Hangfire.States.StateMachine.ApplyState(ApplyStateContext initialContext)
 at Hangfire.Client.CoreBackgroundJobFactory.<>c__DisplayClass14_0.<Create>b__3(Int32 attempt)
 at Hangfire.Client.CoreBackgroundJobFactory.<>c__DisplayClass15_0.<RetryOnException>b__0(Int32 attempt)
 at Hangfire.Client.CoreBackgroundJobFactory.RetryOnException[T](Int32& attemptsLeft, Func`2 action)
--- End of stack trace from previous location ---
 at Hangfire.Client.CoreBackgroundJobFactory.RetryOnException[T](Int32& attemptsLeft, Func`2 action)
 at Hangfire.Client.CoreBackgroundJobFactory.RetryOnException(Int32& attemptsLeft, Action`1 action)
 at Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context)
 at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass12_0.<CreateWithFilters>b__0()
 at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
 at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
 at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
 at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
 at Hangfire.BackgroundJobClient.Create(Job job, IState state)
 --- End of inner exception stack trace ---
 at Hangfire.BackgroundJobClient.Create(Job job, IState state)
 at Infrastructure.BackgroundJobs.HangfireService.Enqueue(Expression`1 methodCall) in C:\Users\6110183\source\repos\ThomsonReuters.HighQIntegration\Infrastructure\BackgroundJobs\HangfireService.cs:line 16
 at Infrastructure.Strategies.Implementations.TimesheetStrategy.RunAsync() in C:\Users\6110183\source\repos\ThomsonReuters.HighQIntegration\Infrastructure\Strategies\Implementations\TimesheetStrategy.cs:line 95

Are you planning to make your project compatible with Hangfire.SQLite? I.e. will the db structure be compatible? In either case, please consider being explicit about the compatibility in the README.

Hi there

is there any permissions needed in order to create the sqlilte db instance?

GlobalConfiguration.Configuration.UseSQLiteStorage();
Thanks
Alex

My ASP.NET Core 6 app shows this error very often:

Unable to update heartbeat on the resource 'HangFire:xxx'. The resource is not locked or is locked by another owner.

I believe this has to do with #68, and may be gone if that error is fixed.

But regardless of #68, if SQLiteDistributedLock cannot update the heartbeat because of that message, what's the point of keep retrying? I think the timer should be stopped in that case - or at the minimum, mute the error log so that it doesn't show up indefinitely in the logs.

Extension method UseSQLiteStorage has a argument named nameOrConnectionString but the sqlite-net library's SQLiteConnection takes databasePath as argument.

Doesn't work

• UseSQLiteStorage($"Data Source={pathToDbFile};Version=3;")
• UseSQLiteStorage("Hangfire")

Works

• UseSQLiteStorage(pathToDbFile)

I add another SQLite provider, like EF Core SQLite to project will throw exception from run project .UseSQLiteStorage().

Exception thrown: 'System.MissingMethodException' in SQLite-net.dll
An exception of type 'System.MissingMethodException' occurred in SQLite-net.dll but was not handled in user code
Method not found: 'System.String SQLitePCL.raw.sqlite3_column_name(SQLitePCL.sqlite3_stmt, Int32)'.

In ASP.NET Core 3.1 API PoC project,

.csproj

<ItemGroup>
  <PackageReference Include="Hangfire.Core" Version="1.7.*" />
  <PackageReference Include="Hangfire.AspNetCore" Version="1.7.*" />
  <PackageReference Include="Hangfire.Storage.SQLite" Version="0.2.4" />
  <PackageReference Include="Microsoft.EntityFrameworkCore.Sqlite" Version="3.1.2" />
</ItemGroup>

startup.cs

public void ConfigureServices(IServiceCollection services)
{
    services.AddHangfire(configuration => configuration
        .UseSimpleAssemblyNameTypeSerializer()
        .UseRecommendedSerializerSettings()
        .UseSQLiteStorage());
    services.AddHangfireServer();
    services.AddControllers();
}

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }

    app.UseRouting();
    app.UseAuthorization();

    app.UseHangfireDashboard();
    BackgroundJob.Enqueue(() => Console.WriteLine("Hello backgroundjobs."));
    RecurringJob.AddOrUpdate("Hello", () => Console.WriteLine("Hello, recurringJob."), Cron.Minutely);

   app.UseEndpoints(endpoints =>
    {
        endpoints.MapControllerRoute(
            name: "default",
            pattern: "{controller=Home}/{action=Index}/{id?}");
        endpoints.MapControllers();
    });
}

Consider the following test:

[Fact, CleanDatabase]
public void OnlySingleLockCanBeAcquired()
{
    var connection = ConnectionUtils.CreateConnection();
    var numThreads = 10;
    long concurrencyCounter = 0;
    var manualResetEvent = new ManualResetEventSlim();
    var success = new bool[numThreads];

    // Spawn multiple threads to race each other.
    var threads = Enumerable.Range(0, numThreads).Select(i => new Thread(() =>
    {
        // Wait for the start signal.
        manualResetEvent.Wait();

        // Attempt to acquire the distributed lock.
        using (new SQLiteDistributedLock("resource1", TimeSpan.FromSeconds(5), connection, new SQLiteStorageOptions()))
        {
            // Find out if any other threads managed to acquire the lock.
            var oldConcurrencyCounter = Interlocked.CompareExchange(ref concurrencyCounter, 1, 0);

            // The old concurrency counter should be 0 as only one thread should be allowed to acquire the lock.
            success[i] = oldConcurrencyCounter == 0;

            Interlocked.MemoryBarrier();

            // Hold the lock for some time.
            Thread.Sleep(100);

            Interlocked.Decrement(ref concurrencyCounter);
        }
    })).ToList();

    threads.ForEach(t => t.Start());

    manualResetEvent.Set();

    threads.ForEach(t => Assert.True(t.Join(TimeSpan.FromMinutes(1)), "Thread is hanging unexpected"));

    // All the threads should report success.
    Interlocked.MemoryBarrier();
    Assert.DoesNotContain(false, success);
}

This test races threads trying to acquire a SQLiteDistributedLock and confirms whether the thread was the only one executing at between acquire and release.

The SQLiteDistributedLock::Acquire() method is unsafe. Consider the following execution through the function:

1. Thread 1 tries to fetch the existing lock and result is null (line 126)
2. Thread 2 tries to fetch the existing lock and result is null (line 126)
3. Thread 1 tries to update the new lock and rowsAffected is 0 (line 135)
4. Thread 2 tries to update the new lock and rowsAffected is 0 (line 135)
5. Thread 1 performs the insert (line 137)
6. Thread 2 performs the insert (line 137)
7. Thread 1 thinks it has the lock
8. Thread 2 thinks it has the lock

Now there are two different locks in the DB for the same resource because there is no uniqueness constraint over the Resource column.

Suggested fixes:

• Perform the entire DB acquire in a single transaction (as the PostgreSQL extension does) instead of 3 separate requests
• Add a uniqueness constraint over the Resource column which would prevent the second insert from succeeding

Aside: I don't know the transactional capabilities nor the uniqueness constraints of LiteDB, but suspect the code this was taken from is similarly flawed.

I discovered this as the DisableConcurrentExecution was failing to prevent concurrent execution of multiple instances of the same job.

Hello,

I noticed that there is a prefix parameter in the HangfireDbContext, but it seems that it is not being utilized, resulting in the inability to customize the table prefix name. I attempted to set a custom prefix, but it doesn't reflect in the generated table names.

Is there a specific way to use the prefix parameter, or is there any other workaround to achieve custom table prefix functionality in HangfireDbContext?

Thank you for your assistance.

• HangfireDbContext

        /// <summary>
        /// Starts SQLite database using a connection string for file system database
        /// </summary>
        /// <param name="databasePath">the database path</param>
        /// <param name="prefix">Table prefix</param>
        private HangfireDbContext(string databasePath, string prefix = "hangfire")
        {
            //UTC - Internal JSON
            GlobalConfiguration.Configuration
                .UseSerializerSettings(new JsonSerializerSettings()
                {
                    DateTimeZoneHandling = DateTimeZoneHandling.Utc,
                    DateFormatHandling = DateFormatHandling.IsoDateFormat,
                    DateFormatString = "yyyy-MM-dd HH:mm:ss.fff"
                });
        
            Database = new SQLiteConnection(databasePath, SQLiteOpenFlags.ReadWrite |
                SQLiteOpenFlags.Create | SQLiteOpenFlags.FullMutex, storeDateTimeAsTicks: true);
        
            ConnectionId = Guid.NewGuid().ToString();
        }

        /// <summary>
        /// Initializes initial tables schema for Hangfire
        /// </summary>
        public void Init(SQLiteStorageOptions storageOptions)
        {
            StorageOptions = storageOptions;

            AutoClean(storageOptions);

            Database.CreateTable<AggregatedCounter>();
            Database.CreateTable<Counter>();
            Database.CreateTable<HangfireJob>();
            Database.CreateTable<HangfireList>();
            Database.CreateTable<Hash>();
            Database.CreateTable<JobParameter>();
            Database.CreateTable<JobQueue>();
            Database.CreateTable<HangfireServer>();
            Database.CreateTable<Set>();
            Database.CreateTable<State>();
            Database.CreateTable<DistributedLock>();

            AggregatedCounterRepository = Database.Table<AggregatedCounter>();
            CounterRepository = Database.Table<Counter>();
            HangfireJobRepository = Database.Table<HangfireJob>();
            HangfireListRepository = Database.Table<HangfireList>();
            HashRepository = Database.Table<Hash>();
            JobParameterRepository = Database.Table<JobParameter>();
            JobQueueRepository = Database.Table<JobQueue>();
            HangfireServerRepository = Database.Table<HangfireServer>();
            SetRepository = Database.Table<Set>();
            StateRepository = Database.Table<State>();
            DistributedLockRepository = Database.Table<DistributedLock>();
        }

Implementing the following features doesn't take much effort

• SQLiteWriteOnlyTransaction.AcquireDistributedLock (Transaction.AcquireDistributedLock)
• HangfireSQLiteConnection.GetSetCount-Limited (Connection.GetSetCount.Limited)
• HangfireSQLiteConnection.GetSetContains (Connection.GetSetContains)
• HangfireSQLiteConnection.GetFirstByLowestScoreFromSet (Connection.BatchedGetFirstByLowestScoreFromSet)
• HangfireSQLiteConnection.GetUtcDateTime (Connection.GetUtcDateTime)

I already got those implemented and will provide a pull request

Hi Felix,

I noticed that the issue still exist for the functions "ScheduledJobs", "ProcessingJobs" and "FailedJobs" on the "SQLiteMonitoringApi"class, It seems that you missed adding validations on "ExceptionDetails", "ExceptionMessage" and "ExceptionType" in the "FailedJobs" function, "StartedAt" in the "ProcessingJobs" function and "ScheduledAt" in the "ScheduledJobs" function.

Can you please include this validations?

I'm sorry I don't know how to reliably reproduce this exception. All I can tell you is that MyJobClass.MyJobMethod has this attribute: [DisableConcurrentExecution(timeoutInSeconds: 10 * 60)] and that I'm using a TransactionScope to create two dependent jobs.

Hangfire.AutomaticRetryAttribute|Failed to process the job '123': an exception occurred. Retry attempt 1 of 10 will be performed in 00:00:39.
Hangfire.Storage.DistributedLockTimeoutException: Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'Could not place a lock on the resource 'HangFire:MyJobClass.MyJobMethod': The lock request timed out.' resource.
   at void Hangfire.Storage.SQLite.SQLiteDistributedLock.Acquire(TimeSpan timeout)
   at new Hangfire.Storage.SQLite.SQLiteDistributedLock(string resource, TimeSpan timeout, HangfireDbContext database, SQLiteStorageOptions storageOptions)
   at IDisposable Hangfire.Storage.SQLite.HangfireSQLiteConnection.AcquireDistributedLock(string resource, TimeSpan timeout)
   at void Hangfire.DisableConcurrentExecutionAttribute.OnPerforming(PerformingContext filterContext)
   at bool Hangfire.Profiling.ProfilerExtensions.InvokeAction<TInstance>(InstanceAction<TInstance> tuple)
   at TResult Hangfire.Profiling.SlowLogProfiler.InvokeMeasured<TInstance, TResult>(TInstance instance, Func<TInstance, TResult> action, string message)
   at void Hangfire.Profiling.ProfilerExtensions.InvokeMeasured<TInstance>(IProfiler profiler, TInstance instance, Action<TInstance> action, string message)
   at PerformedContext Hangfire.Server.BackgroundJobPerformer.InvokePerformFilter(IServerFilter filter, PerformingContext preContext, Func<PerformedContext> continuation)

Anyway I switched to HangFire.SqlServer and this problem did not occur anymore.

Still randomly getting distributed lock exception.
Hangfire.Storage.SQLite version: 0.3.0
Hangfire.Core version: 1.7.22

Timeout expired. The timeout elapsed prior to obtaining a distributed lock on the 'Could not place a lock on the resource 'HangFire:extension:job-mutex:lock:StatusUpdateJob': The lock request timed out.' resource. at Hangfire.Storage.SQLite.SQLiteDistributedLock.Acquire(TimeSpan timeout) at Hangfire.Storage.SQLite.SQLiteDistributedLock..ctor(String resource, TimeSpan timeout, HangfireDbContext database, SQLiteStorageOptions storageOptions) at Hangfire.Storage.SQLite.HangfireSQLiteConnection.AcquireDistributedLock(String resource, TimeSpan timeout)

The following snippet shows a (valid?) usage of SQLiteDistributedLock which leaves/can leave the lock alive forever:

public async Task Method(PerformContext context)
{
    using (context.Connection.AcquireDistributedLock(..., TimeSpan.FromMinutes(1)))
    {
        // await calls here
    }
}

Expected: since the lock is in a using block, it should be released when the using loses scope.

Actual: the lock is held indefinitely.

I believe the problem resides in the use of ThreadLocal in SQLiteDistributedLock.

When using async, there's no guarantee that you resume in the same thread you were before - even less in ASP.NET Core where there's no sync context.

I'll try to add a test case and a fix for this issue.

I am getting this error while looking at the dashboad for the jobs that have been completed.

In the code I am doing something simple:

BackgroundJob.Enqueue(() => System.Console.WriteLine($"Hello world"));

{
"error": {
"code": "ExceptionErrorCode",
"message": "The given key 'SucceededAt' was not present in the dictionary.",
"target": "System.Collections.Generic.KeyNotFoundException",
"details": [
{
"code": "ExceptionStacktraceErrorCode",
"message": " at System.Collections.Generic.Dictionary2.get_Item(TKey key)\r\n at Hangfire.Storage.SQLite.SQLiteMonitoringApi.<>c.<SucceededJobs>b__33_1(JobDetailedDto sqlJob, Job job, Dictionary2 stateData)\r\n at Hangfire.Storage.SQLite.SQLiteMonitoringApi.DeserializeJobs[TDto](ICollection1 jobs, Func4 selector)\r\n at Hangfire.Storage.SQLite.SQLiteMonitoringApi.<>c__DisplayClass33_0.b__0(HangfireDbContext connection)\r\n at Hangfire.Dashboard.Pages.SucceededJobs.Execute()\r\n at Hangfire.Dashboard.RazorPage.TransformText(String body)\r\n at Hangfire.Dashboard.RazorPageDispatcher.Dispatch(DashboardContext context)\r\n at Hangfire.Dashboard.AspNetCoreDashboardMiddleware.Invoke(HttpContext httpContext)\r\n at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\r\n at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)\r\n at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)\r\n at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIIndexMiddleware.Invoke(HttpContext httpContext)\r\n at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext)\r\n at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)\r\n at Enstoa.Application.Core.AspNetCore.Errors.ErrorHandlerMiddleware.Invoke(HttpContext context, ILogger`1 logger, IHostingEnvironment env)",
"details": []
}
]
}
}

Would anyone or the author consider this project to be ready for production environments yet, or are there significant limitations that haven't been worked out?

HangFire sqlitedb file get bigger every day
I want any configuration way to delete records from all tables that created from one week ago

Due to the use of an old version of sqlite-net-pcl, this package does not work on Apple M1 machines. Updating sqlite-net-pcl package to 1.9.172 fixes the issue.

Error message (partly):

"Unable to load shared library 'e_sqlite3' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: \ndlopen(/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.17/e_sqlite3.dylib, 0x0001): tried: '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.17/e_sqlite3.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.17/e_sqlite3.dylib' (no such file), '/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.17/e_sqlite3.dylib' (no such file)

Overview

Ever since upgrading to 0.4.0 with Hangfire at 1.8.7, we have been seeing more errors from the system and some intermittent AccessViolation exceptions on Windows systems stemming from Hangfire. To be clear, on Linux, we don't see the AccessViolation exceptions but we do see new errors that weren't there before. Using Hangfire 1.8.7 with SQLite 0.3.4 seems to have no obvious issues.

Some additional information

I am not specifying any special properties on the SQLiteStorageOptions during setup, so all settings are based on the default in both 0.3.4 and 0.4.0 i.e. using WAL and pool size of 20.

Thinking about the problem

From my perspective, I think it's possible that this could be related to sqlite-net 1003. In 0.4.0, there was a major change to use a pooled DB connection strategy. I think it's possible that, if someone were to not dispose after fetching the IStorageConnection, they could unintentionally lead to a situation where the IStorageConnection (i.e. HangfireSQLiteConnection) is actually garbage collected instead of disposed. That may trigger similar behavior as in the issue seen in the library.

Since this is a library to be consumed by Hangfire itself, it may not always be possible to guarantee correctness on "their side of the fence." From my brief review, I couldn't find anything else that really struck me as a possibility for having the finalizer be called without disposing beforehand.

Thoughts on solutions

From my decidedly narrow perspective, I think there are two easy paths:

1. Do not wait to attain a reference to a created DbConnection. If we grab a reference to it immediately, you can prevent the GC collecting it without a Dispose being called if that reference never came back into the queue via EnqueueOrPhaseOut, this would prove beyond a doubt that something did not call Dispose when it should have on the IStorageConnection.
2. Do not pass the DbConnection straight into the constructor of HangfireSQLiteConnection. Instead, pass either a function/lambda to fetch a DbConnection in the implementations and have a proper using in place each time. This eliminates the need for worrying about other people correctly disposing of things leading to such serious situations.

Things I'm not as confident about:

• Is it possible that these issues are related more to using WAL vs. journal? I think this is unlikely given how well tested SQLite is, but I am not an expert.
• Would avoiding sqlite-net-pcl and using Microsoft.Data.Sqlite be an alternate path? Maybe it doesn't solve correctness issues, but maybe it avoids having AccessViolation exceptions during collection.
• Am I totally off base in thinking it's related to the issue in sqlite-net-pcl when there's an alternate explanation?

I am happy to produce a PR for one of paths, but I'd like to hear which path would be desired.

Errors

Log messages on Linux that weren't there in 0.3.4

2024-01-14 01:24:39.914 +00:00 [ERR] Execution RecurringJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:09
SQLite.SQLiteException: database is locked
   at SQLite.PreparedSqlLiteInsertCommand.ExecuteNonQuery(Object[] source)
   at SQLite.SQLiteConnection.Insert(Object obj, String extra, Type objType)
   at Hangfire.Storage.SQLite.SQLiteWriteOnlyTransaction.<>c__DisplayClass21_0.<SetJobState>b__0(HangfireDbContext _)
   at Hangfire.Storage.SQLite.SQLiteWriteOnlyTransaction.Commit()
   at Hangfire.Server.RecurringJobScheduler.ScheduleRecurringJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, RecurringJobEntity recurringJob, DateTime now)
   at Hangfire.Server.RecurringJobScheduler.TryEnqueueBackgroundJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, DateTime now)
   at Hangfire.Server.RecurringJobScheduler.<>c__DisplayClass18_0.<EnqueueNextRecurringJobs>b__0(IStorageConnection connection)
   at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)
   at Hangfire.Server.RecurringJobScheduler.EnqueueNextRecurringJobs(BackgroundProcessContext context)
   at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context)
   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)
   at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)
2024-01-14 01:24:43.899 +00:00 [ERR] Unable to update heartbeat on the resource 'HangFire:job:12790378:state-lock'. The resource is not locked or is locked by another owner.
2024-01-14 01:24:46.916 +00:00 [ERR] Unable to update heartbeat on the resource 'HangFire:job:12790386:state-lock'. SQLite.SQLiteException: database is locked

2024-01-15 00:01:33.303 +00:00 [ERR] 10 state change attempt(s) failed due to an exception, moving job to the FailedState
SQLite.SQLiteException: database disk image is malformed
   at SQLite.PreparedSqlLiteInsertCommand.ExecuteNonQuery(Object[] source)
   at SQLite.SQLiteConnection.Insert(Object obj, String extra, Type objType)
   at Hangfire.Storage.SQLite.SQLiteWriteOnlyTransaction.<>c__DisplayClass21_0.<SetJobState>b__0(HangfireDbContext _)
   at Hangfire.Storage.SQLite.SQLiteWriteOnlyTransaction.Commit()
   at Hangfire.States.BackgroundJobStateChanger.ChangeState(StateChangeContext context)
   at Hangfire.Server.Worker.TryChangeState(BackgroundProcessContext context, IStorageConnection connection, String jobId, IState state, IReadOnlyDictionary`2 customData, String[] expectedStates, IFetchedJob completeJob,

Exceptions running on Windows with .NET 8.0.1:

All of these stacks ended with an AccessViolation leading to a total application failure.

Stack:
   at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_finalize(IntPtr)
   at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_finalize(IntPtr)
   at SQLite.PreparedSqlLiteInsertCommand.Finalize()

Stack:
   at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_finalize(IntPtr)
   at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_finalize(IntPtr)
   at SQLite.SQLite3.Finalize(SQLitePCL.sqlite3_stmt)
   at SQLite.PreparedSqlLiteInsertCommand.Dispose(Boolean)
   at SQLite.PreparedSqlLiteInsertCommand.Finalize()

Stack:
   at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_open_v2(Byte*, IntPtr ByRef, Int32, Byte*)
   at SQLitePCL.SQLite3Provider_e_sqlite3+NativeMethods.sqlite3_open_v2(Byte*, IntPtr ByRef, Int32, Byte*)
   at SQLitePCL.SQLite3Provider_e_sqlite3.SQLitePCL.ISQLite3Provider.sqlite3_open_v2(SQLitePCL.utf8z, IntPtr ByRef, Int32, SQLitePCL.utf8z)
   at SQLitePCL.raw.sqlite3_open_v2(SQLitePCL.utf8z, SQLitePCL.sqlite3 ByRef, Int32, SQLitePCL.utf8z)
   at SQLitePCL.raw.sqlite3_open_v2(System.String, SQLitePCL.sqlite3 ByRef, Int32, System.String)
   at SQLite.SQLite3.Open(System.String, SQLitePCL.sqlite3 ByRef, Int32, System.String)
   at SQLite.SQLiteConnection..ctor(SQLite.SQLiteConnectionString)
   at SQLite.SQLiteConnection..ctor(System.String, SQLite.SQLiteOpenFlags, Boolean)
   at Hangfire.Storage.SQLite.SQLiteStorage+<>c__DisplayClass8_0.<.ctor>b__0()
   at Hangfire.Storage.SQLite.SQLiteStorage.CreateAndOpenConnection()
   at Hangfire.Server.ServerJobCancellationWatcher.Execute(Hangfire.Server.BackgroundProcessContext)
   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(System.Guid, System.Object)
   at Hangfire.Processing.BackgroundExecution.Run(System.Action`2<System.Guid,System.Object>, System.Object)
   at Hangfire.Processing.BackgroundDispatcher.DispatchLoop()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

NotSupportedException: Current storage doesn't support specifying queues directly for a specific job

Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details.
 ---> System.NotSupportedException: Current storage doesn't support specifying queues directly for a specific job. Please use the QueueAttribute instead.
   at Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context)
   at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass12_0.<CreateWithFilters>b__0()
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
   at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass12_1.<CreateWithFilters>b__2()
   at Hangfire.Client.BackgroundJobFactory.CreateWithFilters(CreateContext context, IEnumerable`1 filters)
   at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
   at Hangfire.BackgroundJobClient.Create(Job job, IState state, IDictionary`2 parameters)
   --- End of inner exception stack trace ---
   at Hangfire.BackgroundJobClient.Create(Job job, IState state, IDictionary`2 parameters)
   at Hangfire.BackgroundJobClient.Create(Job job, IState state)
   at Hangfire.BackgroundJobClientExtensions.Create(IBackgroundJobClient client, String queue, Expression`1 methodCall, IState state)
   at Hangfire.BackgroundJobClientExtensions.Enqueue(IBackgroundJobClient client, String queue, Expression`1 methodCall)
   at HangfirePlugin.Controllers.TestController.Test1()
   at lambda_method15(Closure , Object )

[ApiController]
public class TestController : ControllerBase
{
    private readonly IBackgroundJobClient _backgroundJobClient;
    
    public TestController(IBackgroundJobClient backgroundJobClient)
    {
        _backgroundJobClient = backgroundJobClient;
    }
    
    [HttpGet]
    public async Task<ActionResult> Test1()
    {
        {
            _backgroundJobClient.Enqueue(queue: "test_queue_1", methodCall: () => Test1_NonAction(5));
            _backgroundJobClient.Enqueue(queue: "test_queue_1", methodCall: () => Test1_NonAction(1));
            _backgroundJobClient.Enqueue(queue: "test_queue_1", methodCall: () => Test1_NonAction(3));
        }
        
        return Content("Ok");
    }
    
    [NonAction]
    public async Task Test1_NonAction(int num)
    {
        Thread.Sleep(1000 * num);
        Console.WriteLine("--------------------------------------------------------------------------------");
        Console.WriteLine($"{num} from Hangfire! - {DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss")}");
        Console.WriteLine("--------------------------------------------------------------------------------");
    }
}

<PackageReference Include="Hangfire.Core" Version="1.8.9" />
<PackageReference Include="Hangfire.AspNetCore" Version="1.8.9" />
<!-- https://github.com/raisedapp/Hangfire.Storage.SQLite -->
<PackageReference Include="Hangfire.Storage.SQLite" Version="0.4.0" />

Thanks