APCu crashing on IIS 7.5 about apcu

[weltling]

This issue is definitely related to #19. The locking issue. What is causing it - multiple processes are working on the same shared memory segment, but their locking is not inter process. Thus, it causes race conditions, especially on the higher load. In fact, it's like there were no locking at all. That is the reason why it works fine with mpm_winnt, everything is within the same process. This is actually the same issue the original APC suffers from.

A crash avoiding measure for IIS can be going to the fastcgi settings and setting max instances number to 1 (if the perf expectation permits it). A real solution to this could be creating a comprehensive service so every process would share the locking information. Or, not using any memory sharing between processes. Joe, what do you think? Anyway, any of the solutions might need a huge refactoring.

[flip111]

I would suggest actively notifying the user about this. Throwing errors/warnings when on the IIS platform and explicitly mention it in the documentation.

[weltling]

@samdark i confirm the crashes you've reported. It's not about IIS particularly, it's about the post fork model. It is the only model possible on Windows for a multi-process usage. Unfortunately, it is not possible to use APC and consequently APCu with this model. But, APCu can still be used flawlessly with Apache on Windows.

Today, either pre fork or TS SAPI scenario is only usable on any platform. Both Joe and me believed the crash issues were solvable with an improved locking, but after a deep investigation it turns out it's not the case. The shared memory model in ext/apc that is inherited by ext/apcu won't allow starting separate processes sharing the same memory on any platform, which is now uncovered with the case of FCGI. It is not a big issue on Linux as multiple FCGI fcgi processes can be started using fork(), but it's for sure a blocking issue for using APCu with FCGI on Windows. A fix will require some quite hard refactoring not only for the locking, but for the shared memory model. I could still go for such refactoring, if there are lots of people vouching this thus giving me the reason to plan my times.

Btw the spawner prog you've linked is a very interentisting approach. I think some similar should be integrated with the next PHP versions. That will finally allow to separate FCGI and actual web server hosts.

Thanks.

Anatol

[tengyifei]

Same issue here with Nginx and PHP-FPM. Looks like it only happens on Windows. Non-thread safe apcu 4.0.4 will crash php-cgi on calling apc_exists. Calls to apc_fetch will not crash, but will fail.

Faulting application name: php-cgi.exe, version: 5.4.23.0, time stamp: 0x52a7ad25
Faulting module name: php_apcu.dll, version: 5.4.27.0, time stamp: 0x5346b1da
Exception code: 0xc0000005
Fault offset: 0x000046cd

Interestingly, the crash will only happen out of ~50% server start-ups. If the first apc_fetch does not crash php-cgi, the crash will never happen throughout the lifespan of the php process. The crash will also not happen if only one instance of php-cgi is present.

If one instance of php-cgi crashes, the rest will crash simultaneously.

Unhandled exception at 0x77306ED6 (ntdll.dll) in php-cgi.exe: 0xC0000005: Access violation writing location 0x00B7919C

Call stack:
ntdll.dll!_RtlAcquireResourceShared@8() Unknown
php_apcu.dll!apc_windows_cs_rdlock(_RTL_RWLOCK * lock) Line 105 C
php_apcu.dll!apc_lock_rlock(_RTL_RWLOCK * lock) Line 238    C
php_apcu.dll!apc_cache_exists(_apc_cache_t * cache, char * strkey, unsigned int keylen, long t) Line 938    C
php_apcu.dll!zif_apcu_exists(int ht, _zval_struct * return_value, _zval_struct * * return_value_ptr, _zval_struct * this_ptr, int return_value_used) Line 957   C
php5.dll!562d321f() Unknown

[Andrewpk]

Can't really provide a load of information regarding this issue, but here are the error messages I was receiving with one of our Server 2012 boxes:
Faulting application name: php-cgi.exe, version: 5.5.11.0, time stamp: 0x53447407
Faulting module name: ntdll.dll, version: 6.1.7601.18247, time stamp: 0x521ea8e7
Exception code: 0xc0000005
Fault offset: 0x0004e9e9
Faulting process id: 0x1118
Faulting application start time: 0x01cf80ff4b26a959
Faulting application path: C:\Program Files (x86)\PHP\v5.5\php-cgi.exe
Faulting module path: C:\Windows\SysWOW64\ntdll.dll

[sescandell]

Hi there,

I'm experimenting exactly the same issue on IIS 8.5 running on a Windows Server 2012 R2.

Errors occuring "some times". It's quite difficult to get more information on it. I can run more tests if you tell me how to get more logs on "I don't know what" during the process (IIS logs, FastCGI logs, ... something else?).

Here is the report I got:

Faulting application name: php-cgi.exe, version: 5.5.11.0, time stamp: 0x53447407
Faulting module name: php_apcu.dll, version: 5.5.4.0, time stamp: 0x52651baa
Exception code: 0xc0000005
Fault offset: 0x00003703
Faulting process id: 0xdd0
Faulting application start time: 0x01cf9ac6e6885531
Faulting application path: D:\Program Files (x86)\PHP\v5.5\php-cgi.exe
Faulting module path: D:\Program Files (x86)\PHP\v5.5\ext\php_apcu.dll
Report Id: 5da773d4-06ba-11e4-80bc-00155d585c91
Faulting package full name: 
Faulting package-relative application ID: 


-----

Faulting application name: php-cgi.exe, version: 5.5.11.0, time stamp: 0x53447407
Faulting module name: php_apcu.dll, version: 5.5.4.0, time stamp: 0x52651baa
Exception code: 0xc0000005
Fault offset: 0x00003350
Faulting process id: 0x634
Faulting application start time: 0x01cf9ac7b692fadf
Faulting application path: D:\Program Files (x86)\PHP\v5.5\php-cgi.exe
Faulting module path: D:\Program Files (x86)\PHP\v5.5\ext\php_apcu.dll
Report Id: f4896f7d-06ba-11e4-80bc-00155d585c91
Faulting package full name: 
Faulting package-relative application ID: 

-----

Faulting application name: php-cgi.exe, version: 5.5.11.0, time stamp: 0x53447407
Faulting module name: ntdll.dll, version: 6.3.9600.17114, time stamp: 0x53648f36
Exception code: 0xc0000005
Fault offset: 0x000a747b
Faulting process id: 0x14c
Faulting application start time: 0x01cf9ac7b68bd5c8
Faulting application path: D:\Program Files (x86)\PHP\v5.5\php-cgi.exe
Faulting module path: D:\Windows\SYSTEM32\ntdll.dll
Report Id: f4c2a807-06ba-11e4-80bc-00155d585c91
Faulting package full name: 
Faulting package-relative application ID:  

Running on PHP 5.5.11 + OpCache + APCu 5.5.4

Tnank you for your help,

[oniric85]

Is this solved in 4.0.6?

[JEDIBC]

Hi,

It still crash with APCu 4.0.7 and PHP 5.5.18 NTS. Here's the backtrace :

Report for php-cgi__PID__3352__Date__10_29_2014__Time_08_58_02AM__993__Second_Chance_Exception_C0000005.dmp

Type of Analysis Performed      Combined Crash/Hang Analysis
Machine Name
Operating System                Windows Server 2008Service Pack 2
Number Of Processors
Process ID                      3352
Process Image                   C:\webstack\php\php-cgi.exe
System Up-Time                  01:58:12
Process Up-Time                 00:01:03
Processor Type                  X86
Process Bitness                 32-Bit

Top 5 Threads by CPU time

Note - Times include both user mode and kernel mode for each thread 
Thread ID: 0  Total CPU Time: 00:00:01.078  Entry Point for Thread: php_cgi!mainCRTStartup
Thread ID: 1  Total CPU Time: 00:00:00.000  Entry Point for Thread: msvcr110!_endthreadex+86

Thread report

Thread 0 - System ID 484

Entry point                      php_cgi!mainCRTStartup
Create time                      29/10/2014 08:57:00
Time spent in user mode          0 Days 00:00:01.000
Time spent in kernel mode        0 Days 00:00:00.078

This thread is not fully resolved and may or may not be a problem. Further analysis of these threads may be required.

Function

php_apcu!apc_cache_link_info+3b
php_apcu!zif_apcu_cache_info+3f
php5!zend_do_fcall_common_helper_SPEC+176
php5!execute_ex+295
php5!zend_execute+14b
php5!zend_execute_scripts+de
php5!php_execute_script+14c
0x00a6fb18
msvcr110!getenv+7b

Thread 1 - System ID 3944

Entry point                      msvcr110!_endthreadex+86
Create time                      29/10/2014 08:57:00
Time spent in user mode          0 Days 00:00:00.000
Time spent in kernel mode        0 Days 00:00:00.000

Function

user32!NtUserGetMessage+15
php5!timeout_thread_proc+9f
msvcr110!_beginthreadex+b4
msvcr110!_endthreadex+102
kernel32!BaseThreadInitThunk+e
ntdll!__RtlUserThreadStart+23
ntdll!_RtlUserThreadStart+1b

Exception Information
PHP_APCU!APC_CACHE_LINK_INFO+3BIn php-cgi__PID__3352__Date__10_29_2014__Time_08_58_02AM__993__Second_Chance_Exception_C0000005.dmp the assembly instruction at php_apcu!apc_cache_link_info+3b in C:\webstack\php\ext\php_apcu.dll from The PHP Group has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000000 on thread 0

Module Information 

Image Name:        C:\webstack\php\ext\php_apcu.dll
Base address:      0x00905a4d
Checksum:          0x00000000
COM DLL:           False
ISAPIExtension:    False
ISAPIFilter:       False
Managed DLL:       False
VB DLL:            False
Loaded Image Name: php_apcu.dll
Mapped Image Name: 
Module name:       php_apcu
Single Threaded:   False
Module Size:       72 KBytes
Symbol File Name:  c:\php-debug-pack-5.5.18-nts-win32-vc11-x86\php_apcu.pdb
Symbol Type:       PDB
Time Stamp:        Sat Oct 11 19:59:49 2014 
Comments: 
Company Name:      The PHP Group
File Description:  php_apcu.dll
File Version:      5.5.11
Internal Name:     APCU extension
Legal Copyright:   Copyright © 1997-2013 The PHP Group
Legal Trademarks:  PHP
Original filename: php_apcu.dll
Private Build: 
Product Name:      PHP
Product Version:   5.5.11
Special Build:     &

[JEDIBC]

In addition, the folowing script reproduce the crash under ab -n 1000 -c 5 :

<?php
$datas = [
    'foo' => 'bar',
    'toto' => ['titi', 'tata', 'tutu'],
    'php' => 'roxxx',
    'date' => new \DateTime(),
    'js' => file_get_contents('./test.js'),
    'css' => file_get_contents('./test.css')
];

foreach ($datas as $key => $value)
{
    if (false === apc_fetch($key))
    {
        apc_store($key, serialize($value), 0);
        echo 'store ' . $key . '<br/>';
    } else {
        echo 'load ' . $key . ' = ' . strlen(apc_fetch($key)) . '<br/>';
    }
}

[tengyifei]

How about falling back to an inter-process locking implementation when the plugin detects that it belongs to one of many instances (possibly using mutex?). The Windows implementation of the APC locks using RtlAcquireResourceXXX et. al. depends on undocumented functionality and there is no guarantee if it will break anytime. Could be replaced by global mutexes and semaphores, though along with a slight performance penalty.

[weltling]

@tengyifei yeah, mutex or similar were fine. Why I suggested a service is that something needs to do the initial work, and with FastCGI there is no master process. Whereby as we see from the other issue mpm_prefork is sometimes not happy with that, too. Might be some other Linux specific reason though.

[yurii-github]

setting to 1 instance worked for me on IIS-FastCGI

[fedevegili]

After lots of debugging, I found out that I'm having this very same issue. If I set more than one instance for IIS FastCgi and do some heavy loading, boom, it crashes.

If you guys need, I can provide some really simple testing example (5 lines of code) and even a set up VM.

Is there any updates about this issue? Unfortunately, we'll have to look for another solution for now because we can't lower our fastcgi instantes. :(

[krakjoe]

If it really is the case that the locks we are using on windows are not suitable for sharing among processes (the ones we use on elsewhere are), that's pretty terrible. Isn't there a more suitable lock @weltling ?

I don't know anything about windows ...

[krakjoe]

Please try with the latest release on pecl, and report back ?

[nicolas-grekas]

I don't know if it's related to the issue reported here, but 4.0.8 still segfaults on the Symfony test suite:
https://ci.appveyor.com/project/nicolas-grekas/symfony/build/1.0.8#L1164
Sorry I don't have a better report to give... (cli SAPI with apc.enable_cli=1)

[krakjoe]

Can you try with 4.0.10 or 5.1.2 please ?

[weltling]

May i also ask to check the builds here http://windows.php.net/downloads/pecl/snaps/apcu/5.1.2_global_mutex/ and the branch they're made from https://github.com/krakjoe/apcu/tree/win_global_mutex ?

Those are experimental so not expected to be perfect. Though i see some improvement in the multi-process environment. Thread safe SAPIs like Apache might be hit by some slowdown as mutexes are per se slower, here they're even global. On the other hand, as example, thread safe FCGI bins could be perfectly used if one would want to utilize pthreads extension.

So except we decide to determine we're under a real TS SAPI, a universal locking could make sense and simplify.

Thanks.

[samdark]

4.0.10 with PHP 5.5 nts crashes at apc_cache_insert(_apc_cache_t *, apc_cache_key_t, apc_cache_entry_t *, _apc_context_t *, long, unsigned char) as well with:

Unhandled exception at 0x5D0A311A (php_apcu.dll) in php-cgi.exe: 0xC0000005: Access violation reading location 0x00000004.

[samdark]

@weltling tried mutex branch with PHP 7 ts. Crashes at apc_cache_fetch(_apc_cache_t *, _zend_string *, long, _zval_struct * *) with:

Unhandled exception at 0x0E1C2527 (php_apcu.dll) in php-cgi.exe: 0xC0000005: Access violation reading location 0x15115A28.

[weltling]

@samdark thanks for checking. Were some more info possible, some code and the way you run it? In my checks i was oriented on the repro case posted earlier by @JEDIBC #67 (comment)

Thanks.

[krakjoe]

Also, those trace lines look like 32 bit, is that the case ??

Architecture might be significant somehow ...

[samdark]

@krakjoe yes, these are 32bit.

[samdark]

@weltling happend when I'm doing writes/reads in PHP scripts that are called at the same time via AJAX. I'm using this little thing to spawn 4 cgi processes and nginx with 4 workers to serve them.

[krakjoe]

So, it turns out that the processing model used by IIS is simply incompatible with the way APC(u) shares memory and uses locking.

@weltling has suggested that he may try to lift the limitation for Windows. In my opinion it's not even worth pursuing for non-windows operating systems, so, I'm going to mark the bug closed, though that's not necessarily the end of the conversation ;)

[samdark]

It's not IIS only. Mine crashes are when running multiple cgi processes and serving these via nginx (which is basically because of the same reason but still...)

@weltling is there any way to run nginx + PHP under Windows which would work with APCu except using only one nginx worker and a single cgi process?

[samdark]

To me it's nice to have but not critical. Should be documented though as, I think, it's relatively common.

[Pes8]

Same Issue here:

Faulting application name: php-cgi.exe, version: 7.0.7.0, time stamp: 0x574602f4
Faulting module name: php_apcu.dll, version: 7.0.5.0, time stamp: 0x5756cd47
Exception code: 0xc0000005
Fault offset: 0x0000000000002b80
Faulting process id: 0xd04
Faulting application start time: 0x01d1ecc229758baa
Faulting application path: C:\Program Files\PHP\v7.0\php-cgi.exe
Faulting module path: C:\Program Files\PHP\v7.0\ext\php_apcu.dll
Report Id: 6751c668-58b5-11e6-80e0-005056a23fda

IIS with PHP7 in FastCGI mode.

Is it possible to use apcu on FastCGI PHP ?

[tomolimo]

See krakjoe/apcu-bc#17 where I described the same issue

[pierre-H]

I have the same problem with nginx.
@samdark did you find a solution ? Thanks !

[samdark]

I've personally never experienced it with nginx and on Windows I'm using php-cgi-spawner: https://github.com/samdark/wnmp-dev

[siburny]

I recently added second IIS site with FCGI PHP handler. I set "Max Instances" to 1 hoping it would work as well, but it didn't. Each IIS site uses it's own php-cgi.exe process so first process to start grabs the shared memory block. Others throw an error like this:

[06-Nov-2018 21:45:06 America/New_York] PHP Fatal error:  PHP Startup: apc_shm_create: shmget(0, 1048576, 658) failed: No such file or directory. It is possible that the chosen SHM segment size is higher than the operation system allows. Linux has usually a default limit of 32MB per segment. in Unknown on line 0
[06-Nov-2018 21:45:06 America/New_York] PHP Fatal error:  PHP Startup: apc_shm_attach: shmat failed: in Unknown on line 0

[ifew]

I got same errors, but set "Max Instances" to "1" is work for me

[JEDIBC]

Hi all,

Little report, 7 years later, as I retry APCu preparing the upgrade to PHP8 next year and as wincache is discontinued since PHP 7.2 :

• PHP 7.4.25 VC15 x64 NTS
• IIS 10 Win2019 x64
• APCu 5.1.21 VC15 x64 NTS
• FCGI Max instances set to 0, 1 or 10

All is working great ! Absolutly no crashes 👍

# ab -c 50 -n 5000 http://test.local:8080/apcu-test.php
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking synapse.test.local (be patient)
Completed 500 requests
Completed 1000 requests
Completed 1500 requests
Completed 2000 requests
Completed 2500 requests
Completed 3000 requests
Completed 3500 requests
Completed 4000 requests
Completed 4500 requests
Completed 5000 requests
Finished 5000 requests


Server Software:        Microsoft-IIS/10.0
Server Hostname:        test.local
Server Port:            8080

Document Path:          /apcu-test.php
Document Length:        120 bytes

Concurrency Level:      50
Time taken for tests:   7.881 seconds
Complete requests:      5000
Failed requests:        0
Total transferred:      1420000 bytes
HTML transferred:       600000 bytes
Requests per second:    634.43 [#/sec] (mean)
Time per request:       78.811 [ms] (mean)
Time per request:       1.576 [ms] (mean, across all concurrent requests)
Transfer rate:          175.95 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.4      0       8
Processing:     5   78  73.1     49     595
Waiting:        4   78  73.1     49     595
Total:          6   78  73.1     49     596

Percentage of the requests served within a certain time (ms)
  50%     49
  66%     62
  75%     79
  80%     98
  90%    194
  95%    260
  98%    297
  99%    315
 100%    596 (longest request)

The script used :

<?php
$datas = [
    'foo' => 'bar',
    'toto' => ['titi', 'tata', 'tutu'],
    'php' => 'roxxx',
    'date' => new \DateTime(),
    'js' => file_get_contents('./test.js'),
    'css' => file_get_contents('./test.css')
];

foreach ($datas as $key => $value)
{
    if (false === apcu_fetch($key))
    {
        apcu_store($key, serialize($value), 0);
        echo 'store ' . $key . '<br/>';
    } else {
        echo 'load ' . $key . ' = ' . strlen(apcu_fetch($key)) . '<br/>';
    }
}

Thanks again the the APCu team ! You're doing a great work ! 👍