Alert 0x01000009 memory already free, when calling CRYPTO_free() about amissl

[broadblues]

Crashlog_AWeb retrieve_2017-08-15_15-15-57.txt
Crashlog_python_2017-05-18_14-03-28.txt

[tboeckel]

This is AmiSSL's free() function:

void free(void *p)
{
  if(p != NULL)
  {
    ObtainSemaphore(&__mem_cs);
    p = &((ULONG *)p)[-1];
    FreePooled(__mem_pool, p, *(ULONG *)p);
    ReleaseSemaphore(&__mem_cs);
  }
}

Not much to go wrong here, except the FreePooled() call. This is exactly the line the crashlog points to.

Since you reported similar crashes of other applications when freeing memory from pools in AmigaOS4's BugZilla this points to a bug in AmigaOS4's memory pool handling rather than to a bug in AmiSSL.

[Futaura]

I would think it is more likely misuse of the OpenSSL init functions - a lot changed in OpenSSL between AmiSSL v3 and v4. Initialization is now less ambiguous and less complicated, but as a result less forgiving. A lot of the top level init functions are now just macros that call OPENSSL_init_ssl() - I'm thinking in particular of SSL_load_error_strings(), which may be the problem here. I remember having similar trouble in IBrowse. Now I just call OPENSSL_init_ssl(OPENSSL_INIT_SSL_DEFAULT | OPENSSL_INIT_ADD_ALL_CIPHERS | OPENSSL_INIT_ADD_ALL_DIGESTS, NULL) once directly after InitAmiSSL().

I only have some old AWeb sources to refer to, but if SSL_load_error_strings() and SSLeay_add_ssl_algorithms() are called multiple times, that could be the problem. Both of those ultimately now call OPENSSL_init_ssl(). It is better to use the new API now.

BTW, I'm using all the latest OS4 beta components and am having no issues like this on my A1XE.

[broadblues]

@tboeckel
I'm not sure that these crashes are coming from the same src as my others, they are different in that they conistantly through alert 0x01000009 which is a memeory freed twice error. The crashes I reported against the OS are quite different allways being a DSI on '0x00000000' followed by layer corrupt memeory list errors when the pool is released.

Not dimissing the idea completely, but just expressing my doubts.

[broadblues]

@futura
I compiled the AWeb code without any changes, which seemed to work, at least in terms of compiling without any errors / warnings, so I will review the code in the light of your comments and see if that improves thiings

[Futaura]

@broadblues If I'm reading the code correctly, am I right in thinking that the main AWeb task opens and initialises AmiSSL, and multiple subtasks (for each connection) each create a SSL context, and calls the OpenSSL init stuff?

If it is still crashing after moving the SSLeay_add_ssl_algorithms() and SSL_load_error_strings() (or new API equivalent), then it may possibly be related to creating multiple SSL contexts on subtasks. I guess that should work fine if it did in AmiSSL v3. None of the OpenSSL test programs would test a case like this, and IBrowse does things differently (single SSL context created by main task, and shared by the subtasks).

[broadblues]

AWeb calls Tcpopenssl() which effectively maps to Assl_initamissl() in OS4 for each http process (one for each active connection).

Each http process has it's own 'socketbase' and passes that to Tcpopenssl() so that in turn means InitAmiSSL() gets called once for each process, with the socketbase as the argument. When the task exits Assl_cleanup() and hence CleanupAmiSSL() gets called.

I'm not seeing any way that a call to InitAmiSSL() is matched to the call to CleanipAmiSSL(), by that I mean there no context pointer ar anything, so does that mean the call to CleanupAmiSSL() applies to the whole library instance? Thus possibley causing the memory freed twice type of crash?

[Futaura]

I wouldn't have thought so, no. It is correct to pair calls to InitAmiSSL() and CleanupAmiSSL(). Internally, they use the task address as a context pointer.

Are you able to successfully complete at least one connection before it crashes or is it crashing on the first one? At first glance, where it is crashing in OpenSSL doesn't make much sense as it does set the pointers to NULL after freeing the memory, so even if that code is called twice, it shouldn't crash (unless it is not being initialized properly in the first place, but then I would expect all apps to crash). I'll try making a simple test case for this tomorrow.

[broadblues]

OK so the calls are matched via the Task pointer, that makes sense.

Crashes only occur on subsequent connections the first connection always works at least so far.

I thought it might be tasks running in paralell that causes the crash, but I limited the connections to 1 at a time and still got a crash.

BTW I have reproduce these crashes on both my SAM and my x1000 whereas the MemeoryPool crashes refered to by Tboeckel above only occur on the SAM.

[Futaura]

I'm running out of ideas now - I've attached a simple test program that I knocked up to mimic what AWeb is basically doing, and it will not crash here on my A1XE. Can you try compiling and running it multiple times to see if it crashes on your system.

amissl_test.c.txt

The only other thing that springs to mind is to check that all the latest include files are being used and that calls to InitAmiSSL() and CleanupAmiSSL() are paired. If AWeb (or any other application for that matter) misses calling CleanupAmiSSL(), it can cause any app using AmiSSL to crash.

[broadblues]

Compiled and run that test a number of times, on the x1k no crash, but AWeb was also not crashing, then I ran it 5 times on the SAM and it crashed.

The machine locked up so I couldn't save out the log, but the sashimi output section showing in the console showed it crashed on the Free function again, but in a slightly different way. The crash occured after the SSL_Connect() call in the text program.

[broadblues]

I'm fairly sure my includes etc are all updated to amissl4 (I renamed the amissl3 ones before installing)

[Futaura]

Would you mind e-mailing the compiled test program to me, or uploading here?

I'll check to see if it crashes here also. Just wondering if it is related to your build environment in some way, because when I build the test program, I have been unable to get it to crash here on my A1XE.

It could be something else trashing memory, as Thomas on the OS4 bugzilla suggests. Equally, I wouldn't completely rule out a bug in AmiSSL, but I would also expect there have been similar reports from YAM users at least.

It is frustrating that as OS4 has developed, is has become more and more developer unfriendly. System lockups are not helpful, opposed to a GR, and debugging tools no longer work. In particular, MemGuard was an invaluable tool, which may have been a help in this instance.

[broadblues]

SWorry for delay have now emailed build to you.

[Futaura]

@broadblues Try as I might, but I can't get this to crash on my A1XE using the latest beta components. So, I guess that mean it is compiling perfectly fine on your system, so that rules out a compiler issue. I've also tried the debug and non-debug kernels, with and without the "munge" option. Somehow I managed to get Memguard to work (seemed to be working at least) and that triggered no issues either.

What kernel options are you using? Might be worth trying the debug kernel or vice versa. Do you have any older OS4 installs that you can test it on?

The only other thing that springs to mind is stack size - try increasing the stack size, but I don't think this would be an issue otherwise I could trigger it here. Clutching at straws :).

It is not looking like an AmiSSL issue to me, without more evidence.

[broadblues]

Just an update to this issue.
I have identified the top level cause of my general memory pool issues on my SAM, that being my AORganiser program, bur only when syncing with google calendar. (which means this culprit could be AO itself, python or my port of the python __ssl.so module (which is using amissl but I'm not suggestiong that's the issue at this point, it's just the bottom of the possibilty stack).

Having eliminated that, I retested AWeb and I find that again it remains rock solid with only one connection, but when setting multiple connections https urls crash eventuallt with the same CRYPTO_free() call.

So I've emlinated the third party memory trash option, it's down to AWeb or AmiSSL.

I guess I'm going to have to work out how to build a debug version for myself and see if I can't get some more clues.

[Futaura]

AWeb related issue is fixed in AmiSSL 4.2