UTF-8 path failes to dlopen about dlfcn-win32

[peak3d]

@traversaro you can create a local path by copying the widechar part from here:
xbmc/xbmc#16577 (comment)

Then place any dll into this folder, and try to dlopen it with full path.

[pali]

Some more details and explanation... I hope that this would be clear right now.

All winapi functions which takes char* buffer expects that buffer is in CP_ACP encoding, including fopen(), _open(), CreateFileA(), LoadLibraryExA(), ... . So dlopen() should also takes filename buffer in CP_ACP encoding and not in CP_UTF8. You cannot use different encoding for dlopen() and different for fopen(), it makes no sense, just broke all other things around.

CP_ACP encoding is global system encoding which applies to all running applications and processes. CP_ACP encoding is implicitly derived from option Language for non-Unicode programs available in Regional and Language Options via Control Panel. E.g. For English (United States) is used 1252. This option can be changed and after rebooting Windows it takes effects.

Since Windows 10 insider build 17035 there is a beta option which can set Language for non-Unicode programs to UTF-8 therefore CP_ACP would be UTF-8 globally, see: https://superuser.com/a/1451686

And since Windows 10 Version 1903 (May 2019 Update) there is an option to set CP_ACP to UTF-8 per exe file, but it needs to be done at application compile/build time via manifest, see: https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page

In all other cases you are limited to characters supported by current configuration of CP_ACP, so you can convert subset of Unicode/UTF-8 to CP_ACP via conversion functions. Or you need wdlopen() API which takes file name in Unicode/UTF-16 (and calls Unicode/UTF-16 LoadLibraryExW() API).

[traversaro]

Thanks for reporting the issue. Could you provide an example of a unicode filename for which loading the library fails, so that we can add to the test suite? Thanks!

[peak3d]

I created this patch to get it working inside kodi: https://github.com/peak3d/inputstream.adaptive/blob/master/depends/windows/dlfcn-win32/0001-dlopen_with_widechar.patch

[pali]

API of dlopen is according to current locale. If you have not set current locale to UTF-8, you cannot pass UTF-8 string as its input. IIRC Windows does not support UTF-8 as ANSI codepage (=current locale) since some last update in Windows 10, plus ANSI codepage is system global, not per process.

For this reason WinAPI has for every function two variants. One with A suffix (as ANSI) and one with W suffix (as Wide/Unicode which means UTF-16). POSIX/C functions in MSVC has also two variants, one without any suffix or prefix (mimic POSIX behavior, use current locale = ANSI codepage) and one with w prefix which argument is in wchar_t* (so in UTF-16).
See e.g. open and wopen on WinAPI/MSVC.

Thefore above patch is wrong would completely break opening files.

dlopen must open same files as open function. Not different. And must be in same encoding.

If you need to other encodings, either set it in system, or bring w-prefix function e.g. wdlopen.

[peak3d]

@pali do I read this correct that I should be able to LoadLibraryExA the path from above if locale settings are correct?

[pali]

Yes, exaclty. Also you should be able to use _open or fopen too.

[peak3d]

@pali I have poked arond with locale settings, but there was no success to dlopen the library. All implementations of such wrappers (e.g. chromium) in s/w which are meant to work with unicode / wchar paths use LoadLibrary(Ex)W. One alternative I have seen is using GetShortPathname(), but t.b.h. I want to avoid it.

[pali]

This is reason why there are _wopen or _wfopen functions. If you are unable to use _open or fopen for opening your library file, you also should be unable to use dlopen.

This is also why I suggested to introduce API for wdlopen call to work in same way as MSVC's _wopen.

[traversaro]

If you need to other encodings, either set it in system, or bring w-prefix function e.g. wdlopen.

I think that @pali point makes sense, and dlfcn-win32 should be similar to the rest of Unix-APIs implementation on Windows, and the proper way to solve this is to add a wdlopen non-POSIX call.

cc @seanyen @mxgrey @j-rivero @scpeters this may affect Gazebo loading plugin with UTF-8 names, as dlopen is used in Gazebo and ignition-plugin .

[traversaro]

cc @Gnimuc that has used dlfcn-win32 in JuliaInterop/Cxx.jl#409 and JuliaInterop/Cxx.jl#425 , and so may be interested in this issue as well.