port SLIM to Windows (incl. optional external GSL in CMake) about slim

[rdinnager]

Hi @bhaller,

Wow, thanks so much for the detailed thoughts on my suggested direction for slimr. I am not at all upset or discouraged by your tone, you are absolutely right to voice your concerns strongly, especially when they seem centered around making sure I don't waste a lot of my own time. I have a lot to think about from this. Perhaps I could ask your advice again (by email maybe) after I have reconsidered my ideas? I think there may still be some value in some less extreme integration of SLiM into R than my original plans, but I think you have convinced me already to pull back and rethink. I think probably anything I do in this direction should be discusses with you before I decide to try it, as I agree it should be as 'unbreakable' as possible, and that is probably a bigger challenge than I at first thought without the benefit of insight from inside the SLiM development process. And by the way, thanks so much for all the incredible amount of work you put in to SLiM so far. It really is turning into an unrivaled tool.

For the record, slimr already supports converting slimr scripts into SLiM scripts and opening them automatically in SLiMGUI or QtSLiM. I guess what I was hoping to do was allow users to generate whatever custom visualisation they wanted, but now that I hear your thoughts, perhaps that sort of system would be so complicated anyway, that users might as well learn Qt and make a custom plugin for QtSLiM? Is this possible right now, out of curiousity?

I will open another issue for the Windows compilation project. Just a few comments now. MinGW is also a standard and well-supported compilation toolchain on Windows. It is used exclusively by R for compilation of C++ code in R packages on Windows for example, which is why I am more familiar with it. It also supports Qt5 (the last entry in the table: https://doc.qt.io/qt-5/windows.html : MinGW 8.1), so this shouldn't be an obstacle. Likewise, I believe it can be used in conda packages, so also won't be an obstacle to packaging in conda (but I will have to look into it more, I know it is at least used for R packages on conda). This nicest thing is that MinGW stands for Minimalist GNU for Windows. This means it attempts to mimic the GNU environment as much as possible in Windows, and so usually takes a lot less work to port a Unix project than with Visual Studio. If you are interested in knowing more, here is the webpage: https://www.mingw-w64.org/). I've already done a fair amount of work using it, so I'd be reluctant to switch at the moment. Of course, if I run into trouble with it, I will give Visual Studio a try.

Sticking with the internal SLiM gsl library version should be fine I think, I will start with this and let you know if any issues with it crop up.

Thanks again for your thoughts. I will read them over carefully and think more about this. In the mean time I will keep working on the Windows compilation.

And @jeromekelleher I will definitely want to pick your brain about conda and compiling in general. Thanks so much for the offer.

[bhaller]

I think that ought to be straightforward to add – SLiM's GSL code is just a subset of the standard GSL, without modifications – but I don't really grok CMake, so perhaps @brnorris03 can contribute a PR? The only problem I anticipate is that #includes of GSL headers in the Eidos/SLiM source files might need to change, based on that compile flag, from project includes to system includes, which is a bit annoying but should be manageable.

[brnorris03]

If it's just the paths, then yes, it would be straightforward to add a cmake option, e.g --with_gsl=/path/to/gsl or something like that. If that's acceptable, I'll go ahead and submit a PR.

[jeromekelleher]

Re: Windows, I'd be happy to see SLiM build/run on Windows, but I have no Windows machine to test on, zero Windows development experience, and indeed, pretty much zero experience using Windows either. So if that is going to happen, it will be from PR's submitted by others, which I will be happy to accept; but if someone chooses to do that work, I hope they will also choose to maintain it in the future, as I will not be able to do so. The contribution would be most welcome; but I want to make it clear that I can't commit to doing it or to maintaining it in future, although I'll do my best not to carelessly break it :->.

I don't use windows either @bhaller, so I can't help with the initial changes needed. Setting up CI builds on Appveyor makes maintenance easy though, once the initial (usually pretty trivial) portability problems are sorted out.

[bhaller]

@petrelharp had someone who was interested in porting SLiM to Windows, IIRC. If not, perhaps someone at Cornell wants to do it. I could also ask for a volunteer to do it on the slim-discuss mailing list, maybe someone would come out of the woodwork. :->

[jeromekelleher]

This sounds really interesting @rdinnager. From a packaging perspective, the best outcome would be if we could build a Windows binary as part of the conda-forge package (see here and the recipe here). This would be most straightforward if the code built directly under Visual C++, I think. I'm not sure whether this is possible, but just throwing it in here as a data point.

I have some experience with setting up Windows CI etc as well (although am not actually a windows user, so don't actually know much about it!), so feel free to ping me if you'd like some pointers in that direction.

[bhaller]

@jeromekelleher just commented to me that Qt is in conda-forge, so in principle it ought to be straightforward (once the back end is ported to Windows) to make a slim-gui package on conda that provides SLiMgui to Windows users easily, which would be great. I'm not a conda user so I know very little about it, but Jerome's offer to give you a hand with the Windows port or conda stuff is worth its weight in gold @rdinnager (thanks Jerome!). :->

I'm going to retitle this issue since it seems like the external GSL thing is completely tied to the idea of porting to Windows. If there's a need for more discussion about the idea of wrapping the Eidos APIs in R, injecting R code into SLiM, etc., it would probably be good to start a GitHub Discussion about that; I just enabled Discussions on SLiM. From here, let's keep this issue about a Windows port and external GSL. Thanks!

[rdinnager]

Oops, just saw your new post @bhaller, looks like it came in while I was writing the above. Any further discussion of slimr I will do in Github Discussions. Actually it should be great to have a Github Discussions for SLiM, generally.

[rdinnager]

A small update: I hacked a fix to the /tmp directory issue, just to get the tests to run. For SLiM, 100%:

For Eidos, 17 failures, 6133 successes. All failures had to do with the filesystem bits that don't exist on Windows (e.g. assuming the existence of /tmp or /bin, or else they were calls to system() running a command that doesn't exist on Windows (e.g. printf, expr, etc.). So I think that is pretty good, all of the important guts of SLiM seem to work. I curious now about the performance of this SLiM version now. Do you have a script or set of scripts you use to benchmark? I could start by comparing this installation to my WSL installation.

Getting this into the next release might be possible but I'm not sure I can commit to it definitively, as I have a fair amount on my plate right now. I recently moved overseas from Australia and am starting a Research Assistant Professor position at Florida International University in Miami (to answer your question about where I am). But I will do my best!

I have not used a debugger before (for C++ anyway, at least not since highschool). I have been using VSCode for this project, and it looks like I can use that with a GDB debugger (https://code.visualstudio.com/docs/cpp/cpp-debug). I will give it a try.

I will send a PR now, which I will continue to add to as I clean more stuff up. You might want to have a look at it, especially the bits where I've edited C++ code, because I don't have much experience here and I may have used some inefficient code, or diverged from SLiM's coding style too much. I haven't done this in many places, though I had to add a lot of these:

#ifdef _WIN32
#include "config.h"
#endif

At the beginning of .h. or .cpp (another thing I am sometimes unsure of, is whether to put something in the header, or in the .cpp source). This header sets the environment for gnulib to work (e.g. allowing it to selectively define replacement functions for those missing in windows but not for those that aren't missing.

I'm excited to be the first person to ever use SLiMGUI on Windows natively (I'm assuming?)!

[rdinnager]

Okay no worries. I will clean up the PR first and let you know when that is done.

[bhaller]

Okay, so I am at a point where it makes sense to try and deal with some of the compiler warnings I am getting...

Yep.

The first one is during compilation of tskit:

F:/SLiM/treerec/tskit/core.c: In function 'get_random_bytes':
F:/SLiM/treerec/tskit/core.c:47:29: warning: initialization of 'HCRYPTPROV' {aka 'long long unsigned int'} from 'void *' makes integer from pointer without a cast [-Wint-conversion]
   47 |     HCRYPTPROV hCryptProv = NULL;
      |                             ^~~~
F:/SLiM/treerec/tskit/core.c:57:20: warning: assignment to 'HCRYPTPROV' {aka 'long long unsigned int'} from 'void *' makes integer from pointer without a cast [-Wint-conversion]
   57 |         hCryptProv = NULL;
      |                    ^
F:/SLiM/treerec/tskit/core.c:60:16: warning: assignment to 'HCRYPTPROV' {aka 'long long unsigned int'} from 'void *' makes integer from pointer without a cast [-Wint-conversion]
   60 |     hCryptProv = NULL;
      |                ^
F:/SLiM/treerec/tskit/core.c:63:20: warning: comparison between pointer and integer
   63 |     if (hCryptProv != NULL) {
      |                    ^~

That code builds only on Windows in the first place; it's inside #if defined(_WIN32). According to the Microsoft docs (https://docs.microsoft.com/en-us/windows/win32/seccrypto/hcryptprov) it is supposed to be a pointer to unsigned long:

typedef ULONG_PTR HCRYPTPROV;

It looks like whatever headers you're getting from your toolchain define it as long long unsigned int instead; I have no idea why. I would suggest that maybe tskit ought to change all of those assignments of NULL to assignments of (HCRYPTPROV)NULL so that it builds without warnings regardless of what HCRYPTPROV is defined to be. @benjeffery does that sound reasonable? This warning is on Windows building with the Ming64 compiler, apparently. If the fix sounds good, do you need an issue/PR for it? (You can stop reading here, this is the only tskit-related warning. :->)

Next, I get a warning while compiling slim:

F:/SLiM/core/slim_sim_eidos.cpp: In function 'void PrintBytes(std::ostream&, size_t)':
F:/SLiM/core/slim_sim_eidos.cpp:47:38: warning: integer overflow in expression of type 'long int' results in '0' [-Woverflow]
   47 |  if (p_bytes > 1024L * 1024L * 1024L * 1024L)
      |                ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~

Hmm, that's odd. I'm a bit concerned that standard data type sizes might be different with your compiler. long should be 8 bytes; 1024^4 should fit very easily into long. Check your data type sizes. I just pushed a commit that provides, at the end of SLiM_WarmUp(), three lines that print the size of the major integer data types. Uncomment those lines and run a model, any model. The sizes should be 4 for int (8 is also OK), and at least 8 for long and size_t.

If long is only 4 bytes, then we have a problem that will require a bigger fix. I find that hard to imagine, though; 8 bytes for long is pretty much universal nowadays, I think. So this looks like a compiler bug to me, if long is indeed 8 bytes; you might even try reporting it to the compiler provider! But in any case, the commit I just pushed also changes the 1024L occurrences there to 1024.0 instead; working in floating point instead of integer should be fine there, and should silence the warning.

and another for eidos:

F:/SLiM/core/slim_sim_eidos.cpp: In member function 'EidosValue_SP SLiMSim::ExecuteMethod_rescheduleScriptBlock(EidosGlobalStringID, const std::vector<Eidos_intrusive_ptr<EidosValue> >&, EidosInterpreter&)':
F:/SLiM/core/slim_sim_eidos.cpp:3431:18: warning: 'stage' may be used uninitialized in this function [-Wmaybe-uninitialized]
 3431 |   CheckScheduling(start, stage);

I've just pushed a commit to fix this. This warning is spurious, the compiler is being dumb. I wonder what version of GCC this Ming64 compiler is based on? Recent GCC versions don't warn on this, so it's surprising that it does.

Next is a QtSLiM warning:

F:/SLiM/QtSLiM/QtSLiMExtras.cpp: In function 'QString stringForByteCount(uint64_t)':
F:/SLiM/QtSLiM/QtSLiMExtras.cpp:533:38: warning: integer overflow in expression of type 'long int' results in '0' [-Woverflow]
  533 |     if (bytes > 512L * 1024L * 1024L * 1024L)
      |                 ~~~~~~~~~~~~~~~~~~~~~^~~~~~~

Same problem as the previous very similar warning; again, looks like either long is only 4 bytes, or it's a compiler bug.

That is all of them. Any idea what the problems are here?

Hopefully this clears everything up. The tskit warning will wait for a fix from them, but it's harmless. The others should be fixed. But do check sizeof(long), and if it is only 4 bytes then I will need to go on a search-and-destroy mission finding occurrences of long throughout the code base and changing them to int64_t. Which would be fine, just let me know if it's needed.

[rdinnager]

So I guess you'll want to update your PR to be based on the latest SLiM GitHub head, to get the changes I just committed. I think the usual way to do that is with a rebase? My git-fu is extremely weak, so I'm not sure; hopefully you know how to do this sort of thing!

Yes, I can just fetch upstream. You can actually do it right from github if you want, no need for command line..

[rdinnager]

Unfortunately, wslg (the new WSL graphical mode available in Windows Insider builds) does not work with SLiMgui, I just tried it :( . So only the more complicated Xserver based solution works with WSL so far. Oh well.

Huh. I thought @bernard-kim had tried this and found that it worked. See section 2.4.4 of the SLiM manual.

The instructions in section 2.4.4 do work, they are just very complicated and a little fragile (Windows updates can break the setup). Microsoft has released in the preview version of Windows 10 (and 11) a new WSL that has GUI support built in, called WSLg...

Yep, I'm aware of all this. Section 2.4.4 discusses getting SLiMgui to work with the WSLg, thanks to Bernard. It seems to work for him, and I've gotten no complaints from others. Maybe Microsoft broke it again with a new version of the WSLg, but it used to work.

Anyway, perhaps it doesn't matter so much anymore now that we will soon have a native Windows SLiMgui, but it would have been nice for testing. Anyway, I think I found an alternative, using a remote desktop into my work's linux cluster.

Well, it'd be good for it to get fixed; my guess is that a substantial user base will want to continue using SLiM under the WSL, because Linux. But that's a problem for @bernard-kim, I guess. Maybe you could open a new GitHub issue on this, and provide details regarding your exact setup, and exactly how SLiMgui fails to work under the WSLg? Then maybe I can assign that issue to Bernard, if he's willing. :->

Oh, sorry. I don't see the section on WSLg in the manual. Maybe I'm looking at an old version? Maybe it is just my linux version. I am using Ubuntu-18.04, and I think the manual mentions it is only tested on Ubuntu-20.04. I should have checked this first. I will let you know if this fixes it.

You may be looking at an old version. It starts: "There are two ways to approach running SLiMgui under the WSL. The first way is to use Microsoft’s new GPU-accelerated WSL 2, called the WSLg. At the time of this writing, the WSLg is currently in “preview”, so this is prerelease software. Also, this method of running SLiMgui has, thus far, only been demonstrated to work with Ubuntu 20.04 LTS; Ubuntu 18.04 produced only a black window, and other Linux distros have not been tried. To use the WSLg you will have to...". And yes, 18.04 doesn't work with the WSLg apparently.

Oh great, that is an easy explanation then. Yes, I was just getting a black window. Trying 20.04 now.

[rdinnager]

Side by side, Native WIndows SLiMgui on the left, WSL Ubuntu-20.04 SLiMgui on the right. Actually not that different. The font size in the text editing windows is too big on Windows, but the titles are just a little bit too small. The icons look about the same size to me.. what do you think @bhaller ? The divider lines being nearly invisible on Windows is strange though.. I wonder why?

[bhaller]

Oops, looks like I missed a couple compiler warnings in my last post on this. Here are a few more:

In eidos_zlib I get a warning:
And another very similar one:...

These functions are prototyped in zutil.h at line 191-192, but for some reason those prototypes are inside !defined(_WIN32) so you're getting the function definitions without the prototype declarations, which is producing a warning. Odd that zlib's code is set up that way, but then zlib's code is odd in many ways. In any case, these two functions adler32_combine64() and crc32_combine64() appear to be used by SLiM's version of zlib, so they are not needed to build. I have just #ifed them out to get rid of the warning, and synced that to the windows branch.

That's it. The only other warning now is the tskit one from before. @bhaller, should I open an issue for this with them, or do you want to do it?

I just did.

[bhaller]

Side by side, Native WIndows SLiMgui on the left, WSL Ubuntu-20.04 SLiMgui on the right. Actually not that different. The font size in the text editing windows is too big on Windows, but the titles are just a little bit too small. The icons look about the same size to me.. what do you think @bhaller ?

There are lots of #ifdefed customization points for macOS versus Linux; you should be able to add another branch to them to customize things. You can find these customization points by searching for #ifdef __APPLE__ in the QtSLiM folder. I'd recommend that you look at all of those existing customization points, as they are the likely spots where Windows may need to differ. If you'd like, we can zoom to go through them together one by one, as the purpose of some of them might be unclear (although in general I have tried to comment usefully on them...).

I'd recommend stepping into the model one generation so you see the tick marks labels for the chromosome view. They looked small to me in your previous screenshot, but it's hard to judge since I don't know the size of your screen etc. They are subject to one of these existing customization points. Opening a graph window would also be good to compare, as the font sizes in the graphs are another customization point; there's a scaling factor of 0.75 applied on Linux compared to macOS for them.

The default font and font size used in the editor is tricky since different fonts are available on each platform. The font can be chosen in the Preferences panel, so find a font and point size that you think looks ideal on Windows (and that is a standard font guaranteed to be present on Windows, and is monospace of course) and make that the default. You can find, in the function defaultDisplayFont(), the code that tries to find the best font. The list of fonts between macOS and Linux is almost non-intersecting, so there are no #ifdefs here; if it finds a macOS font it uses that, if it finds a Linux font it uses that. You might want/need to introduce a Windows #ifdef here, though, which is fine. Best would be to provide to platform fonts in the code there, so that there's a fallback if the first one is somehow missing (which does seem to happen for some users even with fonts that are supposed to always be present).

Two customization points that are complicated are (a) handling dark mode, which is very different on Linux (where it is controlled by a checkbox in the preferences panel) versus macOS (where it is a system setting that QtSLiM follows), and (b) what happens when the last main window is closed; on macOS it keeps the app open since that is the standard platform behavior, whereas on Linux it quits the app. These are both controlled by multiple customization points, and are places where platform-specific bugs are likely to occur.

The divider lines being nearly invisible on Windows is strange though.. I wonder why?

Well, this is a complicated part of the code. Qt provides a default appearance for splitters, but it isn't what I wanted. QtSLiM therefore subclasses the classes involved – class QtSLiMSplitterHandle : public QSplitterHandle and class QtSLiMSplitter : public QSplitter – to provide a custom appearance. The relevant drawing code is in QtSLiMSplitterHandle::paintEvent(). I would expect that to draw pretty much the right thing on Windows too, though, so it's a mystery. You might start, if you want to embark on this adventure, by changing the code in QtSLiMSplitterHandle::paintEvent() to just fill the whole bounds of the splitter with red, so you can see where they are, how wide they are, etc. That might help diagnose why they aren't drawing – maybe they're only one pixel wide, or something. Or maybe that method doesn't get called by Qt at all, on Windows! Anyhow, this is one area where I might need to find myself a Windows machine and figure it out myself, since I know you're not conversant with C++ or Qt.

[rdinnager]

Thanks for this detailed information. This should help a lot in getting this graphical stuff fixed up. Cheers!

[rdinnager]

Hi @bhaller. Yes, I just tested the pacman to make sure it works. I even uninstalled qt5 to make sure that pacman installs it automatically as a dependency and it worked without a hitch. So yes, it is ready to go, and I just need to finish the writeup for the manual. the last couple days have been pretty full, but I should have some time tomorrow to work on that. Yes, let's aim to get it into the 3.7 release.

[rdinnager]

@bhaller Thanks, yes, I will finish up those last few things this week (tomorrow hopefully).

[petrelharp]

Where's the PR for this? I can take a look at the failing tskit tests.

[bhaller]

Where's the PR for this? I can take a look at the failing tskit tests.

It has been merged into master, since the failing tests seemed to be the only problem with it, and it seemed clear that they were failing because the new tskit changes (particularly removing the ordering requirement for the individuals table) hadn't been released in a new tskit. So now the tests on master fail; if there's anything that can be done about that before the next tskit gets released, that would be nice. :->

[rdinnager]

Yes, I think I figured almost everything out. There is one or two things that still don't work, Eidos_GetMaxRSS() is currently disabled on Windows because I don't know how to get this information from the windows api. This means that memory tests are also disabled by default. It might be nice to get that working, for completeness. Also, I was never able to get the panel dividers to show up in SLiMgui, so that is another small thing that will require some a bit of trail-and-error fiddling.

[rdinnager]

We are also getting three failures with functions of floats, when comparing with user defined functions:
...
The same failure happens for cumSum and sum. It will be interesting to see if we get the same issues on the compiled version once I get that working.

Huh. I'm not sure I understand; these failures indicate that the compiled version doesn't match the user-defined version, for some input. But then you write "will be interesting to see if we get the same issues on the compiled version once I get that working", which implies that you do not presently have the compiled version working. How are you comparing the compiled version to the user-defined version, then? Anyhow, I don't have a guess on this one so far. What I'd recommend is opening up an Eidos console in SLiMgui and trying out sum()/product()/cumSum()/cumProduct() with some random input, just as these comparison tests do, to see what is going on. Happy to zoom with you if that would be helpful; email me a zoom link if you want to.

Sorry about being unclear. These failures happen on the github actions tests for Windows, when testing the SLiM install from msys2 pacman. This installation is not compiled locally, it simply installs precompiled binaries from the msys2 package repository. I was just curious whether we will see the same when I get the tests where we compile SLiM from source to work. It looks like I fixed the problem and the compilation is running on github actions now. I should see soon what happens when I get to the tests part of the workflow.

[bhaller]

Ah, I can now clarify that these failures happen on 32bit Windows only. I haven't really tested on 32bit Windows because I use 64bit Windows. One option is to just drop support for 32bit Windows (honestly I don't know how many people really still use it, and Microsoft will be abandoning it soon anyway). Otherwise I'm not sure what the consequences of this test failure is, and how to reproduce it without access to a 32bit machine I can play around with..

Aha. I would definitely drop support for 32-bit Windows. I have just added a compile-time check that generates an error message if a compile to 32-bit is attempted (on any platform, not just on Windows); nobody should be doing that. :->

[bhaller]

While we are at it, I am getting a new warning during compilation on Windows:

D:/a/SLiM/SLiM/core/slim_functions.cpp: In function 'double DoSummarizeOperation(Individual*, SummarizeOperation)':
D:/a/SLiM/SLiM/core/slim_functions.cpp:1295:1: warning: control reaches end of non-void function [-Wreturn-type]
 1295 | }
      | ^

Thoughts?

This is new code, recently pushed. The warning is spurious; there's no problem. I've just pushed a fix that should get rid of the warning.

[brnorris03]

If headers are different (e.g., in Windows), then the includes or both would be needed with appropriate macro guards. Is there a Windows example somewhere? I don't develop on Windows, so am not very familiar with what would be considered a system gsl there... The cmake configuration can be fairly easily extended to handle any number of GSL options.

[jeromekelleher]

It depends on how stuff is included. If it's like

#include <gsl/gsl_xyz.h>

then it can all be done by manipulating the "-I" options. This won't be windows specific --- if we can use an external GSL on Linux we should be able to do the same on Windows.

[bhaller]

At present, GSL header includes in SLiM are of the form:

#include "gsl_cdf.h"

And the GSL source/header files in SLiM have been modified to use this style of includes themselves. That's the only modification to them apart from tweaks in a couple of places to get rid of compiler warnings under Xcode. When I said above that "SLiM's GSL code is just a subset of the standard GSL, without modifications" I meant that there are no functional modifications; linking against a system GSL instead should produce no link issues and no change in behavior (IIRC).

So I suppose a CMake switch to use an external GSL would maybe do four things: (1) set a preprocessor flag that would make SLiM switch from "gsl_cdf.h" includes to <gsl/gsl_xyz.h> includes (with #ifdefs to achieve that added to the SLiM source files wherever GSL includes are done, which is not a huge number of places I think), (2) add the appropriate compile-time flag so those headers get found, (3) add the appropriate link option so the external GSL gets linked in, and (4) exclude the internal GSL files in SLiM from the link, in that case, to be certain that they are not used at all. If you are willing to make a PR for this happen, @brnorris03 , that would be great.

The only caveat is: please don't modify the SLiM source files in any way that would break the Xcode build. The present scheme, with the "gsl_cdf.h" style includes, is the only way I could find to get the build to work in Xcode; it's quite possible that there's a better way which I was too clueless to figure out, but, well, I didn't figure that out. :->

Re: Windows, I'd be happy to see SLiM build/run on Windows, but I have no Windows machine to test on, zero Windows development experience, and indeed, pretty much zero experience using Windows either. So if that is going to happen, it will be from PR's submitted by others, which I will be happy to accept; but if someone chooses to do that work, I hope they will also choose to maintain it in the future, as I will not be able to do so. The contribution would be most welcome; but I want to make it clear that I can't commit to doing it or to maintaining it in future, although I'll do my best not to carelessly break it :->.

[petrelharp]

Matt successfully built it on what they call the "windows subsystem for linux" (which is something like a linux subsystem for windows) out-of-the-box. However, we don't have a test machine for this any more, so we won't be much help on this.

[rdinnager]

I am mostly a Windows user @bhaller and I am very interested in getting a native Windows compile working. For slimr, I have been using Windows Subsystem for Linux, which is great, but I have plans that will require compiling on all platforms.
Therefore, I'd like to volunteer to try and get the windows build working and to maintain it. The only issue is that my C++ skills and compiler knowledge is limited, though it has grown by leaps and bounds recently, and I am a fairly quick study. I may need some help here and there though with the C++.

I think the most promising avenue for Windows is to compile with mingw64. I have made some progress already with it. I recently solved a major problem (I think) where mingw's <stdio.h> does not have getline() a POSIX function used a fair bit in SLiM. I finally found a platform-independent implementation in NetBSD, which hopefully will work as a drop-in replacement. I'm still working my way through compiler errors but am reasonably confident it will compile eventually. I generally agree that allowing an external gsl library is a good idea too, so I am happy to work on that too.

Ultimately the plan is to refactor slimr, which currently interfaces with the prebuilt binary, to use Rcpp instead. That is, the R package will compile the C++ code from SLiM during package installation, and I will write some extra C++ that will interface with R. This will mean R users of the package will not have to install SLiM separately, and slimr will run much faster. This will also allow injecting R code into SLiM code, so that callbacks can be written to visualize SLiM objects or output while SLiM is running, without having to drop out of the C++ execution back into R. Anyway, that is longer term plans which I would love to get your thoughts on. slimr is working pretty well as is, but I think it could eventually evolve into a more fully integrated API for SLiM. Kind of like how Pytorch is a python API for libtorch, a C++ engine. Speaking of which, has anyone expressed interest in creating a similar kind of Python API for SLiM? I know there is pyslim, but as far as I know you can't write a SLiM simulation with pyslim, correct?

[bhaller]

Hi @rdinnager,

I am mostly a Windows user @bhaller and I am very interested in getting a native Windows compile working. For slimr, I have been using Windows Subsystem for Linux, which is great, but I have plans that will require compiling on all platforms.
Therefore, I'd like to volunteer to try and get the windows build working and to maintain it. The only issue is that my C++ skills and compiler knowledge is limited, though it has grown by leaps and bounds recently, and I am a fairly quick study. I may need some help here and there though with the C++.

Sure, this sounds great. I'm glad that you mention maintaining this work; I'm not a Windows user at all, and have no Windows machine to build/test on, so if this is going to be a supported platform going forward (which would be great) then it will need someone who is willing to maintain it. I would expect that once you get it ported, maintaining it won't be a lot of additional work; 99.9% of SLiM's code ought to compile and run on Windows without modification, I would expect, and the code that will probably require special treatment is not generally code that's under active development. Hopefully the amount of #ifdef WINDOWS type cruft in the code can be minimized; I'll be happy to work with you on that, and to refactor the code as needed to make it easier to introduce Windows-specific code where it's needed.

I think the most promising avenue for Windows is to compile with mingw64. I have made some progress already with it. I recently solved a major problem (I think) where mingw's <stdio.h> does not have getline() a POSIX function used a fair bit in SLiM. I finally found a platform-independent implementation in NetBSD, which hopefully will work as a drop-in replacement. I'm still working my way through compiler errors but am reasonably confident it will compile eventually. I generally agree that allowing an external gsl library is a good idea too, so I am happy to work on that too.

Jerome mentioned Visual C++ above, and I would second that and very strongly urge you to focus on that as the supported compiler/platform. It's much more standard, I think, which is nothing to sneeze at; but the main reason is that the Qt widget kit supports only Visual C++, I believe (https://doc.qt.io/qt-5/windows.html). Qt is used extensively by SLiMgui for its user interface, and I think SLiMgui ought to be a part of your porting project here. That may sound daunting, but I think it should actually probably be quite easy. SLiMgui already builds on macOS and Linux with almost no platform-specific code; almost everything is handled transparently by Qt. Since Qt fully supports Windows as a platform, I think the port of SLiMgui ought to be trivial – in fact, it might "just work". (The handful of places where platform-specific code was need can be found easily by searching the QtSLiM code for #if.) The only obstacle to this has, in fact, been that the back-end SLiM code didn't build on Windows – but you plan to do that! So I would very strongly urge you to use Visual C++, to allow SLiMgui to be ported as well. I regard SLiMgui as a central part of the SLiM user experience – essential for both learning and using SLiM effectively – so from my perspective there's little point in porting the back end without porting SLiMgui as well.

Ultimately the plan is to refactor slimr, which currently interfaces with the prebuilt binary, to use Rcpp instead. That is, the R package will compile the C++ code from SLiM during package installation, and I will write some extra C++ that will interface with R. This will mean R users of the package will not have to install SLiM separately, and slimr will run much faster. This will also allow injecting R code into SLiM code, so that callbacks can be written to visualize SLiM objects or output while SLiM is running, without having to drop out of the C++ execution back into R. Anyway, that is longer term plans which I would love to get your thoughts on. slimr is working pretty well as is, but I think it could eventually evolve into a more fully integrated API for SLiM. Kind of like how Pytorch is a python API for libtorch, a C++ engine.

First of all, let me say that I'm really happy about your enthusiasm for SLiM and your interest in giving back to the community with new tools. That's really great, and it has been really nice to see this kind of community building around SLiM over time. That said, I hate to be negative since you're clearly excited about this, but to be quite frank: you are of course (!) free to pursue whatever you wish, but I think this is a bad idea and I do not support it. I apologize for my bluntness, but I want to make sure that I am clear here. I will explain my reasons.

The biggest reason is that the C++ APIs inside SLiM's code are not public APIs. They are not documented or supported, they are subject to change at any time without warning, and they do indeed change with great frequency. The internals of SLiM and Eidos are under very active development; even the most fundamental parts of the code change quite often, and more large changes are planned for the future. I cannot make any guarantees at all regarding internal code stability, as it would restrict my freedom to make SLiM better. This means that any attempt to provide an "integrated API" in R, or "injecting R code into SLiM code", or anything of that sort, would be built upon shifting sands, and would constantly be breaking. For the same reason, @jeromekelleher pushes back against people who want to build directly against the msprime C library.

Another reason is that it would restrict the user's freedom in ways that is probably not desirable. SLiM users very commonly build against the GitHub head version of SLiM in order to get a bug fix that they need, and that's important for me in supporting SLiM, too – if a user comes to me with a bug, it's important that I be able to tell them "OK, I think I have committed a fix for your bug to GitHub, please check out that version and verify that your bug has been fixed". If SLiM is built into slimr, it would cut those users off from that level of support. Support is also an issue from another angle: it is crucial, for finding and fixing bugs in SLiM, that I be able to reproduce bugs reported by users under the debugger on my machine. If they are reporting bugs that they observe while using slimr, and it's not straightforward for them to provide me with a reproducible case that runs without slimr, then there's a big problem.

I would also say that when I developed SLiM 2, I made a very deliberate decision to invent Eidos as the front-end language for SLiM rather than trying to base it on R or Python. Philipp and I discussed that for a long time, and I think it was the right decision; the Eidos manual has a page near the beginning that discusses the reasons for that decision in detail, which might interest you. It's great to have wrappers for command-line SLiM like slimr that make it quick and easy to use SLiM from those other languages; but fundamentally, that's the design decision I made and I'm sticking to it, and I can't commit to any kind of support for APIs written in a different language. The Eidos APIs, too, change from version to version; I try to maintain backward compatibility in Eidos, but I cannot commit to any effort to maintain backward compatibility in R or Python wrappers, and the Eidos APIs will often not map well to those other languages, I suspect. Beyond a certain point, I think users of "wrapper" packages will probably want to "graduate" to using SLiM directly, and I don't really see the point of trying to reinvent everything that Eidos does and everything SLiMgui does in another language. Eidos is really not that hard to learn, especially for R users. :-> I would instead focus on helping those advanced users "graduate", by making it easy to emit a SLiM model that contains, ideally, well-commented code that the user can read to understand how their slimr code turned into the corresponding SLiM code. I'm also rather concerned that ultimately using slimr ends up being even more complicated than SLiM already is, because now the user has to understand everything that SLiM does and everything about how slimr repackages and re-presents that functionality in a different way within R; is that really a win? As long as slimr remains simple and easy, the answer can be yes; but if it becomes a full re-implementation of all of SLiM's Eidos APIs in R form, etc., then I think the answer will very definitely be no. Take documentation, for example, of a method like, e.g., addCrossed() in Subpopulation (random example, any would do). The doc in the SLiM manual is extensive, and for the user to really understand what that method does, they will need to read that doc; but to understand that doc, they will really need to understand Eidos, since the doc uses things like Eidos method signatures and refers to Eidos concepts. If you wrapped addCrossed() in R, what would the user experience be? Either you have to write all of your own standalone doc (the SLiM manual is presently 672 pages :->), or the user would end up having to look up addCrossed() in two places now in order to understand what's going on. Not a win.

You also mention wanting to "visualize SLiM objects or output while SLiM is running". That's really what SLiMgui is for, and trying to replicate that sort of thing within R just sounds like reinventing the wheel to me. A tremendous amount of effort has gone into developing SLiMgui, and as I said earlier, I regard it as a core part of the SLiM user experience. I don't think any GUI front end based in R would be likely to provide a comparable experience. Instead of trying to do that, I wonder whether you might instead pursue the approach taken by slendr, another R-based SLiM wrapper package. It actually lets you sublaunch your model into SLiMgui, from a command in R, so that you can watch it run in SLiMgui, use the Eidos console, single-step, etc. I think that is a much more promising approach, and if there are ways that that can be made smoother for the user, I'd be interested in pursuing that. If there are aspects of graphing and visualization in SLiMgui that you think could be improved, I'm all ears, and I would welcome help with development of that app, if you want to get into doing that; I think that is a better path forward.

There are other concerns as well, such as problems with validation and testing with such a complex architecture; you would need to do all of your own validation and testing to ensure that all of your wrapped APIs work in all cases, and I can tell you from experience that that is a huge amount of effort. People publish based on their model results, of course, and such model results impact real-world decisions about things like conservation and management, so it's extremely important to be careful and thorough with validation and testing. Right now there are more than 6000 unit tests for Eidos, all individually hand-written by me, and almost 100,000 unit tests for SLiM; some of those are generated programmatically, but several thousand are again individually hand-written. All this takes enough time that, if we can get funding for it, we'd actually like to hire someone full-time just to handle testing and QA of SLiM. If you get into wrapping APIs and injecting code, you would need to do the same level of testing on your end. You may not realize what a commitment of resources that is.

To be clear: I think that there is value in slimr, and I'm glad you're doing it. I feel badly about the rather negative and discouraging tone I've had to take here; but making my opinion completely clear is the most important thing, I think. I feel quite strongly that there are limits to how far down this road it is wise to go. I hope that my reasons above make sense, and that I've talked you out of pursuing these plans.

Speaking of which, has anyone expressed interest in creating a similar kind of Python API for SLiM? I know there is pyslim, but as far as I know you can't write a SLiM simulation with pyslim, correct?

Yes, someone has been working on a Python front-end to SLiM, somewhat similar to slimr perhaps. They haven't quite gone public with it, so I won't say more here except to say that they plan only a relatively simple wrapper, not code injection with SLiM callbacks written in Python and so forth. And yes, this is not what pyslim is intended for, right; it is only for bridging tree sequences between SLiM and msprime/tskit. :->

Finally, on somewhat of a side note (but it's what this GitHub issue is actually about, I guess – actually it would be good to open a new issue for Windows support if you do intend to go forward with that :->): I'm not sure the "optional external GSL" is a good idea, although I'm open to it. Right now, SLiM is pretty tightly integrated with the GSL's random number generator, for purposes of speed; some code from the version of the GSL that SLiM uses has been copied into SLiM itself, inside inline functions, to allow the C++ compiler to inline aspects of random-number generation inside other functions. That makes a significant speed difference for some common types of models that hit the RNG hard, so I wouldn't really want to change it. It would of course be possible to make that tight integration optional, with a compile switch; but the non-integrated build of SLiM would then run more slowly than the standard integrated build of SLiM. I'm not sure that I see a compelling case for having that be a supported option. Of course if it is necessary in order to do the Windows port, then so be it; I'm just warning you that there will be significant obstacles in the path, and performance will likely suffer somewhat for the non-integrated build. :->

[bhaller]

Hi @bhaller,

Wow, thanks so much for the detailed thoughts on my suggested direction for slimr...

Great, sounds good.

For the record, slimr already supports converting slimr scripts into SLiM scripts and opening them automatically in SLiMGUI or QtSLiM. I guess what I was hoping to do was allow users to generate whatever custom visualisation they wanted, but now that I hear your thoughts, perhaps that sort of system would be so complicated anyway, that users might as well learn Qt and make a custom plugin for QtSLiM? Is this possible right now, out of curiousity?

Well, if someone is sufficiently comfortable hacking away in C++ and Qt, they can do whatever they like, but there's no support for plug-ins and AFAIK nobody has ever tried. Not recommended, really. :-> But in SLiMgui you can do live custom plotting in R that gets displayed in SLiMgui; see section 14.8 of the manual. This works pretty well for simple stuff, but wouldn't work so well for more complex stuff. I might add actual plotting support in Eidos/SLiMgui at some point; it's an idea I've been bouncing around, but implementing a whole plotting library inside Eidos/SLiMgui doesn't really excite me – I dislike reinventing the wheel. I've been thinking that maybe the right path would be to add a few simple plotting functions to make, e.g., a barplot or a scatterplot or a lineplot from a given dataset passed to the function, without getting into making a general-purpose plotting library like R's base plotting library. So far you're the first person to raise it as an issue. Perhaps you'll want to take a crack at this after you finish porting to Windows. ;->

I will open another issue for the Windows compilation project. Just a few comments now. MinGW is also a standard and well-supported compilation toolchain on Windows...

Aha, thanks for the background info, very helpful. If it works well with Qt and conda then it sounds like it may indeed be the best choice! I missed that Qt compatibility note, good that you caught it. :->

Sticking with the internal SLiM gsl library version should be fine I think, I will start with this and let you know if any issues with it crop up.

OK, sounds good. I gather that the full GSL doesn't build on Windows without extensive modifications, but SLiM actually only incorporates a small subset of the full GSL, and the subset that it incorporates might build fine on Windows without modification, so there might turn out to be no issue there, or minor issues might be solvable with #ifdefs taken from some existing Windows-ported version of the GSL code.

Thanks again for your thoughts. I will read them over carefully and think more about this. In the mean time I will keep working on the Windows compilation.

And @jeromekelleher I will definitely want to pick your brain about conda and compiling in general. Thanks so much for the offer.

OK, all sounds excellent. Thank you for taking my comments in the spirit in which they were intended, and let me know if you need anything from me. :->

[rdinnager]

Some progress over here. I have got eidos to compile on Windows (will work on slim next). One issue though, I can't properly test eidos because my Windows system doesn't have a /tmp directory. We will have to fiddle with the eidos source code a bit to get around this. e.g:

The last command was just the following simple script, just to see if it worked at all:

x = 5:6; x = x - 3:4; x;

In order to get this far I have made substantial use of the gnulib project (https://www.gnu.org/software/gnulib/), which is a general GNU portability library. This has helped tremendously. The only problem is that gnulib is designed for use with autoconf, not cmake. To get around this I have imported the required parts of gnulib as an external project in cmake (e.g. using ExternalProject_Add), which runs a configure script and then compiles a static library. This does add considerably to the build time, largely because gnulib does a lot of checks on the user's system to configure the correct compatibility requirements. There may be a way to get around some of this, but in the mean time the extra compile time seems worth it to get it working on Windows relatively quickly.

Next I will work on the slim executable, and then finally QtSLiM. Once it is working, I was thinking I could make a pull request to a side branch, while it is being tested? At that point I will also wrap this up as a msys2 package through the pacman package manager. It should be then straightforward to write a conda package that installs through msys2. For users who want to compile themselves, they would only need to install msys2 on their Windows system, and then the cmake pacman package.

Anyway, I will report back if I run into any problem with compiling slim or QtSLiM, or otherwise if I'm successful.

[rdinnager]

Okay, slim compiled! I have the same testing issue because of the /tmp thing, so that will need some attention. However, I ran Recipe 4.2.1, and it worked seemingly without issue.

Note the Warning about Eidos_GetMaxRSS(), this is one of the compromises I had to make to get it to work, which will need attention eventually. This is because getrlimit() does not exist in windows, and I don't know what the appropriate replacement is, so for now I've basically disabled this function and it prints a warning instead, which is not great, but should be okay for testing purposes.

On to QtSLiM next...

[bhaller]

Hi @rdinnager. This all sounds great. Rapid progress!

Re: the lack of /tmp, if there's somewhere else that can be used as a temporary directory you could use that, otherwise the filesystem tests could be disabled on Windows if you test by hand that all of the functions do what they're supposed to do, I suppose.

Re: gnulib I have no guidance for you. I have little understanding of cmake, and no understanding of autoconf, and I've never heard of gnulib before. Sorry; when it comes to build system stuff I just limp along cluelessly. Figuring out qmake has been an adventure. :-> For QtSLiM you will want to use qmake, BTW, almost certainly, unless it doesn't work on Windows or something.

Re: a PR on a side branch, sure, sounds good to me. Let me know if you need me to do anything.

msys2, pacman, conda: I know nothing about this. Perhaps @jeromekelleher or @bernard-kim has feedback on this? (Bernard, I thought I had tagged you on this before but it looks like I didn't!)

Re: Eidos_GetMaxRSS(), probably not a big deal; this is not crucial to the functioning of anything, but I think it will prevent the -mem command-line option from working, so it'd be nice to fix it since that's quite a useful option.

Recipe 4.2.1 running: great! If one recipe runs then probably most of them run. :-> What about slim -tS?

Once you've got things kinda stable, perhaps the next step will be to get you up to speed on running my suite of SLiM model tests. That lives at https://github.com/MesserLab/SLiM-Tests but probably will require a little explanation, I forget if there's any doc there about how that repo is to be used.

[rdinnager]

Okay, QtSLiM compiled and running on Windows. I've tried a bunch of recipes and they all seem to run fine so far.

I do get a tonne of output to the debug console of this variety:

14:33:06 : ***  method signature mismatch:
old: "ûá(void)setSpatialBounds(numericábounds) "
new: "ûá(void)setSpatialBounds(numericábounds)"
14:33:06 : ***  method signature mismatch:
old: "ûá(void)setSubpopulationSize(integer$ásize) "
new: "ûá(void)setSubpopulationSize(integer$ásize)"
14:33:06 : ***  method signature mismatch:
old: "ûá(string)spatialMapColor(string$áname, numericávalue) "
new: "ûá(string)spatialMapColor(string$áname, numericávalue)"
14:33:06 : ***  method signature mismatch:
old: "ûá(float)spatialMapValue(string$áname, floatápoint) "
new: "ûá(float)spatialMapValue(string$áname, floatápoint)"
14:33:06 : ***  method signature mismatch:
old: "ûá(object<Individual>)subsetIndividuals([No<Individual>$áexcludeá=áNULL], [Ns$ásexá=áNULL], [Ni$átagá=áNULL], [Ni$áminAgeá=áNULL], [Ni$ámaxAgeá=áNULL], [Nl$ámigrantá=áNULL]) "
new: "ûá(object<Individual>)subsetIndividuals([No<Individual>$áexcludeá=áNULL], [Ns$ásexá=áNULL], [Ni$átagá=áNULL], [Ni$áminAgeá=áNULL], [Ni$ámaxAgeá=áNULL], [Nl$ámigrantá=áNULL])"
14:33:06 : ***  method signature mismatch:
old: "ûá(void)takeMigrants(object<Individual>ámigrants) "
new: "ûá(void)takeMigrants(object<Individual>ámigrants)"
14:33:06 : orphan line while reading for top level heading  "SLiM Classes" :  "id => (integer$) "
14:33:06 : orphan line while reading for top level heading  "SLiM Classes" :  "The identifier for this mutation.á Each mutation created during a run receives an immutable identifier that will be unique across the duration of the run, and that identifier is carried over to the Substitution object when the mutation fixes. "
14:33:06 : orphan line while reading for top level heading  "SLiM Classes" :  "fixationGeneration => (integer$) "
14:33:06 : orphan line while reading for top level heading  "SLiM Classes" :  "The generation in which this mutation fixed. "
14:33:06 : orphan line while reading for top level heading  "SLiM Classes" :  "mutationType => (object<MutationType>$) "
14:33:06 : orphan line while reading for top level heading  "SLiM Classes" :  "The MutationType from which this mutation was drawn. "

Besides the encoding issue, which could be a problem of the Windows terminal, I'm not sure, it looks like a lot of mismatches because of a trailing space in one version? I wonder if it is a line-endings thing?

I also get this:

14:33:06 : *** no documentation found for class  "Chromosome"
14:33:06 : *** no documentation found for class  "Individual"
14:33:06 : *** no documentation found for class  "InteractionType"
14:33:06 : *** no documentation found for class  "Genome"
14:33:06 : *** no documentation found for class  "GenomicElement"
14:33:06 : *** no documentation found for class  "GenomicElementType"
14:33:06 : *** no documentation found for class  "LogFile"
14:33:06 : *** no documentation found for class  "Mutation"
14:33:06 : *** no documentation found for class  "MutationType"
14:33:06 : *** no documentation found for class  "SLiMEidosBlock"
14:33:06 : *** no documentation found for class  "SLiMSim"
14:33:06 : *** no documentation found for class  "Substitution"
14:33:06 : *** no documentation found for class  "Subpopulation"
14:33:06 : *** no documentation found for class  "SLiMgui"
14:33:06 : willExecuteScript: Unable to set the working directory to  ~  (error  2 )
14:33:06 : willExecuteScript: Unable to set the working directory to  ~  (error  2 )

The help windows works, but properties of classes seem to be missing?

I've disabled tilde expansion because that isn't really a thing in Windows, and also the code in SLiM to do it uses POSIX functions that don't exist in Windows (e.g. <pwd.h> doesn't exist), so that explains the error, but I'm not sure how to fix it if '~' is used a lot in paths in the source code?

But anyway, it does appear to be mostly working, which is exciting!

[rdinnager]

By the way, my working version of the project is here: https://github.com/rdinnager/SLiM/tree/windows-build
I'm not sure @bhaller but you may have to create a windows-build branch for me to send a pull request to.
I think I will try and solve the /tmp folder issue first so I can run the tests. I'm sure I will find some issues somewhere in the tests.. I have a feeling.

[bhaller]

Hi @rdinnager. I made a new branch for you called windows_port that you can send PRs to. If you want to get stuff off of your machine before you start worrying about cleaning up little details, feel free. That would also allow any other interested parties to give what you've done a whirl.

QtSLiM running, that's great! It looks like there are some font size issues, with some things being too small and others much larger. That's something that Qt unfortunately doesn't standardize across platforms well, because every platform has its own fonts and its own type-rendering layer. You'll find that I've got #ifdefs in some spots already to adjust the font sizes between macOS and Linux; you might see if it is obvious to you how to use the same spots to make the font sizes a bit better on Windows.

It also looks like the vertical splitter in the window looks a little funny, and the horizontal splitter seems to be completely invisible. Those issues may not be terribly easy to fix, but neither are they showstoppers for usability. If buttons track properly, menu items and keyboard shortcuts work as they ought to, simulations run, and the Eidos console work, that's the most important stuff. I may need to get access to a Windows machine at some point to clean up the smaller details. (Where are you located, by the way, Russell? I'm in Ithaca NY.)

It looks like the help panel is pretty broken, though, and that is more of an issue for usability. Hard to guess what's going wrong or whether it'll be hard to fix. It might be a line-ending problem, indeed. You can search the code base for that error message "method signature mismatch:" and you'll find it's inside the help window's processing code. It reads in HTML files and parses out the help items they contain and where in the outline of the help window they should appear. You might be able to just strip off that extra character in a key spot and have everything suddenly start working. There might also be a Unicode issue; some of the characters in the help files are Unicode characters, and Windows might be trying to read the files in in some Windows text encoding rather than in Unicode. Hard to guess. I don't know whether you've ever used a debugger, but if not, now's your chance to learn. :-> Once the HTML files are being correctly read and parsed, all of the console logs should disappear; those logs are just telling you that it isn't working yet.

The ~ thing, well, yes. There will probably be various issues with Windows vs. Linux paths. The slashes go in the opposite direction, too. It may ultimately be the case that Eidos will want to add some new cross-platform APIs that help people with filesystem paths: things like "give me a path that starts with this path and adds this file/folder to the end of it", using the correct / or \ for the platform. See what you think; I'm happy to write such helper functions as needed. IIRC @hyanwong has raised this question before, but at the time I think my answer was "we have no plans to port SLiM to Windows ever, so it doesn't matter". Maybe he'll have some interest in what's going on over here now. :->

The /tmp folder, well, you might just create a folder somewhere that the user has permissions – inside their documents folder or something. To test that the filesystem calls work, it needs to read/write to the filesystem somewhere. :->

Overall, I'm amazed. Qt for the win. But it would be good to use QtSLiM to actually do some stuff, and see whether it behaves in the way that you would expect a Windows app to act, and whether everything looks/acts as it should. You might try all the panels and options – the about box, setting debug points, the Eidos variable browser, etc.

Good stuff, thanks Russell! I made the branch for you; let me know when you need something else. Happy to review a PR any time. When you get to the point where this could be made public, it would be great if you could do a complete writeup about it, following the style of other sections of chapter 2 of the manual, detailing exactly how to build it and run it on Windows, what versions of what software are needed and where to get that software from, how to configure/integrate it with the system, screenshots of any key steps involving panels or whatever, and so forth. Then I can merge that in to the manual. What does the timeframe for this feel like? It seems like it has moved forward very fast and might be ready to go quite soon. I'm planning on doing a SLiM 3.7 release fairly soon (maybe a week or two?), once a few other things are polished up; might this make it into that release?

[bhaller]

Oh, and – if you get to the point where you need my help with the code to understand the right place/way to fix a bug, like the /tmp thing, we could do a zoom with screen-share any time. Happy to help as needed. Thanks!

[bhaller]

Great! The filesystem issues are expected, they'll get sorted out. :->

This is the testbed I use for speed testing:

https://github.com/MesserLab/SLiM-Tests

The R script is not designed to simply be run as-is, but for bits and pieces of it to be selectively run in an interactive environment. It's kind of a weird testbed, let me know if you need guidance on how to use it. There's some info in the README though, maybe that will suffice. It's just hacked together to suit my purposes, there are many ways in which it could be improved and I'm not proud of the code. :->

OK, no worries if it doesn't make version 3.7. It'll just come out in 3.8. :-> Congrats on Florida, I hope the new job goes well! Don't buy a house in a flood zone. :->

I'm in the middle of another thing, so I'll look at the PR in a little while. I'd like it to have all the includes cleaned up first anyway, so that far fewer files are touched, as discussed above; that'll make it much easier to grok the PR. You said you can just add a -include to the compile line, right?

Yep, you are most definitely the first! Congrats! :->

[rdinnager]

I've taken a closer look at SLiMgui in Windows and compared it with the screenshot in the github readme. I think there are a few things missing. Here is my Windows screenshot:

Now here is the readme screenshot:

Most obviously, the fitness values and selection coefficients legends are missing. Also, the little arrow next to the pointing finger is not there. Any thoughts on what could have gone wrong here? Which source files in QtSLiM are responsible for these parts of the window? I can try and see if I can see anything in them that might have led to this issue.

[rdinnager]

Also, just to update you, I managed to solve the help window issues, so the help looks like it is all fine now. I get no more output to the console of errors (except for those relating to the ~, when it tries to set the working directory to "~/Desktop"). The issue was line-endings. Apparently if you just add QIODevice::Text to the open command when opening the html files, this tells Qt to replace linux line-endings with Windows ones. So it was a simple fix. When I did this, all the errors went away.

[bhaller]

I've taken a closer look at SLiMgui in Windows and compared it with the screenshot in the github readme...

The screenshot in the GitHub readme is out of date. :-> None of the differences you noticed are a problem; those UI elements have been removed. Perhaps the best thing for you to do would be to compare your Windows SLiMgui against SLiMgui running under the WSL...?

[bhaller]

Also, just to update you, I managed to solve the help window issues, so the help looks like it is all fine now. I get no more output to the console of errors (except for those relating to the ~, when it tries to set the working directory to "~/Desktop"). The issue was line-endings. Apparently if you just add QIODevice::Text to the open command when opening the html files, this tells Qt to replace linux line-endings with Windows ones. So it was a simple fix. When I did this, all the errors went away.

Yep, I saw your commit fixing that. :-> Good stuff! Nice when these things turn out to be simple.

[rdinnager]

I've taken a closer look at SLiMgui in Windows and compared it with the screenshot in the github readme...

The screenshot in the GitHub readme is out of date. :-> None of the differences you noticed are a problem; those UI elements have been removed. Perhaps the best thing for you to do would be to compare your Windows SLiMgui against SLiMgui running under the WSL...?

Definitely except I haven't managed to get it running under WSL yet (well I did at one point but Windows updates broke it awhile ago). Getting graphical stuff to run in WSL can still a bit of a pain, but maybe I will have another go at it. I haven't tried with the new builtin GUI support for WSL yet. Anyway, good to know those missing things are not actually missing! I won't use that screenshot as reference anymore.

[bhaller]

I've taken a closer look at SLiMgui in Windows and compared it with the screenshot in the github readme...

The screenshot in the GitHub readme is out of date. :-> None of the differences you noticed are a problem; those UI elements have been removed. Perhaps the best thing for you to do would be to compare your Windows SLiMgui against SLiMgui running under the WSL...?

Definitely except I haven't managed to get it running under WSL yet (well I did at one point but Windows updates broke it awhile ago). Getting graphical stuff to run in WSL can still a bit of a pain, but maybe I will have another go at it. I haven't tried with the new builtin GUI support for WSL yet. Anyway, good to know those missing things are not actually missing! I won't use that screenshot as reference anymore.

Bernard's instructions in the manual seem to work well for people to get it running under the WSL (or at least I haven't gotten complaints). Yeah, sorry for the out-of-date screenshot; SLiMgui's UI changes pretty often, so I don't bother updating the screenshot all the time. Funny how these things come back to bite you later! :-O Anyhow, let me know if there's something I ought to do to help; I can certainly update that screenshot if you need me to, but of course there's lots of UI in SLiMgui that isn't shown in the screenshot, so that's only a partial impression of things anyway.

[rdinnager]

Unfortunately, wslg (the new WSL graphical mode available in Windows Insider builds) does not work with SLiMgui, I just tried it :( . So only the more complicated Xserver based solution works with WSL so far. Oh well.

[rdinnager]

Okay, so I am at a point where it makes sense to try and deal with some of the compiler warnings I am getting. So far I have just been ignoring them because the end result seems to work, but we should probably try and figure out what they are and at least quiet them if they are harmless. I need some help with this because I don't have enough C++ experience to know what any of these mean.

The first one is during compilation of tskit:

F:/SLiM/treerec/tskit/core.c: In function 'get_random_bytes':
F:/SLiM/treerec/tskit/core.c:47:29: warning: initialization of 'HCRYPTPROV' {aka 'long long unsigned int'} from 'void *' makes integer from pointer without a cast [-Wint-conversion]
   47 |     HCRYPTPROV hCryptProv = NULL;
      |                             ^~~~
F:/SLiM/treerec/tskit/core.c:57:20: warning: assignment to 'HCRYPTPROV' {aka 'long long unsigned int'} from 'void *' makes integer from pointer without a cast [-Wint-conversion]
   57 |         hCryptProv = NULL;
      |                    ^
F:/SLiM/treerec/tskit/core.c:60:16: warning: assignment to 'HCRYPTPROV' {aka 'long long unsigned int'} from 'void *' makes integer from pointer without a cast [-Wint-conversion]
   60 |     hCryptProv = NULL;
      |                ^
F:/SLiM/treerec/tskit/core.c:63:20: warning: comparison between pointer and integer
   63 |     if (hCryptProv != NULL) {
      |                    ^~

Next, I get a warning while compiling slim:

F:/SLiM/core/slim_sim_eidos.cpp: In function 'void PrintBytes(std::ostream&, size_t)':
F:/SLiM/core/slim_sim_eidos.cpp:47:38: warning: integer overflow in expression of type 'long int' results in '0' [-Woverflow]
   47 |  if (p_bytes > 1024L * 1024L * 1024L * 1024L)
      |                ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~

and another for eidos:

F:/SLiM/core/slim_sim_eidos.cpp: In member function 'EidosValue_SP SLiMSim::ExecuteMethod_rescheduleScriptBlock(EidosGlobalStringID, const std::vector<Eidos_intrusive_ptr<EidosValue> >&, EidosInterpreter&)':
F:/SLiM/core/slim_sim_eidos.cpp:3431:18: warning: 'stage' may be used uninitialized in this function [-Wmaybe-uninitialized]
 3431 |   CheckScheduling(start, stage);

Next is a QtSLiM warning:

F:/SLiM/QtSLiM/QtSLiMExtras.cpp: In function 'QString stringForByteCount(uint64_t)':
F:/SLiM/QtSLiM/QtSLiMExtras.cpp:533:38: warning: integer overflow in expression of type 'long int' results in '0' [-Woverflow]
  533 |     if (bytes > 512L * 1024L * 1024L * 1024L)
      |                 ~~~~~~~~~~~~~~~~~~~~~^~~~~~~

That is all of them. Any idea what the problems are here?

[bhaller]

Unfortunately, wslg (the new WSL graphical mode available in Windows Insider builds) does not work with SLiMgui, I just tried it :( . So only the more complicated Xserver based solution works with WSL so far. Oh well.

Huh. I thought @bernard-kim had tried this and found that it worked. See section 2.4.4 of the SLiM manual.

[rdinnager]

Unfortunately, wslg (the new WSL graphical mode available in Windows Insider builds) does not work with SLiMgui, I just tried it :( . So only the more complicated Xserver based solution works with WSL so far. Oh well.

Huh. I thought @bernard-kim had tried this and found that it worked. See section 2.4.4 of the SLiM manual.

The instructions in section 2.4.4 do work, they are just very complicated and a little fragile (Windows updates can break the setup). Microsoft has released in the preview version of Windows 10 (and 11) a new WSL that has GUI support built in, called WSLg (it doesn't require all the Xserver stuff, it just works). I was hoping this would work with SLiMgui but it seems like it doesn't. I have tried it with other software, like chrome and rstudio and it works great, but something is wrong with SLiMgui. Anyway, perhaps it doesn't matter so much anymore now that we will soon have a native Windows SLiMgui, but it would have been nice for testing. Anyway, I think I found an alternative, using a remote desktop into my work's linux cluster.

[bhaller]

So I guess you'll want to update your PR to be based on the latest SLiM GitHub head, to get the changes I just committed. I think the usual way to do that is with a rebase? My git-fu is extremely weak, so I'm not sure; hopefully you know how to do this sort of thing!

[bhaller]

Unfortunately, wslg (the new WSL graphical mode available in Windows Insider builds) does not work with SLiMgui, I just tried it :( . So only the more complicated Xserver based solution works with WSL so far. Oh well.

Huh. I thought @bernard-kim had tried this and found that it worked. See section 2.4.4 of the SLiM manual.

The instructions in section 2.4.4 do work, they are just very complicated and a little fragile (Windows updates can break the setup). Microsoft has released in the preview version of Windows 10 (and 11) a new WSL that has GUI support built in, called WSLg...

Yep, I'm aware of all this. Section 2.4.4 discusses getting SLiMgui to work with the WSLg, thanks to Bernard. It seems to work for him, and I've gotten no complaints from others. Maybe Microsoft broke it again with a new version of the WSLg, but it used to work.

Anyway, perhaps it doesn't matter so much anymore now that we will soon have a native Windows SLiMgui, but it would have been nice for testing. Anyway, I think I found an alternative, using a remote desktop into my work's linux cluster.

Well, it'd be good for it to get fixed; my guess is that a substantial user base will want to continue using SLiM under the WSL, because Linux. But that's a problem for @bernard-kim, I guess. Maybe you could open a new GitHub issue on this, and provide details regarding your exact setup, and exactly how SLiMgui fails to work under the WSLg? Then maybe I can assign that issue to Bernard, if he's willing. :->

[rdinnager]

Unfortunately, wslg (the new WSL graphical mode available in Windows Insider builds) does not work with SLiMgui, I just tried it :( . So only the more complicated Xserver based solution works with WSL so far. Oh well.

Huh. I thought @bernard-kim had tried this and found that it worked. See section 2.4.4 of the SLiM manual.

The instructions in section 2.4.4 do work, they are just very complicated and a little fragile (Windows updates can break the setup). Microsoft has released in the preview version of Windows 10 (and 11) a new WSL that has GUI support built in, called WSLg...

Yep, I'm aware of all this. Section 2.4.4 discusses getting SLiMgui to work with the WSLg, thanks to Bernard. It seems to work for him, and I've gotten no complaints from others. Maybe Microsoft broke it again with a new version of the WSLg, but it used to work.

Anyway, perhaps it doesn't matter so much anymore now that we will soon have a native Windows SLiMgui, but it would have been nice for testing. Anyway, I think I found an alternative, using a remote desktop into my work's linux cluster.

Well, it'd be good for it to get fixed; my guess is that a substantial user base will want to continue using SLiM under the WSL, because Linux. But that's a problem for @bernard-kim, I guess. Maybe you could open a new GitHub issue on this, and provide details regarding your exact setup, and exactly how SLiMgui fails to work under the WSLg? Then maybe I can assign that issue to Bernard, if he's willing. :->

Oh, sorry. I don't see the section on WSLg in the manual. Maybe I'm looking at an old version? Maybe it is just my linux version. I am using Ubuntu-18.04, and I think the manual mentions it is only tested on Ubuntu-20.04. I should have checked this first. I will let you know if this fixes it.

[bhaller]

Unfortunately, wslg (the new WSL graphical mode available in Windows Insider builds) does not work with SLiMgui, I just tried it :( . So only the more complicated Xserver based solution works with WSL so far. Oh well.

Huh. I thought @bernard-kim had tried this and found that it worked. See section 2.4.4 of the SLiM manual.

The instructions in section 2.4.4 do work, they are just very complicated and a little fragile (Windows updates can break the setup). Microsoft has released in the preview version of Windows 10 (and 11) a new WSL that has GUI support built in, called WSLg...

Yep, I'm aware of all this. Section 2.4.4 discusses getting SLiMgui to work with the WSLg, thanks to Bernard. It seems to work for him, and I've gotten no complaints from others. Maybe Microsoft broke it again with a new version of the WSLg, but it used to work.

Anyway, perhaps it doesn't matter so much anymore now that we will soon have a native Windows SLiMgui, but it would have been nice for testing. Anyway, I think I found an alternative, using a remote desktop into my work's linux cluster.

Well, it'd be good for it to get fixed; my guess is that a substantial user base will want to continue using SLiM under the WSL, because Linux. But that's a problem for @bernard-kim, I guess. Maybe you could open a new GitHub issue on this, and provide details regarding your exact setup, and exactly how SLiMgui fails to work under the WSLg? Then maybe I can assign that issue to Bernard, if he's willing. :->

Oh, sorry. I don't see the section on WSLg in the manual. Maybe I'm looking at an old version? Maybe it is just my linux version. I am using Ubuntu-18.04, and I think the manual mentions it is only tested on Ubuntu-20.04. I should have checked this first. I will let you know if this fixes it.

You may be looking at an old version. It starts: "There are two ways to approach running SLiMgui under the WSL. The first way is to use Microsoft’s new GPU-accelerated WSL 2, called the WSLg. At the time of this writing, the WSLg is currently in “preview”, so this is prerelease software. Also, this method of running SLiMgui has, thus far, only been demonstrated to work with Ubuntu 20.04 LTS; Ubuntu 18.04 produced only a black window, and other Linux distros have not been tried. To use the WSLg you will have to...". And yes, 18.04 doesn't work with the WSLg apparently.

[bhaller]

So I guess you'll want to update your PR to be based on the latest SLiM GitHub head, to get the changes I just committed. I think the usual way to do that is with a rebase? My git-fu is extremely weak, so I'm not sure; hopefully you know how to do this sort of thing!

Yes, I can just fetch upstream. You can actually do it right from github if you want, no need for command line..

I don't think your rebase worked the way it should have. Your PR is now showing changes in EidosScribe/EidosTextView.mm that come from my commit. If your PR is rebased correctly those changes should not be part of it. Let me know when this is resolved, and I'll resume reviewing your diffs. Tread carefully; make sure you don't lose any work while you're trying to fix this problem. Backups are good. :->

[jeromekelleher]

A few small notes here:

1. We're happy to make any changes needed to make tskit compile on mingw - please open an issue on tskit and we'll follow up there.
2. Annoyingly, the C standard only guarantees that longs hold 32 bit numbers. See here e.g. If you want a 64 bit number, you have to use long long, but since fixed width types are part of C99, int64_t is easier to follow, I think. (I assume the same is true of recent C++ standards)
3. My expectation would be that running any linux GUI under WSL will be fragile, and getting a native QT version up and running wouldn't be much more work.

[rdinnager]

SLiM_WarmUp

Hmm, that's odd. I'm a bit concerned that standard data type sizes might be different with your compiler. long should be 8 bytes; 1024^4 should fit very easily into long. Check your data type sizes. I just pushed a commit that provides, at the end of SLiM_WarmUp(), three lines that print the size of the major integer data types. Uncomment those lines and run a model, any model. The sizes should be 4 for int (8 is also OK), and at least 8 for long and size_t.

If long is only 4 bytes, then we have a problem that will require a bigger fix. I find that hard to imagine, though; 8 bytes for long is pretty much universal nowadays, I think. So this looks like a compiler bug to me, if long is indeed 8 bytes; you might even try reporting it to the compiler provider! But in any case, the commit I just pushed also changes the 1024L occurrences there to 1024.0 instead; working in floating point instead of integer should be fine there, and should silence the warning.

I just compiled with your latest commits and uncommenting the lines you added to SLiM_WarmUp(), and this is the print out:

sizeof(int) == 4
sizeof(long) == 4
sizeof(size_t) == 8

So indeed long is 4 bytes. Your change of 1024L to 1024.0 did indeed silence the warning.
The version of gcc that the MinGW compiler in MSYS2 that I am using is version 10.3.0.

I found some information on this here: https://newbedev.com/sizeof-long-in-64-bit-c
It seems like Visual C++ defines long as 4 bytes, and MinGW has followed suite to stay compatible. The reasoning seems to be to maintain compatibility with 32bit code. Now that Microsoft has said they plan to abandon 32bit support for Windows 10, this may eventually change, though probably not for awhile, as I'm sure they will want to keep backwards compatibility for as long as they can.

Anyway, looks like you may have to go on your search and destroy mission after all, sorry!

[rdinnager]

Oops, looks like I missed a couple compiler warnings in my last post on this. Here are a few more:

In eidos_zlib I get a warning:

In file included from C:/Projects/SLiM/eidos_zlib/zlib.h:51,
                 from C:/Projects/SLiM/eidos_zlib/zutil.h:22,
                 from C:/Projects/SLiM/eidos_zlib/adler32.c:8:
C:/Projects/SLiM/eidos_zlib/zconf.h:31:33: warning: function declaration isn't a prototype [-Wstrict-prototypes]
   31 | #  define adler32_combine64     z_adler32_combine64
      |                                 ^~~~~~~~~~~~~~~~~~~
C:/Projects/SLiM/eidos_zlib/adler32.c:180:15: note: in expansion of macro 'adler32_combine64'
  180 | uLong ZEXPORT adler32_combine64(adler1, adler2, len2)
      |               ^~~~~~~~~~~~~~~~~

And another very similar one:

In file included from C:/Projects/SLiM/eidos_zlib/zlib.h:51,
                 from C:/Projects/SLiM/eidos_zlib/zutil.h:22,
                 from C:/Projects/SLiM/eidos_zlib/crc32.c:31:
C:/Projects/SLiM/eidos_zlib/zconf.h:40:33: warning: function declaration isn't a prototype [-Wstrict-prototypes]
   40 | #  define crc32_combine64       z_crc32_combine64
      |                                 ^~~~~~~~~~~~~~~~~
C:/Projects/SLiM/eidos_zlib/crc32.c:436:15: note: in expansion of macro 'crc32_combine64'
  436 | uLong ZEXPORT crc32_combine64(crc1, crc2, len2)
      |               ^~~~~~~~~~~~~~~

That's it. The only other warning now is the tskit one from before. @bhaller, should I open an issue for this with them, or do you want to do it?

[bhaller]

A few small notes here:

1. We're happy to make any changes needed to make tskit compile on mingw - please open an issue on tskit and we'll follow up there.

Logged tskit-dev/tskit#1741.

2. Annoyingly, the C standard only guarantees that longs hold 32 bit numbers. See here e.g. If you want a 64 bit number, you have to use long long, but since fixed width types are part of C99, int64_t is easier to follow, I think. (I assume the same is true of recent C++ standards)

OK, I'll search and destroy long in favor of int64_t, thanks.

3. My expectation would be that running any linux GUI under WSL will be fragile, and getting a native QT version up and running wouldn't be much more work.

Yeah, @rdinnager is making rapid progress on the native build!

[bhaller]

Hi @rdinnager. Looks like this has now gone through on pacman, so that's awesome. Is there anything else that needs to happen besides the writeup for the manual? Let me know if you need an assist with anything. This is so close now, it would be great if it made SLiM 3.7 (which still does not have a definite ship date set, and is just trundling along :->).

[bhaller]

Hi @rdinnager. Just a ping. :->

[rdinnager]

@bhaller Just to refresh my memory, windows_port has not been merged into master yet at any point, correct? Would you be able to merge changes to master into windows_port, so I can try to rebase against that again? I just fixed one last issue and would like to make a pull request but want to be sure everything is up to date first.

[bhaller]

@rdinnager Just did another merge, I believe. You are correct, the windows_port branch has not yet been merged back into master; I wasn't sure whether there would be little things you'd want to do before that happened. I'm happy to do it any time you say you're ready.

[bhaller]

@rdinnager Looks like three out of the four GUI builds failed, perhaps because some server didn't respond to the request to download Qt or something? I didn't look hard at it but that's what it looked like. Interestingly, the CLI builds succeeded, even though they are presently failing for master. I'm guessing you've got the tskit-based post-build tests disabled on Windows? That's what's failing for master, because we need some changes on the tskit side to happen that have not yet been rolled into a release. Anyhow, some messiness there.

[rdinnager]

Yes, looks like the GUI fails are to do with Qt failing to download. I have no idea what is going on with the tskit tests. I didn't disable any post build tests, and even if I did, I would have only disabled them for Windows, but these tests are running on linux and mac... weird..

[bhaller]

OK, well, I'm not worrying about CI too much right now as long as the problems are outside of SLiM's own code.

[bhaller]

Hi @rdinnager. As it happens, there is a CS student at Cornell who is used to working on Windows and is interested in finding some small contribution she could make to SLiM. Are there any remaining loose ends that were beyond your programming knowledge to address? If so, maybe she could take them on. But IIRC you figured out pretty much everything, right?

[bhaller]

Where's the PR for this? I can take a look at the failing tskit tests.

It has been merged into master, since the failing tests seemed to be the only problem with it, and it seemed clear that they were failing because the new tskit changes (particularly removing the ordering requirement for the individuals table) hadn't been released in a new tskit. So now the tests on master fail; if there's anything that can be done about that before the next tskit gets released, that would be nice. :->

Oh, and also tests failing because the Qt download server seemed to be down; that has probably now been fixed on their end.

[rdinnager]

Hey @bhaller, this is just a heads up that testing on master for the windows installation did not go well. The previous 'fix' I merged, which seemed to fix the my CLI build initially (locally), in fact broke all builds. I setup some initial windows tests on github actions, and they are currently all failing, and all for different reasons (https://github.com/rdinnager/SLiM/runs/4082356544). I have leads on what is wrong for most of them, but I will need a little time to sort this all out, so just letting you know in case you were considering making a new release soon.

[bhaller]

@rdinnager OK, thanks. So far no date for SLiM 3.7, I'm just surfing the waves of uncertainty. :-> I'll wait for a fix from you. Let me know if you need a hand with anything; thanks for following up.

[rdinnager]

So for tests I am running using the install from pacman, I am getting 7 failures in eidos similar to this:

strfind('foobarbazxyzzy', 'foo', pos=1); : FAILURE : mismatched values (4294967295), expected (-1)

They are all mismatches of 4294967295 with -1. My understanding is that 4294967295 is the unsigned int equivalent of -1, if I'm not mistaken? Do you have any idea why this might be happening on windows-latest on github actions? All tests still work fine on my local Windows 10 machine.

[rdinnager]

We are also getting three failures with functions of floats, when comparing with user defined functions:

// (numeric)cumProduct(numeric x)
function (numeric)cumProduct_func(numeric x)
{
	return sapply(seqAlong(x), 'product(x[0:applyValue]);');
}

for (iter in 1:100)
{
	x = sample(-100:100, 5, T);	// integer
	xbuiltin = cumProduct(x);
	xuserdef = cumProduct_func(x);
	if (!identical(xbuiltin, xuserdef)) stop('Mismatch in test of cumProduct(i)');
}

for (iter in 1:100)
{
	x = rnorm(10);		// float
	xbuiltin = cumProduct(x);
	xuserdef = cumProduct_func(x);
	if (!identical(xbuiltin, xuserdef)) stop('Mismatch in test of cumProduct(f)');
}


return T;
 : FAILURE : raise during EvaluateInterpreterBlock(): ERROR (Eidos_ExecuteFunction_stop): stop("Mismatch in test of cumProduct(f)") called.

The same failure happens for cumSum and sum. It will be interesting to see if we get the same issues on the compiled version once I get that working.

[bhaller]

So for tests I am running using the install from pacman, I am getting 7 failures in eidos similar to this:

strfind('foobarbazxyzzy', 'foo', pos=1); : FAILURE : mismatched values (4294967295), expected (-1)

They are all mismatches of 4294967295 with -1. My understanding is that 4294967295 is the unsigned int equivalent of -1, if I'm not mistaken? Do you have any idea why this might be happening on windows-latest on github actions? All tests still work fine on my local Windows 10 machine.

This one was a bug; I have just committed a fix. No idea why it only manifested on Windows, but every compiler is different. In this case it seems like the incorrect behavior (i.e., manifesting the bug as a bug) is correct according to the C spec, so Windows seems right here and other platforms maybe seem wrong, but the C type-promotion rules are incredibly arcane, so it's hard to say for sure. :-> Anyway, fix committed but I can't test it on Windows, so please confirm.

[bhaller]

We are also getting three failures with functions of floats, when comparing with user defined functions:

...

The same failure happens for cumSum and sum. It will be interesting to see if we get the same issues on the compiled version once I get that working.

Huh. I'm not sure I understand; these failures indicate that the compiled version doesn't match the user-defined version, for some input. But then you write "will be interesting to see if we get the same issues on the compiled version once I get that working", which implies that you do not presently have the compiled version working. How are you comparing the compiled version to the user-defined version, then? Anyhow, I don't have a guess on this one so far. What I'd recommend is opening up an Eidos console in SLiMgui and trying out sum()/product()/cumSum()/cumProduct() with some random input, just as these comparison tests do, to see what is going on. Happy to zoom with you if that would be helpful; email me a zoom link if you want to.

[rdinnager]

While we are at it, I am getting a new warning during compilation on Windows:

D:/a/SLiM/SLiM/core/slim_functions.cpp: In function 'double DoSummarizeOperation(Individual*, SummarizeOperation)':
D:/a/SLiM/SLiM/core/slim_functions.cpp:1295:1: warning: control reaches end of non-void function [-Wreturn-type]
 1295 | }
      | ^

Thoughts?

[rdinnager]

We are also getting three failures with functions of floats, when comparing with user defined functions:

// (numeric)cumProduct(numeric x)
function (numeric)cumProduct_func(numeric x)
{
	return sapply(seqAlong(x), 'product(x[0:applyValue]);');
}

for (iter in 1:100)
{
	x = sample(-100:100, 5, T);	// integer
	xbuiltin = cumProduct(x);
	xuserdef = cumProduct_func(x);
	if (!identical(xbuiltin, xuserdef)) stop('Mismatch in test of cumProduct(i)');
}

for (iter in 1:100)
{
	x = rnorm(10);		// float
	xbuiltin = cumProduct(x);
	xuserdef = cumProduct_func(x);
	if (!identical(xbuiltin, xuserdef)) stop('Mismatch in test of cumProduct(f)');
}


return T;
 : FAILURE : raise during EvaluateInterpreterBlock(): ERROR (Eidos_ExecuteFunction_stop): stop("Mismatch in test of cumProduct(f)") called.

Ah, I can now clarify that these failures happen on 32bit Windows only. I haven't really tested on 32bit Windows because I use 64bit Windows. One option is to just drop support for 32bit Windows (honestly I don't know how many people really still use it, and Microsoft will be abandoning it soon anyway). Otherwise I'm not sure what the consequences of this test failure is, and how to reproduce it without access to a 32bit machine I can play around with..

[rdinnager]

Okay, all tests Windows tests are working except for the two that run the Eidos tests on 32bit Windows. Even after incorporating your latest changes I still get two -1 mismatches:

strfind(c('foobarbazxyzzy', 'barfoo', 'xyzzyfoobaz', 'xyzzy'), 'foo'); : FAILURE : mismatched values (0 3 5 4294967295), expected (0 3 5 -1)
strfind(c('foobarbazxyzzy', 'barfoo', 'xyzzyfoobaz', 'xyzzy'), 'foo', pos=1); : FAILURE : mismatched values (4294967295 3 5 4294967295), expected (-1 3 5 -1)

But that is down from 6 before, so that is good.

[rdinnager]

Ah, I can now clarify that these failures happen on 32bit Windows only. I haven't really tested on 32bit Windows because I use 64bit Windows. One option is to just drop support for 32bit Windows (honestly I don't know how many people really still use it, and Microsoft will be abandoning it soon anyway). Otherwise I'm not sure what the consequences of this test failure is, and how to reproduce it without access to a 32bit machine I can play around with..

Aha. I would definitely drop support for 32-bit Windows. I have just added a compile-time check that generates an error message if a compile to 32-bit is attempted (on any platform, not just on Windows); nobody should be doing that. :->

Okay, that is good to get clarification on that. I was not sure because I never use 32bit. I'm not surprised that it is wrong to do so. I should probably remove that build from pacman as well..

[bhaller]

...I should probably remove that build from pacman as well..

Definitely. Thanks.

[bhaller]

...Even after incorporating your latest changes I still get two -1 mismatches:

Yes, that's because my previous fix was wrong, dumb error. It's a pain when one can't test code for correctness locally! Fix for the fix now pushed.

[bhaller]

OK, with your new PR I guess all the above is resolved, right? Just need to (a) disable 32-bit in the pacman stuff, (b) add tskit tests for windows once they are working again, and (c) do the documentation work.

[bhaller]

@rdinnager I've just tagged you on #232 regarding the tskit tests. I think everything else about the Windows port is now resolved (although maybe you'll have a few edits for me on the documentation write-up; that doesn't happen on GitHub anyway). So, I'm going to close this issue now! Yay! Of course feel free to create new issues if there are any remaining loose ends to be tracked. Thanks for all your work on this!