Ipcontroller doesn't find ipcontroller-engine.json when '--work-dir' flag for c.SSHEngineSetLauncher.engines is set in ipcluster_config.py about ipyparallel

[minrk]

The gist appears to be that it's using .ipython rather than ~/.ipython, so the path is sensitive to the working directory. This should presumably not be the case.

[simkovic]

You are correct. Note though, that I'm setting absolute path in my config file but the controller does not respect my choice and turns it into rel path. Is there a quick fix? Is there an option to ask controller to use absolute paths? Or do I need to tweak the source code? What would be easier/feasible - postpone the change to working directory after the engines are started loaded or force the controller/engines to keep absolute path? Is there any particular reason why the engines work with relative paths?

In addition what is the purpose of the profile-dir? I thought profile-dir is the directory that specifies the location of the config files while the work-dir is the working directory where the code, sent to the engines, is executed.

[minrk]

It's probably a bug in the launcher code stripping the path prefix, not taking the working dir into account.

[simkovic]

I now have a fix. The solution is simply to set '--profile-dir' for each engine. If the remote profile dir is not set, the launcher guesses that the remote profile dir is a subdirectory of the remote working dir. This is effectively accomplished with the help of SSHClusterLauncher._strip_home which is a clumsy hack and should be removed. strip_home makes it impossible for the user to specify the remote user dir by setting c.SSHClusterLauncher.remote_profile_dir in the config, which initially confused me. In general a much better guess could be constructed based on the location data of each engine.