tdaff / remote_ikernel Goto Github PK

The code in the repo is significantly ahead of the latest release (both on github and PyPi) of 0.4.6 - Can you please release the updated GitHub code onto PyPi (assume it would be version 0.4.7).

This is important as organisations block installing from GitHub and only have access to internal PyPi mirrors, therefore packages need to be released to PyPi.

Original report by Mike Boyle (Bitbucket: moble, GitHub: moble).

I installed from pip with pip install remote_ikernel, as suggested in the readme, and ran remote_ikernel manage. I got an error from bash saying that the command line didn't make sense. I tracked it down to the wheel .whl on pypi, which starts with the #! line

#!/home/tdd20/.local/Virtualenvs/rik/bin/python

Oddly enough, the .tar.gz on pypi does not have this; instead, it starts with /usr/bin/env python. I changed it to that on my installation, and it works correctly.

Original report by Tom Daff (Bitbucket: tdaff, GitHub: tdaff).

If the kernel is producing output that writes directly to the terminal instead of through the standard channels this is lost in the pexpect. There should be some way to see this.

What will be the connection file details for R kernel ?

Original report by Jiří Kunčar (Bitbucket: jirikuncar, GitHub: jirikuncar).

Example:

1. research kernel: --tunnel_cmd="ssh -F ~/research/ssh/config"

2. demo kernel: --tunnel_cmd="ssh -F ~/demo/ssh/config"

(no I can't create different rules easily and I don't want to amend my global ssh config)

Original report by Anonymous.

I tried to set up remote_ikernel to a server, where sshd listened on a non-standard. Kernel died. Connection established, but then the startup script failed when it tried to set up tunnels. After some digging it looked like, in the first ssh command, the host and port were not splitted. My solution is below:

292,293c292,297
<         login_cmd = 'ssh -o StrictHostKeyChecking=no {args} {host}'.format(
<             args=launch_args, host=self.host)
---
>         if ':' in self.host:
>             login_cmd = 'ssh -p {host_port} -o StrictHostKeyChecking=no {args} {host}'.format(
>                 args=launch_args, host=self.host.split(":")[0], host_port=self.host.split(":")[1])
>         else:
>             login_cmd = 'ssh -o StrictHostKeyChecking=no {args} {host}'.format(
>                 args=launch_args, host=self.host)

Original report by Tom Daff (Bitbucket: tdaff, GitHub: tdaff).

Compatibility layer should be working, but the documentation and maybe requirements will need updating

Original report by Jean-Marc Andreoli (Bitbucket: jmandreoli, GitHub: jmandreoli).

The following lines (600-602) in kernel.py, which decode self.host, cause an error with slurm (and probably some other launchers), as the decoding happens too late.

#!python

# zmq needs str in Python 3, but pexpect gives bytes
if hasattr(self.host, 'decode'):
  self.host = self.host.decode('utf-8')

And furthermore, there is no need to do the decoding again and again each time tunnel_cmd is invoked.

My fix: Move them just after launch_* but before start_kernel at line 240

Original report by Scott Field (Bitbucket: sfield83, ).

Hi Tom,

Sometimes I'll need to wait a few minute for my job to start (using the very awesome feature of remotely starting jobs from a batch submission system). If the kernel dies, upon restart a brand new job is submitted. This results in two jobs sitting on the queue.

So far, I've only had a problem on PBS systems.

Best,
Scott

Original report by Scott Field (Bitbucket: sfield83, ).

Hi Tom,

Today I installed the latest version of remote_ikernel .3.1.0 from pip. I was excited to try out the new feature of starting kernels through interactive jobs in PBS. First I tried using an ssh interface which works great. I setup my pbs profile in a few different ways, with the basic command looking like

manage --add --kernel_cmd="ipython kernel -f {connection_file}" --name="pbs python" --interface=pbs --host=zwicky --verbose

When I start the kernel, I see the following error message (copied below) in my terminal. It looks as though qsub is being run on my local machine instead of the remote one.

Traceback (most recent call last):
File "/home/scott/pool/anaconda/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/home/scott/pool/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/scott/pool/anaconda/lib/python2.7/site-packages/remote_ikernel/main.py", line 15, in
start_remote_kernel()
File "/home/scott/pool/anaconda/lib/python2.7/site-packages/remote_ikernel/kernel.py", line 422, in start_remote_kernel
launch_args=args.launch_args, verbose=args.verbose)
File "/home/scott/pool/anaconda/lib/python2.7/site-packages/remote_ikernel/kernel.py", line 115, in init
self.launch_pbs()
File "/home/scott/pool/anaconda/lib/python2.7/site-packages/remote_ikernel/kernel.py", line 181, in launch_pbs
qsub_i = pexpect.spawn(pbs_cmd, logfile=self.log, timeout=600)
File "/home/scott/pool/anaconda/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 194, in init
self._spawn(command, args, preexec_fn, dimensions)
File "/home/scott/pool/anaconda/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 267, in _spawn
'executable: %s.' % self.command)
pexpect.exceptions.ExceptionPexpect: The command was not found or was not executable: qsub.

Original report by Anonymous.

I tried out using a ssh Remote kernel on a host serviced from 3rox.net (Pittsburgh Super-computer center).

After crashing the kernel a few times, my access to the entire 3rox network was blocked and I received an email from the security team claiming my host has been 'intensive SSH Scanning' over an external network, and threating to remove my computer from the network.

Thus using remote_ikernel over the WAN has risks. This needs to be addressed. At least the users shall be warned until a fix or workaround is implemented.

I will post an update when I hear back from the network admins at PSC.

Original report by Tony Hirst (Bitbucket: ajh59, ).

I'm running remote kernels on a Lego EV3 and they can be quite slow to start up; I'm wondering if that causes issues for Jupyter?

It always was hit and miss getting notebooks to receive the Kernel Ready signal (they appear to connect fine), but recently they've stopped getting Kernel Ready at all.

I'm not sure if this is a Jupyter remote kernel connection issue, or something that could be handled by remote_ikernel?

I seem to be able to connect to the remote kernel via jupyter console quite happily.

Original report by zonca (Bitbucket: zonca, GitHub: zonca).

thanks for releasing this project!

I created a remote kernel with:

#!bash

remote_ikernel manage --add --name="NAME" --interface=ssh --host=HOST --cpus=1 --kernel_cmd="ipython kernel -f {connection_file}"

it works fine, however the notebook complaints that it cannot save, neither automatically nor manually.

Is there something wrong in my configuration?

Original report by Anonymous.

Hi,

thanks a lot for this module. I seems quite useful for a multi-user/multi-machine setup of jupyter/ipython.

issue in "short": when I start multiple notebooks/kernels on a remote (interface=ssh) sometimes a kernel/notebook is not responding at startup. Often it also works which makes it hard to catch the issue for me.

long version:

our setup is jupyterhub -> jupyter-notebook ("notebook_server") -> remote_ikernel -> ipython bash ... ("compute_server"); all have the same shared filesystem via NFS, eg. for homes etc.

mainly the whole setup works nicely, but sometimes I start eg. a new remote bash-kernel notebook a this new notebook never gets any command executed.

I started digging a bit into the remote_ikernel code and I'm wondering about this "rik_kernel.json" file and how this should work in principle. I miss myself a bit of understanding of the whole design of the notebook app, so please forgive me...

It seems to me that every new notebook writes such a "rik_kernel.json" file with the connection info (Does each notebook uses its own ports even on the same server?)

What happens if I start multiple notebooks? The the json is overwritten, but what happens if the remote kernel is not ready yet, or has not even read the json?

We use a shared file-system (NFS) and so there might be further timeout/versioning problems.

I was looking for an easy way to get a unique filename per notebook for this "rik_kernel.json" file, but I don't know for example how to get this kernel id (like e2e8f19d-b3ff-4bc0-a1a5-683c03a82778) as string into the remote_ikernel module.

So what additional info could be useful to solve this isssue?
Could you explain me in short how this is expected to work with multiple notebooks/kernels and the "rik_kernel.json" connection info file? In my case they all end up in my home ~/ dir!

Thanks a lot for your help!

Original report by Tom Daff (Bitbucket: tdaff, GitHub: tdaff).

It would be useful to be able to put stuff like '-l h_rt=24:00:00' in the qlogin command arguments

Original report by Dmitriy Serdyuk (Bitbucket: serdyuk_dmitriy, ).

Interrupt signal is never sent. Seems that capitalizing 'c' in self.connection.sendcontrol('C') solves the problem.

Original report by Anonymous.

Hi,

I have problems to get a rnning R remote kernel vio IRkernel. How do I have to config it? is there something different than e.g for bash/python?

I got e.g. bash working with
/.local/bin/remote_ikernel manage --add --kernel_cmd="/install/anaconda/bin/python -m bash_kernel -f {connection_file}" --name="Remote Bash" --interface=ssh --host=MYHOST

How can I get this running?
Thanks a lot!

Original report by Anonymous.

Would it be feasible to add support for Platform LSF scheduler? If so, are there any plans to do so in future releases?

In terms of infrastructure, our cluster works in such a way that there is a login node and interactive compute nodes, which are only accessible via the login node.

Thank you.

Original report by Tom Daff (Bitbucket: tdaff, GitHub: tdaff).

The ssh_tunnel method from pyzmq has a timeout (so the tunnels do not last forever) but this means that tunnels for inactive notebooks exit after a while.

Original report by Leonid Kostrykin (Bitbucket: theV0ID, GitHub: theV0ID).

I'm using a remote IPython kernel which I added as follows:

#!bash
remote_ikernel manage --add \
--remote-precmd="source activate /path/to/conda/envs/ipython" \
--kernel_cmd="ipython kernel -f {connection_file}" --name "remote_host/Python2" \
--interface=ssh --host="remote_host" --workdir="/some/path"

I run an arbitrary computation which takes a few hours to finish. The progress of the computations are printed in the Notebook, and the print-out updates itself every second. If I remain at the computer and keep interacting with the Jupyter notebook (mostly scrolling to see the progress) the computation finishes as expected. Contrary, If I just leave the Jupyter notebook open as-is and go away for a few hours (the browser and the notebook remain open of course) the computation always hangs. From the last print-out I see that it usually hangs after about ~300 to ~3000 seconds. I have automatic suspend/hibernate turned off on my computer. This issue doesn't occur if a local kernel is used instead.

Might that be related to a timeout of the SSH tunnels?

Original report by Anonymous.

File: kernel.py
Function: check_password
lines: 130, 131

    re_passphrase = re.search('Enter passphrase .*:', text)
    re_password = re.search('.*@.* password:', text)

should be changed to

    re_passphrase = re.search(b'Enter passphrase .*:', text)
    re_password = re.search(b'.*@.* password:', text)

since text is a byte object, not an str.

Original report by Tom Daff (Bitbucket: tdaff, GitHub: tdaff).

#!shell

% remote_ikernel manage --add --kernel_cmd="`which ipython3` kernel -f {connection_file}" --name="Python 3 ☂" --interface=local  
Installed kernel rik_local_python3.

% remote_ikernel manage
Traceback (most recent call last):
  File "/home/tdd20/.local/Virtualenvs/rik/bin/remote_ikernel", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/tdd20/mnt/womble/Compile/remote_ikernel/bin/remote_ikernel", line 12, in <module>
    runpy.run_module('remote_ikernel')
  File "/usr/lib64/python2.7/runpy.py", line 180, in run_module
    fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/tdd20/mnt/womble/Compile/remote_ikernel/remote_ikernel/__main__.py", line 12, in <module>
    manage()
  File "/home/tdd20/mnt/womble/Compile/remote_ikernel/remote_ikernel/manage.py", line 196, in manage
    kernel_name=kernel_name, desc=spec.display_name)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2602' in position 15: ordinal not in range(128)

Original report by Tom Daff (Bitbucket: tdaff, GitHub: tdaff).

@rainwoodman mentioned in issue #12 that SSH multiplexing or a SOCKS proxy could be used to consolidate connections. This could be useful, especially to avoid passwords, but still considering whether it should be something managed by the kernels, or just well documented.

Original report by kiudee (Bitbucket: kiudee, GitHub: kiudee).

I have slurm v15.08.7 set up and I am trying to connect to it using remote_ikernel.
After applying the fixes in pull request #2, the initial connection worked and srun was correctly executed on the remote machine:

[I 14:52:47.800 remote_ikernel] Launching kernel through SLURM.
[I 14:52:47.800 remote_ikernel] SLURM command: 'srun  --cpus-per-task 8 -J remote_ikernel  -v -u bash -i'.
[I 14:52:47.949 remote_ikernel] Established session on node: b'hostname'.
[I 14:52:47.950 remote_ikernel] Established connection; starting kernel.
[I 14:52:47.950 remote_ikernel] Current working directory /path/to/notebooks.
[I 14:52:48.251 remote_ikernel] Running kernel command: 'ipython kernel -f ./rik_kernel.json'.

Unfortunately, the local notebook itself is not responding when cells are run. The log is sending this timeout:

[W 14:52:57.512 NotebookApp] Timeout waiting for kernel_info reply from a78ba0c2-4e38-46c9-b546-629f9742dc48
[D 14:52:57.514 NotebookApp] Opening websocket /api/kernels/a78ba0c2-4e38-46c9-b546-629f9742dc48/channels
[D 14:52:57.514 NotebookApp] Connecting to: tcp://127.0.0.1:39352
[D 14:52:57.515 NotebookApp] Connecting to: tcp://127.0.0.1:51203
[D 14:52:57.515 NotebookApp] Connecting to: tcp://127.0.0.1:40005

and then repeating this info message:

[I 14:53:52.552 remote_ikernel] Setting up tunnels on ports: 54638, 39352, 51203, 40005, 42449.

Here is the output of remote_ikernel manage --show ...:

  * Kernel found in: /home/_/.local/share/jupyter/kernels/rik_slurm....
  * Name: SLURM Python 3 Pool 8 CPUs (via username@hostname:port)
  * Kernel command: /path/to/python3 -m remote_ikernel --interface slurm --cpus 8 --tunnel-hosts username@hostname:port --kernel_cmd "ipython kernel -f {host_connection_file}" {connection_file}
  * remote_ikernel command: /path/to/bin/remote_ikernel manage --add "--kernel_cmd=ipython kernel -f {connection_file}" "--name=Python 3 Pool" --cpus=8 --interface=slurm --tunnel-hosts username@hostname:port
  * Raw json: {
[...omitted...]
}

Original report by Tom Daff (Bitbucket: tdaff, GitHub: tdaff).

Some tunnels remain active, but others throw up errors about "channel 5: open failed: administratively prohibited: open failed”

Original report by Anonymous.

On some machines, such as blue waters, ssh access necessarily requires a password (part of the authentication process relies on the one-time-password method). Is it possible to launch a remote kernel on such machines? I tried hacking the member function _spawn in kernel.py without any success.

I've been using the new interactive jobs feature in version 0.4 on both SLURM and PBS machines -- works beautifully!

Original report by Anonymous.

Since version 5.0.0, jupyter_client sets the sticky bit on the connection info file and its parent directory. This is to avoid the file being unexpectedly "garbage collected". See
fixes in 5.0.0
This is harmless since the file is normally placed in /run/user/$uid/jupyter which is only used for that purpose.

Now remote_ikernel copies the connection info file on the remote machine in the current directory, resulting in that directory having its sticky bit set. This may be an unwanted side effect.

How about putting the remote copy of the connection info file also in /run/user/$uid/jupyter (on the remote machine).

Original report by Denis Sergeev (Bitbucket: dennissergeev, GitHub: dennissergeev).

Hi,

Would it be possible to add Windows support?
The main problem seems to be that pexpect module is not fully compatible with Windows, especially pexpect.spawn() function: pexpect.readthedocs.org/en/stable/overview.html#pexpect-on-windows

But there is a windows-ported pexpect: https://gist.github.com/anthonyeden/8488763

Is it possible to plug it in? Are there any other limitations on windows?

Thanks.