torrvision / crayon

A language-agnostic interface to TensorBoard

Crayon

Crayon is a framework that gives you access to the visualisation power of TensorBoard with any language. Currently it provides a Python and a Lua interface, however you can easily implement a wrapper around the provided RESTful API.

---

This system is composed of two parts:

• A server running on a given machine that will be used to display tensorboard and store all the data.
• A client embedded inside your code that will send the datas to the server.

Note that the server and the client do not have to be on the same machine.

Install

Server machine

The machine that will host the server needs to have docker installed. The server is completely packaged inside a docker container. To get it, run:

$ docker pull alband/crayon

Client machine

The client machine only need to install the client for the required language. Detailed instructions can be read by nagivating to their respective directories.

TL;DR:

• Lua / Torch - $ luarocks install crayon
• Python 2 - $ pip install pycrayon
• Python 3 - $ pip3 install pycrayon

Usage

Server machine

To start the server, run the following:

$ docker run -d -p 8888:8888 -p 8889:8889 --name crayon alband/crayon

Tensorboard is now accessible on a browser at server_machine_address:8888. The client should send the data at server_machine_address:8889.

Client

See the documentation for the required language:

• Lua
• Python

crayon's Issues

Can I draw multiple curves in a single figure ?

I use foo.add_scalar_dict({'train_acc':trainacc,'test_acc':testacc}) to send training and testing accuracy to the server, but they ended up in separate figures. How can I draw them in a single figure ?

Create open_experiment() et remove_experiment() on client side

UsageExample: TypeError: request() got an unexpected keyword argument 'json'

Error occurs when I run the python usage example.

from pycrayon import CrayonClient
import time
cc = CrayonClient(hostname="10.1.12.46")
foo = cc.create_experiment("foo")

ERROR:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-be887cc4349c> in <module>()
----> 1 foo = cc.create_experiment("foo")

/usr/local/lib/python2.7/dist-packages/pycrayon/crayon.pyc in create_experiment(self, xp_name, zip_file)
     53     def create_experiment(self, xp_name, zip_file=None):
     54         assert(isinstance(xp_name, basestring))
---> 55         return CrayonExperiment(xp_name, self, zip_file=zip_file, create=True)
     56 
     57     def remove_experiment(self, xp_name):

/usr/local/lib/python2.7/dist-packages/pycrayon/crayon.pyc in __init__(self, xp_name, client, zip_file, create)
     86 
     87         elif create:
---> 88             self.__init_empty()
     89 
     90         else:

/usr/local/lib/python2.7/dist-packages/pycrayon/crayon.pyc in __init_empty(self)
     94     def __init_empty(self):
     95         query = "/data"
---> 96         r = requests.post(self.client.url + query, json=self.xp_name)
     97 
     98         if not r.ok:

/usr/lib/python2.7/dist-packages/requests/api.pyc in post(url, data, **kwargs)
     86     """
     87 
---> 88     return request('post', url, data=data, **kwargs)
     89 
     90 

/usr/lib/python2.7/dist-packages/requests/api.pyc in request(method, url, **kwargs)
     42 
     43     session = sessions.Session()
---> 44     return session.request(method=method, url=url, **kwargs)
     45 
     46 

TypeError: request() got an unexpected keyword argument 'json'

Timeout threshold might be too low ?

I keep getting this error:

Traceback (most recent call last): File "train_viewpoint.py", line 495, in <module> main(args) File "train_viewpoint.py", line 23, in main cc = CrayonClient("focus.eecs.umich.edu") File "/z/home/mbanani/PyTorch2Env/local/lib/python2.7/site-packages/pycrayon/crayon.py", line 29, in __init__ " Server sent: {}.".format(r.text)) RuntimeError: Something went wrong! Server sent: Server: TensorBoard failed to answer request 'logdir'. Done

and I think the reason is that the timeout, specified here is too low when people are training on clusters and trying to connect to a docker on their personal machines. Maybe consider increasing the timeout ?

Missing graph visualisation

We should write an interface for visualising PyTorch (or any other NN graph).

TLS-Enabled

Hello I am trying to install crayon. I have Installed docker. I run docker pull alband/crayon
this command and got this error.

FATA[0000] Post http:///var/run/docker.sock/v1.18/images/create?fromImage=alband%2Fcrayon%3Alatest: dial unix /var/run/docker.sock: permission denied. Are you trying to connect to a TLS-enabled daemon without TLS?

How I can Fixed this?
-Thank You-

Add Python 3.x compatibility

TODO

• Port server code to Python 3 (to handle at least unicode). (Server will stay in python 2 for now as its fully packaged in docker and handles both str and unicodes types)
• Refactor client code so that it's compatible with both 2.x and 3.x.

When do you release the new version?

I am glad that #24 is solved, however I find the lastest version is 0.5 which does not contain this fix...

Exp name containing plus '+' sign does not work

My experiment name is 'myexp+suffix'.
When I call the add_scalar_value function for this experiment, I get the following error message:

Traceback (most recent call last):
  File "plot.py", line 126, in main
    crayon_experiment.add_scalar_value(key, values[key], step = counter)
  File "anaconda/lib/python2.7/site-packages/pycrayon/crayon.py", line 149, in add_scalar_value
    raise ValueError(msg.format(r.text))
ValueError: Something went wrong. Server sent: Unknown experiment name 'myexp suffix'.

It has dropped the + sign from the experiment name and complains experiment does not exist.

Missing image visualisation

Need to write end points and backend for the image tab in tensorboard. To be done after #8.

Testing suite is missing

We should have a testing suite to run on Travis.

TODO

• Set testing suite for [py2.7, py3.*] with tox. For Lua we'll have to check what can be done in terms of test discovery.
• Investigate running tensorboard on Travis.
• Write a travis.yml file.

Code examples are missing

For each client we should add some code examples.

• Python client
• Lua client

How can I use crayon on my own PC? (not a server or client machine)

I have a computer with GPU, and run my programs on it. I don't have a server or client machine, just one PC.
How could I install crayon in this situation?

Crayon might require some form of password protection

Maybe a simple key generated by the server at startup, or something you can add to the docker file startup.

Missing unittests for lua client and docs in README

Adding Embedding methods to crayon.py

Currently there are only scalar and histogram methods. It would be AWESOME to have embedding methods, as that let's people like me who use PyTorch for unsupervised learning make use of Crayon.

If you think about it, Embedding is the most powerful tool on Tensorboard. Histograms and running averages (scalars) aren't hard to implement. PCA/TSNE interactively are not super easy.

Would be happy to work with other devs on adding embedding functionality if anyone's up! Any leads on how I should approach?

Missing load_zip method in lua client

Refactor server

Currently it's a bloody mess. Let's urgently fix that.

TODO

• Clean up commns (and switch to requests)
• Flask class
• pep8

json.decoder.JSONDecodeError

Hi, thanks for making this project open source.

I get the following error,

Traceback (most recent call last): File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap self.run() File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/home/ajay/PythonProjects/PyT_Neural_Arch_Search_v1_2/train_v1.py", line 201, in train foo.get_scalar_values("Mean Reward") File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/site-packages/pycrayon/crayon.py", line 167, in get_scalar_values return json.loads(r.text) File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/json/__init__.py", line 354, in loads return _default_decoder.decode(s) File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/home/ajay/anaconda3/envs/pyphi/lib/python3.6/json/decoder.py", line 357, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

When I run the following in a single thread

cc = CrayonClient( hostname="http://127.0.1.1" , port=8889)
foo = cc.create_experiment("train_" + str(rank))

foo.add_scalar_value("Mean Reward", mean_reward, step = episode_count)
foo.add_scalar_value("Max Reward" , max_reward,  step = episode_count)

foo.get_scalar_values("Mean Reward")
foo.get_scalar_values("Max Reward")

Seems to work fine when I run it from the python interpreter though?

Add lua bindings

Empty experiments are not listed properly

Empty (with no point) experiments are not listed in the GET /data endpoint properly here.
Even though it properly raises an error when trying to create a new one with the same name and opening it works as expected.

attempt to index field 'scalar_steps' (a nil value) when using add_scalar_dict

Hi,

I tried add_scalar_dict as following

local foo = cc:create_experiment("foo")
local d = {}
d['bar'] = 3
d['hoo'] = 4
foo:add_scalar_dict(d)

but got the errors

attempt to index field 'scalar_steps' (a nil value) 

not sure how to use that.

thanks

xp_writers isn't re-initialized after restart

That seems to cause the situation where it becomes impossible to remove experiments after a restart. I guess there's a workaround by writing something to an experiment before deleting it, which I'll try out. Still, should probably be improved.

default port conflicts with jupyter notebook

Crayon should use a more random default port number than 8888 since that conflicts with the default for jupyter notebook.

Install fails unless requests is already installed

This is under a python 3 venv:

$ .venv/bin/pip install pycrayon
Collecting pycrayon
  Downloading pycrayon-0.5.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-4hvqb18r/pycrayon/setup.py", line 2, in <module>
        from pycrayon.crayon import __version__
      File "/tmp/pip-build-4hvqb18r/pycrayon/pycrayon/__init__.py", line 1, in <module>
        from .crayon import CrayonClient
      File "/tmp/pip-build-4hvqb18r/pycrayon/pycrayon/crayon.py", line 1, in <module>
        import requests
    ImportError: No module named 'requests'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-4hvqb18r/pycrayon/

$ .venv/bin/pip install requests pycrayon
Collecting requests
  Downloading requests-2.13.0-py2.py3-none-any.whl (584kB)
    100% |████████████████████████████████| 593kB 2.3MB/s 
Collecting pycrayon
  Using cached pycrayon-0.5.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-xyawjvaw/pycrayon/setup.py", line 2, in <module>
        from pycrayon.crayon import __version__
      File "/tmp/pip-build-xyawjvaw/pycrayon/pycrayon/__init__.py", line 1, in <module>
        from .crayon import CrayonClient
      File "/tmp/pip-build-xyawjvaw/pycrayon/pycrayon/crayon.py", line 1, in <module>
        import requests
    ImportError: No module named 'requests'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-xyawjvaw/pycrayon/

$ .venv/bin/pip install requests 
Collecting requests
  Using cached requests-2.13.0-py2.py3-none-any.whl
Installing collected packages: requests
Successfully installed requests-2.13.0

$ .venv/bin/pip install pycrayon
Collecting pycrayon
  Using cached pycrayon-0.5.tar.gz
Requirement already satisfied: requests in ./.venv/lib/python3.5/site-packages (from pycrayon)
Building wheels for collected packages: pycrayon
  Running setup.py bdist_wheel for pycrayon ... done
  Stored in directory: /home/elis/.cache/pip/wheels/f2/39/f0/b7168e93688c8d36ffaae3c0ca5a9a928c5173d01baf69aeff
Successfully built pycrayon
Installing collected packages: pycrayon
Successfully installed pycrayon-0.5

Implement some form of web interface

It's annoying to write scripts to save / delete runs. We should just run a bootstrap frontend or something.

Probably to do when / after we refactor the server.

comparison with serverless approach (docs)

I started using crayon in my project, and I have been reasonably happy with it although it is a pity that it doesn't support all tensorboard summary types and that it does not seem to be very actively maintained. Later I discovered a different approach of just using a client library to create tensorboard log items (see https://github.com/TeamHG-Memex/tensorboard_logger and, for a particularly simple approach, https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/04-utils/tensorboard/logger.py). I realize that the crayon approach is in a way more flexible since it provides a REST api, but for someone just looking to use tensorboard with pytorch, it is not very clear what the advantages of crayon are over a simpler approach not using a separate server, given that no solution seems to be under active maintenance.

EINTR not handled

Often with the function net.summary.add_scalar_value(...), I intermittently get the following error:

requests.exceptions.ConnectionError: HTTPConnectionPool(host=IP, port=8889): Max retries exceeded with url: /data/scalars?xp=VGG&name=Train%20R%5E2 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at MEMORY_ADDRESS>: Failed to establish a new connection: [Errno 4] Interrupted system call',))

My pycrayon server is on my local machine whereas the computation (that is sending this request) is on a remote machine.

This is in the Python 2.7 version of pycrayon. I think that this request should be retried until it succeeds, as per https://stackoverflow.com/questions/14136195/what-is-the-proper-way-to-handle-in-python-ioerror-errno-4-interrupted-syst, which is the default behavior in Python 3.5.

I'm new to pytorch and crayon, so please forgive me if I'm mistaken.