Comments (14)
@mpmdean this function works now magically at jupyter.nsls2.bnl.gov so I will close this issue.
from csxtools.
is ffmpeg installed on the server?
from csxtools.
@tacaswell I don't know. All I can say is that other kernals on srv1 do work. e.g. CSX on srv1 (legacy 2017-2 v0)
from csxtools.
xref matplotlib/matplotlib#9205
from csxtools.
and xref matplotlib/matplotlib#8760
from csxtools.
I also get these warnings
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/site-packages/matplotlib/animation.py:1021: UserWarning: MovieWriter ffmpeg unavailable
warnings.warn("MovieWriter %s unavailable" % writer)
/opt/conda_envs/analysis-2018-1.0/lib/python3.6/site-packages/matplotlib/animation.py:1026: MatplotlibDeprecationWarning: Support for mencoder is only partially functional, and will be removed entirely in 2.2. Please use ffmpeg instead.
metadata=metadata)
from csxtools.
The issue is that:
a) we are hitting the system ffmpeg
b) this version of debian provides a ffmpeg which is shim around avconv
c) due to avconv being a hostile fork of ffmeg the cli of the two projects is diverging (which is how Matplotlib accesses them to save the movies) Matplotlib tried to ban the usage of the shims.
d) The ban was implemented between 2.0.1 and 2.0.2 (matplotlib/matplotlib#8253) but was a bit too heavy-handed and also broke the AVConvWriter which was fixed in 2.1 (matplotlib/matplotlib#8743)
e) for some reason the version of Matplotlib installed in the current environment has 2.0.2 installed which is the exactly 1 version of Matplotlib that has this bug
f) the memencoder backend is not expecting the inputs that are being passed (which assumes it is talking to ffmpeg/avconv) which is what is blowing up in a less than clear way.
The best solution is to get your enviroment upgraded to a newer version of Matplotlib, the very dirty way to get this working which I will deny having told you:
import matplotlib.animation as ma
ma.FFMpegWriter.isAvailable = lambda: True
ma.writers.register('ffmpeg')(ma.FFMpegWriter)
from csxtools.
During debugging process, we also find conflicts within several libraries in order to update matplotlib to newer version. So the current solution is to update packages at srv1 during beam shutdown time, will keep updated here.
from csxtools.
What happened with this? We have rolled on to a new version of the conda env and still have this issue, but maybe now it is a different issue?
I find this now to be a problem for srv1 and 2 using (current), which I believe to be:
analysis-2018-2.1 /opt/conda_envs/analysis-2018-2.1
I also see this with srv3, same conda environment version.
matplotlib: '2.2.2'
csxtools: '0.1.13'
These systems have now been upgraded to debian 8. I am using %matplotlib widget with other plots okay.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-28-cba7760ebe81> in <module>()
----> 1 image_stack_to_movie(sub_images, vmin=np.percentile(sub_images[1],30), vmax=np.percentile(sub_images[1],90))
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/ipynb/animation.py in image_stack_to_movie(images, frames, vmin, vmax, figsize, cmap, fps)
102 plt.close(anim._fig)
103 # return anim.to_html5_video()
--> 104 return HTML(_anim_to_html(anim, fps))
105
106
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/ipynb/animation.py in _anim_to_html(anim, fps)
115 anim.save(f.name, fps=fps,
116 extra_args=['-vcodec', 'libx264',
--> 117 '-pix_fmt', 'yuv420p'])
118 video = open(f.name, "rb").read()
119 anim._encoded_video = base64.b64encode(video)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/matplotlib/animation.py in save(self, filename, writer, fps, dpi, codec, bitrate, extra_args, metadata, extra_anim, savefig_kwargs)
1198 # TODO: See if turning off blit is really necessary
1199 anim._draw_next_frame(d, blit=False)
-> 1200 writer.grab_frame(**savefig_kwargs)
1201
1202 # Reconnect signal for first draw if necessary
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/contextlib.py in __exit__(self, type, value, traceback)
97 value = type()
98 try:
---> 99 self.gen.throw(type, value, traceback)
100 except StopIteration as exc:
101 # Suppress StopIteration *unless* it's the same exception that
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/matplotlib/animation.py in saving(self, fig, outfile, dpi, *args, **kwargs)
239 yield self
240 finally:
--> 241 self.finish()
242
243
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/matplotlib/animation.py in finish(self)
365 def finish(self):
366 '''Finish any processing for writing the movie.'''
--> 367 self.cleanup()
368
369 def grab_frame(self, **savefig_kwargs):
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/matplotlib/animation.py in cleanup(self)
403 def cleanup(self):
404 '''Clean-up and collect the process used to write the movie file.'''
--> 405 out, err = self._proc.communicate()
406 self._frame_sink().close()
407 _log.debug('MovieWriter -- Command stdout:\n%s', out)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/subprocess.py in communicate(self, input, timeout)
841
842 try:
--> 843 stdout, stderr = self._communicate(input, endtime, timeout)
844 finally:
845 self._communication_started = True
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/subprocess.py in _communicate(self, input, endtime, orig_timeout)
1503 selector.register(self.stdin, selectors.EVENT_WRITE)
1504 if self.stdout:
-> 1505 selector.register(self.stdout, selectors.EVENT_READ)
1506 if self.stderr:
1507 selector.register(self.stderr, selectors.EVENT_READ)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/selectors.py in register(self, fileobj, events, data)
349
350 def register(self, fileobj, events, data=None):
--> 351 key = super().register(fileobj, events, data)
352 poll_events = 0
353 if events & EVENT_READ:
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/selectors.py in register(self, fileobj, events, data)
235 raise ValueError("Invalid events: {!r}".format(events))
236
--> 237 key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
238
239 if key.fd in self._fd_to_key:
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/selectors.py in _fileobj_lookup(self, fileobj)
222 """
223 try:
--> 224 return _fileobj_to_fd(fileobj)
225 except ValueError:
226 # Do an exhaustive search.
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/selectors.py in _fileobj_to_fd(fileobj)
37 except (AttributeError, TypeError, ValueError):
38 raise ValueError("Invalid file object: "
---> 39 "{!r}".format(fileobj)) from None
40 if fd < 0:
41 raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file object: <_io.BufferedReader name=68>
from csxtools.
note that srv3 gives a different traceback
MovieWriter ffmpeg unavailable.
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/PIL/Image.py in save(self, fp, format, **params)
1914 try:
-> 1915 format = EXTENSION[ext]
1916 except KeyError:
KeyError: '.mp4'
During handling of the above exception, another exception occurred:
ValueError Traceback (most recent call last)
<ipython-input-16-cba7760ebe81> in <module>()
----> 1 image_stack_to_movie(sub_images, vmin=np.percentile(sub_images[1],30), vmax=np.percentile(sub_images[1],90))
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/ipynb/animation.py in image_stack_to_movie(images, frames, vmin, vmax, figsize, cmap, fps)
102 plt.close(anim._fig)
103 # return anim.to_html5_video()
--> 104 return HTML(_anim_to_html(anim, fps))
105
106
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/csxtools/ipynb/animation.py in _anim_to_html(anim, fps)
115 anim.save(f.name, fps=fps,
116 extra_args=['-vcodec', 'libx264',
--> 117 '-pix_fmt', 'yuv420p'])
118 video = open(f.name, "rb").read()
119 anim._encoded_video = base64.b64encode(video)
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/matplotlib/animation.py in save(self, filename, writer, fps, dpi, codec, bitrate, extra_args, metadata, extra_anim, savefig_kwargs)
1198 # TODO: See if turning off blit is really necessary
1199 anim._draw_next_frame(d, blit=False)
-> 1200 writer.grab_frame(**savefig_kwargs)
1201
1202 # Reconnect signal for first draw if necessary
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/contextlib.py in __exit__(self, type, value, traceback)
86 if type is None:
87 try:
---> 88 next(self.gen)
89 except StopIteration:
90 return False
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/matplotlib/animation.py in saving(self, fig, outfile, dpi, *args, **kwargs)
239 yield self
240 finally:
--> 241 self.finish()
242
243
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/matplotlib/animation.py in finish(self)
610 self._frames[0].save(
611 self._outfile, save_all=True, append_images=self._frames[1:],
--> 612 duration=int(1000 / self.fps))
613
614
/opt/conda_envs/analysis-2018-2.1/lib/python3.6/site-packages/PIL/Image.py in save(self, fp, format, **params)
1915 format = EXTENSION[ext]
1916 except KeyError:
-> 1917 raise ValueError('unknown file extension: {}'.format(ext))
1918
1919 if format.upper() not in SAVE:
ValueError: unknown file extension: .mp4
from csxtools.
came back to the problem with matplotlib at 2.2.2
ffmpeg was not installed on the computer and as Tom said, we cannot use the ffmpeg in python. This is the workaround that I found because the lines of code he provided still didn't work for me.
- update apt-get and install ffmpeg on the server. (on srv1 only. still need to do this for srv2)
- using the "current" srv1 kernel, add the following to notebook before creating the movie.
plt.rcParams['animation.ffmpeg_path'] = '/usr/bin/ffmpeg'
are we good with closing this for now? Or do we want to wait until there is a perfect solution?
from csxtools.
to download movie, use Classic Notebook View, under Help
from csxtools.
@ambarb verify for current matplotlib version
from csxtools.
I think it's because ffmpeg
is now installed as a system package in Docker/Singularity as instructed in https://github.com/NSLS-II/scipy-binder/blob/a1a7e867e329b18ee18f05687727dff2eb3cc318/binder/apt.txt#L3.
from csxtools.
Related Issues (20)
- make csx tools load multiple scan_ids HOT 6
- image_stack_to_movie is broken HOT 1
- get_fastccd_images cannot process all instances of recorded data HOT 2
- roi argument for get_fastccd_images() is broken ... plus some flatfield dependency discussion HOT 23
- KeyError: 'AD_HDF5' for 1.14 for get_fastccd_flatfield() and get_fastccd_images() HOT 7
- Should we change default behavior get_fastccd_images to use a flatfield of (almost) all ones? HOT 3
- get_fastccd_images() does not work with roi given as input HOT 3
- swmr says it is enabled but it doesn't seem to work HOT 1
- Strange error related to dark image scan with x2 gain HOT 3
- Logging module no longer works
- fccd flatfield function doesn't handle the broken part of the camera very well HOT 5
- Remove hard-coded rows and columns because this changes HOT 1
- Potential place for controlling dark image searching HOT 1
- flat field issue HOT 3
- conda packaging for Python 3.10 HOT 2
- Fix deprecation warnings on tests. HOT 6
- ENH more automation of flatfield correction HOT 1
- homework for new helpers modules HOT 1
- helpers.overscan copy-paste error for right-side of detector
- flake8 style needed for new modules
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from csxtools.