Comments (11)
Dear All,
I also got some issues during the tests (nosetests
)
_warning_: UserWarning: Kpoint weights should sum up to one while sum_weights is 14.000
_warning_: .abinit/abipy/abipyrc does not exist!
_failure_: FAIL: Test the reading of Kpoints from netcdf files (si_scf_GSR.nc)
... pymatgen.util.io_utils.FileLockException
I wonder about the severity of this problems, i.e. whether I can use safely abipy.
I am attaching the full log file of nosetests
With regards,
Temok
#######################
_LOG FILE FOR nosetests_
#######################
OSX
Installation from a source downloaded on 2014.04.14
/sw/bin/python2.7 setup.py install
Dependencies (automatically) installed using:
pip 1.5.4 from /sw/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg (python 2.7)
and/or
/sw/bin/easy_install-2.7 --version
setuptools 3.3
cat nosetests.log
/Users/temok/ApplicationsMine/Installed/abipy-20140420/abipy-20140420/abipy/profile.py:192: UserWarning: Abipy configuration file /Users/temok/.abinit/abipy/abipyrc does not exist!
warn("Abipy configuration file %s does not exist!" % filename)
.................................../Users/temok/ApplicationsMine/Installed/abipy-20140420/abipy-20140420/abipy/core/kpoints.py:740: UserWarning: Kpoint weights should sum up to one while sum_weights is 14.000
The list of kpoints does not represent a homogeneous sampling of the BZ
<class 'abipy.core.kpoints.IrredZone'>
0) [0.500, 0.000, 0.000], weight = 1.000000
1) [0.417, 0.000, 0.000], weight = 1.000000
2) [0.333, 0.000, 0.000], weight = 1.000000
3) [0.250, 0.000, 0.000], weight = 1.000000
4) [0.167, 0.000, 0.000], weight = 1.000000
5) [0.083, 0.000, 0.000], weight = 1.000000
6) [0.000, 0.000, 0.000], weight = 1.000000
7) [0.000, 0.071, 0.071], weight = 1.000000
8) [0.000, 0.143, 0.143], weight = 1.000000
9) [0.000, 0.214, 0.214], weight = 1.000000
10) [0.000, 0.286, 0.286], weight = 1.000000
11) [0.000, 0.357, 0.357], weight = 1.000000
12) [0.000, 0.429, 0.429], weight = 1.000000
13) [0.000, 0.500, 0.500], weight = 1.000000
warnings.warn(err_msg)
F.......................................................................................................................
======================================================================
FAIL: Test the reading of Kpoints from netcdf files.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/temok/ApplicationsMine/Installed/abipy-20140420/abipy-20140420/abipy/core/tests/test_kpoints.py", line 167, in test_reading
self.assertTrue(kpoints.is_path)
AssertionError: False is not true
-------------------- >> begin captured stdout << ---------------------
About to read file: /Users/temok/ApplicationsMine/Installed/abipy-20140420/abipy-20140420/abipy/data/runs/data_si_ebands/outdata/si_scf_GSR.nc
ksampling {'kptopt': None, 'shifts': array([ 9.96920997e+36, 9.96920997e+36, 9.96920997e+36]), 'kptrlatt': None, 'mpdivs': array([8, 8, 8], dtype=int32)}
About to read file: /Users/temok/ApplicationsMine/Installed/abipy-20140420/abipy-20140420/abipy/data/runs/data_si_ebands/outdata/si_nscf_GSR.nc
ksampling {'kptopt': None, 'shifts': array([ 9.96920997e+36, 9.96920997e+36, 9.96920997e+36]), 'kptrlatt': None, 'mpdivs': array([-2147483647, -2147483647, -2147483647], dtype=int32)}
--------------------- >> end captured stdout << ----------------------
----------------------------------------------------------------------
Ran 155 tests in 71.960s
FAILED (failures=1)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/sw/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/sw/lib/python2.7/site-packages/pymatgen/io/abinitio/tasks.py", line 625, in save_lastnode_id
with FileLock(_COUNTER_FILE) as lock:
File "/sw/lib/python2.7/site-packages/pymatgen/util/io_utils.py", line 205, in __enter__
self.acquire()
File "/sw/lib/python2.7/site-packages/pymatgen/util/io_utils.py", line 184, in acquire
self.lockfile)
FileLockException: /Users/temok/.abinit/abipy/nodecounter.lock: Timeout occured.
Error in sys.exitfunc:
Traceback (most recent call last):
File "/sw/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/sw/lib/python2.7/site-packages/pymatgen/io/abinitio/tasks.py", line 625, in save_lastnode_id
with FileLock(_COUNTER_FILE) as lock:
File "/sw/lib/python2.7/site-packages/pymatgen/util/io_utils.py", line 205, in __enter__
self.acquire()
File "/sw/lib/python2.7/site-packages/pymatgen/util/io_utils.py", line 184, in acquire
self.lockfile)
pymatgen.util.io_utils.FileLockException: /Users/temok/.abinit/abipy/nodecounter.lock: Timeout occurred.
from abipy.
A follow up:
the last line of my previous post reads:
pymatgen.util.io_utils.FileLockException: /Users/temok/.abinit/abipy/nodecounter.lock: Timeout occurred.
and I'd like to mention that such file
~/.abinit/abipy/nodecounter.lock
exists but it is empty. Is this ok?
Cheers,
Temok
from abipy.
Hi Bruno,
Is my analysis below correct?
Yes, the problem is that many ETSF-IO files produced by abinit contain
variables that are defined but not initialized.
The decorator returns_None_onfail tries to detect this problem at run-time
so that abipy knows that the value is not initialized. It works on my Mac and
on other linux architectures but obviously this trick is not portable.
I guess that the problematic part is in the test:
# This trick is needed because in many cases we define the netcdf variable
# but we don't write its value.
return value if not isinstance(value, MaskedArray) else None
could you add print(type(value))
before returning so that we know the
kind of object that is returned by netcdf4 on your machine?
If it's a numpy array (and I think so) there's little to do at present.
I have generated the netcdf files on my mac (I don't remember the version of netcdf4
I used) but the error you are reporting clearly shows that there's a portability problem.
The best solution would be to change the code in Abinit so that the netcdf file
contains extra information that allows me to understand if the set of k-points
represent a path or a homogeneous mesh.
from abipy.
A follow up:
the last line of my previous post reads:
pymatgen.util.io_utils.FileLockException: /Users/temok/.abinit/abipy/nodecounter.lock: Timeout occurred.
and I'd like to mention that such file
~/.abinit/abipy/nodecounter.lock
exists but it is empty. Is this ok?
Yep, it's normal. nodecounter.lock is an empty file used to prevent other processes
from writing data to nodecounter in order to avoid race conditions.
pymatgen.util.io_utils.FileLockException: /Users/temok/.abinit/abipy/nodecounter.lock: Timeout occurred.
I will try to find another implementation of FileLock that is more portable
from abipy.
Could you pip install lockfile
Then you have to add the following line at the end of ~pymatgen/util/io_utils.py
from lockfile import FileLock
in order to redefine the class
Let me know it this solves the timeout issue
from abipy.
Thanks Matteo,
I installed lockfile
/sw/bin/pip install lockfile
blablabla
/sw/bin/pip search lockfile
lockfile - Platform-independent file locking module
INSTALLED: 0.9.1 (latest)
zc.lockfile - Basic inter-process locks python-liblockfile - A wrapper around liblockfile, using ctypes. yg.lockfile - Lockfile object with timeouts and context manager
Then ran the test example plot_bands.py
and got the same behaviour.
Then I added
from lockfile import FileLock
at the very end of
./lib/python2.7/site-packages/pymatgen/util/io_utils.py
and now the thing gets a bit worse:
Typing exit
or Ctrl-D
or quit
freezes the prompt. If then I issue Ctrl-C
then it gets released and ipython exits ( a single Ctrl-C
does not exits nor freezes ipython).
I am appending the two outputs I got,
Thanks,
Temok
#######################
LOG FILE 1 of 2:
RUNNING plot_bands.py
and QUITTING IPYTHON
( lockfile has been installed to the latest version)
#######################
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/sw/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(_targs, *_kargs)
File "/sw/lib/python2.7/site-packages/pymatgen/io/abinitio/tasks.py", line 625, in save_lastnode_id
with FileLock(_COUNTER_FILE) as lock:
File "/sw/lib/python2.7/site-packages/pymatgen/util/io_utils.py", line 205, in enter
self.acquire()
File "/sw/lib/python2.7/site-packages/pymatgen/util/io_utils.py", line 184, in acquire
self.lockfile)
FileLockException: /Users/temok/.abinit/abipy/nodecounter.lock: Timeout occured.
Error in sys.exitfunc:
Traceback (most recent call last):
File "/sw/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(_targs, *_kargs)
File "/sw/lib/python2.7/site-packages/pymatgen/io/abinitio/tasks.py", line 625, in save_lastnode_id
with FileLock(_COUNTER_FILE) as lock:
File "/sw/lib/python2.7/site-packages/pymatgen/util/io_utils.py", line 205, in enter
self.acquire()
File "/sw/lib/python2.7/site-packages/pymatgen/util/io_utils.py", line 184, in acquire
self.lockfile)
pymatgen.util.io_utils.FileLockException: /Users/temok/.abinit/abipy/nodecounter.lock: Timeout occurred.
#######################
LOG FILE 2 of 2:
AFTER ADDING
from lockfile import FileLock
TO THE END OF
pymatgen/util/io_utils.py
########################
In [2]: ^D (GETS FROZEN)
^C
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/sw/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/sw/lib/python2.7/site-packages/pymatgen/io/abinitio/tasks.py", line 625, in save_lastnode_id
with FileLock(_COUNTER_FILE) as lock:
File "/sw/lib/python2.7/site-packages/lockfile/__init__.py", line 229, in __enter__
self.acquire()
File "/sw/lib/python2.7/site-packages/lockfile/linklockfile.py", line 49, in acquire
time.sleep(timeout is not None and timeout/10 or 0.1)
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
File "/sw/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/sw/lib/python2.7/site-packages/pymatgen/io/abinitio/tasks.py", line 625, in save_lastnode_id
with FileLock(_COUNTER_FILE) as lock:
File "/sw/lib/python2.7/site-packages/lockfile/__init__.py", line 229, in __enter__
self.acquire()
File "/sw/lib/python2.7/site-packages/lockfile/linklockfile.py", line 49, in acquire
time.sleep(timeout is not None and timeout/10 or 0.1)
KeyboardInterrupt
from abipy.
Remove with FileLock
in the function save_lastnode_id
defined
in pymatgen/io/abinitio/tasks
The new version reads:
def save_lastnode_id():
"""Save the id of the last node created."""
with open(_COUNTER_FILE, "w") as fh:
fh.write("%d" % _COUNTER)
This change is OK if you are using abipy just for post-processing the results of the
calculation. Other parts of the code, in particular the API used to generate and
run calculations in a semi-automated way, rely on this lock to avoid race conditions.
from abipy.
Hello Matteo,
could you add print(type(value)) before returning so that we know the
kind of object that is returned by netcdf4 on your machine?
I indeed get numpy.ndarrays, as you predicted.
The numbers I get in the undefined variables are
-2147483647 (int32)
9.96920997e+36 (float64)
Could we use these numbers as a signal of an unset variable, or will
this also not be portable?
Cheers,
Bruno
On Mon, Apr 21, 2014 at 11:50 AM, gmatteo [email protected] wrote:
Hi Bruno,
Is my analysis below correct?
Yes, the problem is that many ETSF-IO files produced by abinit contain
variables that are defined but not initialized.
The decorator returns_None_onfail tries to detect this problem at run-time
so that abipy knows that the value is not initialized. It works on my Mac
and
on other linux architectures but obviously this trick is not portable.I guess that the problematic part is in the test:
This trick is needed because in many cases we define the netcdf variable# but we don't write its value.
return value if not isinstance(value, MaskedArray) else None
could you add print(type(value)) before returning so that we know the
kind of object that is returned by netcdf4 on your machine?
If it's a numpy array (and I think so) there's little to do at present.
I have generated the netcdf files on my mac (I don't remember the version
of netcdf4
I used) but the error you are reporting clearly shows that there's a
portability problem.The best solution would be to change the code in Abinit so that the netcdf
file
contains extra information that allows me to understand if the set of
k-points
represent a path or a homogeneous mesh.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/27#issuecomment-40945930
.
from abipy.
Thanks Matteo,
ipython now has a clean exit.
Cheers,
Temok
from abipy.
Could we use these numbers as a signal of an unset variable, or will
this also not be portable?
It won't be portable. In the bug tracker of netcdf4 I found a post that may
be related to this problem
which version of netcdf4 do you have?
import netCDF4
netCDF4.__version__
mine is 1.0.6
from abipy.
Hi,
I have version 1.0.8...
cheers,
Bruon
On Mon, Apr 21, 2014 at 4:10 PM, gmatteo [email protected] wrote:
Could we use these numbers as a signal of an unset variable, or will
this also not be portable?It won't be portable. In the bug tracker of netcdf4 I found a post that
may
be related to this problemUnidata/netcdf4-python#168Unidata/netcdf4-python#168
which version of netcdf4 do you have?
import netCDF4netCDF4.version
mine is 1.0.6
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/27#issuecomment-40971599
.
from abipy.
Related Issues (20)
- abistruct.py for eBAND HOT 8
- pytmatgen structure not abipy? HOT 1
- Schedular error
- Erroneous energy conversion?
- python3.6.2 package not found error
- error in import abilib HOT 7
- Private reporting of a potential security vulnerability HOT 1
- Incompatible with `numpy>=1.24.0`
- Incompatible with recent `pymatgen` HOT 5
- pydantic 2.0: `BaseSettings` has been moved to the `pydantic-settings` package.
- pinning the pymatgen version? HOT 3
- ImportError when using abicheck.py with abipy and pymatgen HOT 1
- `which` has been removed from `monty.os.path` HOT 1
- 🐛 `NameError`: name 'TYPE_CHECKING' is not defined
- Reintroduce support for Python 3.8?
- module 'pymatgen.io.abinit.netcdf' has no attribute 'ETSF_Reader' HOT 3
- Abiypy electron ebands.py part problem
- Local manager does not limit the number of running tasks
- Compatibility with pymatgen
- Monty type error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abipy.