GithubHelp home page GithubHelp logo

marshallward / f90nml Goto Github PK

View Code? Open in Web Editor NEW
131.0 12.0 51.0 804 KB

A Python module and command line tool for working with Fortran namelists

License: Apache License 2.0

Python 96.34% Shell 0.46% TeX 3.20%
python fortran namelist parser

f90nml's People

Contributors

aekiss avatar barpaum avatar dalon-work avatar danielskatz avatar exphp avatar guziy avatar howol76 avatar jamesp avatar jenssss avatar letmaik avatar liampattinson avatar malmans2 avatar marshallward avatar martindix avatar naught101 avatar rgieseke avatar thepith avatar warrickball avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

f90nml's Issues

Keeping comments

It would be great to be able to keep comments that are in an original namelist in the following scenario:

  1. load initial file
  2. nml["opt"]["opt"] = "new value"
  3. remove original namelist file
  4. save the modified object to a file with the same name.

Thanks

Empty file raises StopIteration

An empty file should produce an empty Namelist, but appears to be trying to iterate over an empty iterator and prematurely raises StopIteration.

This should be fixed to not iterate (possibly by catching StopIteration) and produce an empty Namelist.

String variables mistmatch

Hello!

'f90nml' - as a great thing! Thanks!

I found some bugs, I suppose.

For example, if we have in namelist

&data_control_on_pressure_levels

params_on_pressure_levels = HGT, TMP, UGRD, VGRD, RH,

/

then on output we get

['HGT', True, 'UGRD', 'VGRD', 'RH']

At second, if

&data_control_on_pressure_levels

params_on_pressure_levels = HGT, FTMP, UGRD, VGRD, RH,

/

then

['HGT', False, 'UGRD', 'VGRD', 'RH']

I think, that bug in the fpy.py, in pybool function.

“TMP” is not “True”.

Test for first symbol is not good.

Thanks!

With regards,

Vladimir Krokhin, researcher, PhD,

Department of Meteorology and Tropical Cyclones,

Far Eastern Regional Hydrometeorological Research Institute,

Vladivostok, Russia.

Obscure real formats

So here's a good one:

I have some namelists with double variables that are formatted like this: blah= 0.1000000000000000+100,
This value is really 0.1D+100, but I think the format statement that originally wrote it didn't have enough digits to put the E maybe? However, Intel reads in this value just fine. f90nml reads it as a string: blah= '0.1000000000000000+100'. The code I have now outputs it properly, but I want to be able to read the older files too.

Would it be possible to include a case where a number with a format like this could be read in properly as a value?

This is a terrific library btw. I'm finally starting to use it for production work. Except for this weird edge case, It is working beautifully!

get not using __getitem__

I have a problem using patch and namelist with upercase.
It's because namelist defines getitem to properly handle lower/uppercase but the code in the paser uses the get method (line 131) which does not call getitem.
I don't know why get does use getitem nor if this applies to any python version... I'm using 2.7.10

Either redefine the get method in namelist obj or use

grp_patch = nml_patch[g_name] if g_name in nml_patch else {}

Command line interface and 1.0 roadmap

This is less of an issue and more of an annoucement that f90nml now includes a command line tool for modifying namelists through the command line.

The tool is not yet in master, but is available in the cli branch.

Basic usage is as follows:

  • f90nml <input> will convert to a namelist if possible. It "detects" this by extension. Currently only JSON and YAML (if installed) are supported. The output is streamed to stdout.
  • f90nml <input> <output> will save the output to the target file. If the extension is .yaml or .json then it will convert the file.
  • f90nml -g <grp> -s var1=val1<output> will either add or replace the variable var1 in namelist grp with value val1
  • f90nml -p ... will apply the change as a patch to an existing file, preserving comments and layout.

The (very minimal) runscript is in the bin directory, but will be useable after a python setup.py install.

I have been pushing out lots of 0.x releases for many years now, but I always had two major goals before considering a 1.0 release. The first was a command line interface, and the second was a native Python tokenizer, which could be improved over time. Now that those two goals have been met, I'd like to set a 1.0 release soon.

Before doing that, I'd like to get feedback on the command line tool, which still has many potential issues, but would also like to get a request for content in 1.0 before releasing.

Also, any suggestions for improving communication (mailing list, slack, etc) would be welcome.

Bump version on PyPi

Bug fix that was merged in #44 is not in the PyPi code repo so
pip install f90nml
installs the old version. Could you please update the version of the code hosted at pypi.

Namelist.__init__ does not convert list of dicts to list of Namelists

in line 40 in namelist.py

        # Convert any internal dicts to Namelists
        for key, val in self.items():
            if isinstance(val, dict):
                self[key] = Namelist(val)

there is a bug as there also needs to be a check for list of dicts so we can parse FORTRAN vectors of derived types correctly.

Write for vector variables

Writing the following namelist with the write method:

&vars
 a%b(2)%c = 1
/

produces the following error:

ValueError: Type <class 'f90nml.namelist.NmlDict'> of NmlDict([('c', 1)]) cannot be converted to a Fortran type.

It seems to be because the first element of b is not there. Note that the read method worked properly, so the bug is only in write.

Furthermore, if the namelist is instead:

&vars
 a%b(1)%c = 1
/

then the write method produces:

&vars
    a%b%c = 1
/

I'm not sure what a compiler would do with this on a namelist read. To be safe, it should preserve the index of b.

Ambiguous output for vectors with v[-1] = None

If we output a vector of value None as its final value, then f90nml only outputs a single trailing comma. For example:

v = [1, 2, 3]   => v = 1, 2, 3
v = [1, 2, None] => v = 1, 2,
v = [1, 2] = > v = 1, 2

As a namelist entry, the second and third are identical. We should output an extra comma if the final value of a vector is None.

Although Fortran will correctly interpret this output by simply ignoring the third value, this could possibly lead to issues if f90nml parses a file multiple times.

Derived type patch doesn't replace values, it just adds them

f90nml.patch doesn't actually modify any of the existing values, if just adds new copies according to the patch.

I'm not sure whether this is a huge problem - it doesn't appear to be a problem for the model I'm patching namelist files for, but it might be for others. Either way, it kind of leaves the namelist a bit messy, and could get bad if you're doing sequential edits.

Of course, removing lines with comments on them makes for potential data loss, so perhaps this in contingent on f90nml supporting comments. Is that ever likely to happen?

Protected Namelist functions in Parser

The Parser class is using some nominally private functions of Namelist.

This is not a major issue, but it does show that our Namelist API is still not hugely mature and needs to be cleaned up to some extent.

Possible bug report for multiple dimension arrays

Hi, this is a useful tool. I found a possible bug when trying to read multiple dimension arrays.
Here is the example namelist file:

&TEST
  a = '12abc'
  ltest = 1
  RBC(0,0) =   1.33100000000000E+00     ZBS(0,0) =   0.00000000000000E+00
  RBC(9,6) =  -8.49639241991135E-05     ZBS(9,6) =  -1.31291865009956E-04
/

What we get by using f90nml.read:

Namelist([('test', Namelist([('a', '12abc'), ('ltest', 1), ('rbc', [[1.331], [], [], [], [], [], [None, None, None, None, None, None, None, None, None, -8.49639241991135e-05]]), ('zbs', [[0.0], [], [], [], [], [], [None, None, None, None, None, None, None, None, None, -0.000131291865009956]])]))])

Apparently, the array data is not correct. Would you pleas help fix the bug? Or if you can point out which part is related to this function. I can help contribute.

String line continuation removal

I'm using f90nml (many thanks for your library!) in https://github.com/openclimatedata/pymagicc

I think this commit in 0.22 broke reading of some nml files for me:
b46fffe

The wrapped Fortran binary is producing multi-line strings. Not sure how common these are in Fortran. I don't think there are necessary in my case.

Would it be worth having this as an option? I try to work around this in my files.

In any case might be good to put this in the changelog as well?

Array support

Arrays inside types don't seem to work properly. Example:

 &test2
 g%b(1) = 599,
 g%b(2) = 504,
 g%b(3) = 502,
 g%b(4) = 503,
 g%b(5) = 501
 /

When read using:

n = f90nml.read('test2.nml')
print json.dumps(n, sort_keys=True, indent=4, separators=(',', ': '))

produces:

{
    "test2": {
        "g": {
            "b": [
                null,
                null,
                null,
                null,
                501
            ]
        }
    }
}

Zero-indexed array indices are ignored

If we have a zero-indexed array, e.g.

&nml_group
    y(0:3) = 1, 2, 3, 4
/

then it incorrectly interprets 0 as None (due to a bug in FIndex) and uses the default value of 1. The above case fails because there are more than 3 values in the array and the iterator runs out of indices.

Comment char inside namelist group

Not sure if this is even legal namelist syntax, but f90nml didn't like this:

# Time stepping parameters
 &PARM03
#startTime=0,
#endTime=311040000,
#endTime=12000.0,
 deltaT=1200.0,
 ntimesteps=10,
 abEps=0.1,
 pChkptFreq=2592000.0,
 chkptFreq=120000.0,
 dumpFreq=2592000.0,
 monitorSelect=2,
 monitorFreq=1.,
 &

and munged it into this:

&parm03
    starttime = 0, '#'
    endtime = 12000.0, '#'
    deltat = 1200.0
    ntimesteps = 10
    abeps = 0.1
    pchkptfreq = 12000.0
    chkptfreq = 0
    dumpfreq = 2592000.0
    monitorselect = 2
    monitorfreq = 1.0
    niter0 = 0
/

Read namelist into Python object

I was wondering if a good feature might be to be able to read the namelist into a Python object, rather than a dictionary. So, for example, rather than accessing the variables like:

z = nml['x']['y']['z']

we could use:

z = nml.x.y.z 

That would be pretty slick, I think.

Empty value with trailing comma: comma is not preserved

f90nml turns this:

# Input datasets
 &PARM05
 bathyFile='topog.box',
 hydrogThetaFile=,
 hydrogSaltFile=,
 zonalWindFile='windx.sin_y',
 meridWindFile=,
 &

into this:

&parm05
    bathyfile = 'topog.box'
    hydrogthetafile =
    hydrogsaltfile =
    zonalwindfile = 'windx.sin_y'
    meridwindfile =
/

and MITgcm doesn't like those empty entries. I'll leave it to you and MITgcm to figure out who is right.

Numpy types

I was wondering if it would be possible to update the representation function in namelist.py to support numpy types? Your code is not using them, but I have a use case where I'm using your code to read a namelist, and then I have APIs to allow manually adjusting some of the values. I'm allowing numpy values if the user wants to pass them in, but your code is only looking for bool, float, etc., so it is crashing in write() with a ValueError.

I think it could be done without creating a numpy dependency in your code, since the numpy types are instances of the default types. For example:

>>> import numpy as np
>>> a = np.float64(1.1)
>>> type(a)
<type 'numpy.float64'>
>>> isinstance(a,float)
True

What do you think?

namelist group order should be preserved

Namelist group order is not preserved in f90nml write, but several Fortran executables seem to expect to read these namelists in sequence (i.e. no IO cache?)

So this can break many read->write operations.

We need to either replace this with an ordered dict or use a list for the namelist groups.

I don't believe that this affects variables inside a namelist group.

Namelist constructor overwrites properties of namelists

When a Namelist is passed through the Namelist constructor, it overwrites its properties with default values, which are usually empty. This is particuarly bad for the start_index property, which deletes all of this information.

The CLI currently has to deal with this bug, and currently avoids the issue by checking if the input argument is a Namelist. But this check should be unncessary and the constructor should not have this issue.

1-based indexing is assumed but incompletely

I'm using f90nml ver. 0.23. Thank you for the nice package.

In the var_strings method of the Namelist class, please consider replacing the expression

i_s = v_start[::-1][len(v_idx)]

with

i_s = v_start[::-1][len(v_idx)] if v_start is not None else 1

The method has two instances of this expression, and one of them sometimes raises an exception when my code calls the write method on its Namelist object:

TypeError: 'NoneType' object has no attribute '__getitem__'

It's quite possible that my own code is entirely at fault for the exception. Only one of my ~1500 test cases triggers the exception, and I haven't yet figured out what's special about this one case.

f90nml.patch: overwriting a longer array produces inconsistent results

#35 reminds me of something confusing about f90nml.patch. Overwriting a longer array does not result in a completely new array in the target namelist, trailing items remain after the leading corresponding items have been replaced, which is inconsistent with the object returned by the patch function.

>>> f90nml.read('test1.nml')
Namelist([('seq', Namelist([('fib', [1, 1, 2, 3, 5])]))])
>>> f90nml.patch('test1.nml', {'seq': {'fib': [1, 2, 3, 5]}}, 'test2.nml')
Namelist([('seq', Namelist([('fib', [1, 2, 3, 5])]))])
>>> f90nml.read('test2.nml')
Namelist([('seq', Namelist([('fib', [1, 2, 3, 5, 5])]))])
>>> print open('test2.nml').read()
&seq
  fib = 1, 2, 3, 5, 5
/

In the target namelist, the array is modified rather than added new copies, so I assume it's a different issue from #35.

token logic error?

Hi. Thanks for your work here! When I read a file with the following contents

&column
   name=Humidity
   units=percent
   type=double
/

I get

Namelist([('column',
           Namelist([('name', 'Humidity'),
                     ('units', 'percent'),
                     ('type', 'double')]))])

as expected.

But when I read a file like this

&column
   name=Humidity
   units=%
   type=double
/

I get something very unexpected:

Namelist([('column',
           Namelist([('name', 'Humidity'),
                     ('units', None),
                     ('=', Namelist([('type', 'double')]))]))])

Is there something you can change to make this parse correctly?
I'm expecting this:

Namelist([('column',
           Namelist([('name', 'Humidity'),
                     ('units', '%'),
                     ('type', 'double')]))])

I guess the problem might be in this line or this line , but I'm not sure.
Thanks!

maximum length of variable names

I'm seeing an issue with very long variable paths:

&a
bbbbbbbbbbbb%ccccccccccccccccccccccc%ddddddddddddddd%eeeeeeeeeeee = 1
bbbbbbbbbbbb%ccccccccccccccccccccccc%ddddddddddddddd%eeeeeeeeeee = 2
bbbbbbbbbbbb%ccccccccccccccccccccccc%ddddddddddddddd%eeeeeeeeee = 3
/

when I read/write it comes out as:

&a
    bbbbbbbbbbbb%ccccccccccccccccccccccc%ddddddddddddddd%eeeeeeeeeeee =
    bbbbbbbbbbbb%ccccccccccccccccccccccc%ddddddddddddddd%eeeeeeeeeee = 2,
    bbbbbbbbbbbb%ccccccccccccccccccccccc%ddddddddddddddd%eeeeeeeeee = 3,
/

Only extract a specified namelist

Idea... Say there was a file with multiple namelists, and you only wanted to read a specified one. Maybe this could be an option, where you specify the one you want, rather than having to parse the whole file. (related to #30 if the file is very large and the parsing is a bottleneck if you only want certain info from the file).

can't parse group names that contain a period

I have a namelist from GFDL that contains this group:

&diag_inESM2_Control_216.xmltegral_nml
file_name = 'diag_integral.out',
time_units = 'days',
output_interval = 1.0
/

for which f90nml.read gives the following error:

File "/Users/andy/bin/nmltab.py", line 49, in nmldict
nmlall[nml] = f90nml.read(nml, strict_logical=False)
File "/Users/andy/Documents/COSIMA/github/aekiss/f90nml/f90nml/init.py", line 53, in read
return parser.read(nml_path)
File "/Users/andy/Documents/COSIMA/github/aekiss/f90nml/f90nml/parser.py", line 108, in read
return self.readstream(nml_file, nml_patch)
File "/Users/andy/Documents/COSIMA/github/aekiss/f90nml/f90nml/parser.py", line 125, in readstream
toks = tokenizer.parse(line)
File "/Users/andy/Documents/COSIMA/github/aekiss/f90nml/f90nml/tokenizer.py", line 71, in parse
raise ValueError
ValueError

I've confirmed that the error goes away when the period is removed from &diag_inESM2_Control_216.xmltegral_nml

Some bug in write

The following namelist triggers some sort of bug in the write routine:

 &blah
  a%vab_bbb%bbb = 1,
  a%vab_fffff%mmmmmmm%i = 399,
  a%c = 10000.0
 /

The code:

import f90nml
n = f90nml.read('test.txt')
n.write('test.nml')

produces the file:

&blah
    a%vab_bbb%bbb = 1
    a%vab_fffff%mmmmmmm%i =
    a%c = 10000.0
/

Interestingly, if you delete one of the m's in mmmmmmm , it works.

Write bug in v0.20

I'm getting an odd bug when trying the write the following namelist:

 &a
 b%c_d_E(1)%ID = 1,
 b%c_d_E(2)%ID = 2,
 /

Code is:

    import f90nml
    data = f90nml.read('test.nml')
    with open('test_RESAVED.nml','w') as f:
        data.write(f, force=True)

Error message is:

Traceback (most recent call last):
  File "test.py", line 309, in <module>
    data.write(f, force=True)
  File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 232, in write
    self.write_nmlgrp(grp_name, grp_vars, nml_file)
  File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 253, in write_nmlgrp
    for v_str in self.var_strings(v_name, v_val, v_start=v_start):
  File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 302, in var_strings
    v_strs = self.var_strings(v_title, f_vals, v_start=v_start_new)
  File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 313, in var_strings
    i_s = v_start[::-1][len(v_idx)]
TypeError: 'NoneType' object is not subscriptable

Oddly, when I change c_d_E to c_d_e, it works fine.

I just updated to the latest release using pip. I've been able to read variables like this before, so something has changed.

Unspecified starting indexes are saved as if using 1-based index during `write`

The default starting index is 1, but if the vector does not specify a starting index, then it should not save the file with an explicit starting index.

This is because we use default_start_index to automatically set an unspecified starting index. The explicit value for start in the Namelist is then used to explicitly write the starting index.

For example,

&nml
    x(:) = 1, 2, 3
/

is, after f90nml.read().write(), saved as

&nml
    x(1:3) = 1, 2, 3
/

The starting index should not be saved and used in the output if it is not explicitly set.

Installation documentation without root privileges

Hi,

Thanks a lot for your work, it's going to make my life way easier!

There's a little typo in the readme for installation without root privileges, you need to invert install and --user:
python setup.py install --user

Cheers

File APIs

Feature request:

Currently f90nml only accepts filenames for its input and output files. This generally restricts the usage of f90nml in comparison to other serialization libraries in Python, which usually have methods for working with arbitrary file-like objects and string data (E.g. json and pickle have dump/load for file-like objects and dumps/loads for strings. xml.etree has parse which accepts both filenames and file-like objects, and fromstring for strings.)

For some examples of how these APIs are useful: Currently, it is not possible to parse an NML file directly from sys.stdin or to write one to sys.stdout, because these are already-open files with no path. It is similarly difficult to apply multiple patches in a row (whereas with a string api it would just be for patch in patches: s = f90nml.patchstr(s, patch)). Currently I work around these limitations using NamedTemporaryFile, which has a fair number of quirks about it and is something I generally consider to be a last resort.

Some thoughts:

  • If it's too much to add both, either one will do. It's easy for client code to write file-based wrappers around string functions (by calling f.read(), f.write()), or to write string-based wrappers around file functions (using StringIO).
  • because there is already patch which takes two files, I feel that the API would be cleanest if read, write, and patch were modified to detect if their arguments are paths or a file-like object (e.g. with hasattr(infile, 'read')). This way a user could mix and match, e.g. read from a file object and write to a path. In any case, though, strings will require their own functions (e.g. readstr(s), writestr(nml), and patchstr(s, patch) or however you want to name them).
  • I suppose there is also the question of whether patchstr should return just a string, or if it should return (string, nml) (since patch returns an NML).
  • One slight hurdle is that, unlike a path-based API, a file-based API should not call close. I bring this up because there are several calls to close() currently embedded inside the parsing functions.

Nested Types

This is probably related to Issue #7. The following namelist with nested types:

 &test4
 f%g%a = 1,
 f%g%b = 399,
 f%g%c = 0.0000000000000000D+00,
 f%g%d = 301,
 f%g%e = 1
 /

reads improperly as:

print json.dumps(f90nml.read('test4.nml'), sort_keys=True, indent=4, separators=(',', ': '))
{
    "test4": {
        "f": {
            "g": {
                "e": 1
            }
        }
    }
}

Although note that

 &test5
 g%a = 1,
 g%b = 399,
 g%c = 0.0000000000000000D+00,
 g%d = 301,
 g%e = 1
 /

reads properly as:

print json.dumps(f90nml.read('test5.nml'), sort_keys=True, indent=4, separators=(',', ': '))
{
    "test5": {
        "g": {
            "a": 1,
            "b": 399,
            "c": 0.0,
            "d": 301,
            "e": 1
        }
    }
}

Preserving index for single-element arrays

As mentioned in issue #10 , the following namelist

&test
    a%b(1)%c = 1
/

is loaded and saved as

&test
    a%b%c = 1
/

We ought to see if there is some way to distinguish between these cases (even if Fortran does not). Compatibility also needs to be investigated.

String-based APIs

Broken off from #25.

Add string versions of read, patch, and write (under different names), comparable to json.loads or xml.etree.fromstring. With the file APIs in place, it should be possible to write all of these as simple wrappers using StringIO.

encoding f90str('string with a backslash \\')

Thank you for publishing f90nml. I'm using v0.17.

In the Namelist class (f90nml/namelist.py), the definition for self.f90str should accommodate character strings with backslash characters. Something like:

            str:
                lambda x: repr(x).replace("\\'", "''").replace('\\"', '""').replace('\\\\', '\\'),

instead of

            str:
                lambda x: repr(x).replace("\\'", "''").replace('\\"', '""'),

To be sure, the definition should also accommodate still other characters that Python escapes, including \a, \b, \f, \n, \r, \t, and \v, but I'm not sufficiently familiar with fortran namelists to know how to correctly encode these.

Namelists from multidimensional list of lists fail

The following dict will fail with a TypeError:

d = {'md': {'x': [[1,2,3],[4,5,6],[7,8,9]]}}
n = f90nml.Namelist(d)
print(n)

I get this error:

  File "~/python/f90nml/f90nml/namelist.py", line 334, in var_strings
    i_s = v_start[::-1][len(v_idx)]
TypeError: 'NoneType' object is not subscriptable

If I "fix" this by assigning a default value to i_s, then I get a garbage index:

&md
    x(, 1) = 1, 2, 3
    x(, 2) = 4, 5, 6
    x(, 3) = 7, 8, 9
/

I think this is related to the issue described in the FIXME comment notes.

Appending to existing file

I was wondering if a good option for write might be to allow for appending to an existing file? This would be an easy way to put some comments in the file first if you wanted, and then write the namelist.

Multi-Dimensional Arrays

I'm interested in using f90nml for my work, but just need the multi-dimensional array support. I'm willing to do the work for it, but was wondering what your thoughts are on how to put the support in. I found the TODO on line 182 of parser.py, so I assume most of it will need to go in there somewhere.
Thanks.

Cosmetic Ideas

Everything seems to work great now. So, now all I have are some possible cosmetic suggestions. These are somewhat minor, so feel free to ignore them.

  • Maybe have the ability to specify for the write method the string to use for the True and False values. Currently it always uses .true. and .false.. Intel (and probably others), also accepts T and F.
  • Maybe also have the option to specify if we want to force a comma to be written after each variable. It looks like currently it only does this for strings that contain commas and to separate array elements? The compiler seems to be OK with that. However, somebody out there may be parsing namelists and expecting the ending commas (I think I had something like that at one point).
  • Maybe allow caller to specify the number of spaces to be used for indenting?

Perhaps these could be optional arguments? Something like:

n = f90nml.read('test.nml')
n.write( 'test2.nml', force=True, indent=2, true_str='T', false_str='F', force_commas=True )

Again: minor stuff, but maybe nice to have (there could be other options as well).

Optimizing performance

I was wondering if there were possibilities for optimizing performance for very large namelist files. I'm in the process of doing some benchmarks for some large files (say around 1500 lines, with multiple namelists and pretty much all variable types). I can only get around 1.7 calls of f90nml.read() per second. I assume that it's the parsing and/or creating of the structures that is the bottleneck (actually reading in the lines should only take a fraction of a second), but I'm going to investigate further.

One thing I was wondering if parsing of different namelists in the same file could take place in parallel? I don't know if such a thing is possible or not, but if it is that might be something to explore. Maybe this is something I can try and contribute to, rather than just reporting bugs and asking for features!

init Namelist from dict

I would like to generate namelists based on (nested) dictionaries.

f = io.StringIO()
n = f90nml.Namelist({'sec': {'entry': [1, 2, 3]}})
n.write(f)

But this does not work due to non-existing start_indices.

>           i_s = v_start[::-1][len(v_idx)]
E           TypeError: 'NoneType' object is not subscriptable

I find it also problematic, that the call to writestream() from write() is in a try-finally clause, so errors are not raised.

I did not find anything like the default_start_index property as in the Parser class for the Namelist class. What is the best way to achieve the overall goal of writing a namelist from dict?

Mismatched vector rank

Two assignments of different rank to a vector raise an error:

&idx_nml
    x(1, 1) = 1
    x(2) = 2
/

This also causes runtime errors in gfortran and Intel fortran, but it would be good to catch and manage this error.

crash

f90nml.read() crashes when reading a file with the following namelist which I think is valid:

&parameter name=dnux/dp, symbol="$gx$r$bx$n", units="1/(2$gp$r)", description="Horizontal chromaticity", type=double, &end

I'm expecting:

Namelist([('parameter',
           Namelist([('name', 'dnux/dp'),
                     ('symbol', '$gx$r$bx$n'),
                     ('units', '1/(2$gp$r)'),
                     ('description', 'Horizontal chromaticity'),
                     ('type', 'double')]))])

I guess the / in the name field is tripping up your state machine.

Verbose array printing

Feature suggestion:

If the final variable within a derived type is an array, currently write will produce a file like this:

    a%b%c = 1, 1, 1, 

Would it be possible to have the option to write them line by line like so:

    a%b%c(1) = 1,
    a%b%c(2) = 1,
    a%b%c(3) = 1,

Note that if a or b are arrays, then it does print them line by line.

f90nml.read() crashes for namelists without a terminating EOL.

I'm trying to transform a list of configuration files into a list of f90nml.Namelist objects.
StopIteration is being raised by Tokenizer.update_chars().

For example, let's say I have two nml files, foo.nml and bar.nml, this bug can be triggered by :

[f90nml.read(s) for s in ['foo.nml','bar.nml']]

Variables with mix of individual and repeated values

Hi, I'm having problems when reading variables for which some, but not all, of the elements are listed using repeat factors. Any values that are read after a repeat factor are also being repeated.

For example, with a namelist file namelist.nml, containing the namelist

&my_namelist
    foo = 3*1,2,3,4
    bar = 3*1,2*2,3
/

Reading it with

import f90nml

filename='namelist.nml'
namelist = f90nml.read(filename)

The variables in the dicts have too many copies of the values that come after the repeat factors

>>> namelist['my_namelist']['foo']
[1, 2, 2, 2, 3, 3, 3, 4, 4, 4]
>>> namelist['my_namelist']['bar']
[1, 1, 1, 2, 2, 3, 3]

It looks like the repeat factor (n_vals) is being stored when reading the next value in the list.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.