drewblount / arff Goto Github PK

I have a fork of this library at

  https://bitbucket.org/sirex/arff

In this fork I implemented these missing features:

* sparse data format support

* ARFF date type support

* missing data marked with question symbol support

* fixed quote handling bug

All features covered with unit tests and tested wit python 2.7 and 3.2.

Hi,

I recently had the need for loading gzipped arff files and added the following 
code to allow file objects (such as those returned by gzip.open) to be directly 
parsed.

Here is the patch for arff/__init__.py:

237,240d236
< # Responsibility for closing fobj is left to the caller
< def load_file_object(fobj):
<     for item in Reader(fobj):
<         yield item
244c240,242
<         load_file_object(fhand)

---
>         for item in Reader(fhand):
>             yield item
> 

What steps will reproduce the problem?
1. Download iris.arff from anywhere 
(http://informatique.umons.ac.be/ssi/teaching/dwdm/iris.arff)
2. arff.load('iris.arff')

What is the expected output? What do you see instead?

Expected to load the data. This is the canonical dataset for ML, and an useful 
testcase. Please consider including into unittests.

Loader fails on
:273                 field_type_text = space_separated[2].strip()

In the file tabs are present in Attributes section. Calling split() with no 
arguments would solve this problem.

What version of the product are you using? On what operating system?

Ubuntu 10.04 Linux 64bit, Python 2.7, arff-0.9

When trying to parse an arff file that contains a line like
@attribute class real
like weka/data/cpu.arff from the weka package
the following error occurs:

Traceback (most recent call last):
  File "./excercise4.py", line 10, in <module>
    for row in arff.load(arff_filename):
  File "/home/moschlar/src/ws11-12/DM/practical_2/arff.py", line 162, in load
    for item in Reader(fhand):
  File "/home/moschlar/src/ws11-12/DM/practical_2/arff.py", line 195, in __iter__
    row_parser = _RowParser(fields)
  File "/home/moschlar/src/ws11-12/DM/practical_2/arff.py", line 139, in __init__
    self.tuple = namedtuple('Row', [f.name for f in fields])
  File "/usr/lib/python2.7/collections.py", line 280, in namedtuple
    raise ValueError('Type names and field names cannot be a keyword: %r' % name)
ValueError: Type names and field names cannot be a keyword: 'class'

I would suggest renaming the keyword then, but I think that's up to the choice 
of the author.

This patch fixes two issues I experienced with a nominal attribute that uses 
quoted strings as options. (It's the glass.arff file from weka's standard data 
directory).
- Using shlex to parse the option strings preserves quoted strings
- Stripping names for both types of quotes (Is propably needed somewhere else, 
too)

What steps will reproduce the problem?
1. Sparse format is not supported


Refer to http://www.mit.edu/~sav/arff/ for a description of what they support. 
Sparse format support would be a nice addition. That mit package is no longer 
maintained.

Request to support the 'hierarchical' attribute type.

What steps will reproduce the problem?
1. import arff
2. for row in arff.load(pathfile):
3.   print row[-1]

What is the expected output? What do you see instead?

Expected to print last element on the row. Insted getting error:
  File "./findCenter.py", line 11, in <module>
    print row[-1]
  File "/usr/lib/python2.7/site-packages/arff/__init__.py", line 105, in __getitem__
    return self._data[key]


What version of the product are you using? On what operating system?

Python 2.7 over Fedora 17

Please provide any additional information below.

The example data from UCI called ionosphere.arff can not be processed by
PyArff, cause this data contains empty line between @data tag and line
with the data. PyArff does not skip this line. It should.

Another problem was with sonar.arff file, which has a parameter called
"class". Cause this field name occurs often, maybe it would be good
approach to perform on-the-fly conversion of 'class' token to e.g.
'class_' ?

A valid arff files with following line in header won't be parsed because of the 
relation name, which contains spaces. Without spaces the file seems to be OK.

@RELATION "notebook usage"

What steps will reproduce the problem?

import arff
for row in arff.load('example.arff'):
    print row[-1]


Where this is the example file:
@relation diabetics_data
@attribute hair_color {blonde, black, blue}
@attribute age real
@attribute patno integer
@data
blonde, 17.2, 1
blue, 27.2, 2
blue, 18.2, 3







What is the expected output? What do you see instead?

Should print:

1
2
3

But this is happening:

Traceback (most recent call last):
  File "test.py", line 3, in <module>
    y = row[-1]
  File "/usr/local/lib/python2.7/dist-packages/arff/__init__.py", line 105, in __getitem__
    return self._data[key]
KeyError: -1

What version of the product are you using? On what operating system?

arff librry version 0.9
python 2.7
ubuntu 12.04

Arff files with following lines won't be parsed:
@ATTRIBUTE description STRING

Should be parsed correctly in the same way as following line is.
@ATTRIBUTE description string

Add support for python decimal type.