elife-asu / datadex

A system for organizing scientific datasets.

datadex

A system for organizing scientific datasets.

datadex's Issues

A way to reindex/rebuild the library

Way to identify existing files

We'd like to come up with a way to check if a file exists within a given folder. We talked about a couple ways of doing this, including:

• updating datadex to include the files within each folder recursively
• keeping a list of all unique file names across any folders
• others

Traverse directory tree for column headers

Description

If no column names are provided, the create_library method will attempt to use a params file in the current directory to determine them. This could easily result in missing column headers when subdirectories have params that the root directory does not.

It would be worthwhile to be able to traverse the directory tree and extract all column headers.

Create a header functionality

This would allow descriptions of the parameters within params to be documented by the user.

Does column header exist?

Description

When iteratively generating data, it would be useful to be able to determine if a column header exists in the library.

At present, the only solution is something like

if 'header_name' in dex.headers():
    pass
else:
    pass

Proposed Resolution

Create a method that queries the database for column header names, e.g.

def header_exists(header_name):
    pass

Select method

Duplicate params names

What happens with duplicate params names and indexing? need test for this

Use JSON for params format

Description

The current params format is atrocious. All we can really do, in an automated way, is append to the end of it. I would be nice to be able to easily modify params files programmatically. For that to be the case, we need to use a standardized file format.

Proposed Resolution

Use the JSON file format for params files.