GithubHelp home page GithubHelp logo

ethall / syphon Goto Github PK

View Code? Open in Web Editor NEW
0.0 0.0 2.0 148 KB

A storage and management engine for CSV data. Moved to:

Home Page: https://github.com/tektronix/syphon

License: MIT License

Python 100.00%
pandas python python3

syphon's People

Contributors

ethall avatar

Watchers

 avatar  avatar

Forkers

tektronix

syphon's Issues

Make a data generation tool

Useful for testing purposes.

Requirements:

  • specify the size, in entries, of the resulting cache
  • generate new import data
  • way to specify headers?

Not sure how the tool will figure out how many data (and optional metadata) files to add.

Keep it simple!

Make command line arguments sane

Delusions of grandeur made the command line argument system a bit unwieldy.

Did you know?
This syphon call will complain about not having metadata files:
py -3 syphon.py import --data /some/import/path
Better add a -q flag to force it to try and find those metadata files! This call will search the import directory for metadata files:
py -3 syphon.py -q import --data /some/import/path

Also, the quiet flag is dumb. Its default state should be quiet. Who cares about super awesome percentage indicators? Dump them both. Use verbosity flags instead.

Look for additional 'features' to dump. Maybe the switch subcommand?

Call out non-existent paths when attempting to archive

If you make a typo in the directory name, it is difficult to understand what went wrong.

Here's the output when I accidentally passed a path that didn't exist:

Traceback (most recent call last):
  File "C:\Python3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Source\syphon\syphon\__main__.py", line 79, in <module>
    exit(_main(argv))
  File "C:\Source\syphon\syphon\__main__.py", line 63, in _main
    archive(this_context)
  File "C:\Source\syphon\syphon\archive\archive.py", line 34, in archive
    lock_list.append(lock_manager.lock(split(data_list[0])[0]))
  File "C:\Python3\lib\site-packages\sortedcontainers\sortedlist.py", line 641, in __getitem__
    raise IndexError('list index out of range')
IndexError: list index out of range

Probably best to raise an exception before we try to create a lock file.

Create a plugin system

Plug-in system shouldn't be too complex. Maybe check the contents of a directory for plugin files?

Plugin system should allow for things like data format verification, format conversion, value mapping, etc.

API documentation would need to be created.

Add multi-file archive unit tests

Tests should include:

  • multiple data files & no metadata files
  • multiple data files & 1 metadata file
  • multiple data files & multiple metadata files (whose names form 1:1 file pairs)
  • 1 data file & multiple metadata files

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.