saulpw / readysetdata

Scripts to make specific datasets cleaner and more convenient

Welcome to Saulville

I live in the terminal, at the intersection of data and fun:

• I created VisiData, a magical little utility for slicing and dicing data.
• The devottys (of which I am a founding member) are a small group of terminal fans who are constantly on the lookout for collaborators on small art projects and games.
• You can sponsor my work on Patreon or via Github Sponsors.
• Fun fact: In 2016, I uncovered a crossword scandal while cleaning the largest collection of crossword puzzles on the planet into my own text crossword format.

If you find this kind of stuff interesting, please reach out. Email me at [email protected], or come chat with us on IRC (libera.chat/#visidata) or on Discord. I'd love to talk with you!

readysetdata's Issues

Error logs

Fixed

title.principles.tsv.gz seems to have been momentarily corrupted. Made a PR with a try/except added, so at least the other tables would get built: #10

anja@allura:git/readysetdata ‹dougb_wpsummaries*›$ time make imdb
scripts/imdb.py -o output
4106s  76.09/408.01MB  (0.02 MB/s)  title.principals.tsv.gzTraceback (most recent call last):
  File "/home/anja/git/readysetdata/scripts/imdb.py", line 15, in <module>
    output_imdb('principals', 'title.principals.tsv.gz')
  File "/home/anja/git/readysetdata/scripts/imdb.py", line 9, in output_imdb
    rsd.output('imdb', tblname, rsd.parse_tsv(rsd.gunzip(fp)))
  File "/home/anja/git/readysetdata/readysetdata/output.py", line 20, in output
    with OutputTable(dbname, tblname) as out:
  File "/home/anja/git/readysetdata/readysetdata/utils.py", line 131, in parse_asv
    for line in Progress(it):
  File "/home/anja/git/readysetdata/readysetdata/utils.py", line 71, in __iter__
    for i, x in enumerate(self.iterator):
  File "/home/anja/miniconda3/envs/deluxedata/lib/python3.9/gzip.py", line 313, in read1
    return self._buffer.read1(size)
  File "/home/anja/miniconda3/envs/deluxedata/lib/python3.9/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/home/anja/miniconda3/envs/deluxedata/lib/python3.9/gzip.py", line 506, in read
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached
make: *** [Makefile:35: imdb] Error 1
make imdb  1959.46s user 551.53s system 19% cpu 3:37:23.65 total

Edit: title.principals.tsv.gz unzipped fine with gzip.