iemejia / catho

A file catalog utility inspired by the awesome Robert Vasicek's Cathy project. Or my excuse to hack something that I really need.

Hi there 👋

catho's Issues

Implement rm option

an option to optimize the databases

like 'git gc', it executes ANALYZE; and then VACUUM; in sqlite

incompatible import with python 3

The import:
from utils import get_file_info
throws an error with python 3

Implement help

--help -h

create path traversal method

similar to os.path, but for catho paths.

Add decent logging

including debug and verbose options.

cathod service

a runtime service that automatically indexes modified files and upgrades the catalog.

Encrypt database

for security reasons in case of use in cloud servers
-e --encrypt
we have to decide a schema

add option to continue indexing

add -c to enable the completion of an unfinished indexing, or the update of a catalog in case of changes.

Operate over the metadata

To add, or change values such as basepath
something like -set-meta key, value

Implement ls option

Memory Error

When indexing really long volumes

implement args parsing with argparse

Error indexing names with quotes in continuous indexing

if a file name includes the character ' in it it throws an error when it builds the query.

Implement cathy import option

Implement a Cathy (.cat) importer

TypeError on listing the catalogs

$ catho/catho.py ls
Traceback (most recent call last):
File "catho/catho.py", line 334, in
logger.info(catalogs_str())
File "catho/catho.py", line 226, in catalogs_str
date = str(datetime.fromtimestamp(timestamp))
TypeError: a float is required

Implement hashing

Add tag support

We have to discuss what to support:

• filesystem tags ?
• user generated tags ?
• inferred tags (e.g. .mp4 = video, or .mp3 = audio ?

Save only relative paths

When saving catalog, save the full path in the metadata.
And only relative paths in the catalog table.
Currently it works for ., but it fails with the system auto completed paths (e.g. ~)

Add fdupes command

Based on the idea of:
https://github.com/seabre/finddupes/blob/master/finddupes.sh
SELECT hash, location FROM files WHERE hash NOT IN (SELECT hash FROM files GROUP BY hash HAVING ( COUNT(hash) = 1 )

create a configuration mechanism

something similar to .cathoconfig
that allows the minimum possible configurations, e.g. string output format

Create PIP Installer

Write results to stdout

Writing with the logger to stderr by default prevent of using unix tools like grep for filtering, for example $ catho/catho.py find \* media | grep Animal just print 133652 records from the test catalog.

There is a workaround, redirecting stderr to stdout, but it is not really straight forward.
$ catho/catho.py find \* media 2>&1 >/dev/null | grep Animal

Error with unicode characters in paths

Create tests

Index metadata (e.g. id3tags, etc)

Fix test so they don't create again the database when it's not needed

Error indexing when a file has the same name of a directory

try to index e.g. /file/file and boom, the problem is in the path_block_iterator, in the way it builds the path names.

Implement scan option

Configure Continous Integration via Travis

refactor to pep-8 standard

Support detection of repeated folders

A function to detect repeated folders (if n hashes are equal per folder there is a possiblitiy).
How to optimize ?

Create a mechanism to ignore paths or filetypes

Probably a file similar to .gitignore, because folders like .git or .svn can return many false positives.

Error when calculating hash of BIG files

I found it trying to index a VM that's bigger than the memory in my local machine:

Python(15742) malloc: *** mmap(size=140645843243008) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
File "catho/catho.py", line 303, in
for files in filesubsets:
File "catho/catho.py", line 77, in file_get_filelist
hash = file_hash(fullpath, hash_type)
File "catho/catho.py", line 50, in file_hash
sha1.update(f.read())
MemoryError

Error parsing hardlinks

Not such file or directory while parsing a hardlink

$ ls -al ~/Dropbox/iPad/
lrwxr-xr-x 1 rgamez staff 38 18 feb 2012 test.pdf -> /Users/rgamez/Documents/test.pdf

$ ./catho.py add iPad ~/Dropbox/iPad
Creating catalog: iPad
An error occurred: [Errno 2] No such file or directory: '/Users/rgamez/Dropbox/iPad/test.pdf'

Return the name of the catalog in the find

Following discussion #7 (comment)

Refactor internal representation

Remove the global catalogs variable and put it in an object
Create class for the DAO operations (maybe the same)

Add progress when adding new catalog

Implement as REST web services

Incremental scan (useful also for cathod)

Support native binaries, (.exe file) via py2exe

check if database exists and abort

Optional: create option -f to force creation.

fix space in catho ls output

output of catho ls should be more consitent in columns, something like ls -l

Something to detect junk-files

something to remove useless files (.DS_Store, *.url, Torrent downloaded from, etc)

List catalog information

with the ls command plus the name of the catalog, it should display metadata and eventually the contents.

create a GUI for catho

It could be a textual or graphical gui

Write docs, tutorial

Tutorials:

• How to use catho
• Encrypting your catalog
• Synchronizing with dropbox

Implement search of find option

Add path in metadata

add before init is not fatal error

if ~/.catho doesn't exist because catho init hasn't been executed, errors are displayed but the parsing of the directory continues.

$ ~/catho/catho/catho.py add Home ~
Creating catalog: Home
An error occurred: unable to open database file
An error occurred: unable to open database file
An error occurred: unable to open database file
$ ls ~/.catho
ls: cannot access ~/.catho: No such file or directory

Two options/suggestions @iemejia

• It's a fatal error that ends with the suggestion of executing catho init.
• ~/.catho it's "touched" before adding any catalog and perhaps any operation.