tbarron / backscratcher

"Every good work of software starts by scratching a developer's personal itch." Eric S. Raymond

                        Backscratcher

"Every good work of software starts by scratching a developer's personal itch."

• Eric Raymond

This is a collection of small programs I have developed over the years to take care of various tasks on the computer. Perhaps the one I use most is 'tps' (turbo-ps), which lets me grep the output of ps with strings rather than pids.

Another one I use a lot is fx (effects). It can do a number of tricks to issue a command on each of a collection of files, even commands that ordinarily only operate on a single file.

align.py
    Read a sequence of text lines and align their contents into
    columns.

ascii.py
    Display the ASCII collating sequence.

calc.py
    Simple calculator/expression evaluator.

chron.py
    Timer. Can count up (like a stopwatch) or down (like a kitchen
    timer).

clps.py
    Command Line Password Safe. Stores (host, user, password)
    tuples in an encrypted file, can copy a password to the
    clipboard for pasting into a password prompt without
    displaying it on the screen. Mac-specific, since it uses
    pbcopy and pbpaste.

dt.py
    Easy date arithmetic.

errno
    Feed it a number from errno.h, get back the symbolic name and
    meaning, or vice versa.

fab.py
    Poor man's make.

filter.py

fl.py
    File manipulations. Copy atime to mtime, mtime to atime; diff file
    against previous revision; revert file to previous revision; remove
    \r from a file (or add them); display all times associated with a
    file; make backup copy of a file; find unreadable files.

fx.py
    Command line effects. Turbo xargs; structured batch renames; etc.

hd.py
    Hexdump.

list.py (list.pl)
    Set arithmetic applied to lists generated by Unix commands.

mag.py (magnitude)
    2384192384283 -> 2.17 Tb

mcal
    Display analog calendar in various formats.

msh
    Start ssh with a control socket so that multiple sessions can
    piggyback over the same connection without repeated
    authentication.

odx.py (odx.pl)
    Report the octal, decimal, and hexadecimal variants of a
    number.

plwhich
    Which for the perl installation. eg., where does Data::Dumper live?

ptidy
    Cleanup up emacs debris.

pytool.py
    Generate python templates.

replay.py (replay.pl)
    Run a command over and over and watch its output.

rxlab
    Play with regular expressions.

scanpath
    Where in my $PATH is foo?

summarize.pl
    Apply "artificial ignorance" to a set of files.

testhelp.py [deprecated -- remove]
    Testing utility routines.

toolframe.py [deprecated -- remove]
    Easy launching for tool-style and simply python programs.

tpbtools.py
    Utility routines.

tps
    Turbo-ps -- find processes.

truth_table
    Generate truth tables for an arbitrary number of variables.

vipath
    Edit $PATH.

wcal
    Wide cal. Three months side by side.

workrpt.py
    Read my work log and generate a report.

wxfr
    Bulk file transfer.

xclean
    Remove emacs backup files (*~) and other debris left behind by work
    processes

You may notice that some of these are written in only perl while others have a python version and some are only in python. I learned perl first and used it alongside the tcl-based expect tool until I discovered python and realized that it provides pretty much everything perl and expect do in a single tool. I'm not a performance wonk, I just want to get the job done, so I didn't worry too much about whether python is as fast or efficient as perl. I just jumped in.

So now I'm in the middle of converting all these programs to python.

Even more recently, I have started using Perl::Expect. So now I know that perl can do expect-type stuff, too.

---

Regression tests now live in directory test.

backscratcher's Issues

toolframe.py tests?

Does toolframe.py have all the tests it could?

filter

2009-02-10 22:17: complete test suite in filter.py
2009-02-10 22:04: write perl version (filter.pm)
2009-02-10 22:04: add to git repo
2010-11-27 12:33: add -L code to link $@ -> $@.py, get rid of $@
as a separate file

add tests to hd

magnitude should use MB, MiB properly

Distinguish between MB (1,000,000 bytes) and MiB (1024 * 1024 bytes) correctly.

Provide a command line option for selecting which set of units to report.

fab: add function diff

This will look at the files called out by 'status -a' and report their diffs from the installed files.

clps: the load and save commands should cache the most recently used filename

so after loading (or saving) a file, saying 'save' with no argument should save the same filename.

pfind -n '*.py' -x 'xxx, yyy' should exclude dir 'yyy' but does '<space>yyy'

when splitting a comma delimited list from the command line, strip each element before using it

wcal

2009-02-10 21:59: add license info
2009-02-10 21:59: write python version
2009-02-10 21:59: add to git repo
2010-11-27 12:33: add -L code to link $@ -&gt; $@.py, get rid of $@
as a separate file

tests for 'workrpt -m'

Also want to be able to limit -m selection by dates. Eg.:

• workrpt -m RGX # match RGX across all dates
• workrpt -m RGX -s 2001.0101 # match RGX after 2001.0101
• workrpt -m RGX -e 2010.1231 # match RGX before 2010.1231
• workrpt -m RGX -s 2001.0101 -e 2010.1231 # match RGX between 2001.0101 and 2010.1231

dt.py: accept an epoch time as a 'whenspec'

add a test for this

add code to satisfy the test

all_tests: script to run all tests

Write a script that globs the current directory for *.py, imports everything, loads all the test classes, and runs all the tests.

msh: tolerate domain suffix on hostname

If the domain suffix is present on the hostname provided by the user, don't add it

clps: it would be helpful to timestamp entries so the user can tell ages and most recent

vipath

2009-02-10 21:59: write python version
2010-11-27 12:33: add -L code to link $@ -&gt; $@.py, get rid of $@
as a separate file

dt

2009-02-10 22:15: implement in python
2009-02-10 22:15: implement in perl
2009-02-10 22:16: add to git repo
2010-11-27 12:33: add -L code to link $@ -&gt; $@.py, get rid of $@
as a separate file

pytool defects

• it appears that the prefix is not put into the tool code correctly
• the example function needs a doc string
• the example should have a sample usage of OptionParser()

Add tox testing

add tox.ini for 2.6.9, 2.7, 3.5.2 and make sure all tests pass for all python versions

add test function to fab

fab's test function will call each of the managed items in fabfile.py's list in the appropriate way to test it

write tests for workrpt

Examples of tests needed:

• should throw exception if times are out of order in the input file
• should not be thrown off by whitespace in the input
• missing COB at end of day should provoke a warning (not an error -- I have been known to work through midnight)
• bad date/time format should get an error, however, we should be more tolerant of a variety of date/time formats
• use first byte of payload as a separator and subtotal on unique supertasks

Add newpkg to pytool that will generate a full python project directory

with setup.py, test directory, README.md, etc.

summarize

• add license info
• write python version
• add to git repo
• add -L code to link $@ -&gt; $@.py, get rid of $@ as a separate file

align

• use toolframe.ezlaunch
• eliminate align as separate file
• support filename on command line

fl

• update the Subfunctions section of the doc string
• write a test for fl_rm_cr
• write fl_save
• write fl_times
• write fl_unreadable

Make pytool newtool use docopt_dispatch

rather than optparse, util.dispatch, etc

toolframe.py

In tf_launch and ez_launch put the call to the payload routine in a try ... except ... structure and let the payload code define its own exception handler

deprecations when running dt.py tests

/Users/tbarron/bin/dt.py:279: SyntaxWarning: assertion is always true, perhaps remove parentheses?
assert(e.message, expected)
/Users/tbarron/bin/dt.py:375: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
self.message = message
/Users/tbarron/bin/dt.py:261: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
assert(e.message == expected)
/Users/tbarron/bin/dt.py:279: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
assert(e.message, expected)

try using python's distutils (i.e., setup.py) for distribution and installation management

tps

2009-02-10 21:59: add license info
2009-02-10 21:59: write python version
2009-02-10 21:59: add to git repo
2010-11-27 12:33: add -L code to link $@ -&gt; $@.py, get rid of $@
as a separate file

ptidy

2009-02-10 21:59: add license info
2009-02-10 21:59: write python version
2009-02-10 21:59: add to git repo
2010-11-27 12:33: add -L code to link $@ -&gt; $@.py, get rid of $@
as a separate file

use 'if __name__ == '__main__''

Some of the programs call toolframe.XX_launch() unconditionally. The launch routine should only be called if name == 'main'.

fl.py

*.py: replace "from x import y" with "import y" and reference imported items as y.z

Best practice is to use the module name to access imported elements rather than polluting the top level namespace with imported material.

failure importing util from bscr

$ stella help
 Traceback (most recent call last):
   File "/Users/tpb/bin/stella", line 4, in <module>
    import bscr.util
  File "/Users/tpb/Dropbox/prj/backscratcher/bscr/bscr.py", line 15, in <module>
    import util
  File "/Users/tpb/Dropbox/prj/backscratcher/bscr/util.py", line 521, in <module>
    bscr = package_module(__name__)
  File "/Users/tpb/Dropbox/prj/backscratcher/bscr/util.py", line 499, in package_module
    rval = sys.modules[sys.modules[module_name].__package__]
KeyError: None


$ cat stella.py
#!/usr/bin/env python
import os
import sys
import bscr.util

def main(args):
    """
    Top entry point
    """
    util.dispatch(__name__, 'stella', args)


def stella_start(args):
    """start - fire up the django server
    """

#     c = util.cmdline([{'opts': ['--repo', '-r'],
#                        'action': 'store',
#                        'default': 'sciwork',
#                        'help': 'which repo to use'}])
#     (o, a) = c.parse(args)

    print("This is stella start")
    path = os.path.expanduser("~/prj/stella/%s/django-fe" % o.repo)
    os.chdir(path)
    print os.getcwd()
    pypath = os.getenv("PYTHONPATH")
    if 'constellationfe' not in pypath:
        os.environ['PYTHONPATH'] = path + '/constellation/constellationfe'
    cmd = "python manage.py runserver 8080"
    os.system(cmd)


if __name__ == '__main__':
    main(sys.argv)

need a way to clean up test debris after running a set of tests

scanpath

2009-02-10 21:59: add license info
2009-02-10 21:59: write python version
2009-02-10 21:59: add to git repo
2010-11-27 12:33: add -L code to link $@ -&gt; $@.py, get rid of $@
as a separate file

pytool: code generated by 'newpy' should include "import unittest"

code generated by 'newpy' should include "import unittest"

Remove nvtool's dependency on xtoolframe

rxlab

add license info
write python version
add to git repo
add -L code to link $@ -&gt; $@.py, get rid of $@ as a separate file

turn dt into a tool-style program to allow adding new functions more easily

The new version is project dtm

clps: replace uses of os.popen* with subprocess.*

The os.popen* routines are deprecated. Use routines from the subprocess module instead.

add ls function to fl.py

I'd like an ls with a long listing that looks like this:

-rwxrwxrwx 1 tpb staff 14.1K 2011.0323 HH:MM:SS fl.py
-rwxrwxrwx 1 tpb staff 25.8K 2011.0209 HH:MM:SS fx.py
-rwxrwxrwx 1 tpb staff 2.7K 2011.0323 HH:MM:SS hd.py
-rwxrwxrwx 1 tpb staff 2.4K 2011.0715 15:33:00 jump.py

Total: 45.0K

That is, the size is given in terms of kilo, mega, giga, etc., the file time is shown in YYYY.MMDD HH:MM:SS format and the total size of the files listed is given.

Ultimately, "fl ls" should support all the options the standard ls does.

move function 'run': fabfile.py -> tpbtools.py

clps: arrange a sequence of files to be opened

In order:

-f from the command
$CLPS_FILENAME
$HOME/.clps/secrets.clps

pytool needs update

pytool needs to be updated in the light of recent work with toolframe.py

truth_table

2009-02-10 21:59: add license info
2009-02-10 21:59: write python version
2009-02-10 21:59: add to git repo
2010-11-27 12:33: add -L code to link $@ -&gt; $@.py, get rid of $@
as a separate file

Add travis continuous integration

rename tpbtools.py to something more general

tpbtools: need tests

deprecate util.dispatch in favor of docopt_dispatch where it makes sense

This should allow for eliminating toolframe.py

ascii

• use toolframe.ezlaunch
• eliminate python launcher file

Expand flake8 checking to include test code

in the "pylint" test, replace "flake8 bscr" with "flake8 bscr test"