jlevy / repren

Rename anything

Hi! My work and interests lie at the intersection of AI, engineering, writing, and knowledge tools.

I believe the purpose of technology is to improve our collective knowledge and intelligence and to solve human problems. A bit more about me:

• Tech: I’ve worked on conversational AI systems since early days at SRI with the original Siri team. I’ve built search engines with prominent ex-Google engineers. I have a broad interest in the theory, the practice, and the human elements of software engineering, and the history of software and AI. Even longer ago, I studied mathematics at Berkeley. Tech stacks I've used the most are a variety of frameworks in Web, React, JS/Node, Python, Java, C/C++, Lisp, and (yes) Prolog.
• Engineering: Have been a founding engineer and eng leader at several startups in AI, SaaS, and web search (some failed, some successful).
• Writing: Wrote and edited few popular guides (2M+ readers), including The Art of Command Line (which is apparently with 140K+ stars a top-40 most popular GitHub repo of all time), equity compensation, and AWS. I’ve also edited (and published) about a dozen books by other authors.
• Recent work: I started and built much of Holloway, a platform that has rebuilt the digital book publishing process with dozens of in-depth, web-based guides in entrepreneurship, creativity, and personal growth. We also publish print books. I love to talk to authors, experts, and anyone who cares about sharing high-value knowledge.
• Inspiration: The technologist I most admire is Doug Engelbart. His work is as relevant today as it was 50 years ago. By sharing a vision of what was possible, he shaped how we use software for decades. I feel privileged to have met him a few times.
• Random: In sometimes random ways, I try to learn a few things about many topics and many things about a few topics. My interests include good books of all kinds, startups, AI, software engineering, obscure programming languages, logic and foundations of mathematics, philosophy, product design, typography, psychology, mental and physical health (and their interactions), fitness and nutrition, trail running and wilderness backpacking, natural history (especially of the California mountains and deserts), fingerstyle guitar, 15th-18th century lute music, early American blues, Brazilian Portuguese, truth, kindness, and some other arcana.

Find me:

• @ojoshe on Twitter (best for DMs) and Bluesky
• LinkedIn
• josh at holloway dot com

Please say hello if you think you should! I'd be glad to hear from you if we share interests. If you're trying to reach me, note I sometimes fall behind on messages. I don't mind reminders and more detail or context in a message helps. :)

repren's Issues

TypeError: can't use a string pattern on a bytes-like object

Trying to replace a string in files inside a directory I get the following error:
Dry run: No files will be changed Using 1 patterns: 'xxx' -> 'yyy' Found 6617 files in: .eclipse/ Traceback (most recent call last): File "/usr/bin/repren", line 534, in <module> dry_run=options.dry_run) File "/usr/bin/repren", line 409, in rewrite_files rewrite_file(path, patterns, do_renames=do_renames, do_contents=do_contents, by_line=by_line, dry_run=dry_run) File "/usr/bin/repren", line 375, in rewrite_file counts = transform_file(transform, path, dest_path, by_line=by_line, dry_run=dry_run) File "/usr/bin/repren", line 339, in transform_file counts = transform_stream(transform, stream_in, stream_out, by_line=by_line) File "/usr/bin/repren", line 307, in transform_stream (new_line, new_counts) = transform(line) File "/usr/bin/repren", line 374, in <lambda> transform = lambda contents: multi_replace(contents, patterns, source_name=path) File "/usr/bin/repren", line 209, in multi_replace for match in regex.finditer(input_str): TypeError: can't use a string pattern on a bytes-like object

The exact command given is: repren --from xxx --to yyy --dry-run .eclipse/

My guess is that this happens because there are binary files in the directory.

Could there be an option to skip binary files?

clearer help about default mode

It is not clear from the documentation or from the --help page which mode is the default one.

I'd suggest either adding this into --help page:

- repren: Multi-pattern string replacement and file renaming
+ repren: Multi-pattern string replacement (default) and file renaming tool

or adding a command-line option:

  -p PAT_FILE, --patterns=PAT_FILE
                        file with multiple replacement patterns (see below)
+  --replace-only         do search/replace on file contents (default)
  --full                do file renames and search/replace on file contents
  --renames             do file renames only; do not modify file contents

or adding a simple example to the README that does nothing but a simple search-and-replace:

repren --from center --to centre .

This is the mode that I use most, but it was tricky to find out the exact syntax from the docs.

"--at-once" option

Hi,

I can't seem to get the "--at-once" option working. My OS is "Linux Mint 17 Qiana".
I'm not really sure what version of Python is being used. "python2.7 -V" gives "Python 2.7.16", but When I do "python -V" gives "Python 2.7.6". I tested on a file containing "hello\nthere". It seems to find the match, but not do any changes:

repren -p toss_repren --at-once -i ./toss.txt
Using 1 patterns:
'hello' -> 'goodbye'
Found 1 files in: ./toss.txt
Read 1 files (12 chars), found 1 matches (0 skipped due to overlaps)
Changed 0 files (0 rewritten and 0 renamed)

When I drop the "--at-once" option, it works:

repren -p toss_repren -i ./toss.txt
Using 1 patterns:
'hello' -> 'goodbye'
Found 1 files in: ./toss.txt

• modify: ./toss.txt: 1 matches
  Read 1 files (12 chars), found 1 matches (0 skipped due to overlaps)
  Changed 1 files (1 rewritten and 0 renamed)

I also did "bash run.sh" to run the tests. Most seemed to work but at the end I got "ls_portable test2
tail: cannot open ‘+2’ for reading: No such file or directory". Thanks in advance for your help.

Support to only include certain files? Support config file?

Hi there! the tool is really useful. I think it would be great if a simple config file would be supported where the settings are specified.

also would be useful to support --include as well as --exclude. In my case for example it is easy to specify what I want repren to work on... only hpp and cpp files

Cheers!

More extensive large-scale test

This is a big omission. Currently testing informally.

Need some end-to-end automated test to cover lots of corner cases by doing some complex renames on a complex tree of files and validating the matches. Can do a sequence of operations including rewrites, renames, and swaps, record the tallies, put them all back, and validate the final result is identical using diff -r.

Perhaps simplest to automate with a silly script harness a bit like https://github.com/jlevy/procdog/blob/master/tests/run.sh

Regular expressions dialect

Hi there, great tool, really useful.

From the documentation, it is implicitly deduced that the expressions may be regular expressions. It would be nice to know exactly what dialect of regular expressions is accepted and what features are available.

Thanks!

Do not rename files, only content?

Possible to only rename the contents of files, and not the file names themselves?

I don't understand how --literal works, e.g.

python /Users/storm/bin/repren --dry-run --literal -i --from='light.mini_sov' --to='light.bedroom_mini' test
Dry run: No files will be changed
Using 1 patterns:
  'light\.mini\_sov' IGNORECASE -> 'light.bedroom_mini'

Why does the from escape the dots, while the to doesn`t?
I've tried including -i and --literal twice, but to no avail :)

Use unicode for internal processing?

Firstly, many thanks for putting this up - I think it's brilliant. In addition to code refactoring this is tremendously useful for people editing and revising books (with LaTeX, markdown and/or reStructured Text sources). For this purpose it would be very helpful to have repren deal also with non-ascii characters (e.g. accented characters in foreign words).

Would you consider adding an --encoding option to enable users to specify file encoding? repren could then decode the inputs read from pattern and input files to unicode, do all the internal processing in unicode and then encode again when writing output.

--dry-run --renames broken

When doing renames with no replacements, --dry-run is broken in 0.3.6.

Out-of-memory errors can be destructive

...unless the user uses backups. Multi-gigabyte stuff can be tricky (lost one XXL-sized file this way, fortunately an intermediate result though), so I would add some words of caution into the help screens/README, viz. after the following passage:

"File operations are done atomically, so interruptions never leave a previously existing file truncated or partly edited."

insert this:

"Users must be careful when processing very large files with --at-once option turned on."

Windows support

Repren works on Mac and Linux. Currently no testing has been done on Windows. Would be great to have some help with this.

`--at-once option

nt5a190812 [zhi:github_issue_aut_zhi_s]:*--at-once option^ (cntn Hi,

I can't seem to get the "--at-once" option working. My OS is "Linux Mint 17 Qiana".
I'm not really sure what version of Python is being used. "python2.7 -V" gives "Python 2.7.16", but When I do "python -V" gives "Python 2.7.6". I tested on a file containing "hello\there". It seems to find the match, but not do any changes:

repren -p toss_repren --at-once -i ./toss.txt
Using 1 patterns:
'hello' -> 'goodbye'
Found 1 files in: ./toss.txt
Read 1 files (12 chars), found 1 matches (0 skipped due to overlaps)
Changed 0 files (0 rewritten and 0 renamed)

When I drop the "--at-once" option, it works:

repren -p toss_repren -i ./toss.txt
Using 1 patterns:
'hello' -> 'goodbye'
Found 1 files in: ./toss.txt

• modify: ./toss.txt: 1 matches
  Read 1 files (12 chars), found 1 matches (0 skipped due to overlaps)
  Changed 1 files (1 rewritten and 0 renamed)

I also did "bash run.sh" to run the tests. Most seemed to work but at the end I got "ls_portable test2
tail: cannot open ‘+2’ for reading: No such file or directory". Thanks i advance for your help.

Python 3 support

Support both or simply migrate to Python 3.

Nondestructive mode can create empty directories

When using dry-run mode (-n), files won't be modified.

However, if we are renaming entire paths, the implementation of transform_file() temporarily creates the parent paths of the new, rewritten path as part of the dry run (to better reproduce real behavior, and because it counts matches).

This means it can leave behind new, empty directories.

This isn't a really big deal, but it's arguably a bug.

is it possible randomly rename all variables in python file

great code thanks
is it possible randomly rename all variables in python file

so have file from github and want to make it not recognizable for others?

Automatic testing

Since repren has a test script, we should run it automatically with Travis CI.

I'm going to take a look at this now.

--preserve-case enhancement: also use underscore-less CamelCase translations

Currently, when --preserve-case is used to translate a CamelCase replacements to additional lowercase and uppercase replacements, it inserts underscores in between the word segments. This is cool, but on some occasions it would be useful to additionally do the replacement without the added underscores.

For example, the current behaviour is like this:

$ repren --full --from CamelCase --to DromedaryCase --preserve-case .
Using 4 patterns:
  'CAMEL_CASE' -> 'DROMEDARY_CASE'
  'CamelCase' -> 'DromedaryCase'
  'camelCase' -> 'dromedaryCase'
  'camel_case' -> 'dromedary_case'

It would be awesome if it could be extended to do this:

$ repren --full --from CamelCase --to DromedaryCase --preserve-case .
Using 6 patterns:
  'CAMEL_CASE' -> 'DROMEDARY_CASE'
  'CAMELCASE' -> 'DROMEDARYCASE'
  'CamelCase' -> 'DromedaryCase'
  'camelCase' -> 'dromedaryCase'
  'camel_case' -> 'dromedary_case'
  'camelcase' -> 'dromedarycase'

Perhaps this could be opt-in.

Backup destination configuration

First off, thanks for the great tool.

I like the fact it auto-creates backups. The only thing it seems to be missing is the ability to configure where backups are stored. Is this currently possible?

How to rename directories?

This is an example:

$ mkdir 1
$ cd 1
$ touch {1,2,3,4,5}.txt
$ mkdir {6,7,8,9}
$ cp *txt 6
$ repren -n --renames --from "^" --to "aaa_" [6789]
Dry run: No files will be changed
Using 1 patterns:
  '^' -> 'aaa_'
Found 5 files in: 6, 7, 8, 9
- rename: 6/4.txt -> aaa_6/4.txt
- rename: 6/1.txt -> aaa_6/1.txt
- rename: 6/3.txt -> aaa_6/3.txt
- rename: 6/5.txt -> aaa_6/5.txt
- rename: 6/2.txt -> aaa_6/2.txt
Read 5 files (0 chars), found 0 matches (0 skipped due to overlaps)
Dry run: Would have changed 5 files (0 rewritten and 5 renamed)

$ repren --renames --from "^" --to "aaa_" [6789]
Using 1 patterns:
  '^' -> 'aaa_'
Found 5 files in: 6, 7, 8, 9
Traceback (most recent call last):
  File "/home/alexey/bin/repren", line 531, in <module>
    dry_run=options.dry_run)
  File "/home/alexey/bin/repren", line 406, in rewrite_files
    rewrite_file(path, patterns, do_renames=do_renames, do_contents=do_contents, by_line=by_line, dry_run=dry_run)
  File "/home/alexey/bin/repren", line 372, in rewrite_file
    counts = transform_file(transform, path, dest_path, by_line=by_line, dry_run=dry_run)
  File "/home/alexey/bin/repren", line 357, in transform_file
    move_file(source_path, dest_path, clobber=False)
  File "/home/alexey/bin/repren", line 299, in move_file
    shutil.move(source_path, dest_path)
  File "/usr/lib/python2.7/shutil.py", line 302, in move
    copy2(src, real_dst)
  File "/usr/lib/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: 'aaa_6/4.txt'

So it is a bug plus feature request (rename folders only, or at least rename them like files)

Add option to skip binary files

./bin/k2pdfopt_linux is a binary file which should not be touched:

repren --dry-run --literal --from agC --to ag_c .
Dry run: No files will be changed
Using 1 patterns:
  'agC' -> 'ag_c'
Found 200 files in: .
- modify: ./bin/k2pdfopt_linux: 1 matches
- modify: ./zsh/auto-load/others/ag.zsh: 3 matches
Read 200 files (14410423 chars), found 4 matches (0 skipped due to overlaps)
Dry run: Would have changed 2 files (2 rewritten and 0 renamed)

--from does not accept pattern starting with #

$ repren --from '#include <a.h>' --to '#include <b.h>' .
error: found no parse patterns

Same with --literal:

$ repren --literal --from '#include <a.h>' --to '#include <b.h>' .
error: found no parse patterns

This is because it interprets the # as a comment character (in function parse_patterns()).

It would be nice if this comment logic did not apply when patterns are supplied directly via --from --to, and/or when --literal is given.

Support non-recursive renames on directories?

repren recurses into directories if they are supplied as arguments. This always makes sense for search/replace.

One feature I've heard: If you're doing renames and not search/replace, you might want to rename directories without recursing. Not sure if this is common or the best solution. Perhaps a --no-recurse option that only applies to files/directories that are listed?