kurohai / flametree

Python file and zip operations made easy

Flametree - Python file operations made easy

Flametree is a Python library which provides a simple syntax for handling files and folders (no os.path.join, os.listdir etc.), and works the same way for different file systems.

Write a Flametree program to read/write files in disk folders, and your code will also be able to read/write in zip archives and virtual (in-memory) archives - which is particularly useful on web servers.

As an illustration, here is how to use Flametree to read a file texts/poems/the_raven.txt, replace all occurences of the word "raven" by "seagull" in the text, and write the result to a new file the_seagull.txt in the same folder:

from flametree import file_tree

with file_tree("texts") as root:
    poem_text = root.poems.the_raven_txt.read()
    new_text = poem_text.replace("raven", "seagull")
    root.poems._file("the_seagull.txt").write(new_text)

Even in this very simple use case, the syntax is clearer than the os way, which would write as follows:

import os

with open(os.path.join("poems", "the_raven.txt"), "r") as f:
    poem_text = f.read()
new_text = poem_text.replace("raven", "seagull")
with open(os.path.join("poems", "the_seagull.txt"), "w") as f:
    content = f.write(new_text)

Moreover, the same Flametree code also works for files inside a zip archive:

with file_tree("my_archive.zip") as root:
    poem_text = root.poems.the_raven_txt.read()
    new_text = poem_text.replace("raven", "seagull")
    root.poems._file("the_seagull.txt").write(new_text)

Now in hard mode: suppose that your server receives binary zip data of an archive containing poems/the_raven.txt, and must return back a new zip containing a file poems/the_seagull.txt. Here again, the syntax of the core operations is the same:

destination_zip = file_tree("@memory") # Create a new virtual zip
with file_tree(the_raven_zip_data) as root:
    poem_text = root.poems.the_raven_txt.read()
    new_text = poem_text.replace("raven", "seagull")
    destination_zip._dir("poems")._file("the_seagull.txt").write(new_text)
destination_zip_data = destination_zip._close()
# Now send the data to the client

See section Usage below for more examples and features.

Installation

Flametree should work on Windows/Max/Linux, with Python 2 and 3, and has no external dependency.

It can be installed by unzipping the source code in one directory and using this command:

sudo python setup.py install

You can also install it directly from the Python Package Index with this command:

sudo pip install flametree

Contribute

Flametree is an open-source software originally written by Zulko and released on Github under the MIT licence (¢ Edinburgh Genome Foundry). Everyone is welcome to contribute ! In particular if you have ideas of new kinds of file systems to add to Flametree.

Usage

Opening a file tree

Here is how you open different kinds of file systems:

from flametree import file_tree

# Open a directory from the disk's file system:
root = file_tree("my_folder/")

# Open a zip archive on the disk:
root = file_tree("my_archive.zip")

# Connect to a file-like object (file handle, StringIO...) of a zip:
root = file_tree(file_like_object)

# Create a virtual 'in-memory' zip file:
root = file_tree("@memory")

# Open some data string representing a zip to read
root = file_tree(some_big_zip_data_string)

In the two first examples, if my_folder or my_archive.zip do not exist, they will be automatically created. If they do exist, it is possible to completely overwrite them with the option replace=True.

Exploring a file tree:

Once you have created the root element with one of the methods above, you can display the whole file tree with root._tree_view() :

>>> print (root._tree_view())
texts/
  poems/
    dover_beach.txt
    the_raven.txt
    the_tyger.txt
  todo_list.txt
figures/
  figure1.png
  figure2.png
Readme.md

The attributes of a directory like root are its files and subdirectories. For instance to print the content of dover_beach.txt you would write:

print( root.texts.poems.dover_beach_txt.read() )

or even simpler:

root.texts.poems.dover_beach_txt.print_content()

Notice that the . before txt was replaced by _ so as to form a valid

attribute name.

This syntactic sugar is particularly useful to explore a file tree in IPython Notebooks or other editors offering auto-completion:

Alternatively, you can access files and directories using dictionary calls:

root["texts"]["poems"]["dover_beach.txt"]

To iterate through the subdirectories of a directory, use the _dirs attribute:

for subdirectory in root._dirs:
    print (subdirectory._name) # Will print 'texts' and 'figures'

To iterate through the files of a directory, use the _files attribute:

for f in root.figures._files:
    print (f._name) # Will print 'figure1.png' and 'figure2.png'

Finally, use _all_files to iterate through all files nested in a directory. The snippet below prints the content of all .txt files in the file tree:

for f in root._all_files:
    if f._name.endswith(".txt"):
        f.print_content()

Creating files and folders

To create a new subdirectory use _dir:

root._dir("data") # create a 'data' folder at the root
root.data._dir("reports") # create a 'reports' folder under `root/data`

To create a new file use _file:

root._file("joke.txt") # create a 'joke.txt' file at the root.
root.texts._file("hello.txt") # create 'hello.txt' in `root/texts`.

To write content in a file, use .write:

root.joke_txt.write("A plateau is the highest form of flattery.")

Writing to a file will use mode a (append) by default. To overwrite the file set the write mode to "w". Let's erase and rewrite that joke.txt:

root.joke_txt.write("'DNA' stands for National Dyslexic Association.", "w")

File and directory creation commands can be chained. Let us create some new folders data/ and data/test_1/, and write to file data/test_1/values.csv, all in a single line:

root._dir("data")._dir("test_1")._file("values.csv").write("1, 15, 25")

Beware that ._dir and ._file overwrite their target by default, which means that if you write:

root._dir("data")._file("values_1.csv").write("1, 4, 7")
root._dir("data")._file("values_2.csv").write("2, 9, 7")

The directory data will only contain values_2.csv, because the second line's _dir("data") erases the data directory and starts a new one. To avoid this, either use root.data in the second line:

root._dir("data")._file("values_1.csv").write("1, 4, 7")
root.data._file("values_2.csv").write("2, 9, 7")

Or use replace=False in _dir:

root._dir("data")._file("values_1.csv").write("1, 4, 7")
root._dir("data", replace=False)._file("values_2.csv").write("2, 9, 7")

Other operations

You can move, copy, and delete a file with .move(folder), .copy(folder), .delete(), and a directory with ._move(folder), ._copy(folder), ._delete().

root.data.values1_csv.delete() # delete file 'values1.csv'
root.data._delete() # delete directory 'data'
# Move folder `plots` from `root/figures` to `other_root/figures`
root.figures.plots._move(other_root.figures)
# Move file `fig.png` from `root/figures` to `other_root/figures`
root.figures.fig_png.move(other_root.figures)

Special rules for ZIP archives

It is not currently possible to modify/delete a file that is already zipped into an archive (because zips are not really made for that, it would be doable but would certainly be a hack).

When creating files and folders in a zip with Flametree, the changes in the actual zip will only be performed by closing the root with root._close() (after which the root can't be used any more). If it is an in-memory zip, root._close() returns the value of the zip content as a string (Python 2) or bytes (Python 3).

Here are a few examples:

root = file_tree("archive.zip")
root._file("hello.txt").write("Hi there !")
root._close()

# Equivalent to the previous, using `with`:
with file_tree("archive.zip") as root:
    root._file("hello.txt").write("Hi there !")

# Getting binary data of an in-memory zip file:
root = file_tree("@memory")
root._file("hello.txt").write("Hi there !")
binary_data = root._close()

Using file writers from other libraries

Some libraries have file-generating methods which expect a file name or a file object to write too. You can also feed Flametree files to these functions. for instance here is how to use Weasyprint to create a PDF pdfs/report.pdf

import weasyprint
from flametree import file_tree
root = file_tree(".") # or 'archive.zip' to write in an archive.
html = weasyprint.HTML(string="<b>Hello</b> world!", base_url='.')
html.write_pdf(root._dir("pdfs")._file("test.pdf"))

And here is how you would save a Matplotlib figure in a zip archive:

import matplotlib.pyplot as plt
from flametree import file_tree
fig, ax = plt.subplots(1)
ax.plot([1, 2, 3], [3, 1, 2])
with file_tree("archive.zip") as root:
    fig.savefig(root._dir("plots")._file("figure.png"), format="png")

That's all folks !