odashi / cloud-storage-mocker

Mocker library of Google Cloud Storage with local filesystem mounting.

cloud-storage-mocker

Mocker library of Google Cloud Storage Python client with local filesystem mounting.

Install

For package users:

pip install cloud-storage-mocker

For package developers:

git clone [email protected]:odashi/cloud-storage-mocker
cd cloud-storage-mocker
python -m venv venv
source venv/bin/activate
pip install -e '.[dev]'

How the package works

Basic usage

This library provides patch context manager, which replaces following classes on the google-cloud-storage package:

• Client
• Bucket
• Blob

patch takes a list of Mount objects, which represent mappings between bucket names and directories on the local filesystem. Each Mount has Boolean configs readable and writable to control read/write permissions to the mounted buckets.

A canonical use-case of this package is writing unit tests to check the behavior of the code that works with Google Cloud Storage:

import pathlib

import google.cloud.storage  # type: ignore[import]

from cloud_storage_mocker import BlobMetadata, Mount
from cloud_storage_mocker import patch as gcs_patch


def test_something(tmp_path: pathlib.Path) -> None:
    # Creates two temporary directories for readable/writable buckets.
    src_dir = tmp_path / "src"
    dest_dir = tmp_path / "dest"
    src_dir.mkdir()
    dest_dir.mkdir()

    # A sample file on the readable bucket.
    (src_dir / "hello.txt").write_text("Hello.")
    # Optionally, object metadata can also be specified by the file beside the
    # content, suffixed by ".__metadata__".
    (src_dir / "hello.txt.__metadata__").write_text(
        BlobMetadata(content_type="text/plain").dump_json()
    )

    # Mounts directories. Empty list is allowed if no actual access is required.
    with gcs_patch(
        [
            Mount("readable", src_dir, readable=True),
            Mount("writable", dest_dir, writable=True),
        ],
    ):
        client = google.cloud.storage.Client()

        # Reads a blob.
        blob = client.bucket("readable").blob("hello.txt")
        assert blob.download_as_text() == "Hello."
        # Metadata is available after downloading the content.
        assert blob.content_type == "text/plain"

        # Writes a blob.
        blob = client.bucket("writable").blob("world.txt")
        blob.upload_from_string("World.")

    # Checks if the file is written appropriately.
    assert (dest_dir / "world.txt").read_text() == "World."

Patched methods/properties

Methods listed below have specialized behavior to mount the local filesystem.

Other methods are mapped to MagicMock.

Client()

Client.bucket()

Bucket()

Bucket.blob()

Blob()

# Blob properties (download only)
Blob.cache_control
Blob.content_disposition
Blob.content_encoding
Blob.content_language
Blob.content_type

Blob.download_to_file()
Blob.download_to_filename()
Blob.download_as_bytes()
Blob.download_as_string()
Blob.download_as_text()
Blob.upload_from_file()
Blob.upload_from_filename()
Blob.upload_from_string()

Caution

This library is basically provided for writing unit tests. DO NOT use this library on any code on the production.

cloud-storage-mocker's Issues

`Blob.from_string(gs_url, client=client)`: TypeError: Client.__init__() got an unexpected keyword argument 'parent'

In the current version, when creating a blob using Blob.from_string() and specifying the gs://bucket/path URL as well as the client, the following error is thrown once we want to retrieve the object (e.g. using blob.download_to_file()):

Traceback (most recent call last):
  File "/workspace/project/foo3.py", line 28, in <module>
    blob.download_to_file(stream)
  File "/workspace/venv/lib/python3.12/site-packages/google/cloud/storage/blob.py", line 1160, in download_to_file
    self._prep_and_do_download(
  File "/workspace/venv/lib/python3.12/site-packages/google/cloud/storage/blob.py", line 4338, in _prep_and_do_download
    download_url = self._get_download_url(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/venv/lib/python3.12/site-packages/google/cloud/storage/blob.py", line 862, in _get_download_url
    hostname = _get_host_name(client._connection)
                              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/python/current/lib/python3.12/unittest/mock.py", line 678, in __getattr__
    result = self._get_child_mock(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python/current/lib/python3.12/unittest/mock.py", line 1055, in _get_child_mock
    return klass(**kw)
           ^^^^^^^^^^^
TypeError: Client.__init__() got an unexpected keyword argument 'parent'

Example code:

import io

import google.cloud.storage
import google.cloud.storage.blob
from cloud_storage_mocker import Mount
from cloud_storage_mocker import patch as gcs_patch

storage_dir = "."
bucket_name = "b"


with gcs_patch([Mount(bucket_name, storage_dir, readable=True, writable=True)]):
    client = google.cloud.storage.Client()

blob_name = "hello.txt"

# Write object
blob = client.bucket(bucket_name).blob(blob_name)
blob.upload_from_string("world")

# Make object using `Blob.from_string()` and `gs://` URL
blob = google.cloud.storage.blob.Blob.from_string(f"gs://{bucket_name}/{blob_name}", client=client)

# Workaround: Recreate blob using `client.bucket().blob()`
# blob = client.bucket(blob.bucket.name).blob(blob.name)

# Read object
stream = io.BytesIO()
blob.download_to_file(stream) # <-- throws exception
stream.seek(0)

There's a workaround though: after parsing the gs:// URL using Blob.from_string(), we can create a new blob using client.bucket(bucket).blob(blob), which works.

TypeError: Blob.upload_from_file() got an unexpected keyword argument 'content_type'

Summary

Mocked methods raise exception if passed kwargs.

I'm uploading blob with content_type like below, but this lib looks not considered.

blob.upload_from_file(
    file_to_upload.stream, content_type=file_to_upload.mimetype, rewind=True
)

The args are ignored, but the kwargs do not seem to be expected.

cloud-storage-mocker/src/cloud_storage_mocker/_core.py

Lines 345 to 349 in a8d89a6

	def upload_from_file(
	self,
	file_obj: io.BufferedReader,
	*args: Any, # Not supported
	) -> None:

So, this resolve this issue.

 def upload_from_file( 
     self, 
     file_obj: io.BufferedReader, 
     *args: Any,  # Not supported 
+    **kwargs: Any,  # Not supported
 ) -> None: 

The same would be necessary for other methods.

Cannot install using Python 3.12.1

The Python requirement in ./pyproject.toml is very restrictive, preventing installation on recent versions of Python.

ERROR: Ignored the following versions that require a different python version: 0.0.1a1 Requires-Python <3.12,>=3.10; 0.0.1a2 Requires-Python <3.12,>=3.10; 0.0.1a3 Requires-Python <3.12,>=3.10; 0.0.1a4 Requires-Python <3.12,>=3.10; 0.0.1a5 Requires-Python <3.12,>=3.10; 0.1.0 Requires-Python <3.12,>=3.10; 0.2.0 Requires-Python <3.12,>=3.10; 0.2.1 Requires-Python <3.12,>=3.10; 0.3.0 Requires-Python <3.12,>=3.10; 0.3.1 Requires-Python <3.12,>=3.10