mdowds / python-mock-firestore Goto Github PK

Firstly, thank you for your work on this library. I have a project that deals with firestore a lot and being able to mock it in tests is super easy with this library.

I make use of the select method on a collection, which enables me to only pull back certain fields. This is implemented in the python library and would be great if this feature could be added.

https://github.com/googleapis/google-cloud-python/tree/master/firestore is broken. I do not know where this package author wanted to point the link.

Updating using the dot notation should update the nested field value

frank_ref = db.collection("users").document("frank")
frank_ref.set(
    {
        "name": "Frank",
        "favorites": {"food": "Pizza", "color": "Blue", "subject": "Recess"},
        "age": 12,
    }
)

# Update age and favorite color
frank_ref.update({"age": 13, "favorites.color": "Red"})

doc = db.collection("users").document("frank").get().to_dict()

assert doc["age"] == 13  # ok
assert doc["favorites.color"] == "Red"  # This shouldnt pass but passes
assert doc["favorites"]["color"] == "Red"  # This fails

Reference :
https://firebase.google.com/docs/firestore/manage-data/add-data#update_fields_in_nested_objects

Hello again!

I also noticed that the delete() function for DocumentReference wasn't added.

I added the following to main.py as a quick fix but would love to hear your thoughts:

def delete(self):
    get_by_path(self._data, self._path).clear()

I also tried:

def delete(self):
    self.set({})

They both seemed to have similar behavior and passed my unit tests

At least in python3.7 collection get method returns generator.
Mock returns list instead.

Propose change in Query class:

def get(self) -> Generator:
        return (DocumentSnapshot(doc) for doc in self._data.values())

Add support for the new collection_group query that was recently added.

# Example Data
parent_1 = db.collection(u'parents').document(u'molly_smith').set({
        u'name': u'Molly Smith',
        u'age': 49
})
child_1a = db.collection(u'parents').document(u'u'molly_smith').collection(u'children').document().set({
        u'name': u'Tom Smith JR',
        u'age': 17
})
parent_2 = db.collection(u'parents').document(bill_jones').set({
        u'name': u'Bill Jones',
        u'age': 56
})
child_2a = db.collection(u'parents').document(u'bill_jones').collection(u'children').document().set({
        u'name': u'Jerry Jones',
        u'age': 17
})

# Collection Group Query
children = db.collection_group(u'children').where(u'age', u'==',17')
docs = children.stream()
for doc in docs:
    print(u'{} => {}'.format(doc.id, doc.to_dict()))

Documentation of Query

GoogleCloudPlatform Example Snippet

Reference:

While mocking fireo ORM, get queries cannot execute due to missing transactions argument for document.get() method. Google Firestore v1 library provides transactions argument: https://github.com/googleapis/python-firestore/blob/2d42731996586fd63e9b8453b0eb627d3e23a310/google/cloud/firestore_v1/document.py#L362

Add support for firestore batch writes.

I noticed the problem in a test that asserts that a specific document does not exist and then retrieves all other documents from a collection. The previously queried document is in the stream, but a .exists check yields False. Hence I'd have to filter all stream()ed documents by checking their exists state, which I don't have to do in the original Firestore.

I have identified the problem to be the handling of empty/non-existing documents in python-mock-firestore, specifically two problems:

1. CollectionReference.document() creates an empty document if there is no existing document for the given name. This empty document is normally treated as doc.exists == False, but CollectionReference.stream() still returns it.
2. mock-firestore treats empty documents as non-existing. This deviates from original Firestore which can hold empty documents (and treats them as existing)

The following to tests reproduce the problem (note that they both work with Google's Firestore but fail with mock-firestore):

def test_does_not_stream_nonexisting_documents(self):
    client = MockFirestore()
    assert client.collection("foo").document("bar").get().exists == False
    assert len(list(client.collection("foo").stream())) == 0 # actual value is 1

def test_can_create_empty_documents(self):
    client = MockFirestore()
    client.collection("foo").document("bar").set({})
    assert client.collection("foo").document("bar").get().exists == True

Since I don't need empty documents in my tests, I have awkardly worked around the first issue by monkeypatching CollectionReference.stream() but I'd like to contribute a real solution to the problem. However, I am unsure how to go forward from here and would like to discuss the issue.

I propose to change document and collection lookup so that they don't create empty dicts anymore, but None values. For documents, treat None as non-existing and treat empty documents as existing. Then, to allow lazy creation of documents, _helpers.set_by_path() would have to iterate over path elements and create them instead of just trusting that the parent path exists and is a dict. This makes _helpers.set_by_path() a little slower, though, but I think it warrants the tradeoff.

WDYT?

Add support for client.collections() that list top-level collections of the client's database. Which was introduced with commit #7494 on firestore client

Documentation of Collections

Firestore supports != operators now: https://firebase.google.com/docs/firestore/query-data/queries#query_operators

I get the following error (which goes away if I use the == operator instead).

doc_snapshots = [doc_snapshot for doc_snapshot in doc_snapshots
>                    if compare(doc_snapshot._get_by_field_path(field), value)]
E   TypeError: 'NoneType' object is not callable

I believe a simple condition just needs to be added here in compare_func: https://github.com/mdowds/python-mock-firestore/blob/master/mockfirestore/query.py#L122

After #13 is merged, at some point we should make Transaction.rollback actually roll the transaction back.

This could be done by making a deep copy of the MockFirestore data before calling the write ops in Transaction._commit, and restoring the DB to that copy when rolling back.

Every new document save requires a deepcopy of the entire dataset, since every document contains the _data field. There must be a way around this. Why is deepcopy necessary? Can a reference to the dataset be stored instead?

Hello!

Thanks for making this library! Saved me a ton of time for my unit testing.

I am having an issue trying to use the library to generate document ids like such:

post_id = db.collection("posts").document().id

I do this using the live Firestore client so I was hoping it'd be possible with the mock library.

The Google Firestore client has an id property available on DocumentSnapshot. id seems to be set regardless of whether the document exists or not.

id = client.collection('test').document('zzz').get().id

id should be 'zzz' in this example, but with mock-firestore, this raises AttributeError: 'DocumentSnapshot' object has no attribute 'id'.

I found out that writing transforms, for example increments, on an empty doc stores the actual objects in the in-memory db, e.g.

# some code writing to the db

print(db.collection("users").document("user_one").get().to_dict())


{'num_matches_joined': <google.cloud.firestore_v1.transforms.Increment object at 0x10f2e8250>, 'scores': {'number_of_scored_games': <google.cloud.firestore_v1.transforms.Increment object at 0x10f2e8b50>, 'total_sum': <google.cloud.firestore_v1.transforms.Increment object at 0x10f2e87c0>}, 'potm_count': <google.cloud.firestore_v1.transforms.Increment object at 0x10f2e8be0>, 'record': {'num_win': <google.cloud.firestore_v1.transforms.Increment object at 0x10f2e8b20>, 'num_draw': <google.cloud.firestore_v1.transforms.Increment object at 0x10f2e8700>, 'num_loss': <google.cloud.firestore_v1.transforms.Increment object at 0x10f2e8d60>}, 'last_date_scores': {'20230503143043': 2.5}}

if instead I populate the doc before

db.collection("users").document("user_one").set({"name": "a"})

# some code writing to the db

print(db.collection("users").document("user_one").get().to_dict())

{'name': 'user_one', 'num_matches_joined': 1, 'scores': {'number_of_scored_games': 1, 'total_sum': 2.5}, 'potm_count': 0, 'record': {'num_win': 1, 'num_draw': 0, 'num_loss': 0}, 'last_date_scores': {'20230503143138': 2.5}}

Add support for using a document snapshot as the query cursor to be able to paginate documents in a collection.

Example:

doc = fs.collection('foo').document('1').get()
fs.collection('foo').order_by('id').start_after(doc).limit(2).stream()

Docs

google-cloud-firestore = "^2.11.1"
mock-firestore = "^0.11.0"

site-packages/google/cloud/firestore_v1/base_collection.py:290: UserWarning: Detected filter using positional arguments. Prefer using the 'filter' keyword argument instead.

i.e. instead of
col.where(_DELETED_ATTR, "==", False)

it is recommended to:

_DELETED_FILTER = FieldFilter(_DELETED_ATTR, "==", False)
col = col.where(filter=_DELETED_FILTER)

current where typesig + docs:

    def where(
        self,
        field_path: Optional[str] = None,
        op_string: Optional[str] = None,
        value=None,
        *,
        filter=None
    ) -> BaseQuery:
        """Create a "where" query with this collection as parent.

        See
        :meth:`~google.cloud.firestore_v1.query.Query.where` for
        more information on this method.

        Args:
            field_path (str): A field path (``.``-delimited list of
                field names) for the field to filter on. Optional.
            op_string (str): A comparison operation in the form of a string.
                Acceptable values are ``<``, ``<=``, ``==``, ``>=``, ``>``,
                and ``in``. Optional.
            value (Any): The value to compare the field against in the filter.
                If ``value`` is :data:`None` or a NaN, then ``==`` is the only
                allowed operation.  If ``op_string`` is ``in``, ``value``
                must be a sequence of values. Optional.
            filter (class:`~google.cloud.firestore_v1.base_query.BaseFilter`): an instance of a Filter.
                Either a FieldFilter or a CompositeFilter.
        Returns:
            :class:`~google.cloud.firestore_v1.query.Query`:
            A filtered query.
        Raises:
            ValueError, if both the positional arguments (field_path, op_string, value)
                and the filter keyword argument are passed at the same time.
        """

Should be fairly simple to extend the mock where call to have same behaviour. Any objections?

Issue

firestore comes with some atomic "transformations" that are used to manipulate data without retrieving the document first, checking the value, then setting it again. These transformations are listed in their documentation: https://googleapis.dev/python/firestore/latest/transforms.html

However, mockfirestore does not support this functionality.

Reproduction Steps

mock_database.collection('user').document('some-random-id').add({'likes': 0})
mock_database.collection('user').document('some-random-id').update({'likes': firestore.Increment(1)})
assert (
    mock_database.collection('user').document('some-random-id').get().to_dict() ==
    {'likes': 1}
)

Instead, I get something like:

{'likes': <google.cloud.firestore_v1.transforms.Increment object at 0x106ee5208>}

The Collection object in the real Firestore client offers the method add. This is the method we use to create new documents. Is there a particular reason that method was omitted from MockFirestore, or did the author just not get to it?

If it's the latter, I can try implementing it.

I got an error, while I'm trying to set not existing document with flag merge=True. The same code works ok for real Firestore.

from google.cloud import firestore
from mockfirestore import MockFirestore


COLLECTION = 'test'

db = firestore.Client.from_service_account_json('firestore.json')
pk = 'test3'

v_not_exist = db.collection(COLLECTION).document(pk).get()
print(f'v_not_exist: {v_not_exist.exists}')

db.collection(COLLECTION).document(pk).set({'first': 1, 'v': 1}, merge=True)

v1 = db.collection(COLLECTION).document(pk).get()
print(f'v1: {v1._data}')

db.collection(COLLECTION).document(pk).set({'second': 2, 'v': 2}, merge=True)
v2 = db.collection(COLLECTION).document(pk).get()
print(f'v2: {v2._data}')

print('---')

mock_db = MockFirestore()
v_not_exist = mock_db.collection(COLLECTION).document(pk).get()
print(f'm v_not_exist: {v_not_exist.exists}')

mock_db.collection(COLLECTION).document(pk).set({'first': 1, 'v': 1}, merge=True)

v1 = mock_db.collection(COLLECTION).document(pk).get()
print(f'm v1: {v1.to_dict()}')


mock_db.collection(COLLECTION).document(pk).set({'second': 2, 'v': 2}, merge=True)
v2 = mock_db.collection(COLLECTION).document(pk).get()
print(f'm v2: {v2.to_dict()}')

Output:

v_not_exist: True
v1: {'second': 2, 'v': 1, 'first': 1}
v2: {'second': 2, 'v': 2, 'first': 1}
---
m v_not_exist: False
Traceback (most recent call last):
  File "/home/peter/projects/af/notes/personal/firestore_learn/personal.py", line 39, in <module>
    mock_db.collection(COLLECTION).document(pk).set({'first': 1, 'v': 1}, merge=True)
  File "/home/peter/.pyenv/versions/firestore_learn_py3.8.1/lib/python3.8/site-packages/mockfirestore/document.py", line 56, in set
    self.update(deepcopy(data))
  File "/home/peter/.pyenv/versions/firestore_learn_py3.8.1/lib/python3.8/site-packages/mockfirestore/document.py", line 63, in update
    raise NotFound('No document to update: {}'.format(self._path))
google.api_core.exceptions.NotFound: 404 No document to update: ['learn', 'test3']

The DocumentSnapshot has a get function, which seems to be missing here. Large sections of our source code uses the get function. It would be great if this could be added.

The current implementation works pretty well, but it has one flaw. It throws a KeyError when the document itself does not exist, but a subcollection does. According to the actual Firestore implementation it considers the document to be non-existent, even when a subcollection for that path exists.

Example:

users/foo
<no data>

users/foo/items
{
  "name": "My Item"
}

then

db.collection("users").document("foo").get().get("name")  # should return None, but throws KeyError instead

I briefly glared at the source code and I think it's because it stores both document and subcollection data in the same _doc property.

Following test will fail:

    def test_collection_start_after_similar_objects(self):
        fs = MockFirestore()
        fs._data = {'foo': {
            'first': {'id': 1, 'value': 1},
            'second': {'id': 2, 'value': 2},
            'third': {'id': 3, 'value': 2},
            'fourth': {'id': 4, 'value': 3}
        }}
        docs = list(fs.collection('foo').order_by('id').start_after({'id': 3, 'value': 2}).stream())
        self.assertEqual({'id': 4, 'value': 3}, docs[0].to_dict())
        self.assertEqual(1, len(docs))

Only last key in document_fields decides if index is set or not because of missing brake in search-compare loop inside Query._apply_cursor:

        for idx, doc in enumerate(doc_snapshot):
            index = None
            for k, v in document_fields.items():
                if doc.to_dict().get(k, None) == v:
                    index = idx
                else:
                    index = None
                    # missing break
            if index:
                ...
    

From the line

arr = self.mock_db.collection('players').stream()

I expected a return type of Iterable[DocumentSnapshot] but instead got

  <generator object CollectionReference.stream at 0x106a1d890>

My setup code:

    def setUp(self) -> None:
            self.mock_db = MockFirestore()
            self.mock_db.collection('players').add({
                'name': 'John Doe',
                'id': 'johndoe',
            })
            self.mock_db.collection('players').add({
               'name': 'Jane Doe',
               'id': 'johndoe',
            })

Screenshot of Pycharm's documentation

It seems like the mocked DocumentReference do not have the create method described in the firestore documentation: https://googleapis.dev/python/firestore/latest/document.html#google.cloud.firestore_v1.document.DocumentReference.create

This results in errors like the following:

> doc_ref = collection.document(doc_id)
> doc_ref.create(doc)
E       AttributeError: 'DocumentReference' object has no attribute 'create'

Most of the code snippets on how to use a transaction use the decorator @firestore.transactional. What is the proper way to use the transactions with this mock?

Official documentation:
https://github.com/GoogleCloudPlatform/python-docs-samples/blob/5238ce34cb92b58ddaa2019706caacfa4d7a2009/firestore/cloud-client/snippets.py#L414

Hello,

Got problem with start_after and after some digging I found that Query._apply_cursor is not working properly when first doc is passed.

Let's take a look at problematic fragment:

        for idx, doc in enumerate(doc_snapshot):
            index = None
            for k, v in document_fields.items():
                if doc.to_dict().get(k, None) == v:
                    index = idx
                else:
                    index = None
            if index:
                if before and start:
                    return islice(docs, index, None, None)
                ...

We search for index of matching document in inner for loop but if found index equals 0 it won't go inside if index:

Following test will fail on current version:

    def test_collection_start_after(self):
        fs = MockFirestore()
        fs._data = {'foo': {
            'first': {'id': 1},
            'second': {'id': 2},
            'third': {'id': 3}
        }}
        docs = list(fs.collection('foo').start_after({'id': 1}).stream())
        self.assertEqual({'id': 2}, docs[0].to_dict())
        self.assertEqual(2, len(docs))

Solution:
In _apply_cursor add explicit check for None

if index is not None:

Really appreciate your work on this library. I'm using it in a suite of unit tests, but I'm missing the merge functionality in DocumentReference.set().

If you're open to PRs, I'm happy to contribute.

Non existent docs should raise an exception.

Example:

try:
    doc = db.collection('users').document('thisDoesNotExist').get()
    print(u'Document data: {}'.format(doc.to_dict()))
except google.cloud.exceptions.NotFound:
    print(u'No such document!')

As black is becoming the de-facto standard formatter for Python code, I think we should use it in this repository. The formatting is wonky in multiple places, and using black would make the code more readable.

Hi, do you guys want a PR with support for asyncio?

I needed async support and spent a whole day implementing it. Just before I went to submit a merge request I found the MyPy page with a newer version containing an async implementation. So sad. :(

According to the documentation ArrayUnion should only add elements that aren't already present in the array. E.g. this test should succeed:

    def test_document_update_transformerArrayUnionDuplicates(self):
        fs = MockFirestore()
        fs._data = {"foo": {"first": {"arr": [1, 3]}}}
        fs.collection("foo").document("first").update(
            {"arr": firestore.ArrayUnion([2, 3, 4])}
        )
        doc = fs.collection("foo").document("first").get().to_dict()
        self.assertCountEqual(doc["arr"], [1, 2, 3, 4])  # fails: doc["arr"] = [1, 3, 2, 3, 4]

....\venv\Lib\site-packages\parameters_validation\validate_parameters_decorator.py:37: in wrapper
return f(*args, **kwargs)
....\svc\db\db.py:94: in get_tails_for_user
docs = tails_ref.stream()
....\venv\Lib\site-packages\mockfirestore\query.py:31: in stream
doc_snapshots = [doc_snapshot for doc_snapshot in doc_snapshots
....\venv\Lib\site-packages\mockfirestore\query.py:32: in
if compare(doc_snapshot._get_by_field_path(field), value)]

x = None, y = 'red'

return lambda x, y: y in x
E TypeError: argument of type 'NoneType' is not iterable

....\venv\Lib\site-packages\mockfirestore\query.py:137: TypeError

In the intended use of different instances to unit test Firestore behavior, this is not available using MockFirestore:

ex:

fire_db_a = MockFirestore()
fire_db_b = MockFirestore()

doc_ref_a = fire_db.collection('test').document('abc')
doc_ref_a.set({ 'test': 'value'})
doc_ref_b = fire_db.collection('test').document('abc')
doc_ref_a.get().to_dict()
// returns {'test': 'value'}
doc_ref_b.get().to_dict()
// returns {}

returns should be the same