AssertionError in test_file_cache.py about codesearch-py

[asankah]

LOL. Fixed. This is why I hate Python.

[asankah]

Interesting. I can't reproduce the issue. Which platform?

Also, does the problem still happen with 9ab1d0f ?

[cpu82]

Tested on FreeBSD 10.3/amd64.

Yes, after applying 9ab1d0f it still fails.

[cpu82]

For the moment, I can skip the failed test case

--- codesearch/test_file_cache.py.orig	2017-06-29 19:07:27 UTC
+++ codesearch/test_file_cache.py
@@ -22,6 +22,7 @@ class TestFileCache(unittest.TestCase):
     finally:
       f.close()
 
+  @unittest.skip("broken on FreeBSD")
   def test_with_cache_dir(self):
     f = None
     g = None

After run the test cases again

===>  Testing for py27-codesearch-py-0.1
===>   py27-codesearch-py-0.1 depends on file: /usr/local/bin/python2.7 - found
test_figment_display_name (codesearch.test_client_api.TestCodeSearch) ... ok
test_figment_display_name_2 (codesearch.test_client_api.TestCodeSearch) ... ok
test_figment_display_name_3 (codesearch.test_client_api.TestCodeSearch) ... ok
test_fixed_cache (codesearch.test_client_api.TestCodeSearch) ... ok
test_get_signature_for_symbol (codesearch.test_client_api.TestCodeSearch) ... ok
test_get_signatures_for_symbol (codesearch.test_client_api.TestCodeSearch) ... ok
test_get_type_1 (codesearch.test_client_api.TestCodeSearch) ... ok
test_get_type_2 (codesearch.test_client_api.TestCodeSearch) ... ok
test_get_type_3 (codesearch.test_client_api.TestCodeSearch) ... ok
test_get_type_4 (codesearch.test_client_api.TestCodeSearch) ... ok
test_search_for_symbol (codesearch.test_client_api.TestCodeSearch) ... ok
test_user_agent (codesearch.test_client_api.TestCodeSearch) ... ok
test_with_cache_dir (codesearch.test_client_api.TestCodeSearch) ... ok
test_display_name (codesearch.test_cs_file.TestCsFile) ... ok
test_path (codesearch.test_cs_file.TestCsFile) ... ok
test_text_range (codesearch.test_cs_file.TestCsFile) ... ok
test_with_cache_dir (codesearch.test_file_cache.TestFileCache) ... skipped 'broken on FreeBSD'
test_with_no_cache_dir (codesearch.test_file_cache.TestFileCache) ... ok
test_class_with_known_keyword (codesearch.test_messages.TestConstructor) ... ok
test_class_with_unknown_keyword (codesearch.test_messages.TestConstructor) ... ok
test_empty_class (codesearch.test_messages.TestConstructor) ... ok
test_from_json_string_1 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_2 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_3 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_4 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_5 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_6 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_7 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_invalid (codesearch.test_messages.TestProto) ... ok
test_from_shallow_dict_1 (codesearch.test_messages.TestProto) ... ok
test_proto_bridge (codesearch.test_messages.TestProto) ... ok
test_get_all_edges (codesearch.test_xref_node.TestXrefNode) ... ok
test_related_annotations (codesearch.test_xref_node.TestXrefNode) ... ok
test_related_definitions (codesearch.test_xref_node.TestXrefNode) ... ok
test_related_definitions_2 (codesearch.test_xref_node.TestXrefNode) ... ok
test_simple_xref_lookup (codesearch.test_xref_node.TestXrefNode) ... ok

----------------------------------------------------------------------
Ran 36 tests in 1.099s

OK (skipped=1)

[asankah]

Have you tried using the library with a cache directory? I wonder if things will break.

[cpu82]

No. I haven't tried the library with a cache directory.

Any suggestions are appreciated.

[asankah]

The cache directory is used when one wants to share a cache across multiple instantiations of the library (think multiple runs of a command line tool). You can enable it by something like:

cs = CodeSearch(should_cache=True, cache_dir="<path to cache directory>", source_root=...)

Just setting should_cache to True without setting cache_dir winds up using an ephemeral cache that's only accessible during a single library session.

If this test is failing, then the persistent cache stuff may need some more work. I just haven't had a chance to look into why this is failing on FreeBSD.

[cpu82]

Well, running a simple test I found a new issue:

Traceback (most recent call last):
  File "test.py", line 7, in <module>
    cs = codesearch.CodeSearch(should_cache=True, cache_dir="/tmp/cache", source_root='.')
  File "/usr/local/lib/python2.7/site-packages/codesearch/client_api.py", line 535, in __init__
    cache_dir=cache_dir, expiration_in_seconds=cache_timeout_in_seconds)
  File "/usr/local/lib/python2.7/site-packages/codesearch/file_cache.py", line 46, in __init__
    os.mkdirs(cache_dir)
AttributeError: 'module' object has no attribute 'mkdirs'

Changing os.mkdirs() to os.makedirs() fixes the problem.

https://docs.python.org/3/library/os.html#os.makedirs

[asankah]

Could you check if the assertion still fails on the master branch? I found a bug where the code wasn't comparing timestamps correctly which could've led to the cache files being marked as expired prematurely.

[cpu82]

I'm using the following code to test the cs library

import codesearch

# The plugin can optionally work with a local Chromium checkout (see the
# documentation for CodeSearch.__init__()), but for this example, we are going
# use the API without a local checkout. This is indicated by setting the
# |source_root| to '.'.
cs = codesearch.CodeSearch(should_cache=True, cache_dir="/tmp/cache", source_root='.')

# Let's look up a class:
# The SearchForSymbol function searches for a symbol of a specific type. In
# this case we are looking for a class named File. There may be more than one
# such class, so the function returns an array. We only need the first one.
file_class = cs.SearchForSymbol('File', codesearch.NodeEnumKind.CLASS)[0]

# SearchForSymbol returns an XrefNode object. This is a starting point for cross
# reference lookups.
assert isinstance(file_class, XrefNode)

# Say we want to look at all the declared members of the File class. This
# includes both member functions and member variables:
members = file_class.GetEdges(codesearch.EdgeEnumKind.DECLARES)

# There'll be a bunch of these.
assert len(members) > 0

# ... and they are all XrefNode objects.
assert isinstance(members[0], XrefNode)

# We can find out what kind it is. The kinds that are known to CodeSearch are
# described in the codesearch.NodeEnumKind enumeration.
print(members[0].GetXrefKind())

but it fails as follows

% python2.7 test.py
Traceback (most recent call last):
  File "test.py", line 17, in <module>
    assert isinstance(file_class, XrefNode)
NameError: name 'XrefNode' is not defined

Some files are generated in /tmp/cache even though the test does not end

% ls /tmp/cache/
19bc07c526459ec4d129dc991382091efc7a6eb5
838facc0f97b0c2d9861abcbda1f48f321f631f7
ba89decef426397c34ada6d3aacc1add94ce2a67

[cpu82]

Could you check if the assertion still fails on the master branch? I found a bug where the code wasn't comparing timestamps correctly which could've led to the cache files being marked as expired prematurely.

Yes, it was fixed in 84f2757

===>  Testing for py27-codesearch-py-0.1
===>   py27-codesearch-py-0.1 depends on file: /usr/local/bin/python2.7 - found
test_figment_display_name (codesearch.test_client_api.TestCodeSearch) ... ok
test_figment_display_name_2 (codesearch.test_client_api.TestCodeSearch) ... ok
test_figment_display_name_3 (codesearch.test_client_api.TestCodeSearch) ... ok
test_fixed_cache (codesearch.test_client_api.TestCodeSearch) ... ok
test_get_signature_for_symbol (codesearch.test_client_api.TestCodeSearch) ... ok
test_get_signatures_for_symbol (codesearch.test_client_api.TestCodeSearch) ... ok
test_get_type_1 (codesearch.test_client_api.TestCodeSearch) ... ok
test_get_type_2 (codesearch.test_client_api.TestCodeSearch) ... ok
test_get_type_3 (codesearch.test_client_api.TestCodeSearch) ... ok
test_get_type_4 (codesearch.test_client_api.TestCodeSearch) ... ok
test_search_for_symbol (codesearch.test_client_api.TestCodeSearch) ... ok
test_user_agent (codesearch.test_client_api.TestCodeSearch) ... ok
test_with_cache_dir (codesearch.test_client_api.TestCodeSearch) ... ok
test_display_name (codesearch.test_cs_file.TestCsFile) ... ok
test_path (codesearch.test_cs_file.TestCsFile) ... ok
test_text_range (codesearch.test_cs_file.TestCsFile) ... ok
test_with_cache_dir (codesearch.test_file_cache.TestFileCache) ... ok
test_with_no_cache_dir (codesearch.test_file_cache.TestFileCache) ... ok
test_class_with_known_keyword (codesearch.test_messages.TestConstructor) ... ok
test_class_with_unknown_keyword (codesearch.test_messages.TestConstructor) ... ok
test_empty_class (codesearch.test_messages.TestConstructor) ... ok
test_from_json_string_1 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_2 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_3 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_4 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_5 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_6 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_7 (codesearch.test_messages.TestProto) ... ok
test_from_json_string_invalid (codesearch.test_messages.TestProto) ... ok
test_from_shallow_dict_1 (codesearch.test_messages.TestProto) ... ok
test_proto_bridge (codesearch.test_messages.TestProto) ... ok
test_get_all_edges (codesearch.test_xref_node.TestXrefNode) ... ok
test_related_annotations (codesearch.test_xref_node.TestXrefNode) ... ok
test_related_definitions (codesearch.test_xref_node.TestXrefNode) ... ok
test_related_definitions_2 (codesearch.test_xref_node.TestXrefNode) ... ok
test_simple_xref_lookup (codesearch.test_xref_node.TestXrefNode) ... ok

----------------------------------------------------------------------
Ran 36 tests in 1.008s

OK

[asankah]

Thanks for confirmation. I updated the example in README.md to fix the other issue you pointed out. I also made it a doctest and included it in the tests run as part of run_tests.sh.

Please feel free to report any other issues you run into, and thanks for your help so far!