statnett / data_cache Goto Github PK
View Code? Open in Web Editor NEWPython data cache decorator
License: MIT License
Python data cache decorator
License: MIT License
Hi, how can I clear all the cache? Thank you
The static-frames python package works like pandas but enforces immutability on its frames, which would benefit greatly from an integration with data_cache (in the hypothesis of a functional data analysis approach).
https://static-frame.readthedocs.io/en/latest/index.html
@pandas_cache
def get_sentiment(text_content: str):
"""
Classifying Content in a String
Args:
text_content The text content to analyze. Must include at least 20 words.
"""
client = language_v1.LanguageServiceClient.from_service_account_json(filename=GOOGLE_APPLICATION_CREDENTIALS)
# text_content = 'That actor on TV makes movies in Hollywood and also stars in a variety of popular new TV shows.'
# Available types: PLAIN_TEXT, HTML
type_ = language_v1.Document.Type.PLAIN_TEXT
# Optional. If not specified, the language is automatically detected.
# For list of supported languages:
# https://cloud.google.com/natural-language/docs/languages
language = "en"
document = {"content": text_content, "type_": type_, "language": language}
response = client.analyze_sentiment(request={"document": document})
out=analyse_response(response)
return out
@pandas_cache
def analyse_response(response)->Tuple[str]:
magnitude=response.document_sentiment.magnitude
score=response.document_sentiment.score
if -.1<magnitude<.1 or -.1<score<.1:
label= "neutral"
elif score>=.1:
label= "positive"
elif score<=-.1:
label= "negative"
# print("magnitude: ",magnitude)
# print("score: ",score)
# print("label: ",label)
return magnitude ,score ,label
text = """i'm soo happy""".strip()
res=get_sentiment(text)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/tmp/ipykernel_6589/3336704012.py in <module>
2 text = """i'm soo happy""".strip()
3
----> 4 res=get_sentiment(text)
~/miniconda3/lib/python3.7/site-packages/data_cache/cache_tools.py in wrapped(*f_args, **f_kwargs)
210 group = "a" + hashlib.md5(inspect.getsource(func).encode("utf-8")).hexdigest()
211 key = "a" + hashlib.md5(json.dumps(extracted_args).encode("utf-8")).hexdigest()
--> 212 return table_getter(group, key, func, f_args, f_kwargs, extracted_args)
213
214 return wrapped
~/miniconda3/lib/python3.7/site-packages/data_cache/cache_tools.py in store_func(func_key, arg_key, func, f_args, f_kwargs, metadata)
172 if isinstance(data, tuple):
173 for i, data_ in enumerate(data):
--> 174 store.create_dataset(f"{path}/data{i}{suffix}", data=data_)
175 add_metadata(store[f"{path}/data{i}"], func, metadata)
176 else:
~/miniconda3/lib/python3.7/site-packages/data_cache/cache_tools.py in create_dataset(self, key, data)
63 class PandasStore(pd.HDFStore):
64 def create_dataset(self, key: str, data: pd.DataFrame) -> None:
---> 65 data.to_hdf(super(), key)
66
67 def __getitem__(self, key: str) -> PandasGroup:
AttributeError: 'float' object has no attribute 'to_hdf'
OS: Manjaro Linux x86_64
Host: Inspiron 5567
Kernel: 5.15.57-2-MANJARO
Packages: 1589 (pacman), 6 (flatpak), 9 (snap)
Shell: zsh 5.9
python: python3.7
Terminal: vscode
CPU: Intel i7-7500U (4) @ 3.500GHz
GPU: AMD ATI Radeon R7 M260/M265 / M340/M360 / M440/M445 / 530/535 / 620/625 Mobile
GPU: Intel HD Graphics 620
Memory: 7031MiB / 15899MiB
Using example decorator code from README.md with pandas 1.2.4 raises ValueError
exception:
ValueError: The file '/tmp/workspace_analysis/data.h5' is already opened. Please close it before reopening. HDF5 v.1.8.5-patch1, FILE_OPEN_POLICY = 'strict'
Somewhere between pandas 1.1.5 and 1.2.4 pytables has been updated, and storage file can be opened only once to prevent corruption.
Pandas==1.2.4
data-cache==0.1.5
tables==3.6.1
h5py==2.10.0
Downgrading to pandas 1.1.5 same example works.
As temporary solution PandasStore.create_dataset() can be augmented to close file before creating new hdf object:
class PandasStore(pd.HDFStore):
def create_dataset(self, key: str, data: pd.DataFrame) -> None:
try:
self.close()
except pandas.io.pytables.ClosedFileError:
pass
data.to_hdf(self, key)
Full stack:
(openbadge) teemu@graygoose:~/projects/startuplab/workspace_analysis3> python cache_test.py
Traceback (most recent call last):
File "/home/teemu/projects/startuplab/workspace_analysis3/cache_test.py", line 13, in <module>
print(simple_func())
File "/home/teemu/opt/miniconda/envs/openbadge/lib/python3.9/site-packages/data_cache/cache_tools.py", line 212, in wrapped
return table_getter(group, key, func, f_args, f_kwargs, extracted_args)
File "/home/teemu/opt/miniconda/envs/openbadge/lib/python3.9/site-packages/data_cache/cache_tools.py", line 177, in store_func
store.create_dataset(f"{path}{suffix}", data=data)
File "/home/teemu/opt/miniconda/envs/openbadge/lib/python3.9/site-packages/data_cache/cache_tools.py", line 65, in create_dataset
data.to_hdf(self, key)
File "/home/teemu/opt/miniconda/envs/openbadge/lib/python3.9/site-packages/pandas/core/generic.py", line 2606, in to_hdf
pytables.to_hdf(
File "/home/teemu/opt/miniconda/envs/openbadge/lib/python3.9/site-packages/pandas/io/pytables.py", line 277, in to_hdf
with HDFStore(
File "/home/teemu/opt/miniconda/envs/openbadge/lib/python3.9/site-packages/pandas/io/pytables.py", line 561, in __init__
self.open(mode=mode, **kwargs)
File "/home/teemu/opt/miniconda/envs/openbadge/lib/python3.9/site-packages/pandas/io/pytables.py", line 710, in open
self._handle = tables.open_file(self._path, self._mode, **kwargs)
File "/home/teemu/opt/miniconda/envs/openbadge/lib/python3.9/site-packages/tables/file.py", line 288, in open_file
raise ValueError(
ValueError: The file '/tmp/workspace_analysis/data.h5' is already opened. Please close it before reopening. HDF5 v.1.8.5-patch1, FILE_OPEN_POLICY = 'strict'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.