Comments (13)
Hi, an export option for the Report class is already on my to-do list! :)
I will come back with a proposal so that we can discuss it before I implement the functionality.
from ranx.
I added Report.to_dict
and Report.save
.
I updated ranx
on PyPi with these new features.
Closing.
from ranx.
Fixed in 0.1.10
. Sorry for the inconvenience.
from ranx.
Hey, sorry for the delay.
This is my proposal for the Report.to_dict
function (I can add a Report.save_as_json
function for convenience too):
{
# metrics and model_names allows to read the report without
# inspecting the json to discover the used metrics and
# the compared models
"metrics": ["metric_1", "metric_2", ...],
"model_names": ["model_1", "model_2", ...],
#
"model_1": {
"scores": {
"metric_1": ...,
"metric_2": ...,
...
},
"comparisons": {
"model_2": {
"metric_1": ..., # p-value
"metric_2": ..., # p-value
...
},
...
},
"win_tie_loss": {
"model_2": {
"W": ...,
"T": ...,
"L": ...,
},
...
},
},
...
}
Let me know what you think. :)
from ranx.
Looks great (and there was not so much delay 😅)!
from ranx.
I’m getting a "TypeError: Object of type int64 is not JSON serializable" which is probably coming from numba or numpy
from ranx.
Yeah, I know about that issue. I will look into it soon.
As a workaround, you can call report.to_dict()
and save the dictionary as a JSON
by yourself with the exact same code I wrote for the report.save
function.
That issue it's kinda weird.
from ranx.
don’t you need to convert int64
to int
in to_dict
?
from ranx.
for example in transformers
they use:
def denumpify_detensorize(metrics):
"""
Recursively calls `.item()` on the element of the dictionary passed
"""
if isinstance(metrics, (list, tuple)):
return type(metrics)(denumpify_detensorize(m) for m in metrics)
elif isinstance(metrics, dict):
return type(metrics)({k: denumpify_detensorize(v) for k, v in metrics.items()})
elif isinstance(metrics, np.generic):
return metrics.item()
elif is_torch_available() and isinstance(metrics, torch.Tensor) and metrics.numel() == 1:
return metrics.item()
return metrics
from ranx.
this fixes it but you probably want to deal with it some other way? If not I can open a PR PaulLerner@7e2218d
from ranx.
I will look into it soon.
from ranx.
I’m still getting TypeError: Object of type int64 is not JSON serializable
from ranx.
oops, looks like I was on the wrong branch, sorry about that
from ranx.
Related Issues (20)
- [BUG] dcg and dcg_burges do not work in the compare function HOT 2
- [Feature Request] Use black to indent the code HOT 1
- [BUG] RBP with multiple relevance levels HOT 3
- [Feature Request] Support gzipped files? HOT 3
- [Feature Request] memory issue / make Run more efficient HOT 2
- Incorrect result for f1 score HOT 13
- Zero-scored documents HOT 10
- [BUG] Misleading exception message on dataframe types HOT 2
- [BUG] Issues when storing/loading Qrels from a dataframe and a parquet file. HOT 6
- [Feature Request] Run.from_df and Run.from_parquet does not allow specifying run name HOT 1
- Question on rank aggregation usage HOT 4
- Getting "Segmentation fault (core dumped)" error HOT 2
- [Feature Request] stddev statistic HOT 3
- Couldn't find any documentation about Qrel and run score range HOT 2
- [Feature Request] Propensity-scored Metrics HOT 1
- How do we compare different runs with multiple folds per run? HOT 1
- [Question] About the correction among multiple hypotheses HOT 1
- [Question] How to compute precision for a retriever operating at passage-level HOT 1
- JIT compilation on serverless (i.e. Modal Labs) HOT 1
- I'm getting NaN for the BPref measurement HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ranx.