Comments (7)
Hello @tuobay , thanks for asking. The F-measurement we use is according to this paper: https://arxiv.org/pdf/1511.06233.pdf , where false-positive is defined as "incorrect classifications on the validation set". I think the validation set in this paper are from seen classes, so that for false positives, labels should not equal to -1. Does that make sense?
from openlongtailrecognition-oltr.
@zhmiao a related question to above: I tried changing the open set threshold as in Fig 8b) of the paper, but I'm getting different results. Specifically, I get a monotonically increasing sequence of F-measure values, rather than decreasing, on increasing open set threshold from 0 to 1. Any ideas why ?
from openlongtailrecognition-oltr.
This is what I get for ImageNet-LT, using the Stage-2 model (its not the latest) :
F_measure (with threshold 0.00) :0.3993
F_measure (with threshold 0.10) :0.4532
F_measure (with threshold 0.20) :0.5793
F_measure (with threshold 0.30) :0.6842
F_measure (with threshold 0.40) :0.7646
F_measure (with threshold 0.50) :0.8261
F_measure (with threshold 0.60) :0.8696
F_measure (with threshold 0.70) :0.9095
F_measure (with threshold 0.80) :0.9348
F_measure (with threshold 0.90) :0.9475
from openlongtailrecognition-oltr.
I think the TP + FP + TN + FN = num of all test samples
So the class (label = -1 and pred = positive) should be put into FP.
from openlongtailrecognition-oltr.
@zhmiao @liuziwei7 awaiting your response
from openlongtailrecognition-oltr.
Hello @tuobay @ssfootball04 . Thank you very much for the discussion. We think the problem is indeed false positive calculation. The new false positive is : false_pos += 1 if preds[i] != labels[i] and labels[i] != -1 else 0
. We removed the preds[i] != -1
since it actually does not make sense. According to the paper we cited, " false positives are incorrect classifications on the validation set". So we think the current calculation is correct. After removing this, the F-measure numbers are normal. We think the reported F-measure numbers are a little bit higher than actual numbers for all baselines. We will update it as soon as possible. I have pushed the new code already. Please check it out. Thanks again.
from openlongtailrecognition-oltr.
Since it has been a while, I will close this issue for now. Please feel free to re-open this issue if any questions raised. Thanks.
from openlongtailrecognition-oltr.
Related Issues (20)
- Reproducing OLTR results HOT 3
- Stage 2 multi GPU
- why fix all parameters except self attention parameters? HOT 4
- Table 2 results HOT 2
- Pretrained Weights for Places_LT?
- the use of fc layer HOT 2
- the accuracy of the train and val HOT 2
- how to compute centroids?
- Why the input dimension of the `fc_spatial` layer in `ModulatedAttLayer` is 7*7*in_channel? HOT 1
- Many_shot_accuracy_top1: nan on my own dataset HOT 1
- Revised F-measure results for other models in your paper
- Applications for face recognition
- Error when running stage_1.py under Places_LT
- Unable to reproduce baseline result on ImageNet-LT HOT 1
- BUG: stage1 test error!!
- Could you please give me an example of arranging ILSVRC2014 dataset? HOT 7
- Implementation on Inat-18
- About Class aware sampler
- The role of untrained FC(add_fc)
- The question about the version of Places_LT
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openlongtailrecognition-oltr.