brain-score / vision Goto Github PK
View Code? Open in Web Editor NEWA framework for evaluating models on their alignment to brain and behavioral measurements (50+ benchmarks)
Home Page: http://brain-score.org
License: MIT License
A framework for evaluating models on their alignment to brain and behavioral measurements (50+ benchmarks)
Home Page: http://brain-score.org
License: MIT License
Using neural fit to go from hvm
to itself (i.e. neural_fit(hvm, hvm)
) yields a comparably low score of around .78
(see this unit test).
Did others experience something similar (@qbilius)? Is that an issue with the regression?
the RDMs seem to be different after the recent assembly re-formatting, suggesting that there's an error in the data.
Make sure that the data is aligned properly.
DataAssembly.reset_index() does not work, because internally DataArray._replace() constructs a new object, and the DataAssembly constructor recreates the MultiIndexes.
A .ini or .yml (or whatever) file, and infrastructure for making config stuff available throughout the project.
Make a config.example.yml, add 'config.yml' to .gitignore so users can add credentials without accidentally committing.
Things to include:
PEP 508 allows the specification of install_requires
requirements like so:
install_requires=[
"result_caching @ git+https://github.com/mschrimpf/result_caching",
]
This functionality is supported since pip 18.1 and removes the need for --process-dependency-links
(which will be removed in pip 19).
We need to update at least our setup.py
accordingly.
There is no convenient method to obtain the names of the MultiIndex levels for a given dimension.
Note that it's not simple, as a dimension may have multiple MultiIndexes, and (I think) a given MultiIndex may apply to more than one dimension.
it seems like there is an issue in the interplay between PCA and the neuralfit.
Specifically, test_metrics.test_neural_fit_metric_pca100
scores very low (<0.15
) and I observed something similar when PCA'ing down to just 167 (= len(neuroids) - 1
).
(Submitting as requested)
If I use the most recent xarray version, it gives an error that 'Score' does not have an attribute 'indexes'. Using older version of xarray won't give the error.
Table doesn't fit on screen yet there is no horizontal scrollbar.
Tested on Firefox.
Possible solution (wild guess):
body {
overflow:auto;
}
test_mkgu
has two unused declarations of test_load
type
is a reserved word in Python; consider replacing with kind
Benchmark.calculate
, Metric.apply
etc might be better served by __call__
method_fetcher_types
might be better declared at the top and in capitals (FETCHER_TYPES
)return 0
(e.g., in metrics.py
) is not a thing in Pythonlookup.db
could be a simple csv file? Since it is unlikely to grow too big, there would be no performance penalty, but there would be an advantage of being able to quickly see dataset names and available assets.right now coord, dims, value
is being yielded
on the same note, make sure that we compute the correlation over neuroid
s and not id
s.
We should automatically run our examples with travis to make sure they still work.
At the moment, they keep getting out-dated because there is no automatic test in place to ensure they run.
There is an example of such a setup here: https://github.com/ghego/travis_anaconda_jupyter
Some of the jupyter notebooks illustrating the use of brainio code are in this repo, and they should be moved to brainio or brainio-contrib repo
given the activations of a model, show how they can be fed into a Benchmark to retrieve a score for that model
ideas for improving:
(transferring @qbilius' issue from https://github.com/mschrimpf/brain-score/issues/5)
df = brainscore.get_stimulus_set('dicarlo.hvm')
df = brainscore.stimuli.StimulusSet(df[df.variation == 6])
df
is now a pandas DataFrame, not StimulusSet.
expected output is that df
remains a StimulusSet.
Profile can then be used with session = boto3.Session(profile_name='mturk')
There is no convenient method to save a DataAssembly.
just saw this and was wondering about it: in https://github.com/dicarlolab/mkgu/blob/d054d65fc2410587c91fac066c21f82f1d32b0df/mkgu/stimuli.py#L10, should this not be moved to the __init__
constructor?
For models such as CORnet-R with many zeros in their activations, PLS regression fails with a numpy.linalg.LinAlgError: SVD did not converge
.
The error originates from NaNs in the regression weights which in turn stem from https://github.com/scikit-learn/scikit-learn/blob/a7a834bdb7a51ec260ff005715d50ab6ed01a16b/sklearn/cross_decomposition/pls_.py#L67 where x_score = 0
and thus y_weights = ... / 0 = NaN
.
Solution approaches
following up on #1, it would be great if we could get some documentation. Ideally we'd create this automatically through readthedocs.io
mkgu gives NoCredentialsError if there are no AWS credentials files, even if the resource is public.
We should have the whole ventral stream in mkgu rather sooner than later, even if chunked.
Right now, we have V4-IT and we can get V1 and V2 data from Jack Gallant's lab.
V1: http://crcns.org/data-sets/vc/pvc-4/about (paper)
V2: http://crcns.org/data-sets/vc/v2-1/about-v2-1 (paper)
(also nice to see the approach of a different data viewer)
To correct add the following field to the stimulus_set_degrees dictionary on brainscore/init.py
'tolias.Cadena2017': 2
It seems like users still have to configure AWS credentials even if they only access public resources.
Can we get rid of that requirement @jjpr-mit? If yes, how long will it take?
It's also okay if we put a note in the README detailing how to configure AWS.
Installing mkgu with python setup.py install
doesn't copy the lookup.db
file to the site-packages
directory. As a result, SQLiteLookup
does not find the table names.
presentation: presentation_id, image_id (currently, it's called id
(double-check it's not the _id
))
neuroid: neuroid_id
rename vars from 'V0' -> 0
filenames -> strip full path
Getting the following errors when creating conda environment with conda env create -f environment.yml
:
ResolvePackageNotFound:
- netcdf4==1.2.4=np113py36_1
After removing that package:
UnsatisfiableError: The following specifications were found to be in conflict:
- libnetcdf==4.4.1=1 -> jpeg=9
- qt==5.6.2=2
Sometimes (!), mkgu raises an sqlite3 disk I/O error.
This does not occur all the time, it seems like it mostly happens when running jobs in batches. Maybe concurrent accesses to sqlite do not work?
Traceback (most recent call last):
File "neural_metrics/compare.py", line 39, in main
hvm = mkgu.get_assembly(name="HvM")
File "/om/user/msch/miniconda3/envs/neural-metrics/lib/python3.6/site-packages/mkgu/__init__.py", line 11, in get_assembly
return fetch.get_assembly(name)
File "/om/user/msch/miniconda3/envs/neural-metrics/lib/python3.6/site-packages/mkgu/fetch.py", line 247, in get_assembly
assy_record = get_lookup().lookup_assembly(name)
File "/om/user/msch/miniconda3/envs/neural-metrics/lib/python3.6/site-packages/mkgu/fetch.py", line 129, in lookup_assembly
cursor.execute(self.sql_lookup_assy, (name,))
sqlite3.OperationalError: disk I/O error
make sure the following works:
import brainscore
brainscore.metrics
contents:
and add a convenience method for getting coords names for a given dimension
A Similarity takes as input assembly1
, assembly2
and outputs a Score
object.
As of now, there are two kinds of similarities:
RDMSimilarity
: compute similarity of two assemblies directlyNeuralFit
: first fit on a training set, then predict on a test set and compute similarity based on the predictionsWe also have additional utility on top of the simple case:
region
).object_name
as part of presentation
There are several ways to organize the code around this:
OuterCrossValidationSimilarity
that all Similarity
classes need to inherit from. This parent class implements (1) and (2) from above and sub-classes only need to implement the simple case. Drawbacks:apply
in our sub-classes but need to adjust the method nameSimilarity
class implements exactly one operation in apply
. A chain operator then takes all these classes, applies them one after another and outputs only the final result. The result here is a list of assemblies which are then fed into a Score
in Similarity.__call__
. However, I don't know how to represent parametric and non-parametric similarities with this approach (one has to fit
, predict
, compare_prediction
, the other just has to compare
)Similarity
: all the specialized handling ((1) and (2) from utilities) goes into Similarity
sub-classes, operation on simplified assemblies goes into a Computor
class. Hard to separate the two though, for instance Similarity
still needs to call fit
, predict
etc.For now, approach (1) works but after NIPS, I would like to revisit the structuring here.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.