Comments (5)
Hi Will,
I spent some time looking into this today, and I think what's happening is that the sample size of the "sleepstudy" dataset is too small for our approximation to work correctly. We approximate the posterior distribution of the coefficients by fitting a Gaussian -- so if there aren't enough samples, the distribution may not be Gaussian enough for the approximation to match the observed variance (I assume lme4 is bootstrapping this).
If you use the larger "InstEval" dataset, the numbers are much closer:
data(InstEval)
m <- lm(y ~ service, data=InstEval)
summary(m)
R lm results:
Column | Estimate | Std. Error |
---|---|---|
(Intercept) | 3.262236 | 0.006527 |
service | -0.130499 | 0.009920 |
Photon-ML Results:
Column | Estimate | Std. Error |
---|---|---|
(Intercept) | 3.262236 | 0.004901 |
service | -0.130499 | 0.007448 |
These still aren't exactly the same, but they're much closer. In our tests with large datasets, the numbers always match within 0.1% or so.
Just to reiterate: The coefficient variance approximation in Photon assumes a very large dataset (let's say at least 50,000 examples), so that the posterior distribution of coefficients is roughly Gaussian.
from photon-ml.
Thank you for the reply @joshvfleming! I loaded up the InstEval data set into my script and I'm still getting some odd results. For your photon-ml results, can you try adding a random effects term in there as well? And are you using the GameEstimator or something else? Thank you for your guidance!
data(InstEval)
# Coercing these to numeric
InstEval$deptNum <- as.numeric(InstEval$dept)
InstEval$serviceNum <- as.numeric(InstEval$service)
m <- lmer(y ~ serviceNum + (serviceNum | deptNum), data=InstEval)
summary(m)
Results from R
Fixed effects:
Column | Estimate | Std Error | T Value |
---|---|---|---|
(Intercept) | 3.34428 | 0.08421 | 39.71 |
serviceNum | -0.07025 | 0.06507 | -1.08 |
Random effects:
Groups | Column | Variance | Std.Dev. |
---|---|---|---|
deptNum | (Intercept) | 0.09424 | 0.3070 |
serviceNum | 0.05676 | 0.2382 |
Results from my Photon-ML script:
Running it with some different hyperparameters gets me closer to the coefficients (so not worried about that) but the variance is still extremely small.
Fixed Effects:
column | coeff | variance | stddev | TStat |
---|---|---|---|---|
INTERCEPT | 2.7221554602438345 | 2.400499E-05 | 0.0048995 | 555.599844 |
service | 0.905410379159576 | 5.5443361E-05 | 0.00744603 | 121.5963934 |
Random Effects:
dept | column | coeff | variance |
---|---|---|---|
12 | INTERCEPT | 0.6153931 | 1.6100E-04 |
12 | service | -0.88572352 | 4.6211E-04 |
2 | INTERCEPT | 0.5510315 | 8.1566E-04 |
2 | service | -1.116591909 | 1.1998E-03 |
7 | INTERCEPT | 0.5953893 | 6.2383E-04 |
7 | service | -1.103209066 | 1.7094E-03 |
8 | INTERCEPT | 0.6417991 | 1.7241E-03 |
8 | service | -1.008190338 | 1.9831E-03 |
6 | INTERCEPT | 0.50042 | 2.6497E-04 |
6 | service | -1.12890574 | 4.9601E-04 |
9 | INTERCEPT | 0.62 | 2.3663E+00 |
9 | service | -1.3549822 | 6.5293E-04 |
5 | INTERCEPT | 0.62422 | 2.7949E+00 |
5 | service | -0.76248481 | 4.9281E-03 |
3 | INTERCEPT | 0.52969 | 2.8153E+00 |
3 | service | -0.5889002 | 1.1144E-03 |
15 | INTERCEPT | 0.50732 | 4.0519E+00 |
15 | service | -0.7092034 | 1.6134E-03 |
4 | INTERCEPT | 0.57968 | 2.2124E+00 |
4 | service | -0.9527013 | 6.7394E-04 |
from photon-ml.
It looks like what's happening there is that lmer
is computing a covariance matrix for the entire model (fixed + random effects), and then taking the diagonal of that to report variances for coefficients for e.g. the Fixed Effect model. We can't really do that with our current setup in Photon, because the models are trained separately. Also it would quickly become very impractical to do this for any real dataset, since the covariance matrix is O(n^2)
in the number of coefficients (which themselves grow as O(nm)
in the number of random-effect groupings n
and average number of coefficients m
).
To sum up -- the variances that Photon produces are a decent approximation to the posterior of each individual (e.g. fixed effect) model, but they're not "correct" in context of the larger additive model. I suppose we could add a post-processing step to compute the joint variances, but as I mentioned before, this would become impractical really fast. I am open to suggestions, though. 😃
from photon-ml.
Ah! Okay. This makes sense. I forgot the A in GAME ;-) Thank you for that clarification!
I would be interested in pursuing the post-processing step but I wouldn't know where to start in the codebase. Any pointers on how I could extract the covariance matrix after the models have been trained?
It looks like model.Coefficients
only contains the means and variances and the actual variance is calculated in SingleNodeOptimizationProblem.
If only there were a simple way of just aggregating the variances and normalizing!
Maybe I just need to try using the ModelDiagnostics for coefficient importance instead.
Thank you for all your help and guidance, Josh!
from photon-ml.
Hello Will - just following up that there's no way right now to pull the covariance matrix out after model training. However, you're always free to contribute any changes you make to Photon ML for your own pusposes. I'm going to close this issue, please re-open it if we can further assist you.
from photon-ml.
Related Issues (20)
- Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class
- error in running job:"Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class" HOT 3
- TRON Optimizer Does Not Support Coefficient Constraints
- Item-item Similarity HOT 1
- GaussianProcessSearch does not substract mean from the prior evaluations HOT 1
- Receiving ClassCastException in getGameDatumFromRow HOT 4
- Getting exception while fitting the Random Effect Model HOT 3
- Example of a Mixed effects model? HOT 4
- Can't run a GAME HOT 5
- 'model-metadata.json' not saved in GCS HOT 1
- GLMix in tensorflow HOT 1
- Value modelsRDD in class RandomEffectModel cannot be accessed HOT 2
- GameTrainingDriver will not run HOT 2
- Examples for hyperparameter tuning HOT 5
- style sheet missing
- featureShardConfigurations when scoring a mixed effects model with random intercept only HOT 1
- The best model's RMSE score doesn't match .. HOT 1
- Using/Refactoring for Standalone Model Runs in Spark-Scala HOT 2
- Cannot build photon-ml with Java 11 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from photon-ml.