Comments (11)
hi, try using the StandardScaler instead of the MinMaxScaler before anything else.
Other things that you can try:
- Use a small sigma, e.g. sigma=1 if you stick with a gaussian neighborhood function.
- If your learning curve is still unexpected, use an inverse_decay for both the decay functions and play with the learning rate.
- Changing the neighborhood function might also lead to good results but you'll have tweak sigma and learning rate accordingly.
from minisom.
@JustGlowing The strange shape of the learning curve can be explained by an oversight in my own code, I was running the visualization process from the BasicUsage.ipynb
example on an already trained SOM instead of a new instance of one. However, the large QE question still remains. I noticed that in the MovieCovers.ipynb
example that there is also quite a large QE, so this could be the real error value for the problem I'm working with.
(The only aspect that has changed about the problem from the last post is I now have a few more samples to work with.)
Using StandardScaler increases QE substantially, is there an interpretive reason as to why standard scaler might be preferred? Here is a side by side comparison with equal hyperparameters:
Learning curve visualization [with linear_decay_to_(zero/one)]
from minisom.
from minisom.
Gotcha, thanks for the explanation! So if I am understanding correctly the high quantization error might have to do with my input size then (combined with the size of the SOM)? If I am originally working with a 420x444 lat/long grid containing pixel values that I flatten down into 'input_len=186480' for input into a SOM then it follows that even small distances between samples might be compounded over the dataset's size leading to higher QE?
# Suppose an average squared difference per feature is around 0.01 (or some small difference due to scaling)
Average Squared Difference = 0.01×186480=1864.80
Quantization Error = sqrt(1864.80) ≈ 43.2
Here is a quick example of the SOM output when I don't scale the data:
[ 200 / 200 ] 100% - 0:00:00 left
quantization error: 157049.84784462157
SOM training took 4.58 seconds!
Begin Learning Curve Visualization
End Learning Curve Visualization
Q-error: 157887.617
T-error: 0.013
SOM LC visualization took 201.26 seconds!
An averaged composite of all samples included within each node (a sanity check):
from minisom.
from minisom.
@JustGlowing Right now the input length is 186480 as I have 1 variable that is flattened into one numpy array (with 149 samples this becomes a vector of length 186480*149. What would be the best way to add another variable on top of this? In the documentation for "train" it states that the input data can be a np.array or list Data matrix.
How does this work when initializing the SOM with input_len
when the data spans multiple rows? Is the way I would incorporate another variable to flatten the new variable and append it onto the first, making the input length 186480 + 186480 = 372960? It would seem more logical to add it as another column in the input variables, but then my question would again be how would this work with input_len
; as one row doesn't correspond to one sample but 186480 rows (and 2 or 3 or more columns) would still correspond to with one input into the SOM.
I am looking for functionality similar to the R "supersom" package. Is that something MiniSOM could support naturally?
from minisom.
Hi, from what I understand you have an input matrix with 149 rows and 186480 columns. This means that you have 149 samples and 186480 variables. Even if you are reshaping objects that in other domains are considered variables, for a SOM the columns are considered variables and the rows samples.
From what I understand, you want to add more variables to your input and it's an easy task. You just need to add more columns to your matrix and set input_len
equal to the number of columns that you have.
from minisom.
@JustGlowing I worded this in a weird way, I apologize. Each variable of the data is spatial and has and X and Z to it. If I were to put it in the format in your previous comment it would have 3 columns (so an input length of 3?). I've tried to implement this in MiniSom given my understanding of this problem and came up with the following discussion:
# | Pressure | Temperature | Humidity |
| [420x444] | [420x444] | [420x444] | (sample 1)
| [420x444] | [420x444] | [420x444] | (sample 2)
...
| [420x444] | [420x444] | [420x444] | (sample 149)
this organization has a shape of (149, 3, 420, 444).
However when this is input into minisom:
ValueError: could not broadcast input array from shape (3,420,444) into shape (3,)
# | Pressure | Temperature | Humidity |
| [186480] | [186480] | [186480] | (sample 1)
| [186480] | [186480] | [186480] | (sample 2)
...
| [186480] | [186480] | [186480] | (sample 149)
This output with the all data being flattened would have a shape of (149, 3, 186480).
Yields the same result when put into Minisom:
ValueError: could not broadcast input array from shape (3,186480) into shape (3,)
When I do this I run into the following errors relating to the shape of the input data, hence why I was looking for a way to represent this in minisom. From the Readme it doesn't seem this is currently supported by MiniSom, but I was wondering if this has ever been encountered by others in the past?
If input_len
could instead accept an input_shape
this could work to represent this sort of multivariable spatial data in a structure compatible for MiniSom, however this wouldn't be such a simple change for the rest of the package.
from minisom.
The only way I can think of to represent this in minisom would be to append the pressure, temperature, and humidity values together into one vector of length 186480 + 186480 + 186480 = 559440. that way this would consider all three variables and I could reshape portions of this vector for the production of a final visualization similar to above?
I.e the first 186480 values within the weights of a SOM node would correspond to pressure, the next 186480 temperature, and the next 186480 humidity?
from minisom.
hi again @vwgeiser, MiniSom only accepts square matrices as input and your last intuition makes sense.
If you find that the dimensionality the problem becomes an issue, you can train a different SOMs for each type of input and then find a way to aggregate the results.
from minisom.
The only way I can think of to represent this in minisom would be to append the pressure, temperature, and humidity values together into one vector of length 186480 + 186480 + 186480 = 559440. that way this would consider all three variables and I could reshape portions of this vector for the production of a final visualization similar to above?
I.e the first 186480 values within the weights of a SOM node would correspond to pressure, the next 186480 temperature, and the next 186480 humidity?
For anyone finding this thread later, vectorization is the process I was looking for and is what I currently am working with for a solution!
from minisom.
Related Issues (20)
- About how you efficiently calculate the winner and the neighbourhood function. HOT 1
- How can I get reproducible results? HOT 2
- Meaning of the position of winning nodes and distance map HOT 3
- considerations on the hexagonal neighborhood used in `distance_map()` HOT 6
- It seems the next run will be influenced by the former run. Will it be more rational to reset after every run? HOT 4
- Is there a way to create clusters of equal size? HOT 2
- What's the order of the coordinates returned by the winner method? HOT 2
- How to save and load trained model for cluster prediction on new data? HOT 1
- PCA initialization HOT 7
- Adjacent nodes in hexagonal distance do not have distance 1 HOT 2
- Question: Interest in a connectivity matrix? HOT 2
- unexpected problem calling the functions HOT 11
- errors in 'AdvancedVisualization.ipnyb' example HOT 2
- Help with clustering and visualization of neurons HOT 1
- There might be an index error in TimeSeries.ipynb HOT 1
- decay for sigma can be improved HOT 5
- error in clustering SST data
- Can i do SOM for multiple variables? HOT 4
- Dashboard shows normalized data HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from minisom.