ba-hanseml / nf_prj_mimii_dataset Goto Github PK

• workflow_improvment.md - threading Jupyter file storing diagrams ... etc.
• imporvments.md - list of left out tech like LSTM autoencoder, DOA filter.... just list .
• application_notes.md - smart sensor train time reaction time sensor aligment etc ... training algorthems memory size ...

We should add pandas and seaborn to the environment

• add to env_baseline_noGPU_TF2.yml
• add to env_baseline_withGPU_TF2.yml

https://stackoverflow.com/questions/28908319/how-to-clear-an-ipython-notebooks-output-in-all-cells-from-the-linux-terminal

make mockup notebook - ID04 and ID06

• feat ext. design - mini EDA
• feat ext. all ... (best feat. version exdia_v1_best_off)
• ensamble performace show of the best version
• result summery

Thisn might be to specific and should not have been commited:
A:\Dev\NF_Prj_MIMII_Dataset\sub_notebooks\clustering_EDA\data\pump_6dB_00020406_full_spectrum_128.pkl

My idea was to keep folder with extracted data with the maine dataset
A:\Dev\NF_Prj_MIMII_Dataset\dataset\PSD_to_Pandas
including alwas the creation screept ...

lets discuss

Scpiy is missing and I guess we eill needd the scipy.signal ata some point

a nice high res pcture with PSD and MEL per MAchine part and ID

machine part ID
normal ,also normal, abnormal, alose abnorma
PSD
MEL
plus mp 3

• Arne - Paper

• Arne - git

• Arne - web

• Bendix- Paper

• Bendix- git

• Bendix- web

4 x 2 example set in mp3 and MEL as example for the intrested reader

I can only find a way to read one channel from wav files

• check with some audio cut programm how many ch are there
• investigate how to read wav file native and see all channels
• not clear now

• Load all Audio files similar to the baseline model and store them within pickle files.
• Change pickle directory to ".\dataset\pickle"
  • Store the pickle files within the dataset-folder to access it easier from different sub_notebooks

• main md and diagram pictures
• jupyter move and clean from sub notebooks
• clean the main folder and notebooks in general
• polish the diagram batch

ileNotFoundError: [Errno 2] No such file or directory: '.\pipes\MEL_den_frame5_6dB_pump_ID00_20200504_132755.pkl'

some wehre here needs to be chack

this is good
os.makedirs(target_folder_full, exist_ok=True)

• Theme and Base - Arne
• Index/Agenda
• Use Case Scenario - Arne
• Dataset and challange transfer to feat. ext. - Arne
• Featextaction - Simple Spectrum -- and list of things you can do... before denoise etc. - ARne
• Predictve Modeling - Diffrent ways to detect anomlys - Bendix
• stochstic - Bendix
• auto encoder NN - Bendix
• pseudo supervised - Arne
• Diverse ensamble and Time Frame ensmable blender- Bendix
• Results - Bendix
• Future Work - feat ext. - Arne
• Future Work - modeling - Bendix ( +feedback augmentation)
• (application notes ) - Arne

clustering -> result summery or general modeling like explored but did not been any good...
basic stats -> result summery explored on spot checks plus future work

conda install -c conda-forge shogun

I have made a base line of README.md please review
Then I think it would be good if the logo would be at the very top ...

We could condense the feature generation notebooks into a python script and makeit easier to use.

This is optional and a reminder for the polishing-phase.

I found pywavelet to be super fast wavelet lib better then scipy - but not yet clear if needed
assumtion is the a CWT maybe a good feature extracor

have a start point wehre the delay is extracted by step scanning very inefficent code but it works

• general function define
• test between microphones
• test for phase stable filted signals
• if looking good to detect sources - work on auto scan through mics
• invetigate efficent algorthmes

in sub_notebooks/DSP_playground is base line

The generation of descriptive dataframes should either:

1. have individual naming for each SNR and machine so they dont overwrite or
2. and preferably append an existing dataframe

This issue acts as discussion collection card until split up into singel tasks:

bug: akku plot can not hue
bug: plot every 10th etc seems to go wrong
bug: some times mel spectra turns into wrong time format ???

task: add ceptrum
task add MFCC

...

wish: combine PSD and PSD per file

wish: zoomable time series , zoomable other plots ..
wish: return all ax handles to use as pre nice plotting some time

Data input

• load data from pickle created by utility function

• PSD-Data

• Mel-Filter-Data

• Beat-Data

Scaling

• Log-Scaling

• StandardScaling

Dimensionality Reduction

• PCA including 2D/3D Visualization

Clustering

• t-SNE for Visualization

• K-Means

• Mean-Shift

• DBSCAN

Evaluation

• Models that handle outlier like DBSCAN or those that calculate a certainty could be used as "prediction" model

Train Test split
Pipline