skjerns / autosleepscorerdev

file dump for training

You can reach me also here:

ORC-iD	0000-0002-9050-9040
ResearchGate	Simon Kern
Twitter	@skjerns
Website	skjerns.de

autosleepscorerdev's Issues

CNN-LSTM model

" I extract the activations of the CNN manually, then use the activations as inputs for training a separate LSTM ".

When you say 'extract the activations' , do you mean extract the predicted values from the CNN layer output and feed them as inputs to LSTM ? Conceptually I wanted to know if the input to the LSTM model will just be the predicted labels from CNN model. Can I use one-hot encoded labels ?
Also, I was unable to find this piece of code in your Github repository. It would be very hepful if you can point me to the python class which feeds the CNN activations into LSTM.

edfx_database.truncate_eeg(sleep)

When I am running run_sample.py, everything seems to work well, until I get to the line:
edfx_database.truncate_eeg(sleep)

I am getting the following error message:

OSError Traceback (most recent call last)
in ()
----> 1 edfx_database.truncate_eeg(sleep)

~/Dropbox/Research/sleep lab/AutoSleepScorerDev-master/edfx_database.py in truncate_eeg(sleepdataset)
106 sleep.data = new_data
107 sleep.hypno = new_hypno
--> 108 sleep.save_object()
109
110 def prepare(datadir = 'edfx'):

~/Dropbox/Research/sleep lab/AutoSleepScorerDev-master/sleeploader.py in save_object(self, filename, path)
359 print('Saving data at {}'.format(filename))
360 with open(os.path.join(path, filename), 'wb') as f:
--> 361 cPickle.dump(self.dict,f,2)
362
363

OSError: [Errno 22] Invalid argument

Error of No such file or directory at the end of model training

Hi Simon,
at the end of the training, the compiler return the following error trail

1.0 min
can't save plots: [Errno 2] No such file or directory: '.\plots\0_edfx-sample_ fc1_ fold_ 0_pure_rnn.png'
Got an error while saving model: Unable to create file (unable to open file: name = '.\weights\0edfx-samplefc1_0_0.390-0.080', errno = 2, error message = 'No such file or directory', flags = 13, o_flags = 302)
fold 0: val acc/f1: 0.44937/0.12432, test acc/f1: 0.39016/0.08039
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm 2020.1.1\plugins\python\helpers\pydev\pydevd.py", line 1438, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "C:\Program Files\JetBrains\PyCharm 2020.1.1\plugins\python\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/rpb/Documents/OneDrive - Universiti Teknologi PETRONAS/Code Development/sleep staging/run_sample.py", line 58, in
plot=True, stop_after=4, balanced=False, cropsize=2800)
File "C:\Users\rpb\Documents\OneDrive - Universiti Teknologi PETRONAS\Code Development\sleep staging\keras_utils.py", line 838, in cv
tools.save_results(save_dict=save_dict)
File "C:\Users\rpb\Documents\OneDrive - Universiti Teknologi PETRONAS\Code Development\sleep staging\tools.py", line 417, in save_results
np.set_printoptions(precision=2,threshold=np.nan)
File "C:\Users\rpb\anaconda3\envs\sleep staging\lib\site-packages\numpy\core\arrayprint.py", line 259, in set_printoptions
floatmode, legacy)
File "C:\Users\rpb\anaconda3\envs\sleep staging\lib\site-packages\numpy\core\arrayprint.py", line 95, in _make_options_dict
raise ValueError("threshold must be non-NAN, try "
ValueError: threshold must be non-NAN, try sys.maxsize for untruncated representation

Process finished with exit code 1

To troubleshot, I had comb the keras_utils.py and tools.py for a line that responsible for creating the directory/ file

\\plots\\0_edfx-sample_ fc1_ fold_ 0_pure_rnn.png

and

'.\weights\0edfx-samplefc1_0_0.390-0.080'

However, I am unable to find anywhere the line of code that responsible for this.

Appreciate if you can shed some light.

About Hypnogram.ground.csv

Hi, Simon. It's Jim again. I'm learning AutoSleepScorerDev, but the convert_hypnogram function in edfx_database.py doesn't work correctly, so I got null hypnogram.groundtruth.csv. Could you please share hypnogram.groundtruth.csv files. Thanks a lot!

Compiler show the UserWarning Update your `predict_generator` call to the Keras 2 API

Hi Simon,
Also, while on training, the compiler repeatedly issue the following warning

C:\Users\rpb\Documents\OneDrive - Universiti Teknologi PETRONAS\Code Development\sleep staging\keras_utils.py:349: UserWarning: Update your predict_generator call to the Keras 2 API: predict_generator(<keras_uti..., 3, max_queue_size=1)
y_pred = np.array(self.model.predict_generator(self.gen, self.gen.n_batches, max_q_size=1))
C:\Users\rpb\anaconda3\envs\sleep staging\lib\site-packages\numpy\core\fromnumeric.py:3335: RuntimeWarning: Mean of empty slice.
out=out, **kwargs)
C:\Users\rpb\anaconda3\envs\sleep staging\lib\site-packages\numpy\core_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)

Despite received this UserWarning for multiple time, the training is still able to continue until the end as indicated by the following report

No improvement after epoch 16.

I am just curious if the RuntimeWarning has serious implication?

Sorry to bugging you, I know this is an old project of yours. But among many example out there, yours is the one I am able to make it run up to the training phase.

What is the difference between target and groups?

Hi Simon,
May I know what is the difference between target and groups in the script run_sample.py of line 35. Quick check the values on these two variables showed no differences between them.

Thanks for the time entertaining this question.

What is the expected output

Hi Simon,
I am just curious what is the expected result or output derived from the line 52 below?

AutoSleepScorerDev/run_sample.py

Line 52 in db26b31

results = keras_utils.cv (data, target, groups, model, rnn=rnn, name='edfx-sample',

Is the figure below is the expected output (ignore the value, as I only train using minimal subject and minimal folds)?

Thanks in advance

Development Environment

Dear Simon, GREAT project you have here.
May I know the environment you have when developing the framework here. Specifically,
The version of the following packages:

1. Phyton x.x.x
2. Tensorflow x.x gpu/cpu?
3. Keras x.x.x

and, if the framework required
CUDA toolkit or CuDNN

Not that important, but helpful you can share what version of Ubuntu xx.xx?

I hope I can troubleshoot more effectively by knowing all these informations.

Just wonder if you are familiar with Docker, maybe you can share an image off the environment when developing this framework.

Best wishes,
Rodney

Run error

Hello :
Thanks for the code. I have a problem running your code recently. If you are free, can you help me answer it? I will be grateful!
Best wishes.
The error is as follows:
`