FFTW Wisdom about bart

[uecker]

Good idea, but please simplify: - by default, this should be off. There are too many scenarios where BART is not supposed to write such files. - please use only one environment variable. BART_FFTW_WISDOM - If this is set it is the file name, if not we do nothing. - Please read the environment variable only once in init.c and store the result it a global variable (which can be read in fft.c) Then please open a pull request or send a patch by email. Thank you! Best, Martin Am Donnerstag, den 06.09.2018, 04:30 +0000 schrieb smyeungx:

…

Hi Team, I have a proposal in adding fftw wisdom feature to BART. Per combination with suggestion #151, it can further reduce the pipeline processing time for l1-wavelet gpu recon, with the wisdom feature of fftw library: http://www.fftw.org/fftw3_doc/Words-of-Wisdom_002dSavin g-Plans.html#Words-of-Wisdom_002dSaving-Plans. It's serves as a cache to store previously optimized plan In order to do this, several minor modification is implemented and trial run on bart-0.4.03. A) In file ./src/num/fft.h, the following line is added after #define __FFT_H: #define FFTW_DEFAULT_WISDOM_CACHE_FILE "./fftw_wisdom_cache" Explanation: it's a default location of the FFTW wisdom file B) In the file ./src/num/init.c, At the end of the function void num_init(void), the following code is added: // Import previously measured fftw plans from environmental path, if any, or default cache path char* env_wisdom_file = getenv("FFTW_WISDOM_FILE"); char *wisdom_file = (NULL != env_wisdom_file)? env_wisdom_file: FFTW_DEFAULT_WISDOM_CACHE_FILE; char* env_disable_wisdom_cache = getenv("FFTW_DISABLE_WISDOM_CACHE"); int fftw_disable_wisdom_cache = (NULL != env_disable_wisdom_cache)? atoi(env_disable_wisdom_cache): 0; if (0 == fftw_disable_wisdom_cache) { if (0 == fftwf_import_wisdom_from_filename(wisdom_file)) { debug_printf(DP_DEBUG1, "fftw wisdcom file import error\n"); } } Explanation: the fftw wisdom file will be determined by the environmental variable FFTW_WISDOM_FILE. It this environmental variable is not available, it will use the file specified in (A). For use cases unfavorable to use a wisdom file. The user will have an option to turn it off explicitly by setting the FFTW_DISABLE_WISDOM_CACHE environmental variable to 1. Afterwards, the fftwf_import_wisdom_from_filename will import the wisdom file. Note: If the file is not available on file system or any error encountered during the wisdom file reading or import process, the fftwf_import_wisdom_from_filename will return a value of 0 (zero). (C) In the file ./src/num/fft.c, after "plan->fftw = fft_fftwf_plan(D, dimensions, flags, strides, dst, strides, src, backwards, true);", the following code is inserted: // Export previously measured fftw plans from environmental path, if any, or default cache path char* env_wisdom_file = getenv("FFTW_WISDOM_FILE"); char *wisdom_file = (NULL != env_wisdom_file)? env_wisdom_file: FFTW_DEFAULT_WISDOM_CACHE_FILE; char* env_disable_wisdom_cache = getenv("FFTW_DISABLE_WISDOM_CACHE"); int fftw_disable_wisdom_cache = (NULL != env_disable_wisdom_cache)? atoi(env_disable_wisdom_cache): 0; if (0 == fftw_disable_wisdom_cache) { if (0 == fftwf_export_wisdom_to_filename(wisdom_file)) { debug_printf(DP_DEBUG1, "fftw wisdom file export error\n"); } } Explanation: similar to import wisdom with the same options, the wisdom file is write back to the same wisdom file on file system. Analysis: Hardware platform: Intel 8700k cpu with 64GB ram, NVidia 1080 Ti Bart command: bart pics -l1 -g -S -r 0.1 ./ksp_cc ./ecalib_map ./image Benchmark tool: NVidia nvvp Data: 256 x 256 x 90 x 7 (compressed from 16 ch) Acceleration: 3.74x with poisson sampling Without fftw wisdom file or 1st time running bart pics without fftw wisdom file Whole process takes ~8.5s. The CPU thread takes around ~5.5s consumed by fftw create plan Enabling fftw wisdom and run bart for the 2nd time Without re-create known fftw plan during previous processes, CPU time reduced from ~5.5s to ~3.1s (43% increase in CPU cycles in this particular case). enable fftw wisdom and executing bart directly on command prompt, without nvvp overhead the whole process actually takes only ~4.69s. Hope this suggestion useful for the BART community. Cheers, Simon — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[sidward]

What if the cache file is saved in the bart/save folder if a Makefile flag is set?

[smyeungx]

Thanks for the recommendation.

________________________________ From: Martin Uecker <[email protected]> Sent: Thursday, September 6, 2018 7:03 AM To: mrirecon/bart Cc: smyeungx; Author Subject: Re: [mrirecon/bart] FFTW Wisdom (#152) Good idea, but please simplify: - by default, this should be off. There are too many scenarios where BART is not supposed to write such files. - please use only one environment variable. BART_FFTW_WISDOM - If this is set it is the file name, if not we do nothing. - Please read the environment variable only once in init.c and store the result it a global variable (which can be read in fft.c) Then please open a pull request or send a patch by email. Thank you! Best, Martin Am Donnerstag, den 06.09.2018, 04:30 +0000 schrieb smyeungx:

Hi Team, I have a proposal in adding fftw wisdom feature to BART. Per combination with suggestion #151, it can further reduce the pipeline processing time for l1-wavelet gpu recon, with the wisdom feature of fftw library: http://www.fftw.org/fftw3_doc/Words-of-Wisdom_002dSavin g-Plans.html#Words-of-Wisdom_002dSaving-Plans. It's serves as a cache to store previously optimized plan In order to do this, several minor modification is implemented and trial run on bart-0.4.03. A) In file ./src/num/fft.h, the following line is added after #define __FFT_H: #define FFTW_DEFAULT_WISDOM_CACHE_FILE "./fftw_wisdom_cache" Explanation: it's a default location of the FFTW wisdom file B) In the file ./src/num/init.c, At the end of the function void num_init(void), the following code is added: // Import previously measured fftw plans from environmental path, if any, or default cache path char* env_wisdom_file = getenv("FFTW_WISDOM_FILE"); char *wisdom_file = (NULL != env_wisdom_file)? env_wisdom_file: FFTW_DEFAULT_WISDOM_CACHE_FILE; char* env_disable_wisdom_cache = getenv("FFTW_DISABLE_WISDOM_CACHE"); int fftw_disable_wisdom_cache = (NULL != env_disable_wisdom_cache)? atoi(env_disable_wisdom_cache): 0; if (0 == fftw_disable_wisdom_cache) { if (0 == fftwf_import_wisdom_from_filename(wisdom_file)) { debug_printf(DP_DEBUG1, "fftw wisdcom file import error\n"); } } Explanation: the fftw wisdom file will be determined by the environmental variable FFTW_WISDOM_FILE. It this environmental variable is not available, it will use the file specified in (A). For use cases unfavorable to use a wisdom file. The user will have an option to turn it off explicitly by setting the FFTW_DISABLE_WISDOM_CACHE environmental variable to 1. Afterwards, the fftwf_import_wisdom_from_filename will import the wisdom file. Note: If the file is not available on file system or any error encountered during the wisdom file reading or import process, the fftwf_import_wisdom_from_filename will return a value of 0 (zero). (C) In the file ./src/num/fft.c, after "plan->fftw = fft_fftwf_plan(D, dimensions, flags, strides, dst, strides, src, backwards, true);", the following code is inserted: // Export previously measured fftw plans from environmental path, if any, or default cache path char* env_wisdom_file = getenv("FFTW_WISDOM_FILE"); char *wisdom_file = (NULL != env_wisdom_file)? env_wisdom_file: FFTW_DEFAULT_WISDOM_CACHE_FILE; char* env_disable_wisdom_cache = getenv("FFTW_DISABLE_WISDOM_CACHE"); int fftw_disable_wisdom_cache = (NULL != env_disable_wisdom_cache)? atoi(env_disable_wisdom_cache): 0; if (0 == fftw_disable_wisdom_cache) { if (0 == fftwf_export_wisdom_to_filename(wisdom_file)) { debug_printf(DP_DEBUG1, "fftw wisdom file export error\n"); } } Explanation: similar to import wisdom with the same options, the wisdom file is write back to the same wisdom file on file system. Analysis: Hardware platform: Intel 8700k cpu with 64GB ram, NVidia 1080 Ti Bart command: bart pics -l1 -g -S -r 0.1 ./ksp_cc ./ecalib_map ./image Benchmark tool: NVidia nvvp Data: 256 x 256 x 90 x 7 (compressed from 16 ch) Acceleration: 3.74x with poisson sampling Without fftw wisdom file or 1st time running bart pics without fftw wisdom file Whole process takes ~8.5s. The CPU thread takes around ~5.5s consumed by fftw create plan Enabling fftw wisdom and run bart for the 2nd time Without re-create known fftw plan during previous processes, CPU time reduced from ~5.5s to ~3.1s (43% increase in CPU cycles in this particular case). enable fftw wisdom and executing bart directly on command prompt, without nvvp overhead the whole process actually takes only ~4.69s. Hope this suggestion useful for the BART community. Cheers, Simon — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#152 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AoLSonKde7BJz1haTUhZMUDHtu3wzuAMks5uYMiygaJpZM4WcNY_>.

[smyeungx]

Hi Martin, Attached is the corresponding implementation. Simon PS: Sorry, cannot create a Pull Request, so I send it as email attachment.

________________________________ From: SM Y <[email protected]> Sent: Friday, September 7, 2018 2:03 PM To: mrirecon/bart; mrirecon/bart Cc: Author Subject: Re: [mrirecon/bart] FFTW Wisdom (#152) Thanks for the recommendation.

________________________________ From: Martin Uecker <[email protected]> Sent: Thursday, September 6, 2018 7:03 AM To: mrirecon/bart Cc: smyeungx; Author Subject: Re: [mrirecon/bart] FFTW Wisdom (#152) Good idea, but please simplify: - by default, this should be off. There are too many scenarios where BART is not supposed to write such files. - please use only one environment variable. BART_FFTW_WISDOM - If this is set it is the file name, if not we do nothing. - Please read the environment variable only once in init.c and store the result it a global variable (which can be read in fft.c) Then please open a pull request or send a patch by email. Thank you! Best, Martin Am Donnerstag, den 06.09.2018, 04:30 +0000 schrieb smyeungx:

Hi Team, I have a proposal in adding fftw wisdom feature to BART. Per combination with suggestion #151, it can further reduce the pipeline processing time for l1-wavelet gpu recon, with the wisdom feature of fftw library: http://www.fftw.org/fftw3_doc/Words-of-Wisdom_002dSavin g-Plans.html#Words-of-Wisdom_002dSaving-Plans. It's serves as a cache to store previously optimized plan In order to do this, several minor modification is implemented and trial run on bart-0.4.03. A) In file ./src/num/fft.h, the following line is added after #define __FFT_H: #define FFTW_DEFAULT_WISDOM_CACHE_FILE "./fftw_wisdom_cache" Explanation: it's a default location of the FFTW wisdom file B) In the file ./src/num/init.c, At the end of the function void num_init(void), the following code is added: // Import previously measured fftw plans from environmental path, if any, or default cache path char* env_wisdom_file = getenv("FFTW_WISDOM_FILE"); char *wisdom_file = (NULL != env_wisdom_file)? env_wisdom_file: FFTW_DEFAULT_WISDOM_CACHE_FILE; char* env_disable_wisdom_cache = getenv("FFTW_DISABLE_WISDOM_CACHE"); int fftw_disable_wisdom_cache = (NULL != env_disable_wisdom_cache)? atoi(env_disable_wisdom_cache): 0; if (0 == fftw_disable_wisdom_cache) { if (0 == fftwf_import_wisdom_from_filename(wisdom_file)) { debug_printf(DP_DEBUG1, "fftw wisdcom file import error\n"); } } Explanation: the fftw wisdom file will be determined by the environmental variable FFTW_WISDOM_FILE. It this environmental variable is not available, it will use the file specified in (A). For use cases unfavorable to use a wisdom file. The user will have an option to turn it off explicitly by setting the FFTW_DISABLE_WISDOM_CACHE environmental variable to 1. Afterwards, the fftwf_import_wisdom_from_filename will import the wisdom file. Note: If the file is not available on file system or any error encountered during the wisdom file reading or import process, the fftwf_import_wisdom_from_filename will return a value of 0 (zero). (C) In the file ./src/num/fft.c, after "plan->fftw = fft_fftwf_plan(D, dimensions, flags, strides, dst, strides, src, backwards, true);", the following code is inserted: // Export previously measured fftw plans from environmental path, if any, or default cache path char* env_wisdom_file = getenv("FFTW_WISDOM_FILE"); char *wisdom_file = (NULL != env_wisdom_file)? env_wisdom_file: FFTW_DEFAULT_WISDOM_CACHE_FILE; char* env_disable_wisdom_cache = getenv("FFTW_DISABLE_WISDOM_CACHE"); int fftw_disable_wisdom_cache = (NULL != env_disable_wisdom_cache)? atoi(env_disable_wisdom_cache): 0; if (0 == fftw_disable_wisdom_cache) { if (0 == fftwf_export_wisdom_to_filename(wisdom_file)) { debug_printf(DP_DEBUG1, "fftw wisdom file export error\n"); } } Explanation: similar to import wisdom with the same options, the wisdom file is write back to the same wisdom file on file system. Analysis: Hardware platform: Intel 8700k cpu with 64GB ram, NVidia 1080 Ti Bart command: bart pics -l1 -g -S -r 0.1 ./ksp_cc ./ecalib_map ./image Benchmark tool: NVidia nvvp Data: 256 x 256 x 90 x 7 (compressed from 16 ch) Acceleration: 3.74x with poisson sampling Without fftw wisdom file or 1st time running bart pics without fftw wisdom file Whole process takes ~8.5s. The CPU thread takes around ~5.5s consumed by fftw create plan Enabling fftw wisdom and run bart for the 2nd time Without re-create known fftw plan during previous processes, CPU time reduced from ~5.5s to ~3.1s (43% increase in CPU cycles in this particular case). enable fftw wisdom and executing bart directly on command prompt, without nvvp overhead the whole process actually takes only ~4.69s. Hope this suggestion useful for the BART community. Cheers, Simon — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#152 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AoLSonKde7BJz1haTUhZMUDHtu3wzuAMks5uYMiygaJpZM4WcNY_>.

[uecker]

Hm. I forgot about this and just merged a pull request from @sidward.
Can you take a look whether this solves your problem?