preview directory in appdata_xxxx has grown over 120GB of Total size about server

[st3iny]

The job mentioned above will only clean previews of deleted files. There is no mechanism for deleting previews of old files.

[kesselb]

i suppose the issue began on 2019/2020 within an earlier version of nextcloud

Please elaborate on the issue, what did you observe, what do you think is not right, what is an obsolete preview?

The expected behavior is: Previews are generated for supported file types. If the source file is deleted, the previews are removed as well.

We don't track the last access for files, and therefore cannot know when a preview was accessed the last time.

[st3iny]

No, this will only replace the preview provider not the storage. Imaginary is an external service that offers great performance and a wide variety of supported preview formats.

It will not store your previews. Instead, it will process them on the fly and send them back to Nextcloud to be stored.

[csware]

Previews are still generated on demand. Nothing has changed about that.

Yep, but they should be only generated on demand and not automatically on every update. I've many users who don't use the webinterface and, thus, generating many previews is useless.

Yesterday I disabled preview generation ('enable_previews' => false), but still new previews are generated (even right now) and showing up in the previews folder. Existing previews are still shown on the browser interface (that's ok) and the photo app shows no previews at all (and does not seem to generate new).

I opened a separate issue: #45654

[joshtrichards]

1 - Is there such an automatic mechanism ?

Yes. It runs hourly: https://github.com/nextcloud/server/blob/master/lib/private/Preview/BackgroundCleanupJob.php

Are you getting any errors in your logs like perhaps was reported in #42540?

2 - Is there a manual mechanism ?

There is no single command that does this currently. Though this would be a reasonable enhancement.

[Githopp192]

@joshtrichards - no - as far i see i do not have those or some of those errors mentioned in #42540

i analysed how many of directories i've got in the preview folder for every year:

[root@server ~]# grep ' 2010' /tmp/file_list_of_preview_dir|wc -l
0
[root@server ~]# grep ' 2017' /tmp/file_list_of_preview_dir|wc -l
357
[root@server ~]# grep ' 2018' /tmp/file_list_of_preview_dir|wc -l
925
[root@server ~]# grep ' 2019' /tmp/file_list_of_preview_dir|wc -l
8382
[root@server ~]# grep ' 2020' /tmp/file_list_of_preview_dir|wc -l
4292
[root@server ~]# grep ' 2021' /tmp/file_list_of_preview_dir|wc -l
345
[root@server ~]# grep ' 2022' /tmp/file_list_of_preview_dir|wc -l
224
[root@server ~]# grep ' 2023' /tmp/file_list_of_preview_dir|wc -l
248

so - i suppose the issue began on 2019/2020 within an earlier version of nextcloud

I did count all image files [jpg/jpeg/png/gif/bmp/tiff/svg] for all users :

Totally 313 GB of used space for all images - Nextcloud Preview will use 38% of space of them !
So, it is assumed, that the cleanup operation will not work as expected.

That means a manual (controlled by the cloud admin) mechanism for cleanup preview space is urgently needed.

[Githopp192]

thanks Daniel for those specific questions.

obsolete - preview files, where no co-existing image is existing
observe - Ratio between the size of the preview folder (121GB) and the actual size of all images (313GB)

Yes, that's my view of things too: "If the source file is deleted, the previews are removed as well."

It would be nice to have a way to find out if files exist for existing previews, something like this:

• create a list of image files
• check if the corresponding image file (still) exists for the preview
• if no, delete preview

It would also be good, if the cloud admin could set the maximum size for the preview folder (so that the system automatically deletes the previews, that have not been used for the longest time - similar to what happens with the recycle bin).

Do you need a feature request for this or do you think this is “useless” :-)

[solracsf]

Please also note that the ratio is not 1 preview per image.
There are quite a few preview sizes generated per file (64x64, 356x356, 1024x768, and so on).
Also, not only image are taken into account (MP3, Vidos, PDF, and other too can count).

So, 38% is not a very big number IMO.

See if needed:

server/config/config.sample.php

Lines 1195 to 1347 in 54afea4

	/**
	* Previews
	*
	* Nextcloud supports previews of image files, the covers of MP3 files, and text
	* files. These options control enabling and disabling previews, and thumbnail
	* size.
	*/
	/**
	* By default, Nextcloud can generate previews for the following filetypes:
	*
	* - Image files
	* - Covers of MP3 files
	* - Text documents
	*
	* Valid values are ``true``, to enable previews, or
	* ``false``, to disable previews
	*
	* Defaults to ``true``
	*/
	'enable_previews' => true,
	/**
	* Number of all preview requests being processed concurrently,
	* including previews that need to be newly generated, and those that have
	* been generated.
	*
	* This should be greater than 'preview_concurrency_new'.
	* If unspecified, defaults to twice the value of 'preview_concurrency_new'.
	*/
	'preview_concurrency_all' => 8,
	/**
	* Number of new previews that are being concurrently generated.
	*
	* Depending on the max preview size set by 'preview_max_x' and 'preview_max_y',
	* the generation process can consume considerable CPU and memory resources.
	* It's recommended to limit this to be no greater than the number of CPU cores.
	* If unspecified, defaults to the number of CPU cores, or 4 if that cannot
	* be determined.
	*/
	'preview_concurrency_new' => 4,
	/**
	* The maximum width, in pixels, of a preview. A value of ``null`` means there
	* is no limit.
	*
	* Defaults to ``4096``
	*/
	'preview_max_x' => 4096,
	/**
	* The maximum height, in pixels, of a preview. A value of ``null`` means there
	* is no limit.
	*
	* Defaults to ``4096``
	*/
	'preview_max_y' => 4096,
	/**
	* Max file size for generating image previews with imagegd (default behavior).
	* If the image is bigger, it'll try other preview generators, but will most
	* likely either show the default mimetype icon or not display the image at all.
	* Set to ``-1`` for no limit and try to generate image previews on all file sizes.
	*
	* Defaults to ``50`` megabytes
	*/
	'preview_max_filesize_image' => 50,
	/**
	* max memory for generating image previews with imagegd (default behavior)
	* Reads the image dimensions from the header and assumes 32 bits per pixel.
	* If creating the image would allocate more memory, preview generation will
	* be disabled and the default mimetype icon is shown. Set to -1 for no limit.
	*
	* Defaults to ``256`` megabytes
	*/
	'preview_max_memory' => 256,
	/**
	* custom path for LibreOffice/OpenOffice binary
	*
	*
	* Defaults to ``''`` (empty string)
	*/
	'preview_libreoffice_path' => '/usr/bin/libreoffice',
	/**
	* custom path for ffmpeg binary
	*
	* Defaults to ``null`` and falls back to searching ``avconv`` and ``ffmpeg`` in the configured ``PATH`` environment
	*/
	'preview_ffmpeg_path' => '/usr/bin/ffmpeg',
	/**
	* Set the URL of the Imaginary service to send image previews to.
	* Also requires the ``OC\Preview\Imaginary`` provider to be enabled.
	*
	* See https://github.com/h2non/imaginary
	*/
	'preview_imaginary_url' => 'http://previews_hpb:8088/',
	/**
	* If you want set a api key for imaginary.
	*/
	'preview_imaginary_key' => 'secret',
	/**
	* Only register providers that have been explicitly enabled
	*
	* The following providers are disabled by default due to performance or privacy
	* concerns:
	*
	* - ``OC\Preview\Font``
	* - ``OC\Preview\HEIC``
	* - ``OC\Preview\Illustrator``
	* - ``OC\Preview\Movie``
	* - ``OC\Preview\MSOffice2003``
	* - ``OC\Preview\MSOffice2007``
	* - ``OC\Preview\MSOfficeDoc``
	* - ``OC\Preview\PDF``
	* - ``OC\Preview\Photoshop``
	* - ``OC\Preview\Postscript``
	* - ``OC\Preview\StarOffice``
	* - ``OC\Preview\SVG``
	* - ``OC\Preview\TIFF``
	* - ``OC\Preview\EMF``
	*
	*
	* Defaults to the following providers:
	*
	* - ``OC\Preview\BMP``
	* - ``OC\Preview\GIF``
	* - ``OC\Preview\JPEG``
	* - ``OC\Preview\Krita``
	* - ``OC\Preview\MarkDown``
	* - ``OC\Preview\MP3``
	* - ``OC\Preview\OpenDocument``
	* - ``OC\Preview\PNG``
	* - ``OC\Preview\TXT``
	* - ``OC\Preview\XBitmap``
	*/
	'enabledPreviewProviders' => [
	'OC\Preview\BMP',
	'OC\Preview\GIF',
	'OC\Preview\JPEG',
	'OC\Preview\Krita',
	'OC\Preview\MarkDown',
	'OC\Preview\MP3',
	'OC\Preview\OpenDocument',
	'OC\Preview\PNG',
	'OC\Preview\TXT',
	'OC\Preview\XBitmap',
	],

And https://github.com/nextcloud/previewgenerator (specially the "I don't want to generate all the preview sizes" part).

[Githopp192]

thx @solracsf - yes, i'm aware about the ratio (not 1 preview per image).

May this could come in place to export the cloud preview to external (interim) storage ==>

/** 
  * Set the URL of the Imaginary service to send image previews to. 
  * Also requires the ``OC\Preview\Imaginary`` provider to be enabled. 
  * 
  * See https://github.com/h2non/imaginary 
  */ 
 'preview_imaginary_url' => 'http://previews_hpb:8088/', 
  
 /** 

[Githopp192]

Thx @st3iny / Richard for this explanation :-)

[csware]

My previews folder grew over 250 GiB.

Was there a change recently in Nextcloud? IIRC Nextcloud used to generate previews only on demand (this is what I'd like to keep), but with (27/28)/29 it seems to automatically generate previews as soon as a file is uploaded/changed (using Nextcloud Client not the browser). May this be related to the (new?) FilesMetadata job?

Is there a way to get the old behavior back?

[st3iny]

Previews are still generated on demand. Nothing has changed about that.