nextcloud-gmbh / mtime_fixer_tool_kit Goto Github PK

Hi!

The source code published here is different from the published version at
htps://github.com/nextcloud/desktop/wiki/Fix-bug-invalid-modification-date

Is this intended, or should/could this stay aligned?

Thanks,
Raoul

Hello,

When creating an executable script (solvable_files.sh) it appears to be a syntax errro in line 31 (see screenshot)
Is this known an how can I fix it? The code has been copy pasted from github. (line 31 should be:
function correct_mtime() {

Greetings

The Commandline to call psql in solvable_files.sh seems to be quite broken.
It for example contains: -E 'UTF-8', which looks like a attempt to set the encoding, however man psql states:

       -E
       --echo-hidden
           Echo the actual queries generated by \d and other backslash commands.
           You can use this to study psql's internal operations.
           This is equivalent to setting the variable ECHO_HIDDEN to on.

Also the way the connect string is put together doesn't seem to allow connecting via socket.

I was able to get the script to work by replacing "postgresql://$db_user:$db_pwd@$db_host/$db_name" with just the name of the database and completely removing -E 'UTF-8'.

When the storage id is too long to fit in the oc_storages table, it is replaced with a md5 of its value.

So the script needs to also md5() the value when longer than 64 chars: https://github.com/nextcloud-gmbh/mtime_fixer_tool_kit/blob/master/list_problematic_files_on_fs.sh#L24

See https://github.com/nextcloud/server/blob/master/lib/private/Files/Cache/Storage.php#L109

@artonge

For me solvable_files.sh fails with

ERROR:  syntax error at or near "base64"
LINE 4: ...mdzdGVjaG5pay9mdy1kdF9LYXBpdGVsMDhfMDFfMjAucGRm, 'base64'), ...
                                                             ^

I'm using pgsql and have groupfolders.

The error seems to occur in the groupfolder branch.

There (line 73) the quoting of the SQL statetment seems wrong to me. It is definitively different than the version for normal users.

After changing the quoting to be similar to the case for normal users, I don't get any errors anymore.

Thank you very much for this script!

When running the script for the first time, an error occurred in the mysql command. I needed to change lines 56 and 86 to the following syntax (compatible with my installation):

nextcloud.mysql-client

> nextcloud.mysql-client \
>        --skip-column-names \
>        --silent \
>        --host="$db_host" \
>        --user="$db_user" \
>        --password="$db_pwd" \
>        --default-character-set=utf8 \
>        --execute="\

Adjusted the script, now when executing in each read file, the following warning occurs:

WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
mysql: [Warning] Using a password on the command line interface can be insecure.
No mtime in database. File not indexed. Skipping /var/snap/nextcloud/common/nextcloud/data/bruno/files/dir/dir/sample.xlsx

And at the end of the scan no files are fixed.

I've also run the command below to scan all files:
# nextcloud.occ files:scan --all
Did not solve this problem.

Any tips on how I can go about fixing the Incorrect Date issue in files?

Note: my NextCloud installation was created via Snapcraft.

Nextcloud Version: 23.0.3 (Snap)
Database: mysql
Linux Fedora 31

Hi,
I have run the unsolvable_files.sh script.
The script lists me a list of files, but at this point I don't know how to proceed to fix the files

All these files are on an external drive connected via USB.

Thank you in advance.

starting from here: https://help.nextcloud.com/t/desktop-client-3-4-0-destroys-local-time-stamp-and-keeps-uploading-data-to-server/128512/54?u=wwe

I didn’t find time to build reliable recovery procedure from this information as I’m not an expert in mysql but it should work as following:

• find files with invalid mtime
• find the most recent version (if many) of the file which has valid mtime
• perform one of the possible recovery options:
  1. safe: touch the file with valid mtime (and remove version in 2nd step)
  2. storage efficient: move version file to the original location and overwrite bad file
• perform occ files:scan all

additional sugar:

• compare the version file you use for recovery with the file in the system (diff/checksum) - AFAIK the contents are unchanged, so file contents and versions should be same
• remove newer versions having invalid mtime (there are github reports when upload with invalid mtime happened multiple times)

I didn’t investigate further - NC devs might know from the top of the head - i think oc_filecache table is not separated by user so if collect the information from DB you don’t need to iterate over each user to identify and recover the files, it should work in one go…

Running the official Nextcloud script on Unraid with db access on port != 3306 required adding "--port="xxxx" " in both the mysql connection sections of the script.

--host="$db_host"
--port="xxxx"
--user="$db_user"
--password="$db_pwd" \

Thank you very much for this script!
When running the script I got the following error:

ERROR 1146 (42S02) at line 1: Table 'nextcloud.oc_storages' doesn't exist

After investigating I found, that the tables in my Nextcloud instance don't have the prefix oc_. After adjusting the script (removing the prefixes) it worked well.

Nextcloud version: 22 (Docker)
Database: MariaDB

I had to modify this script to use php74 executable.

I found more than 60000 files with mtime < 86400 in my instance (SQL count).

solvable_files.sh script is very slow, it has been running for 4 hours in "list mode" and it is not finished, but maybe I can slow it down again :

Why not use "ctime" to correct "mtime" in database?
The last change date is often the same as the last modification date (metadata are rarely changed).

This might avoid losing some data by entering the current date in the ctime.
Am I wrong ?

when i try to run the file, the "find" command throws an error

find: Can't parse date/time: @86400

For future readers who encounter errors like: find: unrecognized: -newermt or environment: line 37: psql: command not found when running the solvable_files.sh script, Alpine Linux (common Docker container distro) is missing two packages you'll need to run this script.

apk update
apk add findutils bash postgresql-client sudo

EDIT: If you're using MySQL as your database instead of Postgres, you'll need:

apk update
apk add findutils bash mysql-client sudo 

Thanks bavarialogy!

Hopefully this helps someone looking at this in the future!

The documentation is not clear about how to handle Nextcloud instances that use S3 as primary storage.

According to Nextcloud's documentation, in such cases, file metadata is stored entirely in the database. However, the solvable_files.sh script expects a local data directory and seems to only address afflicted files that exist in both the data directory and database.

When using S3 as primary storage, there is no local data directory, so the script does not identify or address any files.

Is there a workaround for this?

Hi,

The task browses a single user folder and then stops. it does'nt browse the __groupfolders folder.

Running the script on mysql 5.7.36-0ubuntu0.18.04.1
gives warnings w.r.t. passing a password via the command line.

mysql: [Warning] Using a password on the command line interface can be insecure.

This happens for every file found, which clutters the view.

When listing problematic files, we should exclude storages where the id starts with "shared::xxx".
These are federated shares and the indexed files are located on a remote server, which itself might also have invalid mtimes, that we can't fix automatically.

Instead of reporting these and confusing admins who run it, skips them.

@artonge