nextcloud-gmbh / mtime_fixer_tool_kit Goto Github PK
View Code? Open in Web Editor NEWTool kit to fix the mtime issue on the server state
Tool kit to fix the mtime issue on the server state
Hi,
I have run the unsolvable_files.sh script.
The script lists me a list of files, but at this point I don't know how to proceed to fix the files
All these files are on an external drive connected via USB.
Thank you in advance.
For me solvable_files.sh
fails with
ERROR: syntax error at or near "base64"
LINE 4: ...mdzdGVjaG5pay9mdy1kdF9LYXBpdGVsMDhfMDFfMjAucGRm, 'base64'), ...
^
I'm using pgsql and have groupfolders.
The error seems to occur in the groupfolder branch.
There (line 73) the quoting of the SQL statetment seems wrong to me. It is definitively different than the version for normal users.
After changing the quoting to be similar to the case for normal users, I don't get any errors anymore.
Running the official Nextcloud script on Unraid with db access on port != 3306 required adding "--port="xxxx" " in both the mysql connection sections of the script.
--host="$db_host"
--port="xxxx"
--user="$db_user"
--password="$db_pwd" \
starting from here: https://help.nextcloud.com/t/desktop-client-3-4-0-destroys-local-time-stamp-and-keeps-uploading-data-to-server/128512/54?u=wwe
I didn’t find time to build reliable recovery procedure from this information as I’m not an expert in mysql but it should work as following:
additional sugar:
I didn’t investigate further - NC devs might know from the top of the head - i think oc_filecache table is not separated by user so if collect the information from DB you don’t need to iterate over each user to identify and recover the files, it should work in one go…
The documentation is not clear about how to handle Nextcloud instances that use S3 as primary storage.
According to Nextcloud's documentation, in such cases, file metadata is stored entirely in the database. However, the solvable_files.sh script expects a local data directory and seems to only address afflicted files that exist in both the data directory and database.
When using S3 as primary storage, there is no local data directory, so the script does not identify or address any files.
Is there a workaround for this?
Hi,
The task browses a single user folder and then stops. it does'nt browse the __groupfolders folder.
The Commandline to call psql in solvable_files.sh seems to be quite broken.
It for example contains: -E 'UTF-8'
, which looks like a attempt to set the encoding, however man psql states:
-E
--echo-hidden
Echo the actual queries generated by \d and other backslash commands.
You can use this to study psql's internal operations.
This is equivalent to setting the variable ECHO_HIDDEN to on.
Also the way the connect string is put together doesn't seem to allow connecting via socket.
I was able to get the script to work by replacing "postgresql://$db_user:$db_pwd@$db_host/$db_name"
with just the name of the database and completely removing -E 'UTF-8'
.
Running the script on mysql 5.7.36-0ubuntu0.18.04.1
gives warnings w.r.t. passing a password via the command line.
mysql: [Warning] Using a password on the command line interface can be insecure.
This happens for every file found, which clutters the view.
When the storage id is too long to fit in the oc_storages table, it is replaced with a md5 of its value.
So the script needs to also md5() the value when longer than 64 chars: https://github.com/nextcloud-gmbh/mtime_fixer_tool_kit/blob/master/list_problematic_files_on_fs.sh#L24
See https://github.com/nextcloud/server/blob/master/lib/private/Files/Cache/Storage.php#L109
For future readers who encounter errors like: find: unrecognized: -newermt
or environment: line 37: psql: command not found
when running the solvable_files.sh
script, Alpine Linux (common Docker container distro) is missing two packages you'll need to run this script.
apk update
apk add findutils bash postgresql-client sudo
EDIT: If you're using MySQL as your database instead of Postgres, you'll need:
apk update
apk add findutils bash mysql-client sudo
Thanks bavarialogy!
Hopefully this helps someone looking at this in the future!
I had to modify this script to use php74
executable.
when i try to run the file, the "find" command throws an error
find: Can't parse date/time: @86400
Thank you very much for this script!
When running the script for the first time, an error occurred in the mysql
command. I needed to change lines 56 and 86 to the following syntax (compatible with my installation):
nextcloud.mysql-client
> nextcloud.mysql-client \
> --skip-column-names \
> --silent \
> --host="$db_host" \
> --user="$db_user" \
> --password="$db_pwd" \
> --default-character-set=utf8 \
> --execute="\
Adjusted the script, now when executing in each read file, the following warning occurs:
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
mysql: [Warning] Using a password on the command line interface can be insecure.
No mtime in database. File not indexed. Skipping /var/snap/nextcloud/common/nextcloud/data/bruno/files/dir/dir/sample.xlsx
And at the end of the scan no files are fixed.
I've also run the command below to scan all files:
# nextcloud.occ files:scan --all
Did not solve this problem.
Any tips on how I can go about fixing the Incorrect Date issue in files?
Note: my NextCloud installation was created via Snapcraft.
Nextcloud Version: 23.0.3 (Snap)
Database: mysql
Linux Fedora 31
I found more than 60000 files with mtime < 86400 in my instance (SQL count).
solvable_files.sh script is very slow, it has been running for 4 hours in "list mode" and it is not finished, but maybe I can slow it down again :
Why not use "ctime" to correct "mtime" in database?
The last change date is often the same as the last modification date (metadata are rarely changed).
This might avoid losing some data by entering the current date in the ctime.
Am I wrong ?
When listing problematic files, we should exclude storages where the id starts with "shared::xxx".
These are federated shares and the indexed files are located on a remote server, which itself might also have invalid mtimes, that we can't fix automatically.
Instead of reporting these and confusing admins who run it, skips them.
Hi!
The source code published here is different from the published version at
htps://github.com/nextcloud/desktop/wiki/Fix-bug-invalid-modification-date
Is this intended, or should/could this stay aligned?
Thanks,
Raoul
Thank you very much for this script!
When running the script I got the following error:
ERROR 1146 (42S02) at line 1: Table 'nextcloud.oc_storages' doesn't exist
After investigating I found, that the tables in my Nextcloud instance don't have the prefix oc_
. After adjusting the script (removing the prefixes) it worked well.
Nextcloud version: 22 (Docker)
Database: MariaDB
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.