Bug: Change date referenced instead of Modify date about onedrive

[abraunegg]

@robertschulze
This is not a bug - sorry.

The application does not use the ctime at all - I only was using stat to show and highlight the file timestamps when debugging your issue.

The D function setTimes() as per https://dlang.org/phobos/std_file.html#setTimes details that this function only sets:

• accessTime
• modificationTime

When 'onedrive' runs, and there is a time discrepancy locally, the local file will be updated to set the accessTime and modificationTime as per what is provided in the JSON from online, and for your test file this is:

When you look at this line:

The local item has a different modified time 2024-Feb-04 00:00:14Z when compared to database modified time 2023-Dec-14 23:25:28Z

These times are in UTC and this matches the JSON data.

This also matches your stat data once the local file is updated via setTimes():

robert@atom:/media/3/3 - OneDrive/[email protected]/OneDrive - Constanze/Datenbanken$ stat TestWordfile.docx
  File: TestWordfile.docx
  Size: 10854           Blocks: 24         IO Block: 1048576 regular file
Device: 35h/53d Inode: 281474977227143  Links: 1
Access: (0777/-rwxrwxrwx)  Uid: ( 1000/  robert)   Gid: (  100/   users)
Access: 2023-12-15 00:25:28.000000000 +0100
Modify: 2023-12-15 00:25:28.000000000 +0100   <---------- THIS IS THE IMPORTANT VALUE
Change: 2024-02-04 01:00:14.207692700 +0100
 Birth: 2024-02-03 23:14:52.457050400 +0100

Change & Birth as per stat are ignored and not used by 'onedrive'.

One enhancement is however, is that this line:

The local item has a different modified time ........

should also report that these are UTC times being used for comparison.

[robertschulze]

@abraunegg
Hi Alex, thanks for clarifying. What I am wondering is, why in my case the Change date appears to be used (at least the time mentioned in the log message modified time 2024-Feb-03 22:53:24Z (UTC) is in my case the Change time according to stat: Change: 2024-02-03 23:53:24.353760400 +0100 instead of the Modify time Modify: 2023-12-15 00:25:28.000000000 +0100). Is there any way I can further check on my side?

robert@atom:/media/3/3 - OneDrive/[email protected]/OneDrive - Constanze/Datenbanken$ stat TestWordfile.docx
  File: TestWordfile.docx
  Size: 10854           Blocks: 24         IO Block: 1048576 regular file
Device: 35h/53d Inode: 281474977227143  Links: 1
Access: (0777/-rwxrwxrwx)  Uid: ( 1000/  robert)   Gid: (  100/   users)
Access: 2023-12-15 00:25:28.000000000 +0100
Modify: 2023-12-15 00:25:28.000000000 +0100
Change: 2024-02-03 23:53:24.353760400 +0100
 Birth: 2024-02-03 23:14:52.457050400 +0100

[abraunegg]

@robertschulze
The only theory I have here is:

• You are using a SMB mount from a Windows Server
• The cifs-utils and/or Windows is providing the incorrect date based on the request being asked.
• Timestamp Differences: Linux and Windows have different definitions and uses for file timestamps:

Modify (mtime): The time the file content was last modified.
Change (ctime): The time metadata (e.g., permissions, ownership) of the file was last changed in Linux. In Windows, this concept doesn't exactly match, as Windows uses a single timestamp to track both content and metadata changes.

This is why the application tracks the 'mtime' and uses 'mtime' as the comparison.

Suggestion:

• Move all your data locally - ie - temporarily dont use the SMB mount, use your Linux server and validate what is occurring. If the issue is still the same - then potentially this might be a Ubuntu problem (would not surprise me if that is were the issue lies). Use a different Linux distribution potentially. If the issue is resolved, you know your issue is caused by your environment (the SMB mount).

Point here is - I cannot replicate the issue you are now presenting (when storing data locally, and I dont have the time to test SMB/NFS or anything else at the moment), and the client only looks at the 'Modify' date attribute for the files.

That your OS and/or SMB mount or something else is providing the 'Change' attribute to the application, rather than the 'Modify' as requested .. is not something that I can really assist with and is potentially caused by how Windows uses a single timestamp to track changes.

[robertschulze]

@abraunegg
I have to admit I am right now confused. In order to properly test what you are proposing, I drafted up the following D code

#!/usr/bin/env rdmd
import std.stdio;
import std.file;
import std.datetime : SysTime;

void main(string[] args)
{
    if (args.length != 2)
    {
        writeln("Usage: ./script <file_path>");
        return;
    }

    string filePath = args[1];

    if (!exists(filePath))
    {
        writeln("Error: The specified file does not exist.");
        return;
    }

    SysTime modificationTime = filePath.timeLastModified;

    writeln("File: ", filePath);
    writeln("Last Modified Time: ", modificationTime);
}

but executing on the given file it outputs

(dmd-2.106.0)robert@atom:~/onedrive$ ./fileTime.d /media/3/3\ -\ OneDrive/[email protected]/OneDrive\ -\ Constanze/Datenbanken/TestWordfile.docx                    File: /media/3/3 - OneDrive/[email protected]/OneDrive - Constanze/Datenbanken/TestWordfile.docx
Last Modified Time: 2023-Dec-15 00:25:28

while the log output from onedrive's code

SysTime localModifiedTime = timeLastModified(path).toUTC();
...
addLogEntry("This local file has a different modified time " ~ to!string(localModifiedTime) ~ " (UTC) when compared to " ~ itemSource ~ " modified time " ~ to!string(itemModifiedTime) ~ " (UTC)", ["verbose"]);

is

DEBUG: This local file has a different modified time 2024-Feb-03 22:53:24Z (UTC)

I am not an expert at D so not sure how I can proceed to further deep dive the issue?

[abraunegg]

@robertschulze
Potentially change your code to:

SysTime modificationTime = filePath.timeLastModified.toUTC();

That then puts your modificationTime into UTC and drops any fraction seconds as per the 'onedrive' code.

The only difference post that is this:

SysTime localModifiedTime = timeLastModified(path).toUTC();

versus

SysTime modificationTime = filePath.timeLastModified.toUTC();

To test that this difference is not causing the problem - try this:

#!/usr/bin/env rdmd
import std.stdio;
import std.file;
import std.datetime : SysTime;

void main(string[] args)
{
    if (args.length != 2)
    {
        writeln("Usage: ./script <file_path>");
        return;
    }

    string filePath = args[1];

    if (!exists(filePath))
    {
        writeln("Error: The specified file does not exist.");
        return;
    }

    SysTime modificationTimeMethodA = timeLastModified(filePath);
    SysTime modificationTimeMethodB = filePath.timeLastModified;

    writeln("File: ", filePath);
    writeln("Last Modified Time (Method A) (LOCAL): ", modificationTimeMethodA);
    writeln("Last Modified Time (Method A) (UTC):   ", modificationTimeMethodA.toUTC());
    writeln("Last Modified Time (Method B) (LOCAL): ", modificationTimeMethodB);
    writeln("Last Modified Time (Method B) (UTC):   ", modificationTimeMethodB.toUTC());
}

If this difference is what is causing the issue .. then that is going to be a more difficult issue to solve:

• Compiler bug ?
• 'onedrive' bug and change how last modification time is tracked and consumed ... will be lots of regression testing if that changes as potentially there could be a whole raft of inconsistencies now introduced.

[abraunegg]

@robertschulze
Dropped the fraction second part ..

OK .. testing here using the updated code above:

[alex@onedrive-client-dev file-tests]$ ./fileTime.d ~/lmde-6-cinnamon-64bit.iso 
File: /home/alex/lmde-6-cinnamon-64bit.iso
Last Modified Time (Method A) (LOCAL): 2023-Nov-08 05:37:35
Last Modified Time (Method A) (UTC):   2023-Nov-07 18:37:35Z
Last Modified Time (Method B) (LOCAL): 2023-Nov-08 05:37:35
Last Modified Time (Method B) (UTC):   2023-Nov-07 18:37:35Z
[alex@onedrive-client-dev file-tests]$ stat ~/lmde-6-cinnamon-64bit.iso 
  File: /home/alex/lmde-6-cinnamon-64bit.iso
  Size: 2690580480      Blocks: 5255040    IO Block: 4096   regular file
Device: 811h/2065d      Inode: 67114860    Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/    alex)   Gid: ( 1000/    alex)
Context: unconfined_u:object_r:user_home_t:s0
Access: 2023-11-11 15:32:17.131247745 +1100
Modify: 2023-11-08 05:37:35.000000000 +1100
Change: 2023-11-11 15:33:00.598185174 +1100
 Birth: 2023-11-08 16:27:39.928394524 +1100

The correct modify time is being used.

[abraunegg]

@robertschulze
Mounting a SMB NAS Share (admittedly from another Linux Server, not a Windows box) yields the following results:

[alex@onedrive-client-dev file-tests]$ ./fileTime.d /mnt/samba/newfile.txt        
File: /mnt/samba/newfile.txt
Last Modified Time (Method A) (LOCAL): 2024-Feb-06 05:32:56.0563312
Last Modified Time (Method A) (UTC):   2024-Feb-05 18:32:56.0563312Z
Last Modified Time (Method B) (LOCAL): 2024-Feb-06 05:32:56.0563312
Last Modified Time (Method B) (UTC):   2024-Feb-05 18:32:56.0563312Z
[alex@onedrive-client-dev file-tests]$ stat /mnt/samba/newfile.txt                
  File: /mnt/samba/newfile.txt
  Size: 173             Blocks: 2048       IO Block: 1048576 regular file
Device: 2fh/47d Inode: 11          Links: 1
Access: (0775/-rwxrwxr-x)  Uid: (    0/    root)   Gid: (   10/   wheel)
Context: system_u:object_r:cifs_t:s0
Access: 2024-02-06 05:32:56.056331200 +1100
Modify: 2024-02-06 05:32:56.056331200 +1100
Change: 2024-02-06 05:32:56.056331200 +1100
 Birth: 2024-02-06 05:32:55.951209800 +1100
[alex@onedrive-client-dev file-tests]$ ls -la /mnt/samba/                  
total 2048
drwxrwxr-x. 2 root wheel   0 Feb  6 05:32 .
drwxr-xr-x. 4 root root   30 Jan 19 07:22 ..
-rwxrwxr-x. 1 root wheel  41 Jan 18 08:28 changedfile.txt
-rwxrwxr-x. 1 root wheel 173 Feb  6 05:32 newfile.txt
drwxrwxr-x. 2 root wheel   0 Jan 17 09:44 Source_Folder
drwxrwxr-x. 2 root wheel   0 Jan 19 08:25 Target_Folder

Mount:

//redacted/testshare on /mnt/samba type cifs (rw,relatime,vers=3.1.1,cache=strict,username=root,uid=0,noforceuid,gid=10,noforcegid,addr=redacted,file_mode=0775,dir_mode=0775,iocharset=utf8,soft,nounix,serverino,mapposix,noperm,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)

Edit:
I edited 'newfile.txt' and 'Access' , 'Modify' and 'Change' is being updated at the same time on file write closure using vi

[abraunegg]

@robertschulze
So using an older Windows Server, yields the following:

[alex@onedrive-client-dev file-tests]$ ls -la /mnt/samba/
total 6940
drwxrwxr-x. 2 root wheel       0 Feb  5 18:38 .
drwxr-xr-x. 4 root root       30 Jan 19 07:22 ..
-rwxrwxr-x. 1 root wheel 7106560 Dec  9  2022 FiddlerClassic.msi
[alex@onedrive-client-dev file-tests]$ ./fileTime.d /mnt/samba/FiddlerClassic.msi 
File: /mnt/samba/FiddlerClassic.msi
Last Modified Time (Method A) (LOCAL): 2022-Dec-09 13:13:30.3500984
Last Modified Time (Method A) (UTC):   2022-Dec-09 02:13:30.3500984Z
Last Modified Time (Method B) (LOCAL): 2022-Dec-09 13:13:30.3500984
Last Modified Time (Method B) (UTC):   2022-Dec-09 02:13:30.3500984Z
[alex@onedrive-client-dev file-tests]$ stat /mnt/samba/FiddlerClassic.msi
  File: /mnt/samba/FiddlerClassic.msi
  Size: 7106560         Blocks: 13880      IO Block: 1048576 regular file
Device: 2fh/47d Inode: 32932572275225537  Links: 1
Access: (0775/-rwxrwxr-x)  Uid: (    0/    root)   Gid: (   10/   wheel)
Context: system_u:object_r:cifs_t:s0
Access: 2024-02-05 18:38:54.484469400 +1100
Modify: 2022-12-09 13:13:30.350098400 +1100
Change: 2024-02-05 18:38:54.515471200 +1100
 Birth: 2024-02-05 18:38:54.484469400 +1100
[alex@onedrive-client-dev file-tests]$ 

As per the Windows system:

Edit:
Adding files to the SMB Mounted Windows share yields the same details as the Linux SMB Mount:

[alex@onedrive-client-dev file-tests]$ ls -la /mnt/samba/                  
total 2048
drwxrwxr-x. 2 root wheel   0 Feb  6 05:32 .
drwxr-xr-x. 4 root root   30 Jan 19 07:22 ..
-rwxrwxr-x. 1 root wheel  41 Jan 18 08:28 changedfile.txt
-rwxrwxr-x. 1 root wheel 173 Feb  6 05:32 newfile.txt
drwxrwxr-x. 2 root wheel   0 Jan 17 09:44 Source_Folder
drwxrwxr-x. 2 root wheel   0 Jan 19 08:25 Target_Folder
[alex@onedrive-client-dev file-tests]$ ls -la /mnt/samba/
total 6940
drwxrwxr-x. 2 root wheel       0 Feb  6 05:51 .
drwxr-xr-x. 4 root root       30 Jan 19 07:22 ..
-rwxrwxr-x. 1 root wheel 7106560 Dec  9  2022 FiddlerClassic.msi
[alex@onedrive-client-dev file-tests]$ echo "asdfasdfasdfasdf" > /mnt/samba/newfile.txt
[alex@onedrive-client-dev file-tests]$ ls -la /mnt/samba/                              
total 6941
drwxrwxr-x. 2 root wheel       0 Feb  6 05:52 .
drwxr-xr-x. 4 root root       30 Jan 19 07:22 ..
-rwxrwxr-x. 1 root wheel 7106560 Dec  9  2022 FiddlerClassic.msi
-rwxrwxr-x. 1 root wheel      17 Feb  6 05:52 newfile.txt
[alex@onedrive-client-dev file-tests]$ stat /mnt/samba/newfile.txt                     
  File: /mnt/samba/newfile.txt
  Size: 17              Blocks: 1          IO Block: 1048576 regular file
Device: 2fh/47d Inode: 4785074604163999  Links: 1
Access: (0775/-rwxrwxr-x)  Uid: (    0/    root)   Gid: (   10/   wheel)
Context: system_u:object_r:cifs_t:s0
Access: 2024-02-06 05:52:41.104439200 +1100
Modify: 2024-02-06 05:52:41.104439200 +1100
Change: 2024-02-06 05:52:41.104439200 +1100
 Birth: 2024-02-06 05:52:41.103439100 +1100
[alex@onedrive-client-dev file-tests]$ echo "asdfasdfasdfasdasdfasdfasdff" >> /mnt/samba/newfile.txt
[alex@onedrive-client-dev file-tests]$ stat /mnt/samba/newfile.txt                                  
  File: /mnt/samba/newfile.txt
  Size: 46              Blocks: 1          IO Block: 1048576 regular file
Device: 2fh/47d Inode: 4785074604163999  Links: 1
Access: (0775/-rwxrwxr-x)  Uid: (    0/    root)   Gid: (   10/   wheel)
Context: system_u:object_r:cifs_t:s0
Access: 2024-02-06 05:53:18.190560400 +1100
Modify: 2024-02-06 05:53:18.190560400 +1100
Change: 2024-02-06 05:53:18.190560400 +1100
 Birth: 2024-02-06 05:52:41.103439100 +1100
[alex@onedrive-client-dev file-tests]$ ./fileTime.d /mnt/samba/newfile.txt 
File: /mnt/samba/newfile.txt
Last Modified Time (Method A) (LOCAL): 2024-Feb-06 05:53:18.1905604
Last Modified Time (Method A) (UTC):   2024-Feb-05 18:53:18.1905604Z
Last Modified Time (Method B) (LOCAL): 2024-Feb-06 05:53:18.1905604
Last Modified Time (Method B) (UTC):   2024-Feb-05 18:53:18.1905604Z

[abraunegg]

@robertschulze
If I use the same test data set, now using a SMB Mount from a Windows 2019 Test Server, and using my prior testing configuration to debug your issues:

# @robertschulze Testing
download_only = "true"
cleanup_local_files = "true"

Once initially synced, I run a --resync :

Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application state data: /home/alex/.config/onedrive

ERROR: Unable to create /var/log/onedrive
ERROR: Please manually create '/var/log/onedrive' and set appropriate permissions to allow write access for your user to this location.
ERROR: The requested client activity log will instead be located in your users home directory

Using the following path to store the runtime application log: /home/alex

The usage of --resync will delete your local 'onedrive' client state, thus no record of your current 'sync status' will exist.
This has the potential to overwrite local versions of files with perhaps older versions of documents downloaded from OneDrive, resulting in local data loss.
If in doubt, backup your local data before using --resync

Are you sure you wish to proceed with --resync? [Y/N] y

Deleting the saved application sync status ...
Using IPv4 and IPv6 (if configured) for all network operations
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...

WARNING: Application has been configured to cleanup local files that are not present online.
WARNING: Local data loss MAY occur in this scenario if you are expecting data to remain archived locally.

Application Version:  v2.5.0-alpha-5 GitHub version: v2.4.25-64-g1630ae3
Account Type:         personal
Default Drive ID:     66d53be8a5056eca
Default Root ID:      66D53BE8A5056ECA!101
Remaining Free Space: 2.30 GB (2469606195 bytes)
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /mnt/samba/
Generating a /delta response from the OneDrive API for Drive ID: 66d53be8a5056eca
Adding OneDrive folder details for processing
Adding 9 OneDrive items for processing from the OneDrive 'root' folder
Skipping item - excluded by sync_list config: ./random_25k_files
Adding 10 OneDrive items for processing from random_100_files
Adding 30 OneDrive items for processing from random_100_files/1iN8au6MjOoz4lUvRj11vNrPKBgde5RE
Adding 30 OneDrive items for processing from random_100_files/2wfxyHTgcDmBegcD2FCxaSwgGCq8XH5G
Adding 30 OneDrive items for processing from random_100_files/4WS6Z5Uj75YyE0z6CzuIBwfxzSgwrSTW
Adding 30 OneDrive items for processing from random_100_files/AZ8Iw0gUqxyIFt9pNKV4pYVu0lrKItrl
Adding 30 OneDrive items for processing from random_100_files/c1g4myWX07ATSh3vpLwAKFOYU5qNhfip
Adding 30 OneDrive items for processing from random_100_files/II5RPSP5jexAelUt9GsDWFmN66zK0hM2
Adding 30 OneDrive items for processing from random_100_files/LVmcNeMmhe7swt61sE8sKFFPwuNLiXhl
Adding 30 OneDrive items for processing from random_100_files/sWUd2iAcL7kQttnB8wPbwDQsdZ8ZGINd
Adding 30 OneDrive items for processing from random_100_files/xpA1t3V5kv02wRfxlkFcTdpQfhBDIjks
Adding 30 OneDrive items for processing from random_100_files/Z4tjcEfL5sJ5T2bC6ZpuJQHrUa0sQIf2
Finished processing self generated /delta JSON response from the OneDrive API
Processing 318 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Syncing this OneDrive Personal Shared Folder: OneDrive - Constanze
Generating a /delta response from the OneDrive API for Drive ID: bc7d88ec1f539dcf
Adding OneDrive root details for processing
Adding OneDrive folder details for processing
Adding 4 OneDrive items for processing from the OneDrive 'root' folder
Adding 2 OneDrive items for processing from Datenbanken
Finished processing self generated /delta JSON response from the OneDrive API
Processing 6 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Failed to fetch or interpret quota details from the API response.
Performing a database consistency and integrity check on locally stored data
Processing DB entries for this Drive ID: 66d53be8a5056eca
Processing /mnt/samba
The directory has not changed
Processing OneDrive - Constanze
Processing random_100_files
The directory has not changed
Processing random_100_files/1iN8au6MjOoz4lUvRj11vNrPKBgde5RE
The directory has not changed
Processing random_100_files/1iN8au6MjOoz4lUvRj11vNrPKBgde5RE/file0.data
...
Processing random_100_files/Z4tjcEfL5sJ5T2bC6ZpuJQHrUa0sQIf2/file27.data
The file has not changed
Processing random_100_files/Z4tjcEfL5sJ5T2bC6ZpuJQHrUa0sQIf2/file28.data
The file has not changed
Processing random_100_files/Z4tjcEfL5sJ5T2bC6ZpuJQHrUa0sQIf2/file29.data
The file has not changed
Processing α
The directory has not changed
Processing เอกสาร
The directory has not changed
Processing 1.txt
The file has not changed
Processing 1-onedrive-client-dev.txt
The file has not changed
Processing newfialsdflasdf.txt
The file has not changed
Processing test.txt
The file has not changed
Processing DB entries for this Drive ID: bc7d88ec1f539dcf
Failed to fetch or interpret quota details from the API response.
Processing OneDrive - Constanze
The directory has not changed
Processing OneDrive - Constanze/Datenbanken
The directory has not changed
Processing OneDrive - Constanze/Datenbanken/dummy_file.docx
The file has not changed
Processing OneDrive - Constanze/Datenbanken/new-file.txt
The file has not changed
Processing OneDrive - Constanze/debug_output.log
The file has not changed
Processing OneDrive - Constanze/keepass-minipc-windows.kdbx
The file has not changed
Processing OneDrive - Constanze/newfile.txt
The file has not changed
Scanning the local file system '/mnt/samba' for data to cleanup

Sync with Microsoft OneDrive is complete

No date issue or mismatch.

If I EDIT the file online:

Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application state data: /home/alex/.config/onedrive

ERROR: Unable to create /var/log/onedrive
ERROR: Please manually create '/var/log/onedrive' and set appropriate permissions to allow write access for your user to this location.
ERROR: The requested client activity log will instead be located in your users home directory

Using the following path to store the runtime application log: /home/alex
Using IPv4 and IPv6 (if configured) for all network operations
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...

WARNING: Application has been configured to cleanup local files that are not present online.
WARNING: Local data loss MAY occur in this scenario if you are expecting data to remain archived locally.

Application Version:  v2.5.0-alpha-5 GitHub version: v2.4.25-64-g1630ae3
Account Type:         personal
Default Drive ID:     66d53be8a5056eca
Default Root ID:      66D53BE8A5056ECA!101
Remaining Free Space: 2.30 GB (2469606195 bytes)
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /mnt/samba/
Generating a /delta response from the OneDrive API for Drive ID: 66d53be8a5056eca
Adding OneDrive folder details for processing
Adding 9 OneDrive items for processing from the OneDrive 'root' folder
Skipping item - excluded by sync_list config: ./random_25k_files
Adding 10 OneDrive items for processing from random_100_files
Adding 30 OneDrive items for processing from random_100_files/1iN8au6MjOoz4lUvRj11vNrPKBgde5RE
Adding 30 OneDrive items for processing from random_100_files/2wfxyHTgcDmBegcD2FCxaSwgGCq8XH5G
Adding 30 OneDrive items for processing from random_100_files/4WS6Z5Uj75YyE0z6CzuIBwfxzSgwrSTW
Adding 30 OneDrive items for processing from random_100_files/AZ8Iw0gUqxyIFt9pNKV4pYVu0lrKItrl
Adding 30 OneDrive items for processing from random_100_files/c1g4myWX07ATSh3vpLwAKFOYU5qNhfip
Adding 30 OneDrive items for processing from random_100_files/II5RPSP5jexAelUt9GsDWFmN66zK0hM2
Adding 30 OneDrive items for processing from random_100_files/LVmcNeMmhe7swt61sE8sKFFPwuNLiXhl
Adding 30 OneDrive items for processing from random_100_files/sWUd2iAcL7kQttnB8wPbwDQsdZ8ZGINd
Adding 30 OneDrive items for processing from random_100_files/xpA1t3V5kv02wRfxlkFcTdpQfhBDIjks
Adding 30 OneDrive items for processing from random_100_files/Z4tjcEfL5sJ5T2bC6ZpuJQHrUa0sQIf2
Finished processing self generated /delta JSON response from the OneDrive API
Processing 318 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Failed to fetch or interpret quota details from the API response.
Syncing this OneDrive Personal Shared Folder: OneDrive - Constanze
Generating a /delta response from the OneDrive API for Drive ID: bc7d88ec1f539dcf
Adding OneDrive root details for processing
Adding OneDrive folder details for processing
Adding 4 OneDrive items for processing from the OneDrive 'root' folder
Adding 2 OneDrive items for processing from Datenbanken
Finished processing self generated /delta JSON response from the OneDrive API
Processing 6 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Number of items to download from OneDrive: 1
Downloading file OneDrive - Constanze/Datenbanken/dummy_file.docx ... done   <----- file downloaded without issue
Performing a database consistency and integrity check on locally stored data
Processing DB entries for this Drive ID: 66d53be8a5056eca
Processing /mnt/samba
The directory has not changed
Processing OneDrive - Constanze
Processing random_100_files
The directory has not changed
Processing random_100_files/1iN8au6MjOoz4lUvRj11vNrPKBgde5RE
..............

The details online:

And when checking the file that was modified:

[alex@onedrive-client-dev file-tests]$ ls -la /mnt/samba/OneDrive\ -\ Constanze/Datenbanken/dummy_file.docx
-rwxrwxr-x. 1 root wheel 12100 Feb  6 06:45 '/mnt/samba/OneDrive - Constanze/Datenbanken/dummy_file.docx'
[alex@onedrive-client-dev file-tests]$ ./fileTime.d /mnt/samba/OneDrive\ -\ Constanze/Datenbanken/dummy_file.docx 
File: /mnt/samba/OneDrive - Constanze/Datenbanken/dummy_file.docx
Last Modified Time (Method A) (LOCAL): 2024-Feb-06 06:45:37.4
Last Modified Time (Method A) (UTC):   2024-Feb-05 19:45:37.4Z
Last Modified Time (Method B) (LOCAL): 2024-Feb-06 06:45:37.4
Last Modified Time (Method B) (UTC):   2024-Feb-05 19:45:37.4Z
[alex@onedrive-client-dev file-tests]$ stat /mnt/samba/OneDrive\ -\ Constanze/Datenbanken/dummy_file.docx  
  File: /mnt/samba/OneDrive - Constanze/Datenbanken/dummy_file.docx
  Size: 12100           Blocks: 24         IO Block: 1048576 regular file
Device: 2fh/47d Inode: 562949953422268  Links: 1
Access: (0775/-rwxrwxr-x)  Uid: (    0/    root)   Gid: (   10/   wheel)
Context: system_u:object_r:cifs_t:s0
Access: 2024-02-06 06:45:37.400000000 +1100
Modify: 2024-02-06 06:45:37.400000000 +1100
Change: 2024-02-06 06:46:22.302956900 +1100
 Birth: 2024-02-06 06:46:21.445425800 +1100
[alex@onedrive-client-dev file-tests]$

Performing a further --resync now over the modified download:

[alex@onedrive-client-dev onedrive-v2.5.0-alpha-5]$ ./onedrive -s -v --resync
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application state data: /home/alex/.config/onedrive

ERROR: Unable to create /var/log/onedrive
ERROR: Please manually create '/var/log/onedrive' and set appropriate permissions to allow write access for your user to this location.
ERROR: The requested client activity log will instead be located in your users home directory

Using the following path to store the runtime application log: /home/alex

The usage of --resync will delete your local 'onedrive' client state, thus no record of your current 'sync status' will exist.
This has the potential to overwrite local versions of files with perhaps older versions of documents downloaded from OneDrive, resulting in local data loss.
If in doubt, backup your local data before using --resync

Are you sure you wish to proceed with --resync? [Y/N] y

Deleting the saved application sync status ...
Using IPv4 and IPv6 (if configured) for all network operations
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...

WARNING: Application has been configured to cleanup local files that are not present online.
WARNING: Local data loss MAY occur in this scenario if you are expecting data to remain archived locally.

Application Version:  v2.5.0-alpha-5 GitHub version: v2.4.25-64-g1630ae3
Account Type:         personal
Default Drive ID:     66d53be8a5056eca
Default Root ID:      66D53BE8A5056ECA!101
Remaining Free Space: 2.30 GB (2469606195 bytes)
Sync Engine Initialised with new Onedrive API instance
All application operations will be performed in the configured local 'sync_dir' directory: /mnt/samba/
Generating a /delta response from the OneDrive API for Drive ID: 66d53be8a5056eca
Adding OneDrive folder details for processing
Adding 9 OneDrive items for processing from the OneDrive 'root' folder
Skipping item - excluded by sync_list config: ./random_25k_files
Adding 10 OneDrive items for processing from random_100_files
Adding 30 OneDrive items for processing from random_100_files/1iN8au6MjOoz4lUvRj11vNrPKBgde5RE
Adding 30 OneDrive items for processing from random_100_files/2wfxyHTgcDmBegcD2FCxaSwgGCq8XH5G
Adding 30 OneDrive items for processing from random_100_files/4WS6Z5Uj75YyE0z6CzuIBwfxzSgwrSTW
Adding 30 OneDrive items for processing from random_100_files/AZ8Iw0gUqxyIFt9pNKV4pYVu0lrKItrl
Adding 30 OneDrive items for processing from random_100_files/c1g4myWX07ATSh3vpLwAKFOYU5qNhfip
Adding 30 OneDrive items for processing from random_100_files/II5RPSP5jexAelUt9GsDWFmN66zK0hM2
Adding 30 OneDrive items for processing from random_100_files/LVmcNeMmhe7swt61sE8sKFFPwuNLiXhl
Adding 30 OneDrive items for processing from random_100_files/sWUd2iAcL7kQttnB8wPbwDQsdZ8ZGINd
Adding 30 OneDrive items for processing from random_100_files/xpA1t3V5kv02wRfxlkFcTdpQfhBDIjks
Adding 30 OneDrive items for processing from random_100_files/Z4tjcEfL5sJ5T2bC6ZpuJQHrUa0sQIf2
Finished processing self generated /delta JSON response from the OneDrive API
Processing 318 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Syncing this OneDrive Personal Shared Folder: OneDrive - Constanze
Generating a /delta response from the OneDrive API for Drive ID: bc7d88ec1f539dcf
Adding OneDrive root details for processing
Adding OneDrive folder details for processing
Adding 4 OneDrive items for processing from the OneDrive 'root' folder
Adding 2 OneDrive items for processing from Datenbanken
Finished processing self generated /delta JSON response from the OneDrive API
Processing 6 applicable changes and items received from Microsoft OneDrive
Processing OneDrive JSON item batch [1/1] to ensure consistent local state
Failed to fetch or interpret quota details from the API response.
Performing a database consistency and integrity check on locally stored data
Processing DB entries for this Drive ID: 66d53be8a5056eca
Processing /mnt/samba
The directory has not changed
Processing OneDrive - Constanze
Processing random_100_files
The directory has not changed
Processing random_100_files/1iN8au6MjOoz4lUvRj11vNrPKBgde5RE
The directory has not changed
Processing random_100_files/1iN8au6MjOoz4lUvRj11vNrPKBgde5RE/file0.data
...
The file has not changed
Processing random_100_files/Z4tjcEfL5sJ5T2bC6ZpuJQHrUa0sQIf2/file28.data
The file has not changed
Processing random_100_files/Z4tjcEfL5sJ5T2bC6ZpuJQHrUa0sQIf2/file29.data
The file has not changed
Processing α
The directory has not changed
Processing เอกสาร
The directory has not changed
Processing 1.txt
The file has not changed
Processing 1-onedrive-client-dev.txt
The file has not changed
Processing newfialsdflasdf.txt
The file has not changed
Processing test.txt
The file has not changed
Processing DB entries for this Drive ID: bc7d88ec1f539dcf
Failed to fetch or interpret quota details from the API response.
Processing OneDrive - Constanze
The directory has not changed
Processing OneDrive - Constanze/Datenbanken
The directory has not changed
Processing OneDrive - Constanze/Datenbanken/dummy_file.docx
The file has not changed
Processing OneDrive - Constanze/Datenbanken/new-file.txt
The file has not changed
Processing OneDrive - Constanze/debug_output.log
The file has not changed
Processing OneDrive - Constanze/keepass-minipc-windows.kdbx
The file has not changed
Processing OneDrive - Constanze/newfile.txt
The file has not changed
Scanning the local file system '/mnt/samba' for data to cleanup

Sync with Microsoft OneDrive is complete

Again .. no modification or issue with the timestamp for any file ....

So at this point I can only conclude:

• The issue is due to your Samaba / SMB mount somehow
• The issue is related to your use of Ubuntu potentially

I cannot replicate your issue at all despite repeated and best efforts.

Suggestion for you:

• Dont use Ubuntu at all - build a modern server using a completely different distribution such as RHEL or CentOS or Fedora - ie something that is not Debian based (and stay away from Ubuntu and it's clones as much as possible) and attempt to replicate using that.

If you can replicate with RHEL or CentOS or Fedora - then it is something to do with how you are mounting your SMB Folder or your Windows Server where the SMB mount is coming from.

If you cant replicate .. then the issue is with your Ubuntu platform and potentially will come down to how Ubuntu uses old packages and does not keep things correctly updated (in the guise of 'stability' .. where the bugs they do not fix actually cause stability issues).

[robertschulze]

@abraunegg
Thanks for all your tests. After numerous tests on my side I was now able to pinpoint the issue - at least on onedrive / D level. I monkey-patched the onedrive code to check whether at some point during the run the file date would be modified, and indeed: when checkFileDatabaseItemForConsistency is calling readLocalFile to check whether the file is readable, the read process is modifying the file date. But as if this is not strange enough: this happens only temporarily.

I modified the test script to perform the determination of the modification date before and after a read(fn, 1)

#!/usr/bin/env rdmd
import std.stdio;
import std.file;
import std.datetime : SysTime;

void main(string[] args)
{
    if (args.length != 2)
    {
        writeln("Usage: ./script <file_path>");
        return;
    }

    string filePath = args[1];

    if (!exists(filePath))
    {
        writeln("Error: The specified file does not exist.");
        return;
    }

    SysTime modificationTime = filePath.timeLastModified;

    writeln("File: ", filePath);
    writeln("Last Modified Time: ", modificationTime);

    std.file.read(filePath, 1);

    modificationTime = filePath.timeLastModified;

    writeln("File: ", filePath);
    writeln("Last Modified Time: ", modificationTime);

}

the output being:

(dmd-2.106.0)robert@atom:~/onedrive$ ./fileTime.d /media/3/3\ -\ OneDrive/[email protected]/OneDrive\ -\ Constanze/Datenbanken/TestWordfile.docx
File: /media/3/3 - OneDrive/[email protected]/OneDrive - Constanze/Datenbanken/TestWordfile.docx
Last Modified Time: 2023-Dec-15 00:25:28
File: /media/3/3 - OneDrive/[email protected]/OneDrive - Constanze/Datenbanken/TestWordfile.docx
Last Modified Time: 2024-Feb-09 00:30:29.1690092
(dmd-2.106.0)robert@atom:~/onedrive$ 

but rerunning it just a split second later

(dmd-2.106.0)robert@atom:~/onedrive$ ./fileTime.d /media/3/3\ -\ OneDrive/[email protected]/OneDrive\ -\ Constanze/Datenbanken/TestWordfile.docx
File: /media/3/3 - OneDrive/[email protected]/OneDrive - Constanze/Datenbanken/TestWordfile.docx
Last Modified Time: 2023-Dec-15 00:25:28
File: /media/3/3 - OneDrive/[email protected]/OneDrive - Constanze/Datenbanken/TestWordfile.docx
Last Modified Time: 2023-Dec-15 00:25:28
(dmd-2.106.0)robert@atom:~/onedrive$ 

This only happens for onedrive sync'ed files and only the first script run after the sync. From the second script run on, reported dates are always identical. Strangely, for all other files on the drive identical dates are reported from the first run on. Even if I create a new file manually (touch mytest) both dates are always reported identically.
The only thing I can think of for this particular behaviour could be that after within the onedrive sync the file is somehow not fully closed. Thus when the read() occurs (either by a subsequent onedrive run or the script) the system is closing the file, leading to the intermittent wrong modification date. It would at least explain why it is present for subsequent onedrive runs but not subsequent runs of the But I have to admit my knowledge about the file system details (leave alone the D implementation, ) is too limited.
And yes, it only happens for the Samba mounted drives, in /tmp and my homedir the effect is not present.

[abraunegg]

@robertschulze
Interesting analysis .. however, on all my tests to a Samba mounted Windows share - I cannot replicate even the time difference on a first sync at all.

I will look into if anything is being left open - hightly doubt it - so I really think this is something very unique to your setup.

Can you replicate this without Ubuntu ?

[abraunegg]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

[abraunegg]

@robertschulze
FYI - This bugfix just landed for Samba:

https://bugzilla.samba.org/show_bug.cgi?id=15550

Potentially the cause of your issue when mounting SMB locations?

[robertschulze]

@abraunegg
Yes, good catch, that could well be it. My Ubuntu contains version 4.15.13 with the bug being fixed for 4.18 and 4.19. I will try to upgrade somehow and re-test. Thanks!
In the meanwhile I tested Fedora 39 and I can confirm that the bug was is present there.

[abraunegg]

@robertschulze
Interesting that the original issue was not there on Fedora .... so potentially its a very specific Ubuntu bug due to how Ubuntu always use old package versions.

The Samba bug that was fixed is newish - so if this was the cause, I would have suspected Fedora to exhibit a similar issue - however I was unable to replicate on CentOS or other RHEL / Fedora systems myself - thus I had a strong suspicion it being a Ubuntu specific issue.

[abraunegg]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.