picocrypt only uses the first 1 GB of the key file (and this is not documented) about picocrypt

[hakavlad]

The next question: why picocrypt uses 200% CPU and not read from the disk during "Reading keyfiles..."?

[HACKERALERT]

There is too much "undefined" behaviour here for me to be able to tell what's going on, especially because I don't use Linux so I have no idea how memory management works in there.

Looking back at your initial screenshot in #165, it looks like your host OS correctly moves to swap when main memory is all used up, so I'm not sure why it isn't working here. Since you said the keyfile used was 40 GB, my guess would be that Picocrypt overloaded both the main and swap memory trying to allocate the memory required to store the keyfile. However, this should cause a crash as opposed to only reading the first 1 GB. Try using a 2 GB keyfile and modifying the last half to confirm whether this behaviour also holds or if it only happens when the total memory on the system is exhausted.

Looking at your GitHub profile, it seems like you have a good amount of programming expeience, so you should be able to following along with my code reasonably well even if you don't know Go. Below are two sections of code where the keyfile is hashed. In both cases, each keyfile is read entirely into memory before being hashed, so there's no reason why only the first 1 GB would be used:
https://github.com/HACKERALERT/Picocrypt/blob/main/src/Picocrypt.go#L1473
https://github.com/HACKERALERT/Picocrypt/blob/main/src/Picocrypt.go#L1488

From #165:

In any case, the hashing of the files occurs in a sub-optimal way, and this can be corrected without breaking compatibility.

You are correct. I think the best solution to all of this weird behaviour is to just hash the keyfiles in chunks. It won't break compatibility as you mentioned, and should be much more memory efficient. I think I will keep this issue open (like the one about unencrypted comments) so that I will remember to fix this in the next release. I won't make a new release just yet though since I doubt many people are using 1+ GB keyfiles.

[hakavlad]

Picocrypt overloaded both the main and swap memory trying to allocate the memory required to store the keyfile.

It's not a problem. The same result with 2 GiB file.

this should cause a crash

Encryption and decryption complete successfully with both 2 GiB and 40 GiB files. No difference. In both cases, the disk activity analyzer registers a read of about 1025 MiB.
When disabling the swap partition, the process was killed by OOM killer:

июн 12 02:21:20 PC nohang[921]: Memory status that requires corrective actions:
июн 12 02:21:20 PC nohang[921]:   MemAvailable [436 MiB, 3.7 %] <= soft_threshold_min_mem [589 MiB, 5.0 %]
июн 12 02:21:20 PC nohang[921]:   SwapFree [0 MiB, 0.0 %] <= soft_threshold_min_swap [0 MiB, - %]
июн 12 02:21:20 PC nohang[921]: Found 110 tasks with non-zero oom_score (except init and self) in 7ms
июн 12 02:21:20 PC nohang[921]: TOP-15 tasks by badness:
июн 12 02:21:20 PC nohang[921]:   Name                PID badness
июн 12 02:21:20 PC nohang[921]:   --------------- ------- -------
июн 12 02:21:20 PC nohang[921]:   picocrypt         19984    1146
июн 12 02:21:20 PC nohang[921]:   firefox-esr        9789     725
июн 12 02:21:20 PC nohang[921]:   Isolated Web Co   20088     674
июн 12 02:21:20 PC nohang[921]:   WebExtensions     10265     672
июн 12 02:21:20 PC nohang[921]:   Xorg               1032     670
июн 12 02:21:20 PC nohang[921]:   Privileged Cont    9983     670
июн 12 02:21:20 PC nohang[921]:   dolphin           17618     670
июн 12 02:21:20 PC nohang[921]:   caja               2968     668
июн 12 02:21:20 PC nohang[921]:   wnck-applet        2982     668
июн 12 02:21:20 PC nohang[921]:   applet.py          3032     668
июн 12 02:21:20 PC nohang[921]:   mate-screensave    3040     668
июн 12 02:21:20 PC nohang[921]:   tor                3290     668
июн 12 02:21:20 PC nohang[921]:   mate-system-mon    3974     668
июн 12 02:21:20 PC nohang[921]:   lxterminal        18396     668
июн 12 02:21:20 PC nohang[921]:   dnscrypt-proxy      983     667
июн 12 02:21:20 PC nohang[921]: TOP printed in 1ms; process with highest badness:
июн 12 02:21:20 PC nohang[921]:   PID: 19984, name: picocrypt, badness: 1146
июн 12 02:21:20 PC nohang[921]: Recheck memory levels...
июн 12 02:21:20 PC nohang[921]: Memory status that requires corrective actions:
июн 12 02:21:20 PC nohang[921]:   MemAvailable [418 MiB, 3.5 %] <= soft_threshold_min_mem [589 MiB, 5.0 %]
июн 12 02:21:20 PC nohang[921]:   SwapFree [0 MiB, 0.0 %] <= soft_threshold_min_swap [0 MiB, - %]
июн 12 02:21:20 PC nohang[921]: Victim status (found in 6ms):
июн 12 02:21:20 PC nohang[921]:   PID:       19984, name: picocrypt, state: S (sleeping), EUID: 1000, SID: 2601 (x-session-manag), lifetime: 12min 48s
июн 12 02:21:20 PC nohang[921]:   badness:   1146, oom_score:  1146, oom_score_adj:  0
июн 12 02:21:20 PC nohang[921]:   Vm, MiB:   Size: 46357, RSS: 8439 (Anon: 8429, File: 9, Shmem: 1), Swap: 0
июн 12 02:21:20 PC nohang[921]:   cgroup_v1:
июн 12 02:21:20 PC nohang[921]:   cgroup_v2: /user.slice/user-1000.slice/session-2.scope
июн 12 02:21:20 PC nohang[921]:   ancestry:  PID 19983 (sh) <= PID 19975 (script.sh) <= PID 2864 (mate-panel)
июн 12 02:21:20 PC nohang[921]:   exe realpath: /tmp/.mount_picocrkJtVzB/usr/bin/picocrypt
июн 12 02:21:20 PC nohang[921]:   cwd realpath: /tmp/.mount_picocrkJtVzB/usr
июн 12 02:21:20 PC nohang[921]: Memory info, MiB:
июн 12 02:21:20 PC nohang[921]:   total=11780, used=10409, free=276, available=405, shared=693, buffers=26, cache=1068,
июн 12 02:21:20 PC nohang[921]:   swap_total=0, swap_used=0, swap_free=0
июн 12 02:21:20 PC nohang[921]: Memory pressure (system-wide):
июн 12 02:21:20 PC nohang[921]:   some avg10=0.00 avg60=0.04 avg300=0.68
июн 12 02:21:20 PC nohang[921]:   full avg10=0.00 avg60=0.01 avg300=0.33
июн 12 02:21:20 PC nohang[921]: Implementing a corrective action:
июн 12 02:21:20 PC nohang[921]:   Sending SIGTERM to the victim
июн 12 02:21:20 PC nohang[921]: OK; total response time: 15ms
июн 12 02:21:20 PC nohang[921]: What happened in the last 10h 48min 54s:
июн 12 02:21:20 PC nohang[921]:   [ OK ] Sending SIGTERM to picocrypt: 1

[hakavlad]

each keyfile is read entirely into memory before being hashed, so there's no reason why only the first 1 GB would be used

This is all very strange.

[HACKERALERT]

Yeah I'm not sure what's going on. But with all of this, I agree it'll be worth it to update the code to hash the keyfiles in chunks. So until I do that in the next release, I'll keep this issue open.

[HACKERALERT]

Added this to the changelog for the next release, so I'll close it here to keep things clean and organized.

[hakavlad]

I have also tested this on MS Windows and got the same result. Piсocrypt read exactly 1 GiB of the key file.

[HACKERALERT]

Okay. I will fix this in the next release.