Here are the logs of compiling shake on my system. It includes mostly some deprecation warnings.

[  7%] Building C object CMakeFiles/shake.dir/executive.o
[ 14%] Building C object CMakeFiles/shake.dir/judge.o
[ 21%] Building C object CMakeFiles/shake.dir/linux.o
/home/celelibi/code/shake/linux.c: In function ‘set_ptime’:
/home/celelibi/code/shake/linux.c:175:3: warning: ‘attr_setf’ is deprecated: Use fsetxattr instead [-Wdeprecated-declarations]
  175 |   return attr_setf (fd, "shake.ptime", (char *) &date, DATE_SIZE,
      |   ^~~~~~
In file included from /home/celelibi/code/shake/linux.c:32:
/usr/include/attr/attributes.h:143:12: note: declared here
  143 | extern int attr_setf (int __fd, const char *__attrname,
      |            ^~~~~~~~~
/home/celelibi/code/shake/linux.c: In function ‘get_ptime’:
/home/celelibi/code/shake/linux.c:186:7: warning: ‘attr_getf’ is deprecated: Use fgetxattr instead [-Wdeprecated-declarations]
  186 |       attr_getf (fd, "shake.ptime", (char *) &date, &size, ATTR_DONTFOLLOW))
      |       ^~~~~~~~~
In file included from /home/celelibi/code/shake/linux.c:32:
/usr/include/attr/attributes.h:131:12: note: declared here
  131 | extern int attr_getf (int __fd, const char *__attrname, char *__attrvalue,
      |            ^~~~~~~~~
[ 28%] Building C object CMakeFiles/shake.dir/main.o
/home/celelibi/code/shake/main.c: In function ‘main’:
/home/celelibi/code/shake/main.c:293:13: warning: variable ‘jugement’ set but not used [-Wunused-but-set-variable]
  293 |         int jugement;
      |             ^~~~~~~~
[ 35%] Building C object CMakeFiles/shake.dir/msg.o
[ 42%] Building C object CMakeFiles/shake.dir/signals.o
[ 50%] Linking C executable shake
[ 50%] Built target shake
[ 57%] Building C object CMakeFiles/unattr.dir/executive.o
[ 64%] Building C object CMakeFiles/unattr.dir/linux.o
/home/celelibi/code/shake/linux.c: In function ‘set_ptime’:
/home/celelibi/code/shake/linux.c:175:3: warning: ‘attr_setf’ is deprecated: Use fsetxattr instead [-Wdeprecated-declarations]
  175 |   return attr_setf (fd, "shake.ptime", (char *) &date, DATE_SIZE,
      |   ^~~~~~
In file included from /home/celelibi/code/shake/linux.c:32:
/usr/include/attr/attributes.h:143:12: note: declared here
  143 | extern int attr_setf (int __fd, const char *__attrname,
      |            ^~~~~~~~~
/home/celelibi/code/shake/linux.c: In function ‘get_ptime’:
/home/celelibi/code/shake/linux.c:186:7: warning: ‘attr_getf’ is deprecated: Use fgetxattr instead [-Wdeprecated-declarations]
  186 |       attr_getf (fd, "shake.ptime", (char *) &date, &size, ATTR_DONTFOLLOW))
      |       ^~~~~~~~~
In file included from /home/celelibi/code/shake/linux.c:32:
/usr/include/attr/attributes.h:131:12: note: declared here
  131 | extern int attr_getf (int __fd, const char *__attrname, char *__attrvalue,
      |            ^~~~~~~~~
[ 71%] Building C object CMakeFiles/unattr.dir/signals.o
[ 78%] Building C object CMakeFiles/unattr.dir/unattr.o
/home/celelibi/code/shake/unattr.c: In function ‘strip’:
/home/celelibi/code/shake/unattr.c:114:5: warning: ‘attr_removef’ is deprecated: Use fremovexattr instead [-Wdeprecated-declarations]
  114 |     attr_removef (fd, *attr, ATTR_DONTFOLLOW);
      |     ^~~~~~~~~~~~
In file included from /home/celelibi/code/shake/unattr.c:29:
/usr/include/attr/attributes.h:155:12: note: declared here
  155 | extern int attr_removef (int __fd, const char *__attrname, int __flags)
      |            ^~~~~~~~~~~~
[ 85%] Linking C executable unattr
[ 85%] Built target unattr
[ 92%] Building manpage for shake
[100%] Building manpage for unattr
[100%] Built target doc

Such as baloo or tracker.
In #6, @kakra raised the concern they are likely to break mandatory locks.
#10 is likely to provide a solution for this, but let's create a different bug to track.

Can someone please advise which version of Mint/ or similar Live CD I can use as a base and then install Shake (directly or after installing required dependencies) - And how?

Working steps please. Thanks.

I'll let you be the juge. But I guess that in not too long, the CMakeFiles will stop working. Especially given that the last commit was in 2017, I guess the sooner the update, the better.

Here is the output of cmake 3.22.1.

CMake Deprecation Warning at CMakeLists.txt:19 (CMAKE_MINIMUM_REQUIRED):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Deprecation Warning at CMakeLists.txt:21 (cmake_policy):
  The OLD behavior for policy CMP0005 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.


-- The C compiler identification is GNU 11.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Warning (dev) at CMakeLists.txt:26 (get_target_property):
  Policy CMP0026 is not set: Disallow use of the LOCATION target property.
  Run "cmake --help-policy CMP0026" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  The LOCATION property should not be read from target "shake".  Use the
  target name directly with add_custom_command, or use the generator
  expression $<TARGET_FILE>, as appropriate.

Call Stack (most recent call first):
  CMakeLists.txt:50 (add_help2man_manpage)
This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at CMakeLists.txt:26 (get_target_property):
  Policy CMP0026 is not set: Disallow use of the LOCATION target property.
  Run "cmake --help-policy CMP0026" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  The LOCATION property should not be read from target "unattr".  Use the
  target name directly with add_custom_command, or use the generator
  expression $<TARGET_FILE>, as appropriate.

Call Stack (most recent call first):
  CMakeLists.txt:51 (add_help2man_manpage)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Looking for attr_setf
-- Looking for attr_setf - found
-- Looking for fallocate
-- Looking for fallocate - found
-- Configuring done
-- Generating done

The current version of shake uses error() from error.h, and is therefore unsupported on musl.
Would there be a plan to use more portable alternatives (err.h seems one of them)?

Shouldn't tmpfile location be specified and actually put in /tmp/ or /var/tmp instead of putting it in $HOME ?

$sudo pv -d `pgrep shake`
3:/home/bartus/shakej9LlvJ:  252 B 0:19:05 [17MiB/s] [  <=>  ]

ZFS being a copy on write filesystem has many problems with fragmentation the older the system gets. the standard answer usually is to copy everything (send/receive) which basically is backing up and restoring the filesystem. A user space tool like shake which I've used before on other FS's would greatly help here.

I pulled the current copy from github to see what it would do and pretty much every file handle gets the same error:

shake -v -p /var/log/clientlogs/test

IDEAL START END FRAGC CRUMBC AGE SHOCKED NAME
./shake: /var/log/clientlogs/test/checkpoint/backups/log.20170422010001.tgz: FIBMAP failed: Invalid argument
./shake: /var/log/clientlogs/test/checkpoint/backups/log.20170424010001.tgz: FIBMAP failed: Invalid argument
./shake: /var/log/clientlogs/test/checkpoint/backups/log.20170426010001.tgz: FIBMAP failed: Invalid argument
./shake: /var/log/clientlogs/test/checkpoint/backups/log.20170427010001.tgz: FIBMAP failed: Invalid argument
./shake: /var/log/clientlogs/test/checkpoint/backups/log.20170423010001.tgz: FIBMAP failed: Invalid argument
./shake: /var/log/clientlogs/test/checkpoint/backups/log.20170425010001.tgz: FIBMAP failed: Invalid argument
./shake: /var/log/clientlogs/test/syslog/2017/20170427-kern.log: FIBMAP failed: Invalid argument

This is running under a ubuntu 16.04.2LTS 64bit system.

uname -a

Linux loki 4.4.0-71-generic #92-Ubuntu SMP Fri Mar 24 12:59:01 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

dpkg -l | grep zfs

ii libzfs2 0.6.5.7-1trusty amd64 Native OpenZFS filesystem library for Linux
ii ubuntu-zfs 8trusty amd64 Native ZFS filesystem metapackage for Ubuntu.
ii zfs-dkms 0.6.5.7-1trusty amd64 Native OpenZFS filesystem kernel modules for Linux
ii zfs-doc 0.6.5.7-1trusty amd64 Native OpenZFS filesystem documentation and examples.
ii zfsutils 0.6.5.7-1~trusty amd64 Native OpenZFS management utilities for Linux

I understand the current approach of rewriting the file twice to keep the inode and all its attached attributes intact (xattr, stat, perms, acls). This is especially even more important for hardlinked files (those with nlink > 1) to not break the hardlinks as there seems to be no other way of keeping the hardlink (tho, I found that taking the lease on hardlinks doesn't work here).

But for non-hardlink cases we could try to redo the file from scratch, replicating all its attributes, and then rename it in place of the other to save one rewrite phase.

Additionally, this would get rid of the opportunity for shake to exit uncleanly and leave a damaged file behind (e.g. when killed by lease breaking). What do you think?

When trying to add the ppa it does not work because Ubuntu xenial is not supported

The code uses the GNU style, which few projects use and looks unfamiliar to anyone else.
A pass through clang-format or indent should take care of that.

I would say one of the prime use-cases for "shake" is systems so old that they don't support fe. ext4fs, and hence defrag4fs. Web is full of old articles, which tell you to use "shake" on such old ext3fs machines.

But what happens if you run it now after the patch d984848 ? Well, shake truncates the original file, after which the fallocate() fails. Modern glibc apparently "emulates" fallocate() in such cases, but it really does no good, and in such old systems there appear to be implementations which simply error out. Checking that real fallocate() works would therefore be a good idea. (CMake etc. checks aren't enough, because the syscall exists, it just errors when called on ext3fs)

$> shake --pretend foo.bin
shake: foo.bin: FIBMAP failed: Operation not permitted

Maybe "You have to execute shake as root" would be a more meaningful error message.

$> which shake
/usr/bin/shake

Consequently the binary should be placed into /usr/sbin

I'm not sure if the current locking implementation with leases is the right thing to do. Shake should acquire an unbreakable file lock because otherwise it may mess up with your files in case of a timeout and you cannot let it run unattended.

The number one priority of a defragger should be safety of the users data, and this is currently not implemented at a convenient level. Other defraggers like btrfs-defrag and xfs_fsr use unbreakable locks to ensure the files don't get messed up.

If some other application has to access the file, it has to wait. The current implementation allows the other application to break the lock, and what that application find is a broken, empty, or partial file. At least during rewrite phase the lock has to be unbreakable.

I want to throw in the following idea:

How about separating the judgement and the defrag part into two separate binaries? That way the judgement part would only spill out one filename per line which would be passed into the defrag binary on standard input. In the end, some filesystems require special handling of defragmentation and you probably don't want to put all this logic and heavy lifting into shake. The judgement part of shake itself becomes thus only a clever disk layout analyzer and file sorter. Also, the file system handling code could become a lot cleaner.

Both binaries could then be optimized on their own, shake could spawn multiple defrag binaries per spindle or per filesystem. It could also supply some stub binaries like defrag.xfs, defrag.btrfs to call out for the native tools of those filesystems as an option.

OS: Linux Mint 18.03 64 bit (Ubuntu 16.04)
Dependencies: Installed libattr1-dev.

cmake ..
CMake Error at CMakeLists.txt:44 (message):
Cannot find help2man. Please install it.

CMake Warning (dev) at CMakeLists.txt:26 (get_target_property):
Policy CMP0026 is not set: Disallow use of the LOCATION target property.
Run "cmake --help-policy CMP0026" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.

The LOCATION property should not be read from target "shake". Use the
target name directly with add_custom_command, or use the generator
expression $<TARGET_FILE>, as appropriate.

Call Stack (most recent call first):
CMakeLists.txt:50 (add_help2man_manpage)
This warning is for project developers. Use -Wno-dev to suppress it.

CMake Warning (dev) at CMakeLists.txt:26 (get_target_property):
Policy CMP0026 is not set: Disallow use of the LOCATION target property.
Run "cmake --help-policy CMP0026" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.

The LOCATION property should not be read from target "unattr". Use the
target name directly with add_custom_command, or use the generator
expression $<TARGET_FILE>, as appropriate.

Call Stack (most recent call first):
CMakeLists.txt:51 (add_help2man_manpage)
This warning is for project developers. Use -Wno-dev to suppress it.

CMake Error at CMakeLists.txt:87 (message):
You need libattr headers for Shake to build. They might be in a package
named libattr1-dev or libattr-devel.

-- Configuring incomplete, errors occurred!
See also "/home/antonio/projects/github/others/shake/build/CMakeFiles/CMakeOutput.log".
See also "/home/antonio/projects/github/others/shake/build/CMakeFiles/CMakeError.log".

Modern filesystems like XFS and btrfs support reflinking a file. Before rewriting the file, we could reflink the accused file to the rewrite source, then use that to rewrite the other file. That would save us one complete rewrite and should improve performance by 50% on such filesystems. What do you think?

This should use fiemap to support file system like btrfs or others that do not support fibmap. I've created a patch but I'm not sure that it works as intended. I cannot quite grok your code yet. If interested, I can publish it.

Not every FS support the fallocate system call. Falling back to posix_fallocate would allow more compatibility at the expense of some performance penalty. But I think it's worth it. Maybe with a one-time warning?

└─$ cmake ..
CMake Error at CMakeLists.txt:83 (message):
You need libattr headers for Shake to build. They might be in a package
named libattr1-dev or libattr-devel.

-- Configuring incomplete, errors occurred!
See also "/home/jamy/Downloads/shake/build/CMakeFiles/CMakeOutput.log".
See also "/home/jamy/Downloads/shake/build/CMakeFiles/CMakeError.log".