unbrice / shake Goto Github PK
View Code? Open in Web Editor NEWShake is a defragmenter that runs in userspace
License: Other
Shake is a defragmenter that runs in userspace
License: Other
Shouldn't tmpfile
location be specified and actually put in /tmp/
or /var/tmp
instead of putting it in $HOME
?
Line 276 in 33dd09d
$sudo pv -d `pgrep shake`
3:/home/bartus/shakej9LlvJ: 252 B 0:19:05 [17MiB/s] [ <=> ]
When trying to add the ppa it does not work because Ubuntu xenial is not supported
Not every FS support the fallocate
system call. Falling back to posix_fallocate
would allow more compatibility at the expense of some performance penalty. But I think it's worth it. Maybe with a one-time warning?
I would say one of the prime use-cases for "shake" is systems so old that they don't support fe. ext4fs, and hence defrag4fs. Web is full of old articles, which tell you to use "shake" on such old ext3fs machines.
But what happens if you run it now after the patch d984848 ? Well, shake truncates the original file, after which the fallocate() fails. Modern glibc apparently "emulates" fallocate() in such cases, but it really does no good, and in such old systems there appear to be implementations which simply error out. Checking that real fallocate() works would therefore be a good idea. (CMake etc. checks aren't enough, because the syscall exists, it just errors when called on ext3fs)
Can someone please advise which version of Mint/ or similar Live CD I can use as a base and then install Shake (directly or after installing required dependencies) - And how?
Working steps please. Thanks.
The code uses the GNU style, which few projects use and looks unfamiliar to anyone else.
A pass through clang-format
or indent
should take care of that.
I want to throw in the following idea:
How about separating the judgement and the defrag part into two separate binaries? That way the judgement part would only spill out one filename per line which would be passed into the defrag binary on standard input. In the end, some filesystems require special handling of defragmentation and you probably don't want to put all this logic and heavy lifting into shake. The judgement part of shake itself becomes thus only a clever disk layout analyzer and file sorter. Also, the file system handling code could become a lot cleaner.
Both binaries could then be optimized on their own, shake could spawn multiple defrag binaries per spindle or per filesystem. It could also supply some stub binaries like defrag.xfs, defrag.btrfs to call out for the native tools of those filesystems as an option.
└─$ cmake ..
CMake Error at CMakeLists.txt:83 (message):
You need libattr headers for Shake to build. They might be in a package
named libattr1-dev or libattr-devel.
-- Configuring incomplete, errors occurred!
See also "/home/jamy/Downloads/shake/build/CMakeFiles/CMakeOutput.log".
See also "/home/jamy/Downloads/shake/build/CMakeFiles/CMakeError.log".
I understand the current approach of rewriting the file twice to keep the inode and all its attached attributes intact (xattr, stat, perms, acls). This is especially even more important for hardlinked files (those with nlink > 1) to not break the hardlinks as there seems to be no other way of keeping the hardlink (tho, I found that taking the lease on hardlinks doesn't work here).
But for non-hardlink cases we could try to redo the file from scratch, replicating all its attributes, and then rename it in place of the other to save one rewrite phase.
Additionally, this would get rid of the opportunity for shake to exit uncleanly and leave a damaged file behind (e.g. when killed by lease breaking). What do you think?
This should use fiemap to support file system like btrfs or others that do not support fibmap. I've created a patch but I'm not sure that it works as intended. I cannot quite grok your code yet. If interested, I can publish it.
Modern filesystems like XFS and btrfs support reflinking a file. Before rewriting the file, we could reflink the accused file to the rewrite source, then use that to rewrite the other file. That would save us one complete rewrite and should improve performance by 50% on such filesystems. What do you think?
OS: Linux Mint 18.03 64 bit (Ubuntu 16.04)
Dependencies: Installed libattr1-dev.
cmake ..
CMake Error at CMakeLists.txt:44 (message):
Cannot find help2man. Please install it.CMake Warning (dev) at CMakeLists.txt:26 (get_target_property):
Policy CMP0026 is not set: Disallow use of the LOCATION target property.
Run "cmake --help-policy CMP0026" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.The LOCATION property should not be read from target "shake". Use the
target name directly with add_custom_command, or use the generator
expression $<TARGET_FILE>, as appropriate.Call Stack (most recent call first):
CMakeLists.txt:50 (add_help2man_manpage)
This warning is for project developers. Use -Wno-dev to suppress it.CMake Warning (dev) at CMakeLists.txt:26 (get_target_property):
Policy CMP0026 is not set: Disallow use of the LOCATION target property.
Run "cmake --help-policy CMP0026" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.The LOCATION property should not be read from target "unattr". Use the
target name directly with add_custom_command, or use the generator
expression $<TARGET_FILE>, as appropriate.Call Stack (most recent call first):
CMakeLists.txt:51 (add_help2man_manpage)
This warning is for project developers. Use -Wno-dev to suppress it.CMake Error at CMakeLists.txt:87 (message):
You need libattr headers for Shake to build. They might be in a package
named libattr1-dev or libattr-devel.-- Configuring incomplete, errors occurred!
See also "/home/antonio/projects/github/others/shake/build/CMakeFiles/CMakeOutput.log".
See also "/home/antonio/projects/github/others/shake/build/CMakeFiles/CMakeError.log".
I'm not sure if the current locking implementation with leases is the right thing to do. Shake should acquire an unbreakable file lock because otherwise it may mess up with your files in case of a timeout and you cannot let it run unattended.
The number one priority of a defragger should be safety of the users data, and this is currently not implemented at a convenient level. Other defraggers like btrfs-defrag and xfs_fsr use unbreakable locks to ensure the files don't get messed up.
If some other application has to access the file, it has to wait. The current implementation allows the other application to break the lock, and what that application find is a broken, empty, or partial file. At least during rewrite phase the lock has to be unbreakable.
$> shake --pretend foo.bin
shake: foo.bin: FIBMAP failed: Operation not permitted
Maybe "You have to execute shake as root" would be a more meaningful error message.
$> which shake
/usr/bin/shake
Consequently the binary should be placed into /usr/sbin
ZFS being a copy on write filesystem has many problems with fragmentation the older the system gets. the standard answer usually is to copy everything (send/receive) which basically is backing up and restoring the filesystem. A user space tool like shake which I've used before on other FS's would greatly help here.
I pulled the current copy from github to see what it would do and pretty much every file handle gets the same error:
IDEAL START END FRAGC CRUMBC AGE SHOCKED NAME
./shake: /var/log/clientlogs/test/checkpoint/backups/log.20170422010001.tgz: FIBMAP failed: Invalid argument
./shake: /var/log/clientlogs/test/checkpoint/backups/log.20170424010001.tgz: FIBMAP failed: Invalid argument
./shake: /var/log/clientlogs/test/checkpoint/backups/log.20170426010001.tgz: FIBMAP failed: Invalid argument
./shake: /var/log/clientlogs/test/checkpoint/backups/log.20170427010001.tgz: FIBMAP failed: Invalid argument
./shake: /var/log/clientlogs/test/checkpoint/backups/log.20170423010001.tgz: FIBMAP failed: Invalid argument
./shake: /var/log/clientlogs/test/checkpoint/backups/log.20170425010001.tgz: FIBMAP failed: Invalid argument
./shake: /var/log/clientlogs/test/syslog/2017/20170427-kern.log: FIBMAP failed: Invalid argument
This is running under a ubuntu 16.04.2LTS 64bit system.
Linux loki 4.4.0-71-generic #92-Ubuntu SMP Fri Mar 24 12:59:01 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
ii libzfs2 0.6.5.7-1trusty amd64 Native OpenZFS filesystem library for Linuxtrusty amd64 Native ZFS filesystem metapackage for Ubuntu.
ii ubuntu-zfs 8
ii zfs-dkms 0.6.5.7-1trusty amd64 Native OpenZFS filesystem kernel modules for Linuxtrusty amd64 Native OpenZFS filesystem documentation and examples.
ii zfs-doc 0.6.5.7-1
ii zfsutils 0.6.5.7-1~trusty amd64 Native OpenZFS management utilities for Linux
The current version of shake uses error()
from error.h
, and is therefore unsupported on musl.
Would there be a plan to use more portable alternatives (err.h
seems one of them)?
Here are the logs of compiling shake on my system. It includes mostly some deprecation warnings.
[ 7%] Building C object CMakeFiles/shake.dir/executive.o
[ 14%] Building C object CMakeFiles/shake.dir/judge.o
[ 21%] Building C object CMakeFiles/shake.dir/linux.o
/home/celelibi/code/shake/linux.c: In function ‘set_ptime’:
/home/celelibi/code/shake/linux.c:175:3: warning: ‘attr_setf’ is deprecated: Use fsetxattr instead [-Wdeprecated-declarations]
175 | return attr_setf (fd, "shake.ptime", (char *) &date, DATE_SIZE,
| ^~~~~~
In file included from /home/celelibi/code/shake/linux.c:32:
/usr/include/attr/attributes.h:143:12: note: declared here
143 | extern int attr_setf (int __fd, const char *__attrname,
| ^~~~~~~~~
/home/celelibi/code/shake/linux.c: In function ‘get_ptime’:
/home/celelibi/code/shake/linux.c:186:7: warning: ‘attr_getf’ is deprecated: Use fgetxattr instead [-Wdeprecated-declarations]
186 | attr_getf (fd, "shake.ptime", (char *) &date, &size, ATTR_DONTFOLLOW))
| ^~~~~~~~~
In file included from /home/celelibi/code/shake/linux.c:32:
/usr/include/attr/attributes.h:131:12: note: declared here
131 | extern int attr_getf (int __fd, const char *__attrname, char *__attrvalue,
| ^~~~~~~~~
[ 28%] Building C object CMakeFiles/shake.dir/main.o
/home/celelibi/code/shake/main.c: In function ‘main’:
/home/celelibi/code/shake/main.c:293:13: warning: variable ‘jugement’ set but not used [-Wunused-but-set-variable]
293 | int jugement;
| ^~~~~~~~
[ 35%] Building C object CMakeFiles/shake.dir/msg.o
[ 42%] Building C object CMakeFiles/shake.dir/signals.o
[ 50%] Linking C executable shake
[ 50%] Built target shake
[ 57%] Building C object CMakeFiles/unattr.dir/executive.o
[ 64%] Building C object CMakeFiles/unattr.dir/linux.o
/home/celelibi/code/shake/linux.c: In function ‘set_ptime’:
/home/celelibi/code/shake/linux.c:175:3: warning: ‘attr_setf’ is deprecated: Use fsetxattr instead [-Wdeprecated-declarations]
175 | return attr_setf (fd, "shake.ptime", (char *) &date, DATE_SIZE,
| ^~~~~~
In file included from /home/celelibi/code/shake/linux.c:32:
/usr/include/attr/attributes.h:143:12: note: declared here
143 | extern int attr_setf (int __fd, const char *__attrname,
| ^~~~~~~~~
/home/celelibi/code/shake/linux.c: In function ‘get_ptime’:
/home/celelibi/code/shake/linux.c:186:7: warning: ‘attr_getf’ is deprecated: Use fgetxattr instead [-Wdeprecated-declarations]
186 | attr_getf (fd, "shake.ptime", (char *) &date, &size, ATTR_DONTFOLLOW))
| ^~~~~~~~~
In file included from /home/celelibi/code/shake/linux.c:32:
/usr/include/attr/attributes.h:131:12: note: declared here
131 | extern int attr_getf (int __fd, const char *__attrname, char *__attrvalue,
| ^~~~~~~~~
[ 71%] Building C object CMakeFiles/unattr.dir/signals.o
[ 78%] Building C object CMakeFiles/unattr.dir/unattr.o
/home/celelibi/code/shake/unattr.c: In function ‘strip’:
/home/celelibi/code/shake/unattr.c:114:5: warning: ‘attr_removef’ is deprecated: Use fremovexattr instead [-Wdeprecated-declarations]
114 | attr_removef (fd, *attr, ATTR_DONTFOLLOW);
| ^~~~~~~~~~~~
In file included from /home/celelibi/code/shake/unattr.c:29:
/usr/include/attr/attributes.h:155:12: note: declared here
155 | extern int attr_removef (int __fd, const char *__attrname, int __flags)
| ^~~~~~~~~~~~
[ 85%] Linking C executable unattr
[ 85%] Built target unattr
[ 92%] Building manpage for shake
[100%] Building manpage for unattr
[100%] Built target doc
I'll let you be the juge. But I guess that in not too long, the CMakeFiles will stop working. Especially given that the last commit was in 2017, I guess the sooner the update, the better.
Here is the output of cmake 3.22.1.
CMake Deprecation Warning at CMakeLists.txt:19 (CMAKE_MINIMUM_REQUIRED):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
CMake Deprecation Warning at CMakeLists.txt:21 (cmake_policy):
The OLD behavior for policy CMP0005 will be removed from a future version
of CMake.
The cmake-policies(7) manual explains that the OLD behaviors of all
policies are deprecated and that a policy should be set to OLD only under
specific short-term circumstances. Projects should be ported to the NEW
behavior and not rely on setting a policy to OLD.
-- The C compiler identification is GNU 11.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
CMake Warning (dev) at CMakeLists.txt:26 (get_target_property):
Policy CMP0026 is not set: Disallow use of the LOCATION target property.
Run "cmake --help-policy CMP0026" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.
The LOCATION property should not be read from target "shake". Use the
target name directly with add_custom_command, or use the generator
expression $<TARGET_FILE>, as appropriate.
Call Stack (most recent call first):
CMakeLists.txt:50 (add_help2man_manpage)
This warning is for project developers. Use -Wno-dev to suppress it.
CMake Warning (dev) at CMakeLists.txt:26 (get_target_property):
Policy CMP0026 is not set: Disallow use of the LOCATION target property.
Run "cmake --help-policy CMP0026" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.
The LOCATION property should not be read from target "unattr". Use the
target name directly with add_custom_command, or use the generator
expression $<TARGET_FILE>, as appropriate.
Call Stack (most recent call first):
CMakeLists.txt:51 (add_help2man_manpage)
This warning is for project developers. Use -Wno-dev to suppress it.
-- Looking for attr_setf
-- Looking for attr_setf - found
-- Looking for fallocate
-- Looking for fallocate - found
-- Configuring done
-- Generating done
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.