tex / fusecompress Goto Github PK
View Code? Open in Web Editor NEWThis project is not developed anymore. Transparent (using fuse) compressing filesystem.
Home Page: https://github.com/tex/fusecompress
License: Other
This project is not developed anymore. Transparent (using fuse) compressing filesystem.
Home Page: https://github.com/tex/fusecompress
License: Other
FuseCompress (compressed filesystem in userspace) We are not responsible for any badness that may happen to you during using this software. Todo: Bugs: File becames unreadable when it hits a not enough space condition. So be careful to really have enough free space on the disk to store all the data. This is more a feature than a bug: In CFile *FileManager::GetUnlocked(const char *name, bool create) FuseCompress uses the underlying filesystem inode number. This is a problem when you have multiple partitions underneath. For example /backups/.max (partition 1) /backups/.max/alex (partition 2) /backups/.max -> /backup/max (fusecompress) This would spoil the inode pool and potentially lead to conflicts, especially with hard links. Problems: Do NOT compile FuseCompress with g++-4.3.3! Strange threading problems (broken locking) were detected (thank you Steve!) when using this version of g++. There is a memory leak (not too big, user detects it after milions of copyied files) in 'file' package, more exactly in it's magic library, see: http://mx.gw.com/pipermail/file/2009/000306.html ==13747== 1,356 bytes in 135 blocks are definitely lost in loss record 9 of 12 ==13747== at 0x4C256AE: malloc (in /usr/lib64/valgrind/amd64-linux/vgpreload_memcheck.so) ==13747== by 0x725503F: __vasprintf_chk (in /lib64/libc-2.9.so) ==13747== by 0x7254ED2: __asprintf_chk (in /lib64/libc-2.9.so) ==13747== by 0x63C2993: file_printf (in /usr/lib64/libmagic.so.1.0.0) ==13747== by 0x63BBDC5: (within /usr/lib64/libmagic.so.1.0.0) ==13747== by 0x63BC6A7: file_softmagic (in /usr/lib64/libmagic.so.1.0.0) ==13747== by 0x63C2B8A: file_buffer (in /usr/lib64/libmagic.so.1.0.0) ==13747== by 0x63B6104: magic_buffer (in /usr/lib64/libmagic.so.1.0.0) ==13747== by 0x41F9CF: CompressedMagic::isNativelyCompressed(char const*, int) (CompressedMagic.cpp:140) ==13747== by 0x42A44F: Compress::write(char const*, unsigned long, long) (Compress.cpp:361) ==13747== by 0x43B99E: Memory::write(bool) (Memory.cpp:203) ==13747== by 0x43BC77: Memory::merge(char const*) (Memory.cpp:53) ==13747== This leak should be fixed with the newer 'file' package. Requires: boost >= 1.33.1 libz libbz2 xz-4.999.9beta (lzma library from http://tukaani.org/xz/xz-4.999.9beta.tar.gz) liblzo2 Example (with apt-get): $ sudo apt-get install libboost-dev libz-dev liblzo2-dev libmagic-dev libfuse-dev -y (also install the xz-beta manually, above) Note that 32-bit users may need to recompile boost by hand for long file support (see below). Compile: Typically you only need to run autoreconf -fiv && ./configure && make && sudo make install Boost's libraries may be stored in /usr/lib64 on 64bit systems, configure script needs to be called with: ./configure --with-boost-libdir=/usr/lib64/ Don't use --enable-debug flag. Build fails during linking with boost libraries. If you don't have pkg-tool installed, try following: FUSE_CFLAGS="-D_GNU_SOURCE -D_REENTRANT -D_POSIX_C_SOURCE=200112L -D_POSIX_SOURCE -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=500 -pedantic -Wno-long-long -Wall -fpermissive -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26" FUSE_LIBS="-lfuse" ./configure --with-boost=/usr/include/ --with-boost-libdir=/usr/lib64 If your boost library was compiled for 32bit system without support for long files (bigger than 2GiB) configure script prints this error message: Boost library compiled for 32bit architecture without long file support! It means that any program using this library is limited to work with files smaller than 2GiB. You can either ignore this problem by passing --enable-crippled_boost to FuseCompress' configure script or compile boost with long file support. In both cases, try to report this problem to authors of your Linux distribution. There are switches you can use to build boost library with long file support: ./bjam "define=_FILE_OFFSET_BITS=64" release --toolset=gcc --build-type=minimal --layout=system --libdir=/usr/lib/ --prefix=/usr sudo ./bjam "define=_FILE_OFFSET_BITS=64" release --toolset=gcc --build-type=minimal --layout=system --libdir=/usr/lib/ --includedir=/usr/include/boost --prefix=/usr install Usage: See man pages for fusecompress and fusecompress_offline. Author: Milan Svoboda <[email protected]> (author and project maintainer) Feel free to contact me with suggestions and bug reports.
Various pieces of the documentation as well as the description here on github point to miio.net, however the later doesn't resolve anymore.
If you intend to let go of miio.net it'd be good to all drop the references to it from the docu and possibly move any content concerning fusecompress to a wiki at github?
Thanks,
*t
Is this multithreaded? even with lzma?
I get this problem after upgrading on debian unstable:
fusecompress_offline test_file
Processing file (test_file)
Temporary file (4Rw1jx)
int Compress::open(const char*, int): Failed to restore LayerMap of file 'test_file', exception: requested integer size exceeds type size: 2
File (test_file) cannot be opened! (Input/output error)
fuse2: 2.8.5
fusecompress 2.6.git913897f4-1
libboost1.46-dev 1.46.1-6
linux 2.6.39
Please consider adding FreeArc support. It has fantastic speed and compression, see: http://freearc.org/Maximal-Practical-Compression.aspx and https://plus.google.com/104946298099779869987/posts/NoVxas8ncpE
Using fusecompress 2.6-2 from debian.
On a clean bzr branch / checkout, doing bzr status two times in a row causes:
virtual Compress::~Compress(), picard/.bzr/checkout/conflicts
FuseCompress::open 0x7fd6ac002540 name: picard/.bzr/checkout/dirstate
Compress::open m_refs: 2
FuseCompress::write 0x7fd6ac002540 name: picard/.bzr/checkout/dirstate, size: 0x1000, offset: 0x0
Memory::write(picard/.bzr/checkout/dirstate) | m_FileSize: 0x127b4, size: 0x1000, offset: 0x0
FuseCompress::write 0x7fd6ac002540 name: picard/.bzr/checkout/dirstate, size: 0x1000, offset: 0x1000
Memory::write(picard/.bzr/checkout/dirstate) | m_FileSize: 0x127b4, size: 0x1000, offset: 0x1000
[repeated in 0x1000 increments, until]
FuseCompress::write 0x7fd6ac002540 name: picard/.bzr/checkout/dirstate, size: 0x7b3, offset: 0x12000
Memory::write(picard/.bzr/checkout/dirstate) | m_FileSize: 0x127b4, size: 0x7b3, offset: 0x12000
FuseCompress::truncate file picard/.bzr/checkout/dirstate, to size: 127b3
truncate name: picard/.bzr/checkout/dirstate, m_IsCompressed: 1, size: 127b3
void Compress::DefragmentFast()
offset: 0x0, coffset: 0x19, size: 0x127b3
length: 0x5744
ASSERT off_t Compress::cleverCopy(int, off_t, int, LayerMap&) (r == writeOffset)
Aborted
running "sudo make install" twice results in...
make[3]: Entering directory `/home/rdp/dev/downloads/fusecompress/src' cd /usr/local/bin && \ ln -s fusecompress mount.fusecompress ln: creating symbolic link `mount.fusecompress': File exists make[3]: *** [install-exec-hook] Error 1 make[3]: Leaving directory `/home/rdp/dev/downloads/fusecompress/src' make[2]: *** [install-exec-am] Error 2 make[2]: Leaving directory `/home/rdp/dev/downloads/fusecompress/src' make[1]: *** [install-am] Error 2 make[1]: Leaving directory `/home/rdp/dev/downloads/fusecompress/src' make: *** [install-recursive] Error 1
I believe.
Thanks!
-r
Hi ;
I'm stuck on this error while trying the ./configure
:
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking for style of include used by make... GNU
checking dependency style of g++... gcc3
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking whether ln -s works... yes
checking for boostlib >= 1.33.1... yes
checking build system type... x86_64-unknown-linux-gnu
checking whether the Boost::System library is available... yes
configure: WARNING: BOOST_CPPFLAGS -I/usr/include
checking whether the Boost::Serialization library is available... yes
configure: error: Could not link against !
my configuration : Ubuntu server 14.04 amd64
I've tried all things found on the internet, such as :
./configure --with-boost-libdir=/usr/lib/x86_64-linux-gnu
./configure --with-boost-libdir=/usr/lib/
./configure --with-boost-libdir=/usr/lib64/
but none worked.
could you provide an up-to-date tutorial for compiling please ?
there is no other documentation that this README, all other sites are down.
thanks.
It appears at some point I added lz4 compression at https://github.com/reflectivedevelopment/fusecompress
If you are interested in testing it out and merging it in.
Code needs some minor fix, I get these errors:
In file included from Compress.cpp:39:
./boost/iostreams/device/nonclosable_file_descriptor.hpp: In constructor ‘boost::iostreams::nonclosable_file_descriptor::nonclosable_file_descriptor(const std::string&, std::_Ios_Openmode, std::_Ios_Openmode)’:
./boost/iostreams/device/nonclosable_file_descriptor.hpp:34: error: no matching function for call to ‘boost::iostreams::file_descriptor::file_descriptor(const std::basic_string<char, std::char_traits, std::allocator >&, std::_Ios_Openmode&, std::_Ios_Openmode&)’
/usr/include/boost/iostreams/device/file_descriptor.hpp:81: note: candidates are: boost::iostreams::file_descriptor::file_descriptor(const boost::iostreams::file_descriptor&)
/usr/include/boost/iostreams/device/file_descriptor.hpp:66: note: boost::iostreams::file_descriptor::file_descriptor(const char*, std::_Ios_Openmode)
/usr/include/boost/iostreams/device/file_descriptor.hpp:61: note: boost::iostreams::file_descriptor::file_descriptor(const std::string&, std::_Ios_Openmode)
/usr/include/boost/iostreams/device/file_descriptor.hpp:55: note: boost::iostreams::file_descriptor::file_descriptor(boost::iostreams::detail::file_handle, bool)
/usr/include/boost/iostreams/device/file_descriptor.hpp:52: note: boost::iostreams::file_descriptor::file_descriptor()
can you implement XZ compression, instead of just lzma, or is that redundant.
Hello!
[575211.244134] fusecompress[17389]: segfault at 10 ip 00007f89f66ee00f sp 00007f89a17f9a80 error 4 in libpthread-2.18.so[7f89f66e2000+18000]
Anyone else catch that fun?
kernel 3.13.6, libc6 2.18-4, latest fusecompress.
Hi,
after upgrading to boost 1.71.0 on my Gentoo system, configuration fails.
The error message is not really helpful
configure: error: Could not link against !
Has anybody a fix for the new boost version?
Many thanks,
Helmut
Got this error message when trying to build with boost1.64 (linux,gcc7)
g++ -DHAVE_CONFIG_H -I. -I.. -D_GNU_SOURCE -D_REENTRANT -D_POSIX_C_SOURCE=200112L -D_POSIX_SOURCE -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=500 -Wno-long-long -Wall -fpermissive -D_FILE_OFFSET_BITS=64 -DFUSE_USE_VERSION=26 -DNDEBUG -g -O2 -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.cpp
In file included from /usr/include/boost/assert.hpp:58:0,
from /usr/include/boost/iostreams/detail/adapter/range_adapter.hpp:16,
from /usr/include/boost/iostreams/detail/push.hpp:19,
from /usr/include/boost/iostreams/filtering_stream.hpp:19,
from CompressionType.hpp:23,
from main.cpp:30:
/usr/include/boost/array.hpp: In member function ‘T& boost::array<T, N>::operator[](boost::array<T, N>::size_type)’
/usr/include/boost/array.hpp:118:20: error: expected primary-expression before ‘if’
return BOOST_ASSERT_MSG( i < N, "out of range" ), elems[i];
^
/usr/include/boost/array.hpp:118:20: error: expected ‘;’ before ‘if’
Looks like definition of assert() collide with boost assertion macros.
# define BOOST_ASSERT_MSG(expr, msg) assert((expr)&&(msg))
which in turns cause an error in operator[] in boost array.hpp
reference operator[](size_type i)
{
return BOOST_ASSERT_MSG( i < N, "out of range" ), elems[i];
}
One solution would be to add #define BOOST_DISABLE_ASSERTS
in main.cpp, Compress.cpp
to disable boost assertion all together.
running:
fusecompress_offline -o fc_c:lzma ./folder_with_many_files_in_it
the command eventually ends with:
Processing file (/home/rdp/installs/./mbari_gembox_187/lib/ruby/gems/1.8/cache/cruby-0.0.1.gem)
Temporary file (/home/rdp/installs/./mbari_gembox_187/lib/ruby/gems/1.8/cache/rZdTII)
Not compressed
File (/home/rdp/installs/./mbari_gembox_187/lib/ruby/gems/1.8/cache/rZdTII) cannot be opened! (Too many open files)
Thanks!
configure: WARNING: BOOST_CPPFLAGS -I/usr/include
checking whether the Boost::Serialization library is available... yes
checking for exit in -lboost_serialization... no
configure: error: Could not link against boost_serialization !
fusecompress 0.9x had the "-l" switch to choose the compression level. Fusecompress 2.x doesn't have that option available any more.
Is there a rationale for dropping that feature? Was it maybe dropped because the feature was not considered top priority at the time of the rewrite and was just not added yet? It'd be interesting to re-add the feature to be able to have files compressed at maximum pressure.
Thanks,
*t
Hi,
Built fusecompress on Linux Mint 19.
I do:
$ cd /backup;
$ mkdir fc fcu
$ fusecompress -o fc_d fc fcu
In another terminal, I do:
$ cp x.txt /backup/fcu
==> crash of fusecompress with the following stacktrace:
FuseCompress::mknod name: x.txt
FuseCompress::mknod inode: 41293005
new CFile(..., x.txt)
C (x.txt), raw/user 0x19/0x0 bytes
getattr name: x.txt, m_IsCompressed: 1, m_fh.size: 0x0
Memory::getattr(0x7fc0fc001350) m_FileSize: 0x0, m_FileSizeSet: 0
virtual Compress::~Compress(), x.txt
new CFile(..., x.txt)
C (x.txt), raw/user 0x19/0x0 bytes
FuseCompress::open 0x7fc104001230 name: x.txt
File::open file 'x.txt', inode 41293005, m_refs: 1
Compress::open m_refs: 1
getattr name: x.txt, m_IsCompressed: 1, m_fh.size: 0x0
*** stack smashing detected ***: terminated
Abandon (core dumped)
So I unmounted fcu:
$ fusermount -u /backup/fcu
then restart fusecompress (as above)
then:
$ ls /backup/fcu
x.txt
$ ls /backup/fc
x.txt
$ more /backup/fcu/x.txt
Again, crash with the stack:
new CFile(..., x.txt)
C (x.txt), raw/user 0x19/0x0 bytes
getattr name: x.txt, m_IsCompressed: 1, m_fh.size: 0x0
Memory::getattr(0x7fc7c8001630) m_FileSize: 0x0, m_FileSizeSet: 0
virtual Compress::~Compress(), x.txt
new CFile(..., x.txt)
C (x.txt), raw/user 0x19/0x0 bytes
getattr name: x.txt, m_IsCompressed: 1, m_fh.size: 0x0
Memory::getattr(0x7fc7c0001500) m_FileSize: 0x0, m_FileSizeSet: 0
virtual Compress::~Compress(), x.txt
new CFile(..., x.txt)
C (x.txt), raw/user 0x19/0x0 bytes
getattr name: x.txt, m_IsCompressed: 1, m_fh.size: 0x0
Memory::getattr(0x7fc7c0001500) m_FileSize: 0x0, m_FileSizeSet: 0
virtual Compress::~Compress(), x.txt
new CFile(..., x.txt)
C (x.txt), raw/user 0x19/0x0 bytes
FuseCompress::open 0x7fc7c8001630 name: x.txt
File::open file 'x.txt', inode 41293005, m_refs: 1
Compress::open m_refs: 1
getattr name: x.txt, m_IsCompressed: 1, m_fh.size: 0x0
*** stack smashing detected ***: terminated
Abandon (core dumped)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.