yumeyao / lz4 Goto Github PK
View Code? Open in Web Editor NEWThis project forked from lz4/lz4
lz4 with multi thread support
Home Page: https://github.com/yumeyao/lz4/pull/1
License: Other
This project forked from lz4/lz4
lz4 with multi thread support
Home Page: https://github.com/yumeyao/lz4/pull/1
License: Other
dictionary is read only (const *), so it should be ok for multi-thread compression.
maybe still we can use current model with 1 thread, so that I/O can be done in a separate thread with input prefetched.
or, divide the block_num of input evenly into N threads, then each thread compress the input into a stream containing continous blocks. then combine multiple streams together (skippable streams? compatibility issues?)
Is your feature request related to a problem? Please describe.
I managed to get this to build on macOS14 (ARM) after some wrangling and it does work! It's faster than single threaded version, well done :)
Would be great to document how to build better.
These are the commands to build that worked:
export CXXFLAGS="-std=c++11"
make clean ENABLE_MT=yes
make ENABLE_MT=yes
sudo make uninstall ENABLE_MT=yes
sudo make install ENABLE_MT=yes
But only after creating an endian.h
file at the following location:
sudo mkdir /usr/local/include
sudo vim /usr/local/include/endian.h
with the following content:
#ifndef __FINK_ENDIANDEV_PKG_ENDIAN_H__
#define __FINK_ENDIANDEV_PKG_ENDIAN_H__ 1
/** compatibility header for endian.h
* This is a simple compatibility shim to convert
* BSD/Linux endian macros to the Mac OS X equivalents.
* It is public domain.
* */
#ifndef __APPLE__
#warning "This header file (endian.h) is MacOS X specific.\n"
#endif /* __APPLE__ */
#include <libkern/OSByteOrder.h>
#define htobe16(x) OSSwapHostToBigInt16(x)
#define htole16(x) OSSwapHostToLittleInt16(x)
#define be16toh(x) OSSwapBigToHostInt16(x)
#define le16toh(x) OSSwapLittleToHostInt16(x)
#define htobe32(x) OSSwapHostToBigInt32(x)
#define htole32(x) OSSwapHostToLittleInt32(x)
#define be32toh(x) OSSwapBigToHostInt32(x)
#define le32toh(x) OSSwapLittleToHostInt32(x)
#define htobe64(x) OSSwapHostToBigInt64(x)
#define htole64(x) OSSwapHostToLittleInt64(x)
#define be64toh(x) OSSwapBigToHostInt64(x)
#define le64toh(x) OSSwapLittleToHostInt64(x)
#endif /* __FINK_ENDIANDEV_PKG_ENDIAN_H__ */
(e.g. from https://gist.github.com/yinyin/2027912)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.