tsuraan / jerasure Goto Github PK

Hi,

I appreciate your work. Would it be possible to use Jerasure in the GNU GPL project nubisave? The license in use does not seem to be compatible with GPL, so I would like to ask for your permission first.
Nubisave: (Virtual file system with CRS support and cloud storage integration)
https://github.com/joe42/nubisave/tree/master
www.nubisave.org
I currently use the implementation from jigdfs (10.1109/CICYBS.2009.4925093), but it has some weaknesses. Therefore I should like to substitute it with Jerasure library.

I appreciate your work,
joe42

I know that N+2(2<=N<=16) works well.But I want to know whether Jerasure supports 9+3.

Hi,

It would be nice to have a v2 branch, to separate development from production. The idea is to point to this branch as a submodule of the http://github.com/ceph/ceph/ repository instead of copying.

Cheers

make
make -C src
make[1]: Entering directory /home/loic/software/ceph/jerasure/tsuraan/src' mkdir -p ../lib gcc -g -O3 -fPIC -c galois.c gcc -g -O3 -fPIC -c jerasure.c gcc -g -O3 -fPIC -c reed_sol.c gcc -g -O3 -fPIC -c cauchy.c gcc -g -O3 -fPIC -c liberation.c gcc -shared -Wl,-soname,libJerasure.so.0 \ -o ../lib/libJerasure.so.0 galois.o jerasure.o reed_sol.o cauchy.o liberation.o ln -sf libJerasure.so.0 ../lib/libJerasure.so mkdir -p "../include/" cp *.h ../include/ make[1]: Leaving directory/home/loic/software/ceph/jerasure/tsuraan/src'
make -C Examples
make[1]: Entering directory /home/loic/software/ceph/jerasure/tsuraan/Examples' gcc -g -I../include -L../lib -lJerasure -o jerasure_01 jerasure_01.c /tmp/cc0CalcX.o: In functionmain':
/home/loic/software/ceph/jerasure/tsuraan/Examples/jerasure_01.c:82: undefined reference to galois_single_multiply' /home/loic/software/ceph/jerasure/tsuraan/Examples/jerasure_01.c:85: undefined reference tojerasure_print_matrix'
collect2: error: ld returned 1 exit status
make[1]: *** [jerasure_01] Error 1
make[1]: Leaving directory `/home/loic/software/ceph/jerasure/tsuraan/Examples'
make: *** [all] Error 2

Hi all,

I am using jerasure v 2.0 and gf-complete in a project at my lab.
I have been able to build the libraries.
When I try to link the libraries to my code, I always get an linker error "undefined reference" whenever I try to use any method in the libraries.
My linker command is
g++ -L/media/sathish/6209014a-ae4c-4f58-90be-2085c0162e351/home/sathish/temp/live-proxy/live-core-lib/jerasure/src/.libs/ -L/media/sathish/6209014a-ae4c-4f58-90be-2085c0162e351/home/sathish/temp/live-proxy/live-core-lib/gf-complete/src/.libs/tmp -o "Testing123" ./abcd.o ./timeLogging.o -lgf_complete -lJerasure

Thanks,

Encode using blaum_roth segmentation fault.
k=10, m=4, w=16.

====my patch====
diff -u ../Jerasure-master-ori/Jerasure-master/Examples/decoder.c Examples/decoder.c
--- ../Jerasure-master-ori/Jerasure-master/Examples/decoder.c 2011-11-27 07:24:10.000000000 +0800
+++ Examples/decoder.c 2013-02-04 10:54:34.375865798 +0800
@@ -229,7 +229,7 @@
bitmatrix = liberation_coding_bitmatrix(k, w);
break;
case Blaum_Roth:

•       bitmatrix = blaum_roth_coding_bitmatrix(k, w);
  

•       bitmatrix = blaum_roth_coding_bitmatrix(k, w, m);
      break;
  case Liber8tion:
      bitmatrix = liber8tion_coding_bitmatrix(k);
  

  diff -u ../Jerasure-master-ori/Jerasure-master/Examples/encoder.c Examples/encoder.c
  --- ../Jerasure-master-ori/Jerasure-master/Examples/encoder.c 2011-11-27 07:24:10.000000000 +0800
  +++ Examples/encoder.c 2013-02-04 10:51:58.887225000 +0800
  @@ -455,7 +455,7 @@
  schedule = jerasure_smart_bitmatrix_to_schedule(k, m, w, bitmatrix);
  break;
  case Blaum_Roth:

•       bitmatrix = blaum_roth_coding_bitmatrix(k, w);
  

•       bitmatrix = blaum_roth_coding_bitmatrix(k, w, m);
      schedule = jerasure_smart_bitmatrix_to_schedule(k, m, w, bitmatrix);
      break;
  case Liber8tion:
  

  diff -u ../Jerasure-master-ori/Jerasure-master/src/liberation.c src/liberation.c
  --- ../Jerasure-master-ori/Jerasure-master/src/liberation.c 2011-11-27 07:24:10.000000000 +0800
  +++ src/liberation.c 2013-02-04 10:53:04.594043000 +0800
  @@ -208,15 +208,15 @@
  return matrix;
  }

-int *blaum_roth_coding_bitmatrix(int k, int w)
+int *blaum_roth_coding_bitmatrix(int k, int w, int m0)
{
int *matrix, i, j, index, l, m, p;

if (k > w) return NULL ;

• matrix = talloc(int, 2_k_w*w);
• matrix = talloc(int, m0_k_w*w);
  if (matrix == NULL) return NULL;
• bzero(matrix, sizeof(int)_2_k_w_w);
• bzero(matrix, sizeof(int)_m0_k_w_w);

/* Set up identity matrices */

diff -u ../Jerasure-master-ori/Jerasure-master/src/liberation.h src/liberation.h
--- ../Jerasure-master-ori/Jerasure-master/src/liberation.h 2011-11-27 07:24:10.000000000 +0800
+++ src/liberation.h 2013-02-04 10:54:03.603866097 +0800
@@ -51,6 +51,6 @@

extern int *liberation_coding_bitmatrix(int k, int w);
extern int *liber8tion_coding_bitmatrix(int k);
-extern int *blaum_roth_coding_bitmatrix(int k, int w);
+extern int *blaum_roth_coding_bitmatrix(int k, int w, int m0);

#endif

Hi. Are you still connected with the jerasure.org website? Only it seems to have become contaminated with some weird spam referencing “online casinos” rather cleverly insinuated among the actual mentions of the software.

According to the Wayback Machine, this seems to have happened sometime in 2021.

There are global variables in Galois.c, and lazy initialization in multiple functions.

The above problems make multithreaded coding crash.

I am trying to run the following command :
gf_mult f0f0f0f0f0f0f0f01313131313131313 1313131313131313f0f0f0f0f0f0f0f0 128h .
That is for multiplication for two 128 bit number.
But it gives me error -- Bad a

And if I decrease the 'a' and 'b' to 64 bit than it works fine.

So please suggest me what to do.

it should change the ax_ext.m4 file.

Jerasure/m4/ax_ext.m4
-      ecx=`echo $ax_cv_gcc_x86_cpuid_0x00000001 | cut -d ":" -f 3`
-      edx=`echo $ax_cv_gcc_x86_cpuid_0x00000001 | cut -d ":" -f 4`
+      ecx=0
+      edx=0
+      if test "$ax_cv_gcc_x86_cpuid_0x00000001" != "unknown";
+      then
+        ecx=`echo $ax_cv_gcc_x86_cpuid_0x00000001 | cut -d ":" -f 3`
+        edx=`echo $ax_cv_gcc_x86_cpuid_0x00000001 | cut -d ":" -f 4`
+      fi

kernel version

$ sudo uname -a 
Linux hilton-PC 5.17.3-amd64-desktop #1 SMP PREEMPT Fri Apr 29 15:22:29 CST 2022 x86_64 GNU/Linux

There are global variables in jerasure.c which are accessed without any help of atomic instructions or mutexes:

static double jerasure_total_xor_bytes = 0;
static double jerasure_total_gf_bytes = 0;
static double jerasure_total_memcpy_bytes = 0;

So if someone calls e.g. jerasure_matrix_dotprod() from two different threads in the same time, it's undefined behavior.

Maybe it makes sense to make stats collecting optional by wrapping it in #ifdef-endif?

I just discovered this excellent correction code implementation. I just started to explore it, when I discovered what appears to be a general assumption in the code that sizeof(long int)==32.

For example, in jerasure_05.c there’s a loop that fills the data blocks with psudo-random values:

  srand48(0);
  data = talloc(char *, k);
  for (i = 0; i < k; i++) {
    data[i] = talloc(char, size);
    for(j = 0; j < size; j+=sizeof(long)) {
        l = lrand48();
        memcpy(data[i] + j, &l, sizeof(long));
    }
  }

I’m compiling for 64-bit processors, and sizeof(long int) is 64. When stepping through this loop, the first four bytes get random data (because lrand48() returns a value between 0 and 2^31-1) and the next four bytes are zero (because the upper word of l will always be zero). The implicit assumption seems to be that sizeof(long)==32, consist with lrand48().

I can find nothing in the documentation that explicitly mentions the assumed size of a long int, but it seems telling that in most examples w is restricted one of either 8, 16, or 32 and all size parameters are assumed to be multiples of sizeof(long).

Hi, would you be so kind as to update the description of this repository to set the upstream to be http://jerasure.org/ ? The former upstream does not contain useful information anymore.

after creation of erasure code:
encoder out1.dat 9 2 cauchy_good 9 1024 500000

if you go into /coding directory and delete one of the data file (ie: out1_k1.dat)
the decode actually work fine. (as shown below)

Decoding (MB/sec): 1780.2648556876
De_Total (MB/sec): 302.0309642111

The problem is when you DO NOT delete any data file, basically decode without any missing segment
doing so, it will result in core dump

Core dump happen here, and cdf=0

for (x = 0; x < cdf; x++) {
drive = row_ids[x+ddf+k]-k;
ptr = real_decoding_matrix + k_w_w*(ddf+x);

w = 32 is the largest value supported by Jerasure.
How should I support larger finite fields by cauchy matrix? （such as w = 64）

Use following functions：

int *cauchy original coding matrix(k, m, w); 
int *jerasure matrix to bitmatrix(k, m, w, matrix); 
void jerasure schedule encode(k, m, w, schedule, data ptrs, coding ptrs, size, packetsize); 
int jerasure schedule decode lazy(k, m, w bitmatrix, erasures, data ptrs, coding ptrs, size, packetsize, int
smart);

I need a cross-platform reed-solomon library support linux/osx/bsd/windows/android/ios.
Thank you!

Hi,

I have had problems with the original Makefiles. So, I have fixed them and there is the new ones (I have seperated the headers files in the include directory):

# jerasure/Makefile

PREFIX=/usr/local
LIBDIR=${PREFIX}/lib
INCDIR=${PREFIX}/include

.PHONY: all clean install unistall lib

all: lib
    $(MAKE) -C Examples

clean:
    $(MAKE) -C src clean
    $(MAKE) -C Examples clean

install: lib
    @cp -P lib/libJerasure.so* ${LIBDIR}
    @mkdir -p ${INCDIR}/jerasure
    @cp include/* ${INCDIR}/jerasure

uninstall: lib
    @rm ${LIBDIR}/libJerasure.so*
    @rmdir ${INCDIR}/jerasure

lib:
    $(MAKE) -C src

# jerasure/src/Makefile
CC      = gcc
CFLAGS  = -g -O3 -fPIC -Wall -Werror -I../include
SRCS    = galois.c jerasure.c reed_sol.c cauchy.c liberation.c
OBJS    = $(SRCS:.c=.o)
LIB     = libJerasure.so
LIBDIR  = ../lib

.PHONY: all lib ../lib compile

all: lib

lib: ../lib compile 
    $(CC) -shared -Wl,-soname,$(LIB) \
        -o $(LIBDIR)/$(LIB).0 $(OBJS)
    @ln -sf $(LIB).0 $(LIBDIR)/$(LIB)

../lib:
    @mkdir -p ../lib

clean:
    $(RM) $(RMFLAGS) $(LIBDIR)/$(LIB)* $(OBJS) $(wildcard *.d)

compile: $(OBJS)

%.d: %.c
    @$(SHELL) -ec '$(CC) -M $(CFLAGS) $< \
        | sed '\''s/\($*\)\.o[ :]*/\1.o $@ : /g'\'' > $@; \
        [ -s $@ ] || rm -f $@'

-include $(SRC:.c=.d)

# jerasure/Examples/Makefile
CC      = gcc
CFLAGS  = -g -Wall -Werror -I../include
LDFLAGS = -L../lib
LDLIBS  = -lJerasure
SRCS    = $(wildcard *.c)
OBJS    = $(SRCS:.c=.o)
TARGETS = $(SRCS:.c=)

all: $(TARGETS)

compile: lib $(OBJS)

clean:
    $(RM) $(RMFLAGS) $(OBJS) $(TARGETS) 

lib:
    $(MAKE) -C ../src/

I have tried to take off all warnings too, but one of them let doubt. In jerasure.c file there is the jerasure_smart_bitmatrix_to_schedule function, and the variable bestrow could be used without be initialized, that was the warning that the compiler gave when I use -Wall compiler's flag. So, I have initialized this variable with zero, but could that variable really pass through that two "for" without be initialized?

Thanks for your attention.

After cloning Jerasure from Mac and running the makefile supplied with Jerasure, the following linker error is received:
gcc -shared -Wl,-soname,libJerasure.so.0
-o ../lib/libJerasure.so.0 galois.o jerasure.o reed_sol.o cauchy.o liberation.o
ld: unknown option: -soname
collect2: ld returned 1 exit status
make: *** [../lib/libJerasure.so.0] Error 1

My architecture is (uname -a):
Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64

Hi,

Shouldn't all the .h files under /include be installed to /usr/local/include? Right now only jerasure.h is been installed, and I think it is because of line 10 in src/Makefile.am. Not completely sure since I don't know much of automake.

Thanks,

Alfonso

it would be nice to update the mirror on a regular basis from http://lab.jerasure.org/jerasure/jerasure.git . I would be happy to take care of this if you'd like. When searching for jerasure in google, your repository shows first and it currently gives the impression that the library has not been updated since 2014 :-)