dicl / veloxdfs

DHT-based Distributed File System for MapReduce Jobs

VeloxDFS {#mainpage}

---

VeloxDFS is a decentralized distributed file system based on ChordDHT and HDFS.

This distributed file system serves as the foundation and essential component of the Velox Big Data Framework (VBDF), however, it can be used independently of the other components such as VeloxMR (MapReduce Framework).

Key features of current VeloxDFS implementation includes:

• No central directory service such as in HDFS NameNode.
• Logical block representation where the size adapts to the current workload.
• HADOOP API to be used instead of HDFS.

VELOX ECOSYSTEM

Related projects

Project	Description	URL
VeloxMR	experimental MapReduce engine based on VeloxDFS	https://github.com/DICL/VeloxMR
eclipsed	deployment/debugging helper script writen in RUBY	https://github.com/DICL/eclipsed
velox-hadoop	VeloxDFS JAVA library for Hadoop	https://github.com/DICL/velox-hadoop
velox-deploy-ansible	ansible playbook to automatize velox/hadoop installation	https://github.com/DICL/velox-deploy-ansible
hadoop-etc	Hadoop configuration files to use VeloxDFS	https://github.com/vicentebolea/hadoop-etc
velox-report	Project to benchmark and log VeloxDFS performance	https://github.com/vicentebolea/velox-report

USAGE

To deploy the system please refer to velox command and its following options:

$ veloxd up|down|restart|status

The command line API has the following options:

$ veloxdfs put|get|cat|ls|rm|format|show

The C++ API can be found at vdfs.hh and DFS.hh files, as for the JAVA API, it can be found at src/java directory.

COMPILING & INSTALLING

Further information can be found it in: Installation

Compiling requirements

• C++14 support, this is: GCC >= 4.9, Clang >= 3.4, or ICC >= 16.0.
• Boost library >= 1.56.
• Sqlite3 library.
• GNU Autotools (Autoconf, Automake, Libtool).
• Unittest++ [optional].

For single user installation for developers

$ mkdir -p local_eclipse/{tmp,sandbox}                 # Create a sandbox directories
$ cd local_eclipse                                     # enter in the directory
$ git clone [email protected]:DICL/VeloxDFS.git           # Clone the project from github
$ cd VeloxDFS
$ sh autogen.sh                                        # Generate configure script 
$ cd ../tmp                                            # Go to building folder
$ sh ../VeloxDFS/configure --prefix=`pwd`/../sandbox # Check requirements and generate the Makefile

# If you get a boost error go the FAQ section of the README

### This last command will be needed whenever you want to recompile the source
$ make [-j#] install                                   # Compile & install add -j flag to speed up

Now edit in your ~/.bashrc or ~/.profile:

export PATH="/home/*..PATH/To/eclipse/..*/sandbox/bin":$PATH
export LIBRARY_PATH="/home/*..PATH/To/eclipse/..*/sandbox/lib"
export C_INCLUDE_PATH="/home/*..PATH/To/eclipse/..*/sandbox/include"

Default settings for VELOXDFS

"log" : {
  "type" : "LOG_LOCAL6"
  "name" : "ECLIPSE"
  "mask" : "DEBUG"
},

"cache" : {
  "numbin"      : 100,
  "size"        : 200000,
  "concurrency" : 1
},

"filesystem" : {
  "block"    : 137438953,
  "buffer"   : 512,
  "replica"  : 1
}

Further information can be found it in: Conf reference

FAQ

• Question : configure stops with errors related to boost library.

• Answer : It probably means that you do not have boost library installed in the default location, in such case you should specify the boost library location.

    sh ../VeloxDFS/configure --prefix ~/sandbox --with-boost=/usr/local --with-boost-libdir=/usr/local/lib
  

  In this example we assume that the boost headers are in /usr/local/include while the library files are inside /usr/local/lib.

AUTHORS

• Vicente Adolfo Bolea Sanchez
• MooHyeon Nam
• WonBae Kim
• KiBeom Jin
• Deukyeon Hwang
• Prof. Nam Beomseok
• INSTITUTION: DICL laboratory at UNIST

veloxdfs's Issues

Change usage of std::string to movable object

Constructor of std::string does not allow any moving. Instead, it always copies all data.
Except if pointer std::string is not used. However, that is used minorly in project.
This leads to too many unnecessary copying around data, which can be size of block (64mb).

Proposal, use custom class that will not be copying data, but moving it. Also, it will control number of owners and gets destructed when owners<=0

class Slice {
  char* ptr;
  uint64_t size;
  atomic<int> owners;
}

Other way is to use std::shared_ptrstd::string

Solving this will do some improvement to performance

Fixing dfs bugs

Fixing dfs bugs and changing block hash key part

feature-dfsAPI

Make dfs API

Eclipse Cache

• Cache using ASIO (#1)
• Cache using MPI (#2)

One definition rule in Job and task classes

@zwigul all the definitions should go into the cc file. Otherwise we will have linker errors. Can you make a bugfix branch fix it and pull request to master?

DFSput removes \n between the chunks.

Here I ilustrate how the error occurs, this is the diff of a file called config.h before inserting it with dfsput and after received with dfsget. I believe that the bug is in the dfsput.cc.

--- config.h    2016-03-09 15:34:48.967514994 +0900
+++ build/config.h      2016-03-09 15:31:52.664229099 +0900
@@ -29,6 +29,7 @@

 /* Define to 1 if you have the <memory.h> header file. */
 #define HAVE_MEMORY_H 1
+
 /* Define to 1 if you have the <netdb.h> header file. */
 #define HAVE_NETDB_H 1

@@ -65,6 +66,7 @@

 /* Name of package */
 #define PACKAGE "eclipse"
+
 /* Define to the address where bug reports for this package should be sent. */
 #define PACKAGE_BUGREPORT "[email protected]"

@@ -94,7 +96,8 @@
    #define below would cause a syntax error. */
 /* #undef _UINT32_T */

-/* Define for Solaris 2.5.1 so the uint8_t typedef from <sys/synch.h>,   <pthread.h>, or <semaphore.h> is not used. If the typedef were allowed, the
+/* Define for Solaris 2.5.1 so the uint8_t typedef from <sys/synch.h>,
+   <pthread.h>, or <semaphore.h> is not used. If the typedef were allowed, the
    #define below would cause a syntax error. */
 /* #undef _UINT8_T */

Additional configure option for logical block features

I added some options for logical block enabled build (--enable_lblocks & --enable-boost-static), but after the build completion, the error message is like below

Any classes that I should add?
Thanks.

Change dfsput/dfsget... to dfs [put|get|format|ls]

Adding doxygen

Doxygen is the only software which generates documentation from C++ code, we have to start commenting our functions, making those comment using doxygen syntax will be trivial.

Also github supports creating a website which our project documentation, class diagrams and so on.

Change raw pointers to smart pointers

Whenever it is possible it will be desirable to use smart pointers to avoid strange bugs cause by memory leaks

Remove commented code and use the same coding style

In the previous month we rushed to finish our main parts for EclipseDFS, now that it is working it will be good to clean up our source code.

Also, in order to ease reading our code, we should follow a coding standard. I propose Google C++ style

Getting more done in GitHub with ZenHub

Hola! @olzhabay has created a ZenHub account for the DICL organization. ZenHub is the leading team collaboration and project management solution built for GitHub.

---

How do I use ZenHub?

To get set up with ZenHub, all you have to do is download the browser extension and log in with your GitHub account. Once you do, you’ll get access to ZenHub’s complete feature-set immediately.

What can ZenHub do?

ZenHub adds a series of enhancements directly inside the GitHub UI:

• Real-time, customizable task boards for GitHub issues;
• Burndown charts, estimates, and velocity tracking based on GitHub Milestones;
• Personal to-do lists and task prioritization;
• “+1” button for GitHub issues and comments;
• Drag-and-drop file sharing;
• Time-saving shortcuts like a quick repo switcher.

Add ZenHub to GitHub

Still curious? See more ZenHub features or read user reviews. This issue was written by your friendly ZenHub bot, posted by request from @olzhabay.

dfsrm does not remove certain blocks

@zwigul has observed that sometimes dfsrm does not removes some file blocks in the scratch directory.

Changing the behavior of eclipse when it inits

Professor said that EclipseDFS should read the db when we launch it, not ignore it.

We should implement the logic for that, I think that @kbjin is more familiarize with that, can you make a feature for that?

Network Module

Basic Comm. and Topologies

Master node

Depends on #13

Create dfs [show] to display blocks locations of a file.

This featured was mentioned in #42

[PR:108]Different length formats in two sprintf for size

In update_file_metadata PRIu64 is used while in another functions it uses PRIu32

        size=%" PRIu64 ", num_block=%u where name='%s';",

eclipsed script

From this commit, we can use eclipsed to launch, check status and close eclipseDFS:

To install it run:

$ gem install eclipsed

Here is the repo of the eclipsed:

https://github.com/DICL/eclipsed

Getting more done in GitHub with ZenHub

Hola! @vicentebolea has created a ZenHub account for the DICL organization. ZenHub is the leading team collaboration and project management solution built for GitHub.

---

How do I use ZenHub?

To get set up with ZenHub, all you have to do is download the browser extension and log in with your GitHub account. Once you do, you’ll get access to ZenHub’s complete feature-set immediately.

What can ZenHub do?

ZenHub adds a series of enhancements directly inside the GitHub UI:

• Real-time, customizable task boards for GitHub issues;
• Burndown charts, estimates, and velocity tracking based on GitHub Milestones;
• Personal to-do lists and task prioritization;
• “+1” button for GitHub issues and comments;
• Drag-and-drop file sharing;
• Time-saving shortcuts like a quick repo switcher.

Add ZenHub to GitHub

Still curious? See more ZenHub features or read user reviews. This issue was written by your friendly ZenHub bot, posted by request from @vicentebolea.

Centralized Topology

Problems linking with ASIO serialization

@nammh and I found that in some of our servers Eclipse does not compile since it can not find the boost libraries.

For GCC you must specify in LIBRARY_PATH the location of the the libraries, normally writing in your .bashrc the next line will make the trick:

export LIBRARY_PATH=/usr/local/lib

As for Clang++ compiler, things are a bit harder since it does not read LIBRARY_PATH. In that case, write in your .bashrc:

export CONFIG_SITE="~/.clang_fix"

Then create that ~/.clang_fix file and write inside:

#!/bin/bash
export LDFLAGS+="-L/usr/local/lib"

Dynamic library loader

API for application

Release 1.1.0 task list

There is a few changes to be made for the next minor release:

• #51 : |compatible| Add doxygen to document the code
• #50 : |compatible| Change raw pointers to smart pointers (Many bugs were caused by that #46)
• #52 : |compatible| Remove commented code and adopt unified coding style
• #49 : |compatible| Decide open source license [Critical]
• #41 : |incompatible| Change the way the EclipseDFS initializes [Critical]
• #55 : |incompatible| Change dfsput/dfsget... to dfs [put|get|format|ls]
• #59 : |incompatible| dfsformat [Critical]
• #56 : |incompatible| Create dfs [show] to display blocks locations of a file.

Note that previous releases were pre-releases, version 1.1.0 aims to be a release.
Compatibles changes will be released in versions 1.0.*, incompatibles will wait until
all of them are done to relase 1.1.0

MPI Eclipse Cache

Release 1.2.0 task list

• : |compatible| Add replica support [Critical]
• : |compatible| Add Persistence of the network in case of failure [Critical]

Open source license

Which open source license should we use in EclipseDFS?

About dfsget and retrieve operation in DFS

I am implementing the logic for dfsget inside the eclipse network. The main ideas is that the leader node request file blocks to the neighbors, and file blocks arrives in a asynchronous manner. So in order to send them to the client i have to wait to collect all the block so that i can send them in order. So should i store in disk the blocks?

Application API

Add file system support

Thread pool for std::future

When calling std::async, it creates new thread. Better having thread pool and enqueue to it functions and lambdas with return of std::future.

Logical block configuration

How can I configure logical block features?
Just adding adding --enable-lblocks options as configuration, /eclipse_node abort as I execute the program
Is there any additional options that I should handle?

LocalIO Update

In localio function update should pass value variable by reference, so it will not double copy.

MapReduce framework

• Job and Task classes #8
• FS support #10
• Executor class #9
• Dynamic lib loader #16
• Application API #11

Add Task and Job classes

Add executor

Include list function for the Directory API

We need a member function inside directory class to display all the files inside its local sqlite db.

ASIO Eclipse Cache

• Network Class (#6)

Iterative map workflow discussion

We already determined the way of implement the iterative map workflow. However, @kbjin mentioned that not having shuffling phase in between the map iterations might lead to an inconsistent since the key values created from the previous iteration might be scattered in different nodes. @kbjin can you explain here your question?