presto412 / storev

Storage Virtualization implemented as a Distributed FS with de-duplication and replication

StoreV

A Storage Virtualization implementation, that considers backups and de-duplication.

Introduction

In this project, I had deployed four storage servers via DigitalOcean in Amsterdam, Bangalore, Toronto and Singapore data centres. The central server is in New York. The aim was to replicate a newly uploaded file and store it in two of the four servers. When there is a download request, the file is downloaded from the server that is the closest geographically. Using this technique, users do not have to be worried about the specific location of their file. It allows the pooling of physical storage from multiple storage devices into what appears to be a single storage device being managed from a central console. The use of hashes in this project prevents the user from adding redundant data, ie, before uploading a file, hashing is performed, to check if it exists in the system. This solves data-deduplication. An example metadata storage is given here

Architecture

Stack

• Node.js(Express), EJS Templates
• Docker, Docker Swarm

Development Steps

• Create a .env file like the .env.example file
• Get an access key from IPSTACK (It's free)
• Set the NODE_ENV value to dev
• Run ./scripts/dev-master.sh
• Go to localhost

Deploying to production

• Clone the repository in a local system
• Make sure you've ssh installed and configured
• Set up all the servers, and add your public key found in .ssh/
• Create a .env file like the .env.example file, with all corresponding IPs
• Set the NODE_ENV value to prod
• Setup docker, pull the images and start the network

  ./scripts/prod-master.sh

• Check out port 3000 on your central IP and it should be running

storev's Issues

Add some cool CSS to the UI

Currently, the UI is pure HTML and solves functionality. You can add some CSS to make it look good!
Here's how it currently looks:

Any improvements appreciated!

Add Authentication to the project

You can utilise Passport and add signup/login functionality to the project. The file list view should then be able to display only what the user has uploaded. Since auth requires a database, you can pick up issue #5 as well. Happy hacking!

Add delete option for files (issue looking for assignee)

Currently, the files uploaded can only be viewed/downloaded. You can add a delete option for the same(deletes the file across the cluster).

Convert JSON based storage of file metadata to use MongoDb/ any other NOSQL DB

The metadata being stored is in the form of a JSON file that is updated for every upload. You can work on adding DB backed storage to the same!
Here's the way it is stored.

Refactor index.js and clean-up code

All the backend code is in a single file! This stinks. Here's your chance to do some cool refactoring and celebrate Hacktoberfest! The code is fully functional, all you have to do is run a linting program and move code around. If you have tons of time, you can migrate it to TypeScript as well!