amccurry / pack Goto Github PK
View Code? Open in Web Editor NEWHdfs Block Storage System
License: Apache License 2.0
Hdfs Block Storage System
License: Apache License 2.0
Volumes should be allowed to be put into a readonly mode or created as readonly from a snapshot of another volume.
This needed for writing blocks into hdfs. If clients are using multipathing then one and only one server should deliver the wal data to hdfs.
Add create volume form in admin application.
Read blocks from external store when a sequential read is detected.
This command will allow thin provisioning of data in the remote storage by allowing fstrim to be used on the mounted OS.
This should help clean up old disk space in the block storage system.
Create CSD and Parcel for Cloudera Manager deployment.
Needs a cli for managing volumes and getting status from pack.
Local block cache and state should be removed on volume detachment.
The idea is to prime the write ahead log with data before enabling the new log for normal writes. This might allow for more seamless WAL roles.
Currently pack copies all the files on the server's classpath before it start a mount. That way upgrades won't disrupt current running mounts. We can improve this by having the server maintain the copies and when all the files are not in use by a running mount the old libraries can be GCed.
The compactor should be able to operate without the need to be restarted from time to time. This could have been an issue with where the compactors were running at the time of the issue. The servers were running out of memory from time to time. So this could be a non-issue.
Check volume not in use before detachment of volume.
The current block file implementation requires more reading during block file open than is required. If the block file indexes starting positions were calculated and stored in the meta data the logic required to open the block file would simplified and should be faster.
The current WAL implementation should be made into a replicated service to avoid single points of failure.
Investigate using TOKEN authentication for performance improvement.
Actively repair blocks from WAL after volume has been shutdown so that WAL entries can be released.
Deal with failed local storage for both WAL and block cache stores.
Create new volumes from snapshots.
Cache removal listener is now a blocking call. Some aspects of the call should be made async.
Snapshots of volumes should be allowed, both online and offline. Online snapshots may produce unexpected results given that the FS may be changing. So for online snapshots FS freeze and sync should be employed.
Create test external error generation framework to test error handling on the iscsi and wal services.
Store the current cache state (which blocks are currently cached) for the given volume so that on reopen the state can be restored. This should allow similar performance after a volume is moved to a new machine.
Investigate compressing blocks in remote store.
Blocks in external store that are no longer referenced in either the current blocks or in snapshots should be deleted.
Add multiple host directory support for block cache and WAL storage.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.