couchbaselabs / cbfs

Distributed Blobstore using Couchbase Server

cbfs is a distributed file server built on top of Couchbase

How do I get the stuff

go get github.com/couchbaselabs/cbfs

And you'll find the source in $GOPATH/src/github.com/couchbaselabs/cbfs (and a cbfs binary should be in your path)

How do I build the stuff

cd $GOPATH/src/pkg/github.com/couchbaselabs/cbfs
go build

How do I run the stuff

mkdir -p /tmp/localdata
./cbfs -nodeID=$mynodeid \
       -bucket=cbfs \
       -couchbase=http://$mycouchbaseserver:8091/
       -root=/tmp/localdata \
       -viewProxy

The server will be empty at this point, you can install the monitor using cbfsclient (go get github.com/couchbaselabs/cbfs/tools/cbfsclient)

cbfsclient http://localhost:8484/ upload \
    $GOPATH/src/github.com/couchbaselabs/cbfs/monitor monitor

Then go to http://localhost:8484/monitor/

Running on Docker / CoreOS

See Traun Leyden's blog post on Running CBFS + Couchbase Cluster on CoreOS.

cbfs's Issues

Clobber blob info on empty reg

If nobody has a blob, don't store it. This requires enhancements to my cas interface

Cache proxied objects

Since I'm playing with this right now (and it's not super-exciting), I'm going to go ahead and do this, @mschoch

View for showing storage in use.

Interesting side-effect: you get the storage in-use by subdirectory at a significantly lower cost than I get with du on local filesystems.

Include blob ID in file http response

Let the user know what he's looking at.

I might want to use etag for this, but I'm not entirely sure yet.

"master" service elector

"master" is probably not the best term, but we do want a few things that only run in one place. We could possibly separate them by service, but there are two main things we need:

1. Have the nodes elect a master and record it into a key (e.g. /master).
2. Have the master heartbeat itself with a CAS operation.

The election is either a CAS or ADD operation to ensure there's no conflict. Other nodes will need to occasionally verify this by pulling the document and checking the timestamp.

Report recent health and performance data

Some recent histos or something to help make decisions on sourcing and distribution.

Validate hashes on reconcile

Instead of just lobbing up all the files we find on the filesystem, we should also verify their hashes.

File versioning

This is too easy to not do.

Randomly delay jobs at start time.

Don't need everything happening at once.

Storage node object reconciliation

We should occasionally walk our local object store and make sure all of the local objects are represented in the metadata store.

Ensure blobs are replicated sufficiently.

After a blob appears in the system, make sure it's on as many nodes as necessary.

Centralize some configuration properties

There are too many things that have to be configured identically on all the nodes.

Support range requests and conditional get

Should be trivial to pull this out of the file info and will make the content serving do the right thing.

Limiting replicas at startup seems broken

It was over replicating when bringing up new nodes even though a limit was set and already met.

Node list

Probably /.cbfs/nodes/

Include heartbeat time, sizes, etc...

Susceptible to older time parsing bug

Marty noticed this trying to read some of the data from one of my nodes.

I don't think this is a view problem as much as it's likely to be a bug in the time parser. I tried it out on play.golang.org and it broke: http://play.golang.org/p/F4q7Rzx1B9

But locally:

dhcp-110:/tmp 716% go run thing.go 
2012-09-17 22:12:09.894702 +0000 UTC - <nil>

I think I worked around this in seriesly with a custom time parser (well, and I made it slightly faster). I'm going to add this test to the suite. I think it'll also fail on goci (where we have CI running outside of my box).

Record blob length.

While I'm storing the blob info, I might as well store the length of the blob. This would make it really easy to calculate the total size of the store, the size of a store within any range, amount of data being served, etc...

GC unused blobs

Find objects on a node that shouldn't exist and unexist them.

Need some way to package up and install views

I assume this needs a gocouchbase change, but we have a lot of views that are required for this thing to work and we need a way to distribute them with the storage node.

Probabilistic Eager Caching

Instead of always copying a blob to a secondary server when requesting and proxying, we should do this on a sliding probability scale. This will prevent excessive overcaching while still getting the hot stuff distributed widely.

Unify internal interfaces under a common path.

I think we should possibly have a /.cbfs/ path wherein we keep all the useful things. e.g.

• PUT/GET/DELETE /cbfs/blob/[sha]
• POST /.cbfs/blob/ (for the raw streaming thing)
• GET /.cbfs/blob/ (list of all stored blobs)

etc...

Move URL generation up to StorageNode

This only happens in one place now, but it makes sense to have different types of storage nodes, e.g. an S3 bucket.

Record blob frequency.

For balance management, let's keep a counter near the object that represents the blob that records how frequently the blob is accessed.

@mschoch, these should also be blindly deleted along with GC. I'm thinking something like /blobid/r or something for a read counter.

Redistribute blobs to new nodes

Nodes can do this when starting up. Just get the node list, blob counts per node and start pulling from hot ones. Let them trim down.

Find blobs by name through couchbase

Currently, we only work with local blobs. We can try that, but also retrieve the blob remotely if we don't have it locally by looking up a recent location through couchbase.

Proxying it should be a trivial http request against the raw blob and an io.Copy. You'll already have the headers.

This obviously depends on #1

Automatically store data on a second node on initial put

I should be able to issue a request and stream the data to the local disk and remote server simultaneously, link both objects and then link the file.

Clean up after stale nodes.

If a node is stale, we should notice this and perhaps consider invalidating the "ownership" mappings that this node claims.

Don't overwrite user data on PUT

We have a proper protocol for managing user data. It makes sense to preserve this data across object mutations. Users who don't want this can always remove it on change. It can be tracked by recording object IDs or something in the data such that a view can pull out objects that haven't been processed or whatever.

@apage43, does this make sense to you?

Need some notion of available space on nodes.

We store enough information to know how much data each node has stored. We just need to report how much total is available in the heartbeats.

This is somewhat related to #2.

Arbitrary file metadata storage

I think you just need to add userdata as a json.RawMessage and you're good to go.

Provide interface to user meta

After #32, we should provide direct access to file meta. Something like:

• POST/GET/DELETE /.cbfs/meta/some/path

Trim over-replication

Find objects that are excessively replicated and trim them down. Probably want to have a minimum lifetime on these over-replicated blobs. Ideally, we validate hotness more intelligently before trimming.

Record node access frequency

For balance measurement, let's record how often a request is sent to a particular node.

File listing interface

Someway to pull down a list of files, possibly between two ranges. Possibly simulating directories and excluding depth traversal.

blob metadata

We have file (name) metadata, but no blob (content) metadata. Sometimes you want to mark up the name and sometimes you want to mark up the contents. Both should be possible.

It's not complete clear what a metadata response would look like incorporating both. I guess I could do:

{"file": {}, "blob": {}}

and include whatever the user asked for.

@apage43 opinions?

Expires header to expire server-side data?

Another trivial change since the metadata server supports the concept. The file object will disappear and the blobs will GC.

conditional put/delete for docs and meta

We've got eTags, let's do it.

Configurable default object

From a conversation with @apage43:

/ can map to a specifiable default object iff it's not a cbfs control request. Something like -default=/index.html

Implement http DELETE

Change node object type to "node"

It's currently "storage"

Reconciliation should verify metadata doesn't think we have things we don't.

This cleans itself up when it's asked for, but we need to find it sooner than that.

View proxy

Access your couchbase views for custom app awesomeness

Support HTTP HEAD

Might as well return headers if someone asks for it. It's a cheap operation.

When tearing down object references from a node, copy them elsewhere

If we think a node is going offline and we start revoking object ownership, we should copy any of these objects that would be under-replicated onto other nodes to keep the replication count high.

Command endpoint to request a node take on a blob

This is related to #2 as it's an efficient way to accomplish it without having to copy the data through the requestor and will simplify the current takedown path by having the caller only have to deal with a destination.

Unhardcode some params

Health check freq, few more

Proper content-encoding/transfer-encoding handling

Right now, everything is sent back and forth verbatim. We should probably canonicalize compression and strip those headers out, then recompress or something to make sure we don't confuse clients.

Don't remove node records when views partially error

System is designed to self-repair, but let's make sure that the view result is both empty and accurate.

Record oid locations when storing files

This can be a bit lazy, though it might make it harder to find files instantly after storing them. Not too worried about that. It can also reconcile easily with the list functionality.

It would be nice with a README.markdown to tell me how to get/build/start the stuff