cloudmazing / lfs-server-go Goto Github PK

View Code? Open in Web Editor NEW

55.0 5.0 8.0 2.7 MB

LFS server with multiple file stores and backing stores

License: MIT License

Go 98.95% JavaScript 0.09% Shell 0.96%

lfs-server-go's Introduction

LFS Server Go!

NOT yet ready for primetime and is going through some iterations, specifically regarding access control.

LFS Server Go! a server that implements the Git LFS API

This is based off of lfs-test-server

This server provides access to
The meta store is offloaded to

BoltDB
Cassandra

There is a notion of project -> OID membership, which is lacking from the original. This is wired up but still a WIP. It will allow for validating a user's membership to a project and the project's associated OID to the user, thus ensuring a user's access to a project will allow for access to an OID

##TODO:

Update access

~~Rename user to namespace~~
Implement namespace and project based access

Remove/Clean up old objects on delete
When an object is public and AWS is enabled, offload GETs directly to AWS
Adopt verification of uploads
Redo the UI so it is abstracted into its own app

Installing

Use the Go installer, this will install all dependencies tracked:

  $ go install github.com/cloudmazing/lfs-server-go

To use a specific config file, set LFS_SERVER_GO_CONFIG=/path/to/config

Then start with ./scripts/start

Stop with ./scripts/stop

Running

Set your GO_ENV. Options are prod or dev or test

Running the binary will start an LFS server on localhost:8080 by default. All of the configuration settings are stored in config.ini.

You'll want to copy config.ini.example to config.ini

A running database server, if desired. One of MySQL or Cassandra are the external database options. BoltDB is the local option and is not suggested for production use

An example usage:

Generate a key pair

openssl req -x509 -sha256 -nodes -days 2100 -newkey rsa:2048 -keyout mine.key -out mine.crt

Update the config to point at your new keys

Build the server

go build

S3

If using S3, you'll need to set AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY in your environment AWS_ACCESS_KEY is your AWS access key ID AWS_SECRET_ACCESS_KEY is your AWS secret key

Example:

export AWS_ACCESS_KEY_ID=someLongString
export AWS_SECRET_ACCESS_KEY=someLongerString/withMoreStuff

Or set it in the config file via the AccessKeyId and SecretAccessKey config settings

Start it

./scripts/start

Client

Further client documentation on the client is available at https://git-lfs.github.com/

To use the LFS test server with the Git LFS client, configure it in the repository's .gitconfig file:

  [lfs]
    url = "http://localhost:8080/janedoe/lfsrepo"

This file MUST be checked into git inside of your project.

HTTPS:

NOTE: If using https with a self signed cert also disable cert checking in the client repo.

	[lfs]
		url = "https://localhost:8080/jimdoe/lfsrepo"

	[http]
		sslfverify = false

Security Design

Namespaces -> projects Users are given access to a namespace: read, write, or both Users are given access to a project: read, write, or both

Building

To build from source, use the Go tools + godep:

  $ go get github.com/cloudmazing/lfs-server-go
  $ go install ./...
  $ godep restore

Making changes

To build from source, use the Go tools:

  $ go get github.com/cloudmazing/lfs-server-go
  $ go install ./...
  <edit files>
  $ godep save ./...
  $ godep update ./...

Testing

MUST have AWS S3 credentials (public and secret keys)

MUST create a database and user in mysql used for testing:

create database lfs_server_go_test;
grant all privileges on lfs_server_go_test.* to 'lfs_server'@'localhost' identified by 'pass123';

MUST have Cassandra

brew install cassandra

lfs-server-go's People

Contributors

Stargazers

Watchers

Forkers

itsmikeq rjaylyn pradeepbbl zhanglei ksurent emi80 prichey lcodemakerl

lfs-server-go's Issues

Add License

I couldn't find a license. It would be great if this had an MIT license like lfs-test-server.

Dockerfile and start are out of date.

Just trying this out and I have steps to make this usable with Docker. A couple minor issues with the code as it stands, which I outline at the top. Hopefully this helps anyone else out there kicking the tires.

On the host machine:

mkdir git-lfs && cd git-lfs
git clone  https://github.com/cloudmazing/lfs-server-go.git .

Change Dockerfile RUN line to look like this:
RUN ./scripts/start

Insert this line right after export GO_ENV in scripts/start:
which goreman > /dev/null || go get github.com/mattn/goreman

Then back out to the main directory and run this command:
docker build git-lfs

This will run for a bit and should generate an image which you can then find with:
docker images

Cut and paste your image hash below:

docker run -i -t --name cloudmazing-git-lfs [paste-image-hash-here] /bin/bash
exit

Find the hash of the container you just exited with this command:
docker ps -a

Then commit the container to be a new image that you can launch locally by pasting here:
docker commit -m "Initial build of cloudmazing-git-lfs" -a "Me" [paste-image-hash-here] cloudmazing-git-lfs

Now, you can launch a docker image with this command (where /var/git_lfs is a host directory that survives container death):
docker run -d -it --rm -p 9999:9999 --name git_lfs -v /var/git_lfs:/data cloudmazing-git-lfs

Dead or just dormant?

This project looks rather dead. Is it?

Does not handle newer versions of cassandra

Need to implement the following in config settings:
https://github.com/gocql/gocql/blob/6c910921faab44389a426721cf57599fcd0a10d5/cluster.go#L53

Using this for GitLab? PostgreSQL support

How would you feel about GitLab using this as well?

Would you be open to us adding PostgreSQL support for the metadata store?

Different behavior between git-lfs client 0.6.0 and 0.5.4

When doing a git push with 0.5.4:

$git push origin
ismaels@gitlfs-client:/local/mnt/gitrepos/reponame.git$ git push --set-upstream origin master
Username for 'http://gitlfs-server:9999': repo
Password for 'http://repo@gitlfs-server:9999': 
(283 of 283 files) 22.29 MB 29s
Counting objects: 8132, done.
Compressing objects: 100% (4205/4205), done.
Writing objects: 100% (8132/8132), 231.18 MiB | 200.97 MiB/s, done.
Total 8132 (delta 3348), reused 8132 (delta 3348)
Username for 'http://gitlfs-server:80': repo
Password for 'http://repo@gitlfs-server:80': 
Everything up-to-date
$ git lfs version
git-lfs/0.5.4 (GitHub; linux amd64; go 1.3; git b00fa51)

But when using 0.6.0 it does not even try to connect to the git lfs server:

$ git push origin
Counting objects: 8135, done.
Compressing objects: 100% (4208/4208), done.
Writing objects: 100% (8135/8135), 231.18 MiB | 142.66 MiB/s, done.
Total 8135 (delta 3349), reused 8131 (delta 3348)
Username for 'http://gitlfs-server:80': repo
Password for 'http://repo@gitlfs-server:80': 
Everything up-to-date
$ git lfs version
git-lfs/0.6.0 (GitHub; linux amd64; go 1.5.1; git 183bdc8)

Should all the clients work?
Should have I different configuration depending on the client?

Repository config:

[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true

[lfs]
        url = "http://gitlfs-server:9999/repo/reponame.git"
    batch = true

[http]
        sslverify = false
    postBuffer = 524288000
[remote "origin"]
    url = "http://gitlfs-server:80/repo/reponame.git"
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "test"]
    remote = origin
    merge = refs/heads/test
[branch "test2"]
    remote = origin
    merge = refs/heads/test2
[lfs "http://gitlfs-server:9999/repo/reponame.git"]
    access = private

func (a *App) searchOidHandler(w http.ResponseWriter, r *http.Request) {
    searchedOid := r.URL.Query().Get("oid")
    if len(searchedOid) < 1 {
        writeStatus(w, r, 404)
    }
    sess := NewCassandraSession()

That's not great if someone is using a different meta store. Unless this functionality is somehow only exposed when using Cassandra.