clusterhq / fli

Fli, the CLI to FlockerHub, is like Git for data. Just like you can use Git to copy, branch, push and pull code, you can use Fli to copy, branch, push and pull your data volumes.

FLI

Fli, the CLI to FlockerHub, is like Git for data. Just like you can use Git to copy, branch, push and pull code, you can use Fli to copy, branch, push and pull your data volumes.

Fli lets you take incremental snapshots of any database or data volume that runs on Linux, and push those snapshots to a hosted volume repository called FlockerHub where they can be accessed by any person or machine to whom you’ve granted access.
Fli can also be used without FlockerHub to manage data volumes locally.

Fli uses ZFS for taking and managing snapshots. Because the snapshots are incremental, each additional push or pull to the FlockerHub sends only part of the data, speeding up transfer times and reducing bandwidth consumption.

Additionally, just like you can use Git to organize your code into repos and branches, Fli gives you the ability to organize your data into volumesets and branches. You can use Fli to manage these snapshots locally without using FlockerHub at all, or push them to FlockerHub to share with others.

Installation

It's easy to get started using Fli because Fli is distributed on the DockerHub as a container image called clusterhq/fli so you just need to run docker pull clusterhq/fli. You can also download the Go binary directly by running curl -L "https://download.clusterhq.com/releases/fli/<version>" > /usr/local/bin/fli chmod +x /usr/local/bin/fli<

Using Fli requires a few dependencies so make sure you have these on your system before getting started.

• A supported Linux distribution– (Ubuntu 16.04 Xenial Xerus, RHEL 7.2, CentOS 7.2, or Fedora 24.)
• Docker Engine if you are using the Fli container.
• ZFS kernel modules and ZFS utilities

View the complete documentation for Fli or read on for some more background on the project.

Build Fli from source

Fli can also be built from source. Instructions to build from source can be found here.

Why use Fli?

With increased popularity of microservices and container-based architectures, greater and greater emphasis is being placed on automated testing to ensure that separately created and updated microservices all work together when deployed to production. Manual QA is no longer sufficient, there are simply too many variables to test. As a result, a new set of requirements are emerging to address data management needs across the entire DevOps lifecycle. Even before getting to production, container data management (CDM) needs to span a number of use cases including:

• Making realistic test data available to the entire development team to help improve developer productivity
• Providing realistic production data to staging and QA environments for accurate pre-prod testing
• Integrating test data into Continuous Integration / Delivery (CI/CD) environments to accelerate application development

Solving these problems is why we created Fli.

Fli, a Git-like CLI, lets Developers and Ops capture the state of a data volume running on a Linux server (usually this means a database, but Fli also works for snapshotting files, plugins, binaries and other test fixtures), and move it to a cloud based repository similar to GitHub where it can then be shared and then pulled by any other authorized user or machine.

At the present, Fli works best for testing, but these generic capabilities to snapshot and move data will one day enable:

• Disaster Recovery (DR) and backup resiliency for production stateful services to ensure required uptime availability
• Enabling movement of workloads across a multi-cloud environment for cost, performance and security reasons

Demo

Get involved

• Submit a issue or PR on GitHub. We'd love your help.
• Join our Slack community at http://slack.clusterhq.com/
• You can also join us on the #clusterhq channel on the irc.freenode.net
• Ask a question on the flockerhub-fli-users Google group
• Send us an email at [email protected]
• Tweet us at @ClusterHQ

fli's Issues

Support for docker.app on Mac

Would be great to have support for the official docker app on mac. I did attempt to install ZFS in the xhyve VM but stopped short of getting the ZFS kernel module working.

fli push fails with error "Differ error too many levels of symbolic links"

While following examples in https://github.com/ClusterHQ/examples I encounter the following error:

docker@fli-vm:~$ fli push richardw-volumeset-1
Upload snapshot 0f0a9aa6-768b-401b-8c1f-e44e59ed6992 failed: Put http://ec2-35-160-189-40.us-west-2.compute.amazonaws.com:80/upload?token=98ec9168-3307-4517-927f-87f308b0acba: Differ error too many levels of symbolic links

The volumeset, volume and snapshots show up at www.flockerhub.clusterhq.com.

docker@fli-vm:~$ fli version
Version:    0.6.0
Git commit: a4c0d93

docker@fli-vm:~$ sudo cat /root/.fli/config
current: /root/.fli/mds_current
initial: /root/.fli/mds_initial
url: https://data.flockerhub.clusterhq.com
token: /root/token.txt
zpool: chq

docker@fli-vm:~$ fli search --all --full
VOLUMESET ID                         CREATOR                              CREATED         SIZE  DESCRIPTION ATTRIBUTES NAME
97244daf-0b65-4737-8a40-495c34f2888a fe59a254-bd67-4f05-a423-b99592a2cb73 Dec  6 11:18:11 29 KB -           -          My-VolumeSet
     BRANCH SNAPSHOT TIP
     -      My-Snapshot
     SNAPSHOT ID                          OWNER                                CREATOR                              CREATED         SIZE   DESCRIPTION ATTRIBUTES NAME
     8050111a-20b1-412f-aaff-1a37f3baf0aa fe59a254-bd67-4f05-a423-b99592a2cb73 fe59a254-bd67-4f05-a423-b99592a2cb73 Dec  6 11:18:37 9.5 KB -           -          My-Snapshot
     VOLUME ID                            CREATED         SIZE MOUNT POINT                                                                    NAME
     4d1dc3a1-4fa2-468d-a52e-d22323f59a14 Dec  6 11:18:17 0    /chq/97244daf-0b65-4737-8a40-495c34f2888a/4d1dc3a1-4fa2-468d-a52e-d22323f59a14 My-Volume
VOLUMESET ID                         CREATOR                              CREATED         SIZE  DESCRIPTION ATTRIBUTES                                           NAME
b5c553c1-e73b-4b2d-9751-8af4066eb57c fe59a254-bd67-4f05-a423-b99592a2cb73 Dec  6 11:34:22 29 KB foo bar baz reason=https://github.com/ClusterHQ/fli-docs/pull/85 richardw-volumeset-1
     BRANCH SNAPSHOT TIP
     -      snapshot-1
     SNAPSHOT ID                          OWNER                                CREATOR                              CREATED         SIZE   DESCRIPTION ATTRIBUTES                                           NAME
     0f0a9aa6-768b-401b-8c1f-e44e59ed6992 fe59a254-bd67-4f05-a423-b99592a2cb73 fe59a254-bd67-4f05-a423-b99592a2cb73 Dec  6 11:37:37 9.5 KB -           reason=https://github.com/ClusterHQ/fli-docs/pull/85 snapshot-1
     VOLUME ID                            CREATED         SIZE MOUNT POINT                                                                    NAME
     b48af388-1795-4561-bba8-526adeaa0eb2 Dec  6 11:35:25 0    /chq/b5c553c1-e73b-4b2d-9751-8af4066eb57c/b48af388-1795-4561-bba8-526adeaa0eb2 volume-1

Support for static filesystem volume(set) names?

Running fli create volumesetname volumename creates uuid paths inside the mounted zpool like /chq/fc3d6776-d76a-4d90-8ec3-89f348b3536d/ba603e7a-e529-4aa4-b126-7989432c3662.

For automation it's not suitable, because you have to manually change the docker volume path to point to a volume in the volumeset(I guess?). Is there going to be support for static volumeset/volume paths? Or am I confused as to how docker volumes should work together with fli?

"Invalid VHUT format" during fli config

I have created an account on flockerhub and created a token.

Then I followed the guide and configured fli to use the token:

fli config --token $HOME/token.txt

Problem is I get the following response from fli:

Invalid VHUT format

Typo in Build from source section

https://github.com/ClusterHQ/fli#build-fli-from-source

Fli can also be built from source. Instruction to build from source can be found here.

Should be

Fli can also be built from source. Instructions to build from source can be found here.