kingces95 / lfx

Teaching git how to restore nuget packages

git-lfx

Teaching git how to restore nuget packages

Lfx is an adaption of Lfs. Lfs allows git to manage binary resources but requires a server be maintained to host the binaries. Lfx only supports restoring files already hosted either individually or as part of a zip archive (e.g. a nuget package).

Quickstart

Before any of the following quickstart scenarios, install git-lfx:

1. Install chocolatey
2. choco install git-lfx
3. relaunch cmd.exe
4. git-lfx verify installation; dump help

####Sync a repository and restore lfx pointers automatically:

1. git lfx clone lfx-sample
2. git lfx files list files lfx downloaded

####Sync a repository and restore lfx pointers manually:

1. git clone lfx-sample
2. git lfx files list files lfx will download
3. type dls\tools\nuget.exe observe the binary content has been replaced with a lfx pointer
4. git lfx checkout restore binary content for lfx tracked files
5. dir dls\tools\nuget.exe observe the binary content is restored

####Initialize a repository with lfx support:

1. git lfx init

####Add\Remove lfx filters to\from an existing repository:

1. git init
2. git lfx config --list observe lack of filter.lfx.* config settings
3. git lfx config --set add lfx filter config settings
4. git lfx config --list observe addition of filter.lfx.* config settings
5. update .gitattributes to specify files using lfx filter (see below)
6. add .lfxconfig to specify how how pointers are constructed as a function of file path (see below)
7. git lfx config --unset clear filters

####Initialize a repository with lfx support + samples + exploration:

1. git lfx init --samples
2. git lfx env
3. type dls\tools\NuGet.exe
4. git lfx smudge dls\tools\NuGet.exe > %TEMP%\Nuget.exe resolve binary content given lfx pointer
5. git lfx checkout smudge filter replaces dls\tools\nuget.exe lfx pointer with binary content
6. dls\tools\NuGet.exe restore dls\packages\packages.config -PackagesDirectory dls\packages
7. git lfx clean dls\packages\NUnit.2.6.4\lib\nunit.framework.dll generate lfx pointer given binary
8. git add . clean filter converts nuget package file content to lfx pointers
9. git lfx show dls\packages\NUnit.2.6.4\lib\nunit.framework.dll dump example of content actually staged for commit
10. git commit -m "Add nunit v2.6.4" commit lfx pointers

History

In the beginning, there was darkness, then light, then centralized source control systems (e.g. TFS) which stored all the dependencies necessary to build. Building was a simple matter of syncing the repository, build tools and all, and building. It was good. And worked well for those teams that could afford the time and energy of maintaining a TFS server. However, it didn't work so well for those who couldn't or for open source projects with thousands of loosely associated collaborators (e.g. Linux). They needed a different solution.

So, banished from centralized source control, along came distributed version control (e.g. GIT) which solved those pesky server maintenance issues by not having a central server at all. Instead the entire history of the repository was distributed to everyone. However, this solution came at with a cost. Distributing the history of every binary was prohibitively expensive and so the binaries had to be removed. And so, in this new world, a new step to restore the removed binaries was introduced after syncing the source but before building (e.g. nuget restore). It was ok.

Naturally, efforts were made to hide the extra restore step and return to the utopic sync/build experience of centralized source control. Usually, this involved having the build preform the restore. However, this proved to be problematic. Some nuget packages included msbuild files which could not be restored before they were needed while, at the same time, modifying other msbuild project files which had already been processed. Yuck. And so restoring during the build was abandoned and the official advice became:

when building from the command line, you need to run 'nuget restore' yourself before msbuild

Not exactly helpful for those trying to automate their build. So, what to do?

Binary files could not be stored in GIT however no one said we couldn't store pointers to binary files. If GIT could recognize these pointers during a sync and download the referenced file then the build could proceed directly after the sync! And if GIT could replace binary files with a pointer before pushing changes then those pointers could be distributed instead of the binaries themselves. Wonderful! But how do we teach GIT to recognize and generate pointers?

A Beautiful Hack

Stay tuned...

Actual Impetus

Builds composed of multipule C# projects often suffer from duplication of common project settings. Among other things, this makes enforcement of policy challenging (e.g. enforcing warnings as errors is enabled for all projects). Builds typically solve this problem by extracting common settings to a single location where they can be centrally administered (e.g. coreclr and corefx extract their common settings to a set of dir.proj files).

Unfortunately, most teams roll their own variant of the dir.proj file solution as their build becomes larger and larger. Xamarin.Form libraries, however, are born needing dozens of projects in order to generate the zoo of binaries needed for each supported platform. In this case, it makes sense to invest in a generalized solution which can be packaged up and reused by Xamarin.Forms library authors. Nuget seemed natural distribution vehicle but executing a nuget restore to pull down msbuild files felt wrong. So lfx.