dwyl / gogs

⚙️ Interface with a Gogs (Git Server) from any Elixir App

Interface with a Gogs instance from Elixir.

Why? 💡

We needed an easy way to interact with our Gogs (GitHub Backup) Server from our Elixir/Phoenix App. This package is that interface.

Note: We were briefly tempted to write this code inside the Phoenix App that uses it, however we quickly realized that having it separate was better for testability/maintainability. Having a separate module enforces a separation of concerns with a strong "API contract". This way we know this package is well-tested, documented and maintained. And can be used and extended independently of any Elixir/Phoenix app. The Elixir/Phoenix app can treat gogs as a logically separate/independent entity with a clear interface.

What? 📦

A library for interacting with gogs (git) from Elixir apps.

Hopefully this diagram explains how we use the package:

For the complete list of functions, please see the docs: https://hexdocs.pm/gogs 📚

Who? 👤

This library is used by our (Phoenix) GitHub Backup App.
If you find it helpful for your project, please ⭐ on GitHub: github.com/dwyl/gogs

How? 💻

There are a couple of steps to get this working in your project. It should only take 2 mins if you already have your Gogs Server deployed (or access to an existing instance).

If you want to read a step-by-step complete beginner's guide to getting gogs working in a Phoenix App, please see: github.com/dwyl/gogs-demo

Install ⬇️

Install the package from hex.pm, by adding gogs to the list of dependencies in your mix.exs file:

def deps do
  [
    {:gogs, "~> 1.0.2"}
  ]
end

Once you've saved the mix.exs file, run:

mix deps.get

Config ⚙️

If you are writing tests for a function that relies on gogs (and you should!) then you can add the following line to your config/test.exs file:

config :gogs, mock: true

e.g: config/test.exs#L2-L4

Setup 🔧

For gogs to work in your Elixir/Phoenix App, you will need to have a few environment variables defined.

There are 3 required and 2 optional variables. Make sure you read through the next section to determine if you need the optional ones.

Required Environment Variables

See: .env_sample

There are 3 required environment variables:

1. GOGS_URL - the domain where your Gogs Server is deployed, without the protocol, e.g: gogs-server.fly.dev

2. GOGS_ACCESS_TOKEN - the REST API Access Token See: https://github.com/dwyl/gogs-server#connect-via-rest-api-https

3. GOGS_SSH_PRIVATE_KEY_PATH - absolute path to the id_rsa file on your localhost or Phoenix server instance.

@SIMON: this last env var currently not being picked up. So it will just use ~/simon/id_rsa You will need to add your public key to the Gogs instance for this to work on your localhost see: https://github.com/dwyl/gogs-server#add-ssh-key

Optional Environment Variables

GOGS_SSH_PORT

If your Gogs Server is configured with a non-standard SSH port, then you need to define it: GOGS_SSH_PORT
e.g: 10022 for our Gogs Server deployed to Fly.io

You can easily discover the port by either visiting your Gogs Server Config page:
https://your-gogs-server.net/admin/config

e.g: https://gogs-server.fly.dev/admin/config

Or if you don't have admin access to the config page, simply view the ssh clone link on a repo page, e.g: https://gogs-server.fly.dev/nelsonic/public-repo

In our case the GOGS_SSH_PORT e.g: 10022.
If you don't set it, then gogs will assume TCP port 22.

GIT_TEMP_DIR_PATH

If you want to specify a directory where you want to clone git repos to, create a GIT_TEMP_DIR_PATH environment variable. e.g:

export GIT_TEMP_DIR_PATH=tmp

Note: the directory must already exist. (it won't be created if it's not there ...)

Usage

If you just want to read the contents of a file hosted on a Gogs Server, write code similar to this:

org_name = "myorg"
repo_name = "public-repo"
file_name = "README.md"
{:ok, %HTTPoison.Response{ body: response_body}} = 
  Gogs.remote_read_raw(org_name, repo_name,file_name)
# use the response_body (plaintext data)

This is exactly the use-case presented in our demo app: dwyl/gogs-demo#4-create-function

Here's a more real-world scenario in 7 easy steps:

1. Create a New Repo on the Gogs Server

# Define the params for the remote repository:
org_name = "myorg"
repo_name = "repo-name"
private = false # boolean
# Create the repo!
Gogs.remote_repo_create(org_name, repo_name, private)

⚠️ WARNING: there is currently no way to create an Organisation on the Gogs Server via REST API so the org_name must already exists. e.g: https://gogs-server.fly.dev/myorg We will be figuring out a workaround shortly ... #17

2. Clone the Repo

git_repo_url = Gogs.Helpers.remote_url_ssh(org_name, repo_name)
Gogs.clone(git_repo_url)

Provided you have setup the environment variables, and your Elixir/Phoenix App has write access to the filesystem, this should work without any issues. We haven't seen any in practice. But if you get stuck at this step, open an issue

3. Read the Contents of Local (Cloned) File

Once you've cloned the Git Repo from the Gogs Server to the local filesystem of the Elixir/Phoenix App, you can read any file inside it.

org_name = "myorg"
repo_name = "public-repo"
file_name = "README.md"
{:ok, text} == Gogs.local_file_read(org_name, repo_name, file_name)

4. Write to a File

file_name = "README.md"
text = "Your README.md text"
Gogs.local_file_write_text(org_name, repo_name, file_name, text)

This will create a new file if it doesn't already exist.

5. Commit Changes

{:ok, msg} = Gogs.commit(org_name, repo_name, 
  %{message: "your commit message", full_name: "Al Ex", email: "[email protected]"})

6. Push to Gogs Remote

# Push to Gogs Server this one is easy.
Gogs.push(org_name, repo_name)

7. Confirm the File was Update on the Remote repo

# Confirm the README.md was updated on the remote repo:
{:ok, %HTTPoison.Response{ body: response_body}} = 
    Gogs.remote_read_raw(org_name, repo_name, file_name)
"Your README.md text"

Full Function Reference / Docs? 📖

Rather than duplicate all the docs here, please read the complete function reference, on hexdocs: https://hexdocs.pm/gogs/Gogs.html

Tests!

By default, the tests run with "mocks", this means that:

1. Functional tests run faster (0.2 seconds)
2. Tests that require filesystem access will run on GitHub CI.
3. We know that functions are appropriately ["Test Doubled"] so that a downstream Elixir/Phoenix app can run in mock: true and tests will be mocked (and thus fast!)

To alter this setting to run the tests without mocks, simply change the boolean from:

config :gogs, mock: true

To:

config :gogs, mock: false

You should still see the same output as all the functions should be tested.

Test Coverage

When you run the command:

mix c

(an alias for mix coveralls.html)
You will see output similar to the following:

Finished in 0.1 seconds (0.1s async, 0.00s sync)
3 doctests, 27 tests, 0 failures

Randomized with seed 715101
----------------
COV    FILE                                        LINES RELEVANT   MISSED
100.0% lib/git_mock.ex                                55        7        0
100.0% lib/gogs.ex                                   212       41        0
100.0% lib/helpers.ex                                131       17        0
100.0% lib/http.ex                                   119       18        0
100.0% lib/httpoison_mock.ex                         124       20        0
[TOTAL] 100.0%
----------------

If you want to run the tests without mocks (i.e. "end-to-end"), update the line in config/test.exs:

config :gogs, mock: false

When you run end-to-end tests with coverage tracking:

mix c

You should see the same output:

Finished in 5.5 seconds (5.5s async, 0.00s sync)
3 doctests, 27 tests, 0 failures

Randomized with seed 388372
----------------
COV    FILE                                        LINES RELEVANT   MISSED
100.0% lib/git_mock.ex                                55        7        0
100.0% lib/gogs.ex                                   212       41        0
100.0% lib/helpers.ex                                131       17        0
100.0% lib/http.ex                                   119       18        0
100.0% lib/httpoison_mock.ex                         124       20        0
[TOTAL] 100.0%
----------------

The only difference is the time it takes to run the test suite.
The outcome (all tests passing and 100% coverage) should be identical.

If you add a feature to the package, please ensure that the tests pass in both mock: true and mock: false so that we know it works in the real world as well as in the simulated one.

Roadmap

We are aiming to do a 1:1 feature map between GitHub and Gogs so that we can backup our entire organisation, all repos, issues, labels & PRs.

We aren't there yet and we might not be for some time. The order in which we will be working on fleshing out the features is:

1. Git Diff - using the Git module to determine the changes made to a specific file between two Git commits/hashes. This will allow us to visualize the changes made and can therefore derive the contents of a Pull Request without having the PR feature exposed via the Gogs API. See: #27
2. Issues: https://github.com/gogs/docs-api/tree/master/Issues

• Comments - this is the core content of issues. We need to parse all the data and map it to the fields in Gogs.
• Labels - the primary metadata we use to categorize our issues, see: https://github.com/dwyl/labels
• Milestones - used to group issues into batches, e.g. a "sprint" or "feature".

3. Repo Stats: Stars, watchers, forks etc.
4. Your Feature Request Here! Seriously, if you spot a gap in the list of available functions, something you want/need to use Gogs in any a more advanced/custom way, please open an issue so we can discuss!

I'm Stuck! 🤷

As always, if anything is unclear or you are stuck getting this working, please open an issue! github.com/dwyl/gogs/issues we're here to help!

---

⚠️ Disclaimer! ⚠️

This package is provided "as is". We make no guarantee/warranty that it works.
We cannot be held responsible for any undesirable effects of it's usage. e.g: if you use the Gogs.delete/1 it will permanently/irrecoverably delete the repo. Use it with caution!

With the disclaimer out of the way, and your expectations clearly set, here are the facts: We are using this package in "production". We rely on it daily and consider it "mission critical". It works for us an and we have made every effort to document, test & maintain it. If you want to use it, go for it! But please note that we cannot "support" your usage beyond answering questions on GitHub. And unless you have a commercial agreement with [dwyl Ltd.]

If you spot anything that can be improved, please open an issue, we're very happy to discuss!

gogs's Issues

Setup Required Environment Variables

Build is failing because Environment Variables are not set: 🤦‍♂️
https://github.com/dwyl/gogs/runs/6256003701?check_suite_focus=true

This is a friendly error message from Envar not the noise we get from Elixir ... 🙄
This tells us exactly want we need to do next. 📝

Todo ⏭️

• see: https://github.com/dwyl/gogs/blob/main/.env_sample
• Access token should be secret: https://github.com/dwyl/gogs/settings/secrets/actions

https://github.com/dwyl/auth_plug/blob/e6b3fd039dbe1510981a29a83309cdc9190ae4af/.github/workflows/ci.yml#L31-L33

Use `Logger` to Log Info + Errors!

At present I've got a few IO.inspect statements peppered in the code (all commented out)
We can/should do better for debugging our apps in our staging (and even prod) environments!

Todo

• Add Logger statements in the code in strategic places e.g. errors and info.
• Borrow heavily from:
  • https://github.com/dwyl/envar/blob/main/lib/envar.ex#L10
  • https://github.com/dwyl/auth_plug/blob/main/lib/auth_plug.ex#L19

Feat: Create an Organisation on Gogs (Fly.io) via REST API

My reading of the Org API: https://github.com/gogs/docs-api/tree/master/Organizations
is that there is no way to create an Org, e.g:

PUT /orgs/:orgname

or

POST /orgs/:orgname

So going to try and use the "Edit" endpoint:
https://github.com/gogs/docs-api/tree/master/Organizations#edit-an-organization

PATCH /orgs/:org

If that fails, I will be opening an issue on: https://github.com/gogs/docs-api/issues ⏳
And I might end up writing some Go code today. 💭

Wish me luck! 🤞

Write up readme what, why how

Get list of Orgs from Gogs

Sadly, creating a new Organisation on the Gogs Server via the REST API doesn't appear to be possible: #8 (comment) so the next best thing is to:
a) create the org that we need manually via the Web UI: https://gogs-server.fly.dev/org/create
b) list all the orgs we have access to via REST API e.g: GET /user/orgs
c) if the org we need already exists, then we create the repo in that org.
otherwise we create the repo in the "global" org but name-space it to the desired org.
This isn't ideal, obviously, but I think it might be better than nothing. 💭

Create Local Branch of Remote Repo

I've got local_branch_create/2:

gogs/lib/gogs.ex

Lines 169 to 173 in c05cd77

	def local_branch_create(repo_name, _branch_name \\ "draft") do
	path = local_repo_path(repo_name)
	repo = %Git.Repository{path: path}
	Git.checkout(repo, ~w(-b draft)) # ["-b", branch_name])
	end

working on my machine:

But it fails on GitHub CI:

https://github.com/dwyl/gogs/runs/6297415703?check_suite_focus=true#step:6:58

I need to either

• create a stub/mock function
  or
• figure out how to write to the GitHub Actions CI filesystem

update links

http=>https

Feat: Git Diff should return the diff between two commits

This is a significantly more advanced feature than what we have already built
and it's one of the reasons we wanted to have this package separate from the App that uses it;
it needs to be extensively independently tested.

We will need to map this out carefully before diving in.

1. Diff a specific file or return a list of all the files with the diff nested?
2. Diff the current branch against the main (or master) branch?
3. Run the diff on localhost as there is no REST API endpoint for this.
4. Parse the result and return as plaintext so that we can visualise it

Create Repo on Fly.io Gogs instance

The first step (after creating an Org #8 ...) in using Gogs as a Backup for GitHub is to create a Repository!
Reading the API: https://github.com/gogs/docs-api/tree/master/Repositories#create

It appears this should be straightforward:

POST /org/:org/repos

Todo

• Create an interface to this endpoint that easily allows us to create a Repository e.g. Gogs.remote_repo_create/3
• Expose a Test Double that returns the desired payload when invoked during testing.
  • e.g: https://github.com/dwyl/elixir-auth-google/blob/main/lib/httpoison_mock.ex

Add dialyxir

Use dialyxir to detect errors in the code

see https://github.com/jeremyjh/dialyxir

At the moment running mix dialyzer returns

Total errors: 7, Skipped: 0, Unnecessary Skips: 0

We could also run dialyzer on CI, however from https://github.com/jeremyjh/dialyxir#continuous-integration there are a few points to be aware of:

Building the PLT file may take a while if a project has many dependencies
The PLT should be cached using the CI caching system
The PLT will need to be rebuilt whenever adding a new Erlang or Elixir version to build matrix

So it might be easier to run it on a pre-commit hook.

Chore: Add Typespec to all Functions/Arguments

I don't have time to do this right now but it's a really nice to have.

• See: https://hexdocs.pm/elixir/1.13/typespecs.html

Get Raw Contents of File from Git Repo

While building the demo: dwyl/gitea-demo#3
I've spotted a glaring omission: We don't currently have a way to read the contents of a file.
We're doing this with custom code in the App we're building.
But it's an obvious use case so should be included in the library.
Thankfully it won't be too much work to create it because it's just local filesystem access.

Todo

• Create remote_read_raw/4 - with org_name, repo_name and file_name as arguments
• local_file_read/3 same args. self-explanatory
• Should return the raw contents of the file for display.
• Test
• Docs
• Publish!

Delete all Repos in an Org During Testing

During testing we are creating a lot of "test-123" repos: https://gogs-server.fly.dev/myorg

The reason I'm opening this issue now is that there are enough repos in the org that my tests are failing:

If we attempt to drop the whole org (so we can re-create it empty) it doesn't work:
https://gogs-server.fly.dev/org/myorg/settings/delete

I don't want to include a function in the library that would allow someone to accidentally/carelessly "nuke" an entire Org ...
It would be irresponsible of us to build such a weapon never mind include it in the lib.

So instead I propose that we create a bash script that calls the List organization repositories REST API endpoint
https://github.com/gogs/docs-api/tree/master/Repositories#list-organization-repositories

GET /orgs/:orgname/repos

Then runs the delete a repo command in a loop for each repo.
https://stackoverflow.com/questions/32791663/how-to-run-curl-command-with-parameter-in-a-loop-from-bash-script
i.e. we can still use it for development purposes but it wouldn't be exported in the library on Hex.pm

"fatal: in unpopulated submodule 'test-repo'"

https://github.com/dwyl/gogs/runs/6315570116?check_suite_focus=true#step:6:63

Going to try this: https://github.community/t/github-action-not-pulling-dependant-submodules/17310/5 🤞

 - name: include stub submodule to base repo
   run: git submodule update --init

Gogs.push/1 Should Push the Current Branch to Gogs Server

We've already got most of the functions we need documented, tested and written: https://hexdocs.pm/gogs 🚀
This is the penultimate piece in the puzzle. And without it all the other functions are just "Work in Progress" ...

Todo

• Push current branch e.g. master, main, draft etc. to the remote Gogs Server
• Mock the function for GitHub/Test so that (a) it can run on CI and (b) it runs faster during tests.
• Write tests.
• Document it!
• Release new version of the package to Hex.pm

Inconsistency: Some Functions don't Require `org_name` ... 🤦‍♂️

while writing usage example docs, I noticed that Gogs.push/1 does not require the org name as an argument.
While this currently works, because of the "flat" cloning ... and would probably work fine for the project we're using this package in - because we are actually going to abstract the concept of "orgs" away in practice ... the issue is we don't want a published module/package to have an inconsistent API. 🤦‍♂️ (my bad!)

Todo ✅

• review all the functions in https://github.com/dwyl/gogs/blob/main/lib/gogs.ex and https://github.com/dwyl/gogs/blob/main/lib/helpers.ex 🔍
  • If the org_name is implicit, we need to update the function to require it to be passed in explicitly! 📝
• Update all the tests, typespecs, @doc comments, usage example [README.md] and actual implementation code to reflect that we want to always require both org_name and repo_name so that we use the org_name for name-spacing on the Elixir/Phoenix server. ⬆️
• Publish a new version to Hex.pm e.g: v1.0.0 🚀

@SimonLab, I know you're working in Elm Land right now ... 🌳
so there is a context-switch involved in me dragging you into looking at this. 🙏
But this really needs a fresh pair of eyes on it. Mine eyes have stared at it too long. 🙄
I think once this is done and we've published a new version, we can ship it in the "consuming" app. 🍕

Remove tests related module from Hex

I think we can remove at least the mocks module from hex.
I would maybe just left the Gogs module as it is the only one used when the package is installed on other project.

I think we can also convert some function to private functions. This will reduce the number of function in the public api

Errors on CI due to /tmp dir ...

https://github.com/dwyl/gogs/runs/6330998579?check_suite_focus=true#step:8:88

Will look at this in the morning with fresh eyes ...

Library guidelines

Review guidelines for building library: https://hexdocs.pm/elixir/main/library-guidelines.html

• Check for possible error inside the library:
  For example:
  

  gogs/lib/gogs.ex

  Lines 162 to 165 in 2c55a5b

  	def local_file_read(org_name, repo_name, file_name) do
  	file_path = Path.join([local_repo_path(org_name, repo_name), file_name])
  	File.read(file_path)
  	end

  
  Here we call the File.read function without checking for errors. We can add a case statement here to make sure we read the file without any errors or report the error from the library itself to make is easier to debug
  see: https://hexdocs.pm/elixir/main/library-guidelines.html#avoid-working-with-invalid-data

• Application configuration
  At the moment we are using module constant to defined from environment variable. We had a few issues linked to this and it might be better to let the application using the library define these values in the config file.
  see https://hexdocs.pm/elixir/main/library-guidelines.html#avoid-application-configuration

• Rename modules
  Create the Gogs namespace and use it for other modules:
  

  gogs/lib/helpers.ex

  Line 1 in 2c55a5b

  defmodule GogsHelpers do

  
  become Gogs.Helpers
  see: https://hexdocs.pm/elixir/main/library-guidelines.html#avoid-defining-modules-that-are-not-in-your-namespace

These are the main points that we can update for now.

Is the GITHUB_WORKSPACE env required

I'm not sure we still used the GITHUB_WORKSPACE environment variable:

gogs/lib/gogs_helpers.ex

Line 8 in e41e744

@github Envar.is_set?("GITHUB_WORKSPACE")

The only place we use it is here:

gogs/lib/gogs_helpers.ex

Lines 73 to 81 in e41e744

	def local_repo_path(repo_name) do
	# coveralls-ignore-start
	if @github || @mock do
	Path.join([temp_dir(@git_dir), "test-repo"])
	else
	Path.join([temp_dir(@git_dir), repo_name])
	end
	# coveralls-ignore-stop
	end

Could we remove it completely?

Delete Repo on Fly.io Gogs instance

Following on from the Create Repo #9 we need a way to programatically
We need a way to programmatically delete a repo so that we can clean-up during our E2E tests.

• Create remote_repo_delete/2
  • accepts: org_name and repo_name as arguments
  • DELETE /repos/:owner/:repo see: https://github.com/gogs/docs-api/tree/master/Repositories#delete
• Export TestDouble (HTTPMock) for testing

Clone an existing repo hosted on Gogs Fly https://gogs-server.fly.dev/

• e.g: https://gogs-server.fly.dev/nelsonic/public-repo

This works fine on my localhost:

git clone ssh://[email protected]:10022/nelsonic/public-repo.git

• Wrap the Git.Clone command in some syntactic sugar

Gogs.remote_render_markdown_html/4 should use Gogs' built-in Markdown renderer to return HTML!

https://github.com/gogs/docs-api/tree/master/Miscellaneous
Render an arbitrary Markdown document:

POST /markdown

e.g:

{
  "text": "Hello world gogs/gogs#1 **cool**, and #1!",
  "context": "https://github.com/gogs/gogs"
}

Returns:

Status: 200 OK
Content-Type: text/html

With a response body:

<p>Hello world <a href="https://try.gogs.io/gogs/gogs/issues/1" rel="nofollow">gogs/gogs#1</a> <strong>cool</strong>, and <a href="https://github.com/gogs/gogs/issues/1" rel="nofollow">#1</a>!</p>

This could be a really good way of:
a) simplifying the demo/example/tutorial to render HTML instead of having to include a JS library.
b) it could be used to create a "Blog" backed by Git i.e. totally version controlled!

The only "gotcha" could be a lack of more advanced HTML features. But we'd need to explore those!

Either way, I think it's worth exploring as a "spike".

#HelpWanted