waarzitjenu / server

Back-end server (API) of waarzitje.nu.

waarzitjenu/server

Waarzitje.nu is a basic application for submitting and retrieving live location updates. Originally focused on OsmAnd, but basically any application that supports REST could use it.

The back-end (REST API) of this application is made with Go, the front-end (map interface) is made with modern Vue 3 and TypeScript. Because of its modular setup, it is possible to run the back-end, front-end or a combination of both. 🎉

Image: OsmAnd app submitting location updates to the back-end, front-end retrieving the current location and showing it on a map. Learn how to set up OsmAnd.

---

Resources

Getting started

Please read the installation instructions to learn how to get started. It explains how to run the back-end, front-end or both, with and without Docker. 😉

API Documentation

To learn more about the API and see some examples of using it, take a look at the API Documentation.

Contributing

The contribution guidelines document briefly explains how to contribute to this repository. Also consider taking a look at the open issues to know what needs to be done and the kan-ban board to see who's currently working on what.

License

The license this project uses is the EUPL v1.2 or later. See LICENSE.txt. Available in other languages as well.

server's Issues

Improve maintainability of the project

I found out that the current source code layout makes it difficult to contribute concurrently.

Or, in other words, when receiving multiple pull requests at a given time, the source code of main.go changes so drastically that merge conflicts happen. That takes time, I need to open pull requests for others to keep their code up-to-date, etc. In other words: Very bad efficiency.

In order to fix this, it's mandatory that each component of go-osmand-tracker gets their own source code file(s). For example, the type definitions, database logic and web server logic should have their own files. This improves maintainability of the project, as each individual could master their own "piece of art".

To do

• Create folder structure for components
• Move type definitions to their own files
• Move database logic to their own files
• Move web server logic to their own files
• Move auxiliary code to their own files

Important: Project gets its own GitHub organization after Hacktoberfest

An important message to all users and contributors!

We're just halfway past Hacktoberfest 2020 and this project is already successful. Awesome! 🥇

---

What now?

After Hacktoberfest, this project will be moved to @waarzitjenu. "Waar zit je nu?" is Dutch for "Where are you now?".

What is it?

@waarzitjenu is a fully open-source location tracking platform, with a solid and fast back-end, an open API and a website with a great user interface. Inclusivity and an open culture are very important values.

For who?

It will be useful to both technical and non-technical users. So whether you're a software engineer working on the code or just a person who wants to share your live location with friends and family, you just go to @waarzitjenu.

---

Good to know

• The name of the repository will be changed to go-open-location-tracker (it's not limited to OsmAnd anymore).
• Contributions will not be altered (i.e. people will still see the names of all contributors).
• License stays the same (EUPL v1.2 or newer).
• We will continue to collaborate via GitHub, just like we did during Hacktoberfest.

And, as the project progresses, I might also create a separate repository for the map interface.

Save configuration data to local configuration file

The app should get a functionality to save and retrieve configuration data from a local file, for example, settings.json. This way, it doesn't always have to rely on command-line flags.

The functionality should be added to internal/settings/settings.go.

It should...

• Look in the user's home folder, i.e. ~/.go-osmand-tracker/settings.json
• Look in the folder where the app has been started (current working directory)
• ... or use default values and write them to the current working directory

It should not...

• Crash because the settings file(s) did not exist
• Crash because a settings file is invalid

Bonus points for...

• If the user's personal settings.json overrules and/or extends the rules in the current working directory. (a.k.a. local settings over global settings)

Authentication

The current version of go-osmand-tracker is just an HTTP endpoint, without authentication, practically providing no security at all.

There should be some sort of authentication added to go-osmand-tracker. Either Basic HTTP, Bearer Token Authentication or JWT.

Write contribution guidelines

The project should have CONTRIBUTING.md in its root folder to tell people how they could contribute to this project.

To do

1. Create an outline to have an idea of the global contents of CONTRIBUTING.md
2. Write out the document according to the outline

Create Dockerfile

Now that the project has both a back-end and a front-end interface (#15), it is recommended to create a Dockerfile that can be used to build the web interface (yarn i && yarn run build) and then run the Go back-end (go run main.go), so that the app can be started easily (docker build -t ricardobalk/go-osmand-tracker . && docker run [...] ricardobalk/go-osmand-tracker

Operation flashback

I found out that splitting the components to individual repositories is unnecessary and has a lot of inconvenient side effects, like having to manually replace local packages. It's not future-proof.

I also found out that you can make Go apps that are importable without separate repositories, take flashmob/go-guerrilla as an example. You could use that as a Go package.

So I basically need to partially revert 3d98b85, take a look at Guerrilla Mail how they did it and then move all separate repositories back here, but in a way that they can be used separately, if someone would need that. E.g. when someone only needs the database functionality, it would be cool to be able to import that package only.

It's probably something not that hard, like using package/package.go instead of internal/package/package.go for non-internal functionalities.

But hey, let's start Operation Flashback 😎

• New branch feature/issues/#56-operation-flashback
• Partially revert 3d98b85
• Merge into develop

Use the same endpoint for /retrieve and /retrieve/multi

Although /retrieve and /retrieve/multi looks good, retrieving location data could be made more practically.

• /submit should be the endpoint to submit location updates
• /retrieve should be the endpoint to retrieve location updates

That being said, how would someone retrieve multiple entries? Easy: they would use the count parameter. Like this: /retrieve?count=10 would return the last 10 location entries, /retrieve would just return the last location.

For now, we might even suboptimize /retrieve (without count parameter) to respond with the in-memory value, so it doesn't have to use the database for fetching the last submitted location. This could make the app slightly faster.

Multi-user support

The app can currently only receive location updates from a single user. When authentication (see #1) is built-in, it becomes possible to provide multi-user support as well.

Map doesn't work

Hi there!
I've spined up the whole server (back + front end) with docker-compose, and filled .env.local with correct values:

VUE_APP_LOCATION_API_ENDPOINT="https://waar.matih.duckdns.org/retrieve"
VUE_APP_MAPBOX_ACCESS_TOKEN="fsdhsdajkghsajgsahlfg..."

But map doesn't work:

I have it public, so you can even check this yourself: https://waar.matih.duckdns.org/

Saving tracks (database integration)

The current app only saves the last known location. There is no database that can save tracks.

Authentication using API keys

While testing the location tracker with a simple nginx reverse proxy set-up, I found out that the OsmAnd does not support Basic Authentication nor Digest Authentication. Bummer. However, TLS/SSL works. 👍

So, instead of using Basic Authentication, authentication with API keys would be a good alternative.

As TLS/SSL has been added (#44), the key can be added to the query parameters of a GET request (GET /submit?api_key=...). If used with TLS/SSL, it won't compromise the security. Only thing is that the server, when started in debug mode, might show the full URL with this "sensitive" data. Still, considering the server runs in a production environment, this is fine.

An even better way of sending an API key via a GET request would be by using a cookie or request header, like X-API-KEY: ..., so there won't be a URL containing an API key written to the log files. However, setting cookies or request headers not possible with OsmAnd.

Remove CLI arguments, only keep a --config

So, it seemed that passing CLI arguments were not the best way to configure this application, and a configuration file like settings.json is much more convenient.

We're aiming to solely rely on a single configuration file, either one that's in the apps' root directory or one passed via --config. Less (double) code makes the application more maintainable and less prone to errors.

Update README.md

README.md should get updated.

These things have been added:

• Saving tracks (#3)
• Choosing a custom port for the server (#4)

Custom port CLI flag

It's currently not possible to set the port from the command line. Should become possible by giving a command-line flag like --port 9000

Add CI/CD

Now we have Dockerfiles (#25), it would be cool if the project has some kind of CI/CD, like Travis, Jenkins, GitHub Actions, etc.

Docker provides an environment and a default setup for the application, CI/CD can take advantage of that.

Main goals for CI/CD

• Runs common linters and formatting tools on source tools (e.g. go fmt, go vet, npx prettier)
  — and commit automatically fixed lines to the PR
• Checks the resulting PR for compatibility (i.e. failed Docker builds should not pass)

Bonus points for

• If the CI/CD checks availability of the API endpoint (i.e. does the API really work?)
• If the CI/CD checks availability of the map interface by making screenshots with something like Puppeteer
  — but this stuff probably involves complex stuff like xvfb, as most CI/CD systems are headless and don't have a "real display".

Correct REST verbs for CRUD

The current version of go-osmand-tracker uses GET instead of POST, so it is not a correct REST-y way to receive location updates, but the OsmAnd uses GET requests for location updates, so that's why GET is used.

go-osmand-tracker should accept both GET (for backward compatibility) and POST (with a JSON body) in the future.

Operation	HTTP	SQL	DDS
Create	PUT / POST	INSERT	write
Read	GET	SELECT	read / take
Update	PUT / PATCH	UPDATE	write
Delete	DELETE	DELETE	dispose

Remove serving front-end from back-end

waarzitjenu/map@5bf0190 has been updated with a revamped Docker setup which includes nginx. However, the server part (this repo) still contains some code that allows to serve the front-end via the back-end, made possible by the following piece of code:

server/internal/engine/engine.go

Lines 41 to 50 in 0f4fa5d

	if filesystem.DoesDirExist("./web/dist") {
	ginEngine.Use(static.Serve("/", static.LocalFile("./web/dist", false)))
	} else {
	ginEngine.GET("/", func(c *gin.Context) {
	c.Header("Server", ServerIdentifier)
	c.Header("Content-Type", "text/plain")
	c.Writer.WriteHeader(501)
	c.String(http.StatusNotImplemented, "Sorry, %s is not implemented.", c.Request.RequestURI)
	})
	}

(It basically means: if the directory ./web/dist exists, point http://localhost:8080/ to that directory so that the front-end map is displayed)

I like the inclusion and use of nginx in the front-end more. I also think that it is better to use the back-end solely for saving and retrieving location updates. Less code to maintain, fewer things that could go wrong.

---

So here's the idea

• Remove the Git submodule
• Remove the piece of code shown above

Why

• Git Submodules are a bit overkill for a project like this.
• Back-end needs to do less things — which makes it require less resources and run faster.
• Less code to maintain, fewer things that could go wrong.

Impact of this change

Well, technically it is more difficult to serve both the back-end and front-end from a single endpoint, i.e. showing the map when going to http://localhost:8080/ and also use the REST endpoints (e.g. http://localhost:8080/retrieve), but there are two alternatives: setting up a reverse proxy or agree with using different ports. Nginx is flexible and well-maintained — if needed, it could be set up as a reverse proxy, basically providing the same functionality 🎉

Fix panic on empty database

When starting the server for the very first time, without locations added to a database, the application panics:

2020/10/11 18:23:41 [Recovery] 2020/10/11 - 18:23:41 panic recovered:
runtime error: index out of range [0] with length 0
/usr/lib/go-1.14/src/runtime/panic.go:88 (0x432732)
	goPanicIndex: panic(boundsError{x: int64(x), signed: true, y: y, code: boundsIndex})
/home/undisclosed/Local/Git/GitHub/ricardobalk/go-osmand-tracker/internal/server/server.go:132 (0x938e2a)
	Listen.func3: lastDatabaseAddition[0] = entry
/home/undisclosed/.golang/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0x922b6a)
	(*Context).Next: c.handlers[c.index](c)
/home/undisclosed/.golang/pkg/mod/github.com/gin-gonic/[email protected]/recovery.go:83 (0x935f3f)
	RecoveryWithWriter.func1: c.Next()
/home/undisclosed/.golang/pkg/mod/github.com/gin-gonic/[email protected]/context.go:161 (0x922b6a)
	(*Context).Next: c.handlers[c.index](c)
/home/undisclosed/.golang/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:409 (0x92c615)
	(*Engine).handleHTTPRequest: c.Next()
/home/undisclosed/.golang/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:367 (0x92bd2c)
	(*Engine).ServeHTTP: engine.handleHTTPRequest(c)
/usr/lib/go-1.14/src/net/http/server.go:2807 (0x7495d2)
	serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
/usr/lib/go-1.14/src/net/http/server.go:1895 (0x744f4b)
	(*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
/usr/lib/go-1.14/src/runtime/asm_amd64.s:1373 (0x4642d0)
	goexit: BYTE	$0x90	// NOP

It is caused by https://github.com/ricardobalk/go-osmand-tracker/blob/bb156cdc8e7977ae6e79fcefe4a57b94486b03e4/internal/server/server.go#L132

because the lastDatabaseAddition array contains no keys when starting the app for the first time - so it is impossible to place something to the non-existing keyindex 0.

Who is affected by this bug?

• Anyone starting the app for the first time, trying to put the first location into the database via /submit

Use request framework

A nice suggestion by @raydwaipayan was to use a framework like Gin, see #1. It would make the code easier to read and also easier to maintain.

Possible solutions include

• gin-gonic/gin
• gorilla/mux

@raydwaipayan I'm giving this one to you 😄

Add debug CLI flag

There should be a --debug CLI flag.

Default behaviour

• Default behaviour is to not output anything to the console, except informative messages ("Server started on port xxxx").
• The Gin router runs in "release" mode. (gin.SetMode(gin.ReleaseMode))

Debug mode

• Allows more verbose messages.
• The Gin router runs in "debug" mode

Add OpenAPI Specification

I created a Postman collection for running tests against go-osmand-tracker, and also published Postman Documentation.

The API requests and response examples should be converted into OpenAPI format (formerly Swagger) and added to this repository.

TLS (SSL) support

The app should get support for TLS certificates, provided via the command line and/or local configuration file (see #12).

For example, by using command line flags like --cert '/path/to/pubkey.pem' --cert-ca '/path/to/ca.pem' --cert-private '/path/to/privkey.pem'

TLS should be enabled by default, because location data is privacy sensitive and it's a good practice to use TLS by default. If no certificate paths are provided, the user will see a warning and the app will exit. To override this behaviour, the user could add the --no-tls flag to disable it.

Update README docs

The documentation should be updated now the project moved forward and new things like have been added (e.g. database for saving tracks (#3), a fancy map interface (#15), Gin framework (#17), configuration via a settings file (#24), dockerfiles (#25), ...)

I'll take the responsibility to update the README docs.

Change the tag of this project to #openstreetmap instead of #openstreetmapS

Hi,
This is a very small thing, but I noticed your github project has the # openstreetmaps (with an S at the end, but it should actually be openstreetmap without the S), it's an easy typo to make.

Write test cases and update CI/CD file

Now the core of the go-osmand-tracker Go application has been turned into the waarzitjenu/core module, it should get test cases.

The reason for adding test cases to a Go module is because a module can't be tested by simply running it, like one would do with a Go application. i.e. you can't simply do go run ., as the package is not main. However, it is possible to build test cases for each function to check their actual behaviour against expected behaviour. (= Test Driven Development).

An example of test cases for a Go module can be found at one of my previous Go projects: https://github.com/LighthouseLab/go-maidenhead.

I will take care of these test cases.

Map interface

A nice web interface with a map showing the last location would be awesome.

Basically, my idea comes down to this:

• To see a map with the current location, visit / from a browser.
• To submit location updates, use the /submit endpoint, as usual.
• To retrieve raw location updates, use /retrieve or /retrieve/multi.

Going to make something for it with React or Vue. However, I need to wait for #14 first.