heidihoward / ios

Reliable distributed agreement service for the cloud

Welcome to Ios, a reliable distributed agreement service for cloud applications. Built upon a novel decentralised consensus protocol, Ios provides vital services for your cloud application such as distributed locking, consistent data structures and leader election as well as distributed configuration and coordination.

This repository is pre-alpha and under active development. APIs will be broken. This code has not been proven correct and is not ready for production deployment.

Getting Started

These instructions will get you a simple Ios server and client up and running on your local machine. See deployment for notes on how to deploy Ios across a cluster.

Prerequisites

Ios is built in Go version 1.6.2 and currently supports Go version 1.6 and 1.7. The Golang site details how to install and setup Go on your local machine. Don't forget to add GOPATH to your .profile.

Installation

After installing Go, run:

go get github.com/heidi-ann/ios/...

This command will copy the Ios source code to $GOPATH/src/github.com/heidi-ann/ios and then fetch and build the following dependancies:

• glog - logging library, in the style of glog for C++
• gcfg - library for parsing git-config style config files It will then build and install Ios, the server and client binaries will be placed in $GOPATH/bin.

Up & Running

You can now start a simple 1 node Ios cluster as follows:

$GOPATH/bin/ios -id 0

This will start an Ios server providing a simple key-value store. The server is listening for clients on port 8080.

You can now start an Ios client as follows:

$ $GOPATH/bin/clientcli
Starting Ios client in interactive mode.

The following commands are available:
	get [key]: to return the value of a given key
	exists [key]: to test if a given key is present
	update [key] [value]: to set the value of a given key, if key already exists then overwrite
	delete [key]: to remove a key value pair if present
	count: to return the number of keys
	print: to return all key value pairs

Enter command: update A 1
OK
Enter command: get A
1
...

You can now enter commands for the key value store, followed by the enter key. These commands are being sent to the Ios server, executed and the result is returned to the user.

The Ios server is using files called persistent_log_0.temp, persistent_snap_0.temp and persistent_data_0.temp to store Ios's persistent state. If these files are present when a server starts, it will restore its state from these files. You can try this by killing the server process and restarting it, it should carry on from where it left off.

When you would like to start a fresh server instance, use rm persistent*.temp first to clear these files and then start the server again.

Building in Docker

Alternatively, you can build and run in Ios using Docker. Make sure Docker is installed and running, then clone this repository and cd into it.

Build an image named 'ios' using the following command

docker build -t ios .

You should now be able to run docker images and see the Ios image you just created. If you run, docker ps you will see that no docker containers are currently running, start a simple 1 node Ios cluster as follows:

docker run --name ios-server -d ios -id 0

Running docker ps we can now see that is Ios server is running. We can test this by communicate with it using an Ios command line client:

docker run --net=container:ios-server -it --name ios-client --entrypoint clientcli ios

Note that this will only use storage local to the container instance. If you want persistence/recoverability for instances you will need to store persistence logs on a mounted data volume

Next steps

In this section, we are going to take a closer look at what is going on underneath. We will then use this information to setup a 3 server Ios cluster on your local machine and automatically generate a workload to put it to the test. PS: you might want to start by opening up a few terminal windows.

Server configuration

The server we ran in previous section was using the default configuration file found in example.conf. The first section of this file lists the Ios servers in the cluster and how the peers can connect to them and the second section lists how the client can connect to them. The configuration file example3.conf shows what this looks like for 3 servers running on localhost. The same configuration file is used for all the servers, at run time they are each given an ID (starting from 0) and use this to know which ports to listen on. The rest of the configuration file options are documented at https://godoc.org/github.com/heidi-ann/ios/config. After removing the persistent storage, start 3 Ios servers in 3 separate terminal windows as follows:

$GOPATH/bin/ios -id [ID] -config $GOPATH/src/github.com/heidi-ann/ios/configfiles/simple/server3.conf -stderrthreshold=INFO

For ID 0, 1 and 2

Client configuration

Like the servers, the client we ran in the previous section was using the default configuration file found in example.conf. The first section lists the Ios servers in the cluster and how to connect to them. The configuration file example3.conf shows what this looks like for 3 servers currently running on localhost.

We are run a client as before and interact with our 3 servers.

$GOPATH/bin/clientcli -config $GOPATH/src/github.com/heidi-ann/ios/client/example3.conf

You should be able to kill and restart the servers to test when the system is available to the client. Since the Ios cluster you have deployed is configured to use strict majority quorums then the system should be available whenever at least two servers are up.

Workload configuration

Typing requests into a terminal is, of course, slow and unrealistic. To help test the system, Ios provides test clients which can automatically generate a workload and measure system performance. To run a client in test mode use:

$GOPATH/bin/test -config $GOPATH/src/github.com/heidi-ann/ios/client/example3.conf -auto $GOPATH/src/github.com/heidi-ann/ios/test/workload.conf

This client will run the workload described in test/workloads/example.conf and then terminate. It will write performance metrics into a file called latency.csv. Ios currently also support a REST API mode which listens for HTTP on port 12345.

Contributing

Debugging

We use glog for logging. Adding -logtostderr=true -v=1 when running executables prints the logging output. For more information, visit https://godoc.org/github.com/golang/glog.

Likewise, the following commands work with the above example and are useful for debugging:

sudo tcpdump -i lo0 -nnAS "(src portrange 8080-8092 or dst portrange 8080-8092) and (length>0)"

sudo strace -p $(pidof server) -T -e fsync -f
sudo strace -p $(pidof server) -T -e trace=write -f

Benchmarking

The benchmarking scripts for Ios can found here https://github.com/heidi-ann/consensus_eval

License

This project is licensed under the MIT License - see the LICENSE.md file for details

ios's Issues

concurrent map writes in window.go

Bug found on OSX running the 3 server deployment from example3.conf with test clients.

fatal error: concurrent map writes

goroutine 51 [running]:
runtime.throw(0x28a3b0, 0x15)
	/usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc820b175e0 sp=0xc820b175c8
runtime.mapassign1(0x1c60c0, 0xc82000a600, 0xc820b17748, 0xc820b1771e)
	/usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc820b17688 sp=0xc820b175e0
github.com/heidi-ann/ios/consensus.runMaster(0x0, 0xffffffffffffffff, 0x1, 0xc8200fc000, 0x0, 0x3, 0x186a0, 0x0, 0x64, 0x0, ...)
	/Users/heidi/go/src/github.com/heidi-ann/ios/consensus/master.go:145 +0xc6b fp=0xc820b17b50 sp=0xc820b17688
github.com/heidi-ann/ios/consensus.monitorMaster(0xc820094300, 0xc8200fc000, 0x0, 0x3, 0x186a0, 0x0, 0x64, 0x0, 0xa, 0x64, ...)
	/Users/heidi/go/src/github.com/heidi-ann/ios/consensus/master.go:14 +0x1cd fp=0xc820b17f20 sp=0xc820b17b50
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820b17f28 sp=0xc820b17f20
created by github.com/heidi-ann/ios/consensus.Init
	/Users/heidi/go/src/github.com/heidi-ann/ios/consensus/consensus.go:62 +0x6b9

goroutine 1 [chan receive, 1 minutes]:
main.main()
	/Users/heidi/go/src/github.com/heidi-ann/ios/server/server.go:90 +0x11f3

I will look into it further now

Arguments after glog ignored

Arguments passed to /bin/server/ after -logtostderr true are ignored

Docker 3 server test script failing

The output of deploy3serversfailed (previously PR #21) is not as expected. After node0 fails, the client seems unable to recover.

runtime: out of memory due to JSON marshal/unmarshal

There are various memory issues with JSON marshal/unmarshal. At the moment, we do this either in a strange verbose manner or has been replicated by a placeholder to prevent errors like this:

~/go/bin/server -id=0
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x5ddf0c, 0x16)
	/usr/local/go/src/runtime/panic.go:566 +0x95
runtime.sysMap(0xc440500000, 0x20000000, 0xa00001f700, 0x6d64b8)
	/usr/local/go/src/runtime/mem_linux.go:219 +0x1d0
runtime.(*mheap).sysAlloc(0x6bda40, 0x20000000, 0x34e3f)
	/usr/local/go/src/runtime/malloc.go:407 +0x37a
runtime.(*mheap).grow(0x6bda40, 0x10000, 0x0)
	/usr/local/go/src/runtime/mheap.go:726 +0x62
runtime.(*mheap).allocSpanLocked(0x6bda40, 0x10000, 0x0)
	/usr/local/go/src/runtime/mheap.go:630 +0x4f2
runtime.(*mheap).allocStack(0x6bda40, 0x10000, 0x40f0fc)
	/usr/local/go/src/runtime/mheap.go:597 +0x62
runtime.stackalloc(0x20000000, 0xc41fd57b00, 0x100, 0x40, 0x2000, 0x7f0d9b218710)
	/usr/local/go/src/runtime/stack.go:395 +0x2ed
runtime.copystack(0xc4200001a0, 0x20000000, 0x1)
	/usr/local/go/src/runtime/stack.go:839 +0x83
runtime.newstack()
	/usr/local/go/src/runtime/stack.go:1070 +0x370
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:366 +0x7f

goroutine 1 [copystack]:
encoding/json.(*decodeState).object(0xc42550ee10, 0x5aa960, 0xc4254fd668, 0x16)
	/usr/local/go/src/encoding/json/decode.go:557 fp=0xc430500488 sp=0xc430500480
encoding/json.(*decodeState).value(0xc42550ee10, 0x5aa960, 0xc4254fd668, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430500508 sp=0xc430500488
encoding/json.(*decodeState).unmarshal(0xc42550ee10, 0x5aa960, 0xc4254fd668, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430500580 sp=0xc430500508
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd668, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc4305005c8 sp=0xc430500580
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd650, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430500610 sp=0xc4305005c8
encoding/json.(*decodeState).object(0xc42550ed20, 0x5aa960, 0xc4254fd650, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc4305008a0 sp=0xc430500610
encoding/json.(*decodeState).value(0xc42550ed20, 0x5aa960, 0xc4254fd650, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430500920 sp=0xc4305008a0
encoding/json.(*decodeState).unmarshal(0xc42550ed20, 0x5aa960, 0xc4254fd650, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430500998 sp=0xc430500920
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd650, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc4305009e0 sp=0xc430500998
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd638, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430500a28 sp=0xc4305009e0
encoding/json.(*decodeState).object(0xc42550ec30, 0x5aa960, 0xc4254fd638, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430500cb8 sp=0xc430500a28
encoding/json.(*decodeState).value(0xc42550ec30, 0x5aa960, 0xc4254fd638, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430500d38 sp=0xc430500cb8
encoding/json.(*decodeState).unmarshal(0xc42550ec30, 0x5aa960, 0xc4254fd638, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430500db0 sp=0xc430500d38
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd638, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430500df8 sp=0xc430500db0
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd620, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430500e40 sp=0xc430500df8
encoding/json.(*decodeState).object(0xc42550eb40, 0x5aa960, 0xc4254fd620, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc4305010d0 sp=0xc430500e40
encoding/json.(*decodeState).value(0xc42550eb40, 0x5aa960, 0xc4254fd620, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430501150 sp=0xc4305010d0
encoding/json.(*decodeState).unmarshal(0xc42550eb40, 0x5aa960, 0xc4254fd620, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc4305011c8 sp=0xc430501150
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd620, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430501210 sp=0xc4305011c8
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd608, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430501258 sp=0xc430501210
encoding/json.(*decodeState).object(0xc42550ea50, 0x5aa960, 0xc4254fd608, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc4305014e8 sp=0xc430501258
encoding/json.(*decodeState).value(0xc42550ea50, 0x5aa960, 0xc4254fd608, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430501568 sp=0xc4305014e8
encoding/json.(*decodeState).unmarshal(0xc42550ea50, 0x5aa960, 0xc4254fd608, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc4305015e0 sp=0xc430501568
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd608, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430501628 sp=0xc4305015e0
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd5f0, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430501670 sp=0xc430501628
encoding/json.(*decodeState).object(0xc42550e960, 0x5aa960, 0xc4254fd5f0, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430501900 sp=0xc430501670
encoding/json.(*decodeState).value(0xc42550e960, 0x5aa960, 0xc4254fd5f0, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430501980 sp=0xc430501900
encoding/json.(*decodeState).unmarshal(0xc42550e960, 0x5aa960, 0xc4254fd5f0, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc4305019f8 sp=0xc430501980
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd5f0, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430501a40 sp=0xc4305019f8
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd5d8, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430501a88 sp=0xc430501a40
encoding/json.(*decodeState).object(0xc42550e870, 0x5aa960, 0xc4254fd5d8, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430501d18 sp=0xc430501a88
encoding/json.(*decodeState).value(0xc42550e870, 0x5aa960, 0xc4254fd5d8, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430501d98 sp=0xc430501d18
encoding/json.(*decodeState).unmarshal(0xc42550e870, 0x5aa960, 0xc4254fd5d8, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430501e10 sp=0xc430501d98
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd5d8, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430501e58 sp=0xc430501e10
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd5c0, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430501ea0 sp=0xc430501e58
encoding/json.(*decodeState).object(0xc42550e780, 0x5aa960, 0xc4254fd5c0, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430502130 sp=0xc430501ea0
encoding/json.(*decodeState).value(0xc42550e780, 0x5aa960, 0xc4254fd5c0, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc4305021b0 sp=0xc430502130
encoding/json.(*decodeState).unmarshal(0xc42550e780, 0x5aa960, 0xc4254fd5c0, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430502228 sp=0xc4305021b0
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd5c0, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430502270 sp=0xc430502228
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd5a8, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc4305022b8 sp=0xc430502270
encoding/json.(*decodeState).object(0xc42550e690, 0x5aa960, 0xc4254fd5a8, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430502548 sp=0xc4305022b8
encoding/json.(*decodeState).value(0xc42550e690, 0x5aa960, 0xc4254fd5a8, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc4305025c8 sp=0xc430502548
encoding/json.(*decodeState).unmarshal(0xc42550e690, 0x5aa960, 0xc4254fd5a8, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430502640 sp=0xc4305025c8
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd5a8, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430502688 sp=0xc430502640
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd590, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc4305026d0 sp=0xc430502688
encoding/json.(*decodeState).object(0xc42550e5a0, 0x5aa960, 0xc4254fd590, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430502960 sp=0xc4305026d0
encoding/json.(*decodeState).value(0xc42550e5a0, 0x5aa960, 0xc4254fd590, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc4305029e0 sp=0xc430502960
encoding/json.(*decodeState).unmarshal(0xc42550e5a0, 0x5aa960, 0xc4254fd590, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430502a58 sp=0xc4305029e0
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd590, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430502aa0 sp=0xc430502a58
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd578, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430502ae8 sp=0xc430502aa0
encoding/json.(*decodeState).object(0xc42550e4b0, 0x5aa960, 0xc4254fd578, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430502d78 sp=0xc430502ae8
encoding/json.(*decodeState).value(0xc42550e4b0, 0x5aa960, 0xc4254fd578, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430502df8 sp=0xc430502d78
encoding/json.(*decodeState).unmarshal(0xc42550e4b0, 0x5aa960, 0xc4254fd578, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430502e70 sp=0xc430502df8
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd578, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430502eb8 sp=0xc430502e70
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd560, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430502f00 sp=0xc430502eb8
encoding/json.(*decodeState).object(0xc42550e3c0, 0x5aa960, 0xc4254fd560, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430503190 sp=0xc430502f00
encoding/json.(*decodeState).value(0xc42550e3c0, 0x5aa960, 0xc4254fd560, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430503210 sp=0xc430503190
encoding/json.(*decodeState).unmarshal(0xc42550e3c0, 0x5aa960, 0xc4254fd560, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430503288 sp=0xc430503210
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd560, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc4305032d0 sp=0xc430503288
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd548, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430503318 sp=0xc4305032d0
encoding/json.(*decodeState).object(0xc42550e2d0, 0x5aa960, 0xc4254fd548, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc4305035a8 sp=0xc430503318
encoding/json.(*decodeState).value(0xc42550e2d0, 0x5aa960, 0xc4254fd548, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430503628 sp=0xc4305035a8
encoding/json.(*decodeState).unmarshal(0xc42550e2d0, 0x5aa960, 0xc4254fd548, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc4305036a0 sp=0xc430503628
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd548, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc4305036e8 sp=0xc4305036a0
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd530, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430503730 sp=0xc4305036e8
encoding/json.(*decodeState).object(0xc42550e1e0, 0x5aa960, 0xc4254fd530, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc4305039c0 sp=0xc430503730
encoding/json.(*decodeState).value(0xc42550e1e0, 0x5aa960, 0xc4254fd530, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430503a40 sp=0xc4305039c0
encoding/json.(*decodeState).unmarshal(0xc42550e1e0, 0x5aa960, 0xc4254fd530, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430503ab8 sp=0xc430503a40
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd530, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430503b00 sp=0xc430503ab8
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd518, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430503b48 sp=0xc430503b00
encoding/json.(*decodeState).object(0xc42550e0f0, 0x5aa960, 0xc4254fd518, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430503dd8 sp=0xc430503b48
encoding/json.(*decodeState).value(0xc42550e0f0, 0x5aa960, 0xc4254fd518, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430503e58 sp=0xc430503dd8
encoding/json.(*decodeState).unmarshal(0xc42550e0f0, 0x5aa960, 0xc4254fd518, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430503ed0 sp=0xc430503e58
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd518, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430503f18 sp=0xc430503ed0
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd500, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430503f60 sp=0xc430503f18
encoding/json.(*decodeState).object(0xc42550e000, 0x5aa960, 0xc4254fd500, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc4305041f0 sp=0xc430503f60
encoding/json.(*decodeState).value(0xc42550e000, 0x5aa960, 0xc4254fd500, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430504270 sp=0xc4305041f0
encoding/json.(*decodeState).unmarshal(0xc42550e000, 0x5aa960, 0xc4254fd500, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc4305042e8 sp=0xc430504270
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd500, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430504330 sp=0xc4305042e8
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd4e8, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430504378 sp=0xc430504330
encoding/json.(*decodeState).object(0xc42550def0, 0x5aa960, 0xc4254fd4e8, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430504608 sp=0xc430504378
encoding/json.(*decodeState).value(0xc42550def0, 0x5aa960, 0xc4254fd4e8, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430504688 sp=0xc430504608
encoding/json.(*decodeState).unmarshal(0xc42550def0, 0x5aa960, 0xc4254fd4e8, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430504700 sp=0xc430504688
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd4e8, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430504748 sp=0xc430504700
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd4d0, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430504790 sp=0xc430504748
encoding/json.(*decodeState).object(0xc42550de00, 0x5aa960, 0xc4254fd4d0, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430504a20 sp=0xc430504790
encoding/json.(*decodeState).value(0xc42550de00, 0x5aa960, 0xc4254fd4d0, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430504aa0 sp=0xc430504a20
encoding/json.(*decodeState).unmarshal(0xc42550de00, 0x5aa960, 0xc4254fd4d0, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430504b18 sp=0xc430504aa0
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd4d0, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430504b60 sp=0xc430504b18
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd4b8, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430504ba8 sp=0xc430504b60
encoding/json.(*decodeState).object(0xc42550dd10, 0x5aa960, 0xc4254fd4b8, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430504e38 sp=0xc430504ba8
encoding/json.(*decodeState).value(0xc42550dd10, 0x5aa960, 0xc4254fd4b8, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc430504eb8 sp=0xc430504e38
encoding/json.(*decodeState).unmarshal(0xc42550dd10, 0x5aa960, 0xc4254fd4b8, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430504f30 sp=0xc430504eb8
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd4b8, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430504f78 sp=0xc430504f30
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd4a0, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc430504fc0 sp=0xc430504f78
encoding/json.(*decodeState).object(0xc42550dc20, 0x5aa960, 0xc4254fd4a0, 0x16)
	/usr/local/go/src/encoding/json/decode.go:562 +0x1cb5 fp=0xc430505250 sp=0xc430504fc0
encoding/json.(*decodeState).value(0xc42550dc20, 0x5aa960, 0xc4254fd4a0, 0x16)
	/usr/local/go/src/encoding/json/decode.go:368 +0x2cc fp=0xc4305052d0 sp=0xc430505250
encoding/json.(*decodeState).unmarshal(0xc42550dc20, 0x5aa960, 0xc4254fd4a0, 0x0, 0x0)
	/usr/local/go/src/encoding/json/decode.go:169 +0x1fa fp=0xc430505348 sp=0xc4305052d0
encoding/json.Unmarshal(0xc42008029c, 0x2, 0x4, 0x5aa960, 0xc4254fd4a0, 0x4, 0xc42008029c)
	/usr/local/go/src/encoding/json/decode.go:97 +0x102 fp=0xc430505390 sp=0xc430505348
github.com/heidi-ann/ios/services.(*dummy).UnmarshalJSON(0xc4254fd488, 0xc42008029c, 0x2, 0x4, 0x0, 0x7f0d9dce8a78)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/services/dummy.go:26 +0x6c fp=0xc4305053d8 sp=0xc430505390
...additional frames elided...

Segmentation violation in test suite

Test suite is currently failing due to the following:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x4eb15b]

goroutine 19 [running]:
panic(0x5a6120, 0xc4200100a0)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/heidi-ann/ios/app.(*StateMachine).Apply(0xc4200143c0, 0x0, 0x1, 0x0, 0xc4200148a0, 0x11, 0x0, 0x0, 0x0, 0x0, ...)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/app/app.go:32 +0x3bb
github.com/heidi-ann/ios/consensus.runParticipant(0xc420017940, 0xc4200d2000, 0x0, 0x1, 0x186a0, 0x0, 0x1, 0x0, 0x1, 0x3e8, ...)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/consensus/participant.go:69 +0x1379
github.com/heidi-ann/ios/consensus.Init(0xc4200d2000, 0x0, 0x1, 0x186a0, 0x0, 0x1, 0x0, 0x1, 0x3e8, 0x5daa3e, ...)
	/home/ubuntu/go/src/github.com/heidi-ann/ios/consensus/consensus.go:63 +0x575
created by main.main
	/home/ubuntu/go/src/github.com/heidi-ann/ios/server/server.go:77 +0xe4b

I will take a look into it now

Message dispatch issue on master failure

There exists a serious bug in the IO abstraction used in server.go.

To reproduce, set scripts/failure_testing.sh to kill node 0, run it and observe that the New View RPC is not sent by node 1 to node 2.

Switch configuration file parsing library

Currently, we use gvfg for parsing configuration files however it does not support optional parameters. The result is that Ios configuration files have many parameters most of which are useless to a standard developer

Add support for Windows

Currently Ios only builds Linux due to use of syscalls such as Fallocate, Fdatasync and directI/O

Add -fresh option to server

Add -fresh option to server command line to instruct the Ios server to ignore and overwrite persistent storage if present. This will save the need to run rm persistent_* before starting a fresh Ios instance.

Performance degradation once write ahead log exceeds 64MB

Currently, the write ahead log grows in infinitely, in particular we observe performance degradation once the write ahead log exceeds 64MB, the size allocated by Fallocate.

Efficient representation of NO-OPS

Currently, NO-OPS are represented explicitly like all other log entries, this is not necessary and could be done much more efficiently

[question] consensus protocol for `ios`

[question]

@Heidi-Ann

is ios the implementation of A Generalised Solution to Distributed Consensus?

which consensus protocol it is using?

Quick negative response for malformed requests

Currently, if a request for an application state machine is malformed then it must be replicated by consensus system before being applied to the state machine and detected. Server instead could check formatting before replication

package github.com/heidi-ann/ios: no buildable Go source file

Resolve package github.com/heidi-ann/ios: no buildable Go source file to avoid confusion

add a configuration file validator

Sometimes client, server or workload configuration files are wrong/incorrectly formatted. Currently, we need to start Ios and wait for some exception to discover this. Would be useful to add a configuration verifier which hooks into functions like config.ParseServerConfig to check config files.

Add cluster ID to all I/O

Currently, multiple deployments of Ios running across the same hosts risk interfering with each other. This could be address by assigning each deployment at random ID and checking this ID when receiving messages/reading from disk

Fix builds on OSX

/unix/wal.go uses syscalls such as Fallocate, Fdatasync which are not available on OSX so Ios will not build. This could be fixed by adding an wal_osx.go which doesn't use these calls and building the correct version depending on the host

Add process manager/docker to make setup easier for newcomers

Currently, setting up an Ios cluster locally can require many tabs/window and managing quite a few processes. This could be greatly improved using Docker or a tools like Foreman.

Replace JSON with Protobuf for serializing state

Currently, we use JSON for serialising data as it was easy to implement and is human readable. This should be switched to Protobuf or similar before a production deployment.

Skip recovery/leader election for 1 node clusters

There is no need to execute full recovery and leader election when there is only one member in a Ios cluster, this can be skipped