zegl / goriak

goriak - Go language driver for Riak KV

goriak

Current version: v3.2.1.
Riak KV version: 2.0 or higher, the latest version of Riak KV is always recommended.

What is goriak?

goriak is a wrapper around riak-go-client (version 1.9.0 or newer is required) to make it easier and more friendly for developers to use Riak KV.

• Documentation

Installation

As a Go module (requires Go 1.11 or later):

go get github.com/zegl/goriak/[email protected]

As a Go package:

go get -u gopkg.in/zegl/goriak.v3

Maps (Riak Data Types)

The main feature of goriak is that goriak automatically can marshal/unmarshal your Go types into Riak data types.

Set (Riak Data Types)

In the example below Name will be saved as a register, and Aliases will be a set.

type User struct {
    Name    string
    Aliases []string
}

user := User {
    Name:   "Foo",
    Alises: []string{"Foo", "Bar"},
}

goriak.Bucket("bucket-name", "bucket-type").Set(user).Key("key").Run(c)

Tags

Struct tags can be used to change the name of the item, or to ignore it.

type User struct {
    Name    string   `goriak:"-"`       // Ignore field
    Aliases []string `goriak:"aliases"` // Save as "aliases" in Riak KV
}

Get (Riak Data Types)

The map can later be retreived as a whole:

var res User
goriak.Bucket("bucket-name", "bucket-type").Get("key", &res).Run(c)

Supported Go types

Go Type	Riak Type
struct	map
string	register
[n]byte	register
[]byte	register
[]slice	set
[]slice	set
[][]byte	set
map	map
time.Time	register
int [1]	register

1: All signed and unsigned integer types are supported.

Golang map types

Supported key types: all integer types, string.
Supported value types: string, []byte.

Helper types

Some actions are more complicated then necessary with the use of the default Go types and MapOperations.

This is why goriak contains the types Counter, Set, Flag and Register. All of these types will help you performing actions such as incrementing a value, or adding/removing items.

Counters

Riak Counters is supported with the special goriak.Counter type.

Example:

type Article struct {
    Title string
    Views *goriak.Counter
}

// Get our object
var article Article
goriak.Bucket("articles", "map").Get("1-hello-world", &article).Run(con)

// Increase views by 1
err := article.Views.Increase(1).Exec(con)

// check err

Counter.Exec(con) will make a lightweight request to Riak, and the counter is the only object that will be updated.

You can also save the changes to your counter with SetMap(), this is useful if you want to change multiple counters at the same time.

Check godoc for more information.

Sets

You can chose to use goriak.Set to help you with Set related actions, such as adding and removing items. goriak.Set also has support for sending incremental actions to Riak so that you don't have to build that functionality yourself.

Example:

type Article struct {
    Title string
    Tags *goriak.Set
}

// Get our object
var article Article
goriak.Bucket("articles", "map").Get("1-hello-world", &article).Run(con)

// Add the tag "animals"
err := article.Tags.AddString("animals").Exec(con)

// check err

Check godoc for more information.

Values

Values can be automatically JSON Marshalled/Unmarshalled by using SetJSON() and GetJSON(). There is also SetRaw() and GetRaw() that works directly on []bytes.

JSON

// Set object
goriak.Bucket("bucket-name", "bucket-type").SetJSON(obj).Key("key").Run(con)

// Get object
goriak.Bucket("bucket-name", "bucket-type").GetJSON("key", &obj).Run(con)

MapOperation

There is a time in everyones life where you need to perform raw MapOperations on your Riak Data Values.

Some operations, such as RemoveFromSet requires a Context to perform the operation. A Context can be retreived from Get by setting a special context type.

type ourType struct {
    Aliases []string

    // The context from Riak will be added if the tag goriakcontext is provided
    Context []byte `goriak:"goriakcontext"`
}

// ... GetMap()

// Works with MapOperation from github.com/basho/riak-go-client
operation := goriak.NewMapOperation()
operation.AddToSet("Aliases", []byte("Baz"))

goriak.MapOperation("bucket-name", "bucket-type", "key", operation, val.Context)

Secondary Indexes

You can set secondary indexes automatically with SetJSON() by using struct tags.

Strings and all signed integer types are supported. Both as-is and in slices.

type User struct {
    Name    string `goriakindex:"nameindex_bin"`
    Aliases []string
}

Indexes can also be used in slices. If you are using a slice every value in the slice will be added to the index.

type User struct {
    Aliases []string `goriakindex:"aliasesindex_bin"`
}

When saved the next time the index will be updated.

KeysInIndex

Keys in a particular index can be retreived with KeysInIndex.

callback := func(item goriak.SecondaryIndexQueryResult) {
    // use item
}

goriak.Bucket("bucket-name", "bucket-type").
    KeysInIndex("nameindex_bin", "Value", callback).
    Run(con)

AddToIndex

An alternative way of setting Secondary Indexes is by using AddToIndex().

goriak.Bucket("bucket-name", "bucket-type").
    SetRaw(data).
    AddToIndex("indexname_bin", "value").
    Run(con)

goriak's Issues

Riak Counters in AutoMap

Allow query builder arguments in any order

Currently methods such as Key() and AddToIndex() needs to be performed before SetRaw() to take affect. If they are performed afterwards they will not do anything.

Make sure that all query builder methods can be performed in any order.

• SetRaw
• SetJSON
• KeysInIndex (Limit)
• MapOperation (Key)
• Set (key)

Register helper type

Add an additional helper type for Registers, similar to how the Set and Counter helper types already work.

Proposed API:

// Functions
NewRegister() *Register

// *Register methods
Set([]byte) *Register
SetString(string) *Register
Value() []byte
String() string
Exec(*Session) error

Remove the "Empty" values saved by the Set helper type

Remove either Context or VClock from Result{}

What is the meaning of `NotFound` field in `Result` struct ?

Hello,

Could you please explain why there is a NotFound field in Result struct ?
For instance I see that here

goriak/map_get.go

Line 58 in 853c1d1

return &Result{

a tuple returned that consists of struct with NotFound: true field and a "not found" error.

It seems that these two are counterintuitive — if we already return NotFound error, why do we have to return ... a basically invalid Result ?

Hope my question finds you well.
Thank you!

All map features

• Registers
• Sets
• Flags
• Counters
• Maps

Secondary Index: Int slices

It should be possible to use int slices as automatic index values.

Connection tools

Currently we're always connecting to localhost. Allow to bring your own Riak connector?

Flag helper type

New helper type for Flags.

Proposed API:

// Functions
NewFlag() *Flag

// *Flag methods
Get() bool
Set(bool) *Flag
Exec(*Session) error

Ignore fields in auto map

Add the option to ignore a field in auto map, so that it won't be saved to Risk.
Probably via a strict tag.

HyperLogLog

http://docs.basho.com/riak/kv/2.2.0/developing/data-types/hyperloglogs/

KeysInIndexRange()

http://docs.basho.com/riak/kv/2.2.0/developing/usage/secondary-indexes/#range

Auth options

https://github.com/basho/riak-go-client/blob/master/connection.go#L37

Do connections need management?

I've looked at the examples folder and I saw how the connection is created:

con, err := goriak.Connect(goriak.ConnectOpts{Address: "127.0.0.1"})

Do we have to manage this somehow like defer conn.Close() (I have not seen such API call) or is it entirely managed via underlying riak-go-client?

Their (Basho's) code does something like this:

defer func() { stopping = true close(sc) if serr := c.Stop(); serr != nil { util.ErrExit(serr) } close(sdc) close(fdc) }()

Nice repo BTW, I'm playing with it right now, good work!

Documentation

• GetValue
• SetValue
• Delete
• SetMap
• GetMap
• MapOperation
• Secondary Indexes

AutoMap: More types

• Go: struct
• Riak counters
• Riak flags

Tests for int slices in SetValue and GetValue

Update helper type values after `Exec()`

• Set
• Counter

Middleware

General middleware that can run on all commands

type RunMiddlewarer interface {
	Key() string
	Bucket() string
	BucketType() string
}

• GetRaw
• SetRaw
• Set
• Get
• SetJSON
• GetJSON

Simpler API

The API is too complicated to use, and not that flexible. Find a better solution.

Add to Secondary Index from slice

Nested structures

I'm curious how to store and restore nested structures, for example slice/array of structs, or map where key is string and value is some other struct (arbitrarily nested). What would be the best strategy for this?

I was able to store and restore JSON as is, but what If I have some already mapped structs, is it possible to reuse them somehow?
For example:

type TB struct {
B string
DataTB map[string]string
}

type A struct {
A string
DataTA map[string]string
B TB
DataTC map[string]TC
}

At the moment, all thins except DataTC which is map[string]TC, will be stored and restored properly. Only DataTC will end up as an empty map. What is the best solution for this?

Split command builder into multiple types

Single type -> Multiple types

Conflict resolver middleware

type ConflictResolverMiddlewarer interface {
	Key() string
	Bucket() string
	BucketType() string
        Values() []Value
}

Create tests for riak security

Tuneable consistency

Allow use for setting the consistency options executing Riak commands.

Client methods

• WithPW (Primary Writes)
• WithDW (Durable Writes)
• WithW (Writes)
• WithRW (Durable Deletes)
• WithPR (Primary Reads)
• WithR (Reads)

Use in query builder

• Set()
• SetRaw()
• SetJSON()
• Get()
• GetRaw()
• GetJSON()
• Delete()

Simple time series support?

Support for Sibling documents / conflict resolving

https://godoc.org/github.com/basho/riak-go-client#ConflictResolver

Counter: Initialise with Set()

Updated and improved documetation

Counter: Update after Exec()

JSON int indexes

Add support for all types of ints in automatic JSON indexes

Custom types for Sets

I want to have a custom *goriak.Set type that makes it simpler to deal with sets.

Methods:

• Exec(*goriak.Client)
• Bytes() [][]byte
• Strings() []string
• AddString(string)
• AddBytes([]byte)
• RemoveString(string)
• RemoveBytes([]byte)

The type should keep track of it's position and key (the same way as Counter), so that it's simple to only update this set without using SetMap()

API for only saving a subset of data

With AutoMaps, the only ways to not update the whole object is to either use Helper Types, or Map Operations.

In a situation where the data of a field (lets say that it is a Register) is not suitable to used in a helper type, and the map operation conversion is to cumbersome (or unreliable), there should instead be a better way.

Internally in Set() a "path" to each object is created. This path consists of the struct field name (which can be overwritten by a struct tag), and the names of it's parents (other structs, or maps).

Proposed API endpoints

• FilterInclude(path... string)
• FilterExclude(path... string

Example

Assuming this struct:

type User struct {
    Name string
    Location struct {
        Country string
        City        string
    }
}

The commands:

• To only save the Name field: Set(val).FilterInclude("Name")
• To only save the City field: Set(val).FilterInclude("Location", "City")
• To save the Name and the City: Set(val).FilterInclude("Name").FilterInclude("Location", "City")

You can use FilterExclude and FilterInclude in combination.

If a field is specified in both an Exclude and Include, the command will be rejected, and an error is returned.

Release v3

Command should be *Command in public API