andytitu / go-sdk

Extism Go SDK

Extism Go SDK

This repo houses the Go SDK for integrating with the Extism runtime. Install this library into your host Go applications to run Extism plugins.

Join the Extism Discord and chat with us!

Installation

Install via go get:

go get github.com/extism/go-sdk

Reference Docs

You can find the reference docs at https://pkg.go.dev/github.com/extism/go-sdk.

Getting Started

This guide should walk you through some of the concepts in Extism and this Go library.

Creating A Plug-in

The primary concept in Extism is the plug-in. You can think of a plug-in as a code module stored in a .wasm file.

Plug-in code can come from a file on disk, object storage or any number of places. Since you may not have one handy let's load a demo plug-in from the web. Let's start by creating a main func and loading an Extism Plug-in:

package main

import (
	"context"
	"fmt"
	"github.com/extism/go-sdk"
	"os"
)

func main() {
	manifest := extism.Manifest{
		Wasm: []extism.Wasm{
			extism.WasmUrl{
				Url: "https://github.com/extism/plugins/releases/latest/download/count_vowels.wasm",
			},
		},
	}

	ctx := context.Background()
	config := extism.PluginConfig{}
	plugin, err := extism.NewPlugin(ctx, manifest, config, []extism.HostFunction{})

	if err != nil {
		fmt.Printf("Failed to initialize plugin: %v\n", err)
		os.Exit(1)
	}
}

Note: See the Manifest docs as it has a rich schema and a lot of options.

Calling A Plug-in's Exports

This plug-in was written in Rust and it does one thing, it counts vowels in a string. As such, it exposes one "export" function: count_vowels. We can call exports using extism.Plugin.Call. Let's add that code to our main func:

func main() {
    // ...

	data := []byte("Hello, World!")
	exit, out, err := plugin.Call("count_vowels", data)
	if err != nil {
		fmt.Println(err)
		os.Exit(int(exit))
	}

	response := string(out)
	fmt.Println(response)
    // => {"count": 3, "total": 3, "vowels": "aeiouAEIOU"}
}

Running this should print out the JSON vowel count report:

$ go run main.go
# => {"count":3,"total":3,"vowels":"aeiouAEIOU"}

All exports have a simple interface of optional bytes in, and optional bytes out. This plug-in happens to take a string and return a JSON encoded string with a report of results.

Plug-in State

Plug-ins may be stateful or stateless. Plug-ins can maintain state between calls by the use of variables. Our count vowels plug-in remembers the total number of vowels it's ever counted in the "total" key in the result. You can see this by making subsequent calls to the export:

func main () {
    // ...

    exit, out, err := plugin.Call("count_vowels", []byte("Hello, World!"))
    if err != nil {
        fmt.Println(err)
        os.Exit(int(exit))
    }
    fmt.Println(string(out))
    // => {"count": 3, "total": 6, "vowels": "aeiouAEIOU"}

    exit, out, err = plugin.Call("count_vowels", []byte("Hello, World!"))
    if err != nil {
        fmt.Println(err)
        os.Exit(int(exit))
    }
    fmt.Println(string(out))
    // => {"count": 3, "total": 9, "vowels": "aeiouAEIOU"}
}

These variables will persist until this plug-in is freed or you initialize a new one.

Configuration

Plug-ins may optionally take a configuration object. This is a static way to configure the plug-in. Our count-vowels plugin takes an optional configuration to change out which characters are considered vowels. Example:

func main() {
    manifest := extism.Manifest{
        Wasm: []extism.Wasm{
            extism.WasmUrl{
                Url: "https://github.com/extism/plugins/releases/latest/download/count_vowels.wasm",
            },
        },
        Config: map[string]string{
            "vowels": "aeiouyAEIOUY",
        },
    }

    ctx := context.Background()
    config := extism.PluginConfig{}

    plugin, err := extism.NewPlugin(ctx, manifest, config, []extism.HostFunction{})

    if err != nil {
        fmt.Printf("Failed to initialize plugin: %v\n", err)
        os.Exit(1)
    }

    exit, out, err := plugin.Call("count_vowels", []byte("Yellow, World!"))
    if err != nil {
        fmt.Println(err)
        os.Exit(int(exit))
    }

    fmt.Println(string(out))
    // => {"count": 4, "total": 4, "vowels": "aeiouAEIOUY"}
}

Host Functions

Let's extend our count-vowels example a little bit: Instead of storing the total in an ephemeral plug-in var, let's store it in a persistent key-value store!

Wasm can't use our KV store on it's own. This is where Host Functions come in.

Host functions allow us to grant new capabilities to our plug-ins from our application. They are simply some Go functions you write which can be passed down and invoked from any language inside the plug-in.

Let's load the manifest like usual but load up this count_vowels_kvstore plug-in:

manifest := extism.Manifest{
    Wasm: []extism.Wasm{
        extism.WasmUrl{
            Url: "https://github.com/extism/plugins/releases/latest/download/count_vowels_kvstore.wasm",
        },
    },
}

Note: The source code for this is here and is written in rust, but it could be written in any of our PDK languages.

Unlike our previous plug-in, this plug-in expects you to provide host functions that satisfy our its import interface for a KV store.

We want to expose two functions to our plugin, kv_write(key string, value []bytes) which writes a bytes value to a key and kv_read(key string) []byte which reads the bytes at the given key.

// pretend this is Redis or something :)
kvStore := make(map[string][]byte)

kvRead := extism.NewHostFunctionWithStack(
    "kv_read",
    func(ctx context.Context, p *extism.CurrentPlugin, stack []uint64) {
        key, err := p.ReadString(stack[0])
        if err != nil {
            panic(err)
        }

        value, success := kvStore[key]
        if !success {
            value = []byte{0, 0, 0, 0}
        }

        stack[0], err = p.WriteBytes(value)
    },
    []ValueType{ValueTypePTR},
    []ValueType{ValueTypePTR},
)

kvWrite := extism.NewHostFunctionWithStack(
    "kv_write",
    func(ctx context.Context, p *extism.CurrentPlugin, stack []uint64) {
        key, err := p.ReadString(stack[0])
        if err != nil {
            panic(err)
        }

        value, err := p.ReadBytes(stack[1])
        if err != nil {
            panic(err)
        }

        kvStore[key] = value
    },
    []ValueType{ValueTypePTR, ValueTypePTR},
    []ValueType{},
)

Note: In order to write host functions you should get familiar with the methods on the extism.CurrentPlugin type. The p parameter is an instance of this type.

We need to pass these imports to the plug-in to create them. All imports of a plug-in must be satisfied for it to be initialized:

plugin, err := extism.NewPlugin(ctx, manifest, config, []extism.HostFunction{kvRead, kvWrite});

Now we can invoke the event:

exit, out, err := plugin.Call("count_vowels", []byte("Hello, World!"))
// => Read from key=count-vowels"
// => Writing value=3 from key=count-vowels"
// => {"count": 3, "total": 3, "vowels": "aeiouAEIOU"}

exit, out, err = plugin.Call("count_vowels", []byte("Hello, World!"))
// => Read from key=count-vowels"
// => Writing value=6 from key=count-vowels"
// => {"count": 3, "total": 6, "vowels": "aeiouAEIOU"}

Enabling Compilation Cache

While Wazero (the underlying Wasm runtime) is very fast in initializing modules, you can make subsequent initializations even faster by enabling the compilation cache:

ctx := context.Background()
cache := wazero.NewCompilationCache()
defer cache.Close(ctx)

manifest := Manifest{Wasm: []Wasm{WasmFile{Path: "wasm/noop.wasm"}}}

config := PluginConfig{
    EnableWasi:    true,
    ModuleConfig:  wazero.NewModuleConfig(),
    RuntimeConfig: wazero.NewRuntimeConfig().WithCompilationCache(cache),
}

_, err := NewPlugin(ctx, manifest, config, []HostFunction{})

Enable filesystem access

WASM plugins can read/write files outside the runtime. To do this we add AllowedPaths mapping of "HOST:PLUGIN" to the extism.Manifest of our plugin.

package main

import (
	"context"
	"fmt"
	"os"

	extism "github.com/extism/go-sdk"
)

func main() {
	manifest := extism.Manifest{
		AllowedPaths: map[string]string{
			// Here we specifify a host directory data to be linked
			// to the /mnt directory inside the wasm runtime
			"data": "/mnt",
		},
		Wasm: []extism.Wasm{
			extism.WasmFile{
				Path: "fs_plugin.wasm",
			},
		},
	}

	ctx := context.Background()
	config := extism.PluginConfig{
		EnableWasi: true,
	}
	plugin, err := extism.NewPlugin(ctx, manifest, config, []extism.HostFunction{})

	if err != nil {
		fmt.Printf("Failed to initialize plugin: %v\n", err)
		os.Exit(1)
	}

	data := []byte("Hello world, this is written from within our wasm plugin.")
	exit, _, err := plugin.Call("write_file", data)
	if err != nil {
		fmt.Println(err)
		os.Exit(int(exit))
	}
}

Note: In order for filesystem APIs to work the plugin needs to be compiled with WASI target. Source code for the plugin can be found here and is written in Go, but it could be written in any of our PDK languages.

Build example plugins

Since our example plugins are also written in Go, for compiling them we use TinyGo:

cd plugins/config
tinygo build -target wasi -o ../wasm/config.wasm main.go