blend / go-sdk

A composable toolbox of libraries to build everything from CLIs to enterprise applications.

go-sdk

go-sdk is our core library of packages. These packages can be composed to create anything from CLIs to fully featured web apps.

The general philosophy is to provide loosely coupled libraries that can be composed as a suite of tools, vs. a do it all framework.

Requirements

This repository requires golang version 1.16+ to be installed.

To run tests, it is required that you have Docker installed, with docker-compose, or have postgres running locally.

Addtional CLI Tools

We also provide the following CLI tools to help with development that leverage some of these packages:

• cmd/ask : securely input secrets and output to a file to be read by templates.
• cmd/copyright : injects and verifies copyright headers are present in files.
• cmd/cover : allows for project level coverage reporting and enforcement.
• cmd/job : run a command on a cron schedule; useful for writing jobs as kubernetes pods.
• cmd/profanity : profanity rules checking (i.e. fail on grep match).
• cmd/recover : recover crashed processes (to be used when debugging panics).
• cmd/semver : semver manipulation and validation.
• cmd/shamir : securely partition secrets using shamir's sharing scheme.
• cmd/template : commandline template generation using golang text/template.

Repository Organization Notes

• The repository is organized into composible packages. They are designed to be as loosely coupled as possible, but are all in a single repo to facilitate breaking change management.
• Any commandline programs should live under cmd/** with the core library code in a top level package. The CLI should just be the bare minimum to run the library from the cli.

Contributing

We currently don't accept PRs as this repository at this time, but feel free to log issues and we'll address when we can.

Code Style Notes

The "Options" Pattern

• The "Options" pattern is a variadic set of arguments as functions that mutate the returned object, typically in the constructor.
  • This lets callers add their own mutators to be used in constructors.
  • It also lets callers establish a set of "default" options that can be combined / overridden later.
  • It also reduces the amount of code required in the go-sdk repo itself.

Other General Notes

• Where possible, follow the golang proverbs.
• Make the zero value useful. Some situations require pointers, and are noted exceptions.
• Export all fields unless strictly internal state and would never be set by calling code.
• Where possible, packages should export configuration objects that can be used to create the core types of that package. Those configuration objects should be readable from both JSON and YAML.
• Anything that can return an error, should. Anything that needs to return a single value (but would return an error) should panic on that error and should be prefixed by Must....
• Minimize dependencies between packages as much as possible; add external dependencies with extreme care.
  • Notable exceptions include:
    • airbrake
    • sentry
    • lib/pq
    • aws sdk
    • datadog

Version Management

We follow calendar versioning. What that means in practice:

[major].[year][month][day].[patch]

Major version changes are changes that break backwards-compatibility. A breaking change is defined as a change that would cause code written against the current major version having a build failure of any type. That's even for a trivial find and replace. Once you merge a new api or name for an object after CR, that's it. Major changes are rolled up into a release, at most, once-per-year.

Minor version changes are additions to exported types, values, or functions that don't cause break backwards compatibility. These types of changes are rolled into a new minor version at an approximate monthly cadence.

Patch versions is the used to distinguish different versions from the same calendar day.

The current version is stored in the VERSION file at the root of the package.

Currently we support 1 major version branches of the go-sdk.

• v1.xxxxxxxx.x

All other major versions are deprecated. The previous semver scheme is deprecated.

Version Release Cycle

We will not be releasing more than 1 major version in any calendar year, and major versions will recieve at least 2 years of support before being deprecated.

Bugs and Feature Requests

Please use the issues page to report any bugs or request new features be added to the SDK. We welcome contributions to any issue reported therein, including those you report. Please ping us on the issue itself for things like access requests.

Maintainers

This repo is maintained by a core group of Blend employees, including:

• Addie Bendory
• Alex Kaplan
• Chinmaya Sharma
• Igor Morozov
• Justin Fan
• Kevin Wang
• Michael Turner

go-sdk's Issues

Outdated modules that contains volunerabilities

Found vulnerability (SNYK-GOLANG-GITHUBCOMDGRIJALVAJWTGO-596515)

github.com/spf13/cobra v1.1.1 uses github.com/dgrijalva/jwt-go which is affected by a known vulnerability and is no longer maintaned so it should probably be upgraded to a newer version.

v3: `web.Ctx.CookieDomain()` includes port

This leads to

...
2019-12-16 11:17:14.441421 I | net/http: invalid Cookie.Domain "localhost:8080"; dropping domain attribute
...

See an example, we are using u.Host rather than u.Hostname():

=========================================
u.Host:       "localhost:8080"
u.Hostname(): "localhost"
Raw URL:      "http://localhost:8080"
Parsed URL:   &url.URL{Scheme:"http", Opaque:"", User:(*url.Userinfo)(nil), Host:"localhost:8080", Path:"", RawPath:"", ForceQuery:false, RawQuery:"", Fragment:""}
=========================================
u.Host:       "web.invalid"
u.Hostname(): "web.invalid"
Raw URL:      "https://web.invalid"
Parsed URL:   &url.URL{Scheme:"https", Opaque:"", User:(*url.Userinfo)(nil), Host:"web.invalid", Path:"", RawPath:"", ForceQuery:false, RawQuery:"", Fragment:""}

(It's going to take me less time to write a fix than it did filing this.)

db.Connection.CreateMany should set IDs

What happened:

What you expected to happen:
IDs to be set in array of objects after calling CreateMany()

How to reproduce it (as minimally and precisely as possible):
CreateMany() any array of objects. Print out the IDs, which will remain 0.

Anything else we need to know?:
https://stackoverflow.com/a/5876411 seems to provide a solution:
INSERT INTO my_table VALUES ('a'), ('b'), ('c') RETURNING id;

Environment:

• go-sdk version: v1.1.1

• go version (e.g go version): go version go1.12.12 darwin/amd64

• OS (e.g: cat /etc/os-release): uname -mrsv -> Darwin 18.6.0 Darwin Kernel Version 18.6.0: Thu Apr 25 23:16:27 PDT 2019; root:xnu-4903.261.4~2/RELEASE_X86_64 x86_64

• Others:

r2.New() doesn't invoke http.NewRequest()

The current approach is to just manually construct an instance via http.Request{ ... }, but this misses out on goodies like setting the default protocol or the host.

We don't strictly need to invoke it in the same way, but any option that would correspond to one of the four arguments of NewRequestWithContext (ctx, method, url, body) should have similar behavior.

(This came about because @PaulMH wrote some perfectly normal looking r2 request code that didn't set the content length on a post.)

Provide custom `CheckRedirect` for `r2.Request.Client`.

By default, if a (net/http).Client hits 10 redirects, it will give up with:

func defaultCheckRedirect(req *Request, via []*Request) error {
	if len(via) >= 10 {
		return errors.New("stopped after 10 redirects")
	}
	return nil
}

but this default can be overridden on a Client.

v2 logger swallows panic caused by `nil` HTTP request

Running

package main

import (
	"net/http"

	"github.com/blend/go-sdk/logger"
)

func main() {
	log := logger.All()
	log.SyncTrigger(logger.NewHTTPResponseEvent(nil).WithRoute("foo").WithStatusCode(http.StatusOK))
}

results in

2019-11-25T23:47:04.421687Z    [fatal] runtime error: invalid memory address or nil pointer dereference

However, the equivalent code with the v3 API

package main

import (
	"context"
	"net/http"

	"github.com/blend/go-sdk/logger"
	"github.com/blend/go-sdk/webutil"
)

func main() {
	log := logger.All()
	event := webutil.NewHTTPResponseEvent(
		nil,
		webutil.OptHTTPResponseRoute("foo"),
		webutil.OptHTTPResponseStatusCode(http.StatusOK),
	)
	log.Trigger(context.TODO(), event)
}

"correctly" panics

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1278173]

goroutine 1 [running]:
github.com/blend/go-sdk/webutil.HTTPResponseEvent.WriteText(0x0, 0x131b668, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc8, 0x0, ...)
        .../src/github.com/blend/go-sdk/webutil/http_response_event.go:107 +0x53
github.com/blend/go-sdk/logger.TextOutputFormatter.WriteFormat(0x0, 0x1325691, 0x23, 0xc0000b4e10, 0x1383b40, 0xc0000160e0, 0x137ec00, 0xc0000b2380, 0x137f560, 0xc0000a6300, ...)
        .../src/github.com/blend/go-sdk/logger/text_output_formatter.go:145 +0x2de
github.com/blend/go-sdk/logger.(*Logger).Write(0xc0000c4090, 0x1383b40, 0xc0000160e0, 0x137f560, 0xc0000a6300)
        .../src/github.com/blend/go-sdk/logger/logger.go:225 +0xce
github.com/blend/go-sdk/logger.(*Logger).Trigger(0xc0000c4090, 0x1383b40, 0xc0000160e0, 0x137f560, 0xc0000a6300)
        .../src/github.com/blend/go-sdk/logger/logger.go:209 +0x22e
main.main()
        .../main.go:18 +0x19f
exit status 2

`log.Trigger()` buries panics

Consider the following example:

wg := sync.WaitGroup{}
listener := makeWaitListener(&wg)
log.Listen(webutil.HTTPResponse, "stats", webutil.NewHTTPResponseEventListener(listener))
// ...
wg.Add(1)
log.Trigger(context.Background(), *event)
wg.Wait()

wg.Add(1)
log.Trigger(context.Background(), *event)
wg.Wait()

// This should panic.
log.Trigger(context.Background(), *event)
wg.Wait()

But running this we get:

$ go run ./main.go
::: Listener fired (1)
::: Listener fired (2)
$ echo "${?}"
0

---

Full script that snippet above came from:

package main

import (
	"bytes"
	"context"
	"fmt"
	"net/http"
	"os"
	"sync"

	"github.com/blend/go-sdk/logger"
	"github.com/blend/go-sdk/webutil"
)

func main() {
	err := runMain()
	if err != nil {
		fmt.Fprintf(os.Stderr, "%v\n", err)
		os.Exit(1)
	}
}

func runMain() error {
	var logBuffer bytes.Buffer
	log := inMemoryLog(&logBuffer)

	// Register a listener.
	wg := sync.WaitGroup{}
	listener := makeWaitListener(&wg)
	log.Listen(webutil.HTTPResponse, "stats", webutil.NewHTTPResponseEventListener(listener))

	event, err := makeEvent("/foo/bar")
	if err != nil {
		return err
	}

	wg.Add(1)
	log.Trigger(context.Background(), *event)
	wg.Wait()

	wg.Add(1)
	log.Trigger(context.Background(), *event)
	wg.Wait()

	// This should panic.
	log.Trigger(context.Background(), *event)
	wg.Wait()

	return nil
}

func inMemoryLog(logBuffer *bytes.Buffer) *logger.Logger {
	return logger.MustNew(
		logger.OptAll(),
		logger.OptOutput(logBuffer),
		logger.OptFormatter(logger.NewTextOutputFormatter(
			logger.OptTextNoColor(),
			logger.OptTextHideTimestamp(),
		)),
	)
}

func makeEvent(path string) (*webutil.HTTPResponseEvent, error) {
	r, err := http.NewRequest("GET", "http://web.invalid:8080/foo/bar", nil)
	if err != nil {
		return nil, err
	}

	event := webutil.NewHTTPResponseEvent(
		r,
		webutil.OptHTTPResponseRoute(path),
		webutil.OptHTTPResponseStatusCode(http.StatusOK),
	)
	return &event, nil
}

func makeWaitListener(wg *sync.WaitGroup) func(context.Context, webutil.HTTPResponseEvent) {
	callCount := 0
	return func(_ context.Context, _ webutil.HTTPResponseEvent) {
		defer wg.Done()
		callCount++
		if callCount == 3 {
			panic("Really do panic")
		}
		fmt.Printf("::: Listener fired (%d)\n", callCount)
		return
	}
}

dependency error go sdk

What happened:
go get error

$ go get golearn
...
go: github.com/sjwhitworth/[email protected] requires
        github.com/rocketlaunchr/[email protected] requires
        github.com/blend/[email protected]: reading github.com/blend/go-sdk/go.mod at revision v1.1.1: unknown revision v1.1.1

What you expected to happen:

a successful go get of golearn

How to reproduce it (as minimally and precisely as possible):

$ go get github.com/rocketlaunchr/dataframe-go

Anything else we need to know?:

It looks as though rocketlaunchr/dataframe-go is a bit unmaintained; there is a bug report that looks as though it's been sitting for a few months; the suggestion from that bug report would be that (at least for now) you could remove the 'legacy-' prefix from version 1.1.1 please

Environment:

• go-sdk version: 1.1.1
• go version (e.g go version): go1.18.6 linux/amd64
• OS (e.g: cat /etc/os-release): Fedora 36
• Others:

SNYK-GOLANG-GITHUBCOMJACKCPGPROTO3V2-1316251

The following modules are affected by volunerability SNYK-GOLANG-GITHUBCOMJACKCPGPROTO3V2-1316251:

• github.com/jackc/pgproto3/[email protected]
• github.com/jackc/pgproto3/[email protected]
• github.com/jackc/pgproto3/[email protected]
• github.com/jackc/pgproto3/[email protected]
• github.com/jackc/pgproto3/[email protected]

 $ ggdh 'github.com/jackc/pgproto3/[email protected]'
                          github.com/blend/go-sdk
                                     ⬇
                      github.com/jackc/pgx/[email protected]
                                     ⬇
                       github.com/jackc/[email protected]
                                     ⬇
        github.com/jackc/pgx/[email protected]
                                     ⬇
        github.com/jackc/[email protected]
                                     ⬇
        github.com/jackc/pgx/[email protected]
                                     ⬇
                       github.com/jackc/[email protected]
                                     ⬇
         github.com/jackc/[email protected]
                                     ⬇
         github.com/jackc/[email protected]
                                     ⬇
     github.com/jackc/pgx/[email protected]
                                     ⬇
         github.com/jackc/[email protected]
                                     ⬇
         github.com/jackc/pgx/[email protected]
                                     ⬇
  github.com/jackc/pgproto3/[email protected]

 $ ggdh 'github.com/jackc/pgproto3/[email protected]'
                          github.com/blend/go-sdk
                                     ⬇
                      github.com/jackc/pgx/[email protected]
                                     ⬇
                       github.com/jackc/[email protected]
                                     ⬇
        github.com/jackc/pgx/[email protected]
                                     ⬇
        github.com/jackc/[email protected]
                                     ⬇
        github.com/jackc/pgx/[email protected]
                                     ⬇
                       github.com/jackc/[email protected]
                                     ⬇
         github.com/jackc/[email protected]
                                     ⬇
         github.com/jackc/[email protected]
                                     ⬇
     github.com/jackc/pgx/[email protected]
                                     ⬇
  github.com/jackc/pgproto3/[email protected]

 $ ggdh 'github.com/jackc/pgproto3/[email protected]'
                         github.com/blend/go-sdk
                                   ⬇
                     github.com/jackc/pgx/[email protected]
                                   ⬇
                     github.com/jackc/[email protected]
                                   ⬇
      github.com/jackc/pgx/[email protected]
                                   ⬇
      github.com/jackc/[email protected]
                                   ⬇
      github.com/jackc/pgx/[email protected]
                                   ⬇
                     github.com/jackc/[email protected]
                                   ⬇
       github.com/jackc/[email protected]
                                   ⬇
  github.com/jackc/pgproto3/[email protected]

 $ ggdh 'github.com/jackc/pgproto3/[email protected]'
                    github.com/blend/go-sdk
                               ⬇
                github.com/jackc/pgx/[email protected]
                               ⬇
                 github.com/jackc/[email protected]
                               ⬇
  github.com/jackc/pgx/[email protected]
                               ⬇
  github.com/jackc/[email protected]
                               ⬇
  github.com/jackc/pgx/[email protected]
                               ⬇
              github.com/jackc/pgproto3/[email protected]

 $ ggdh 'github.com/jackc/pgproto3/[email protected]'
        github.com/blend/go-sdk
                  ⬇
    github.com/jackc/pgx/[email protected]
                  ⬇
  github.com/jackc/pgproto3/[email protected]

Typo in workqueue readme

It says:

wq := workqueue.New()
workQueue.Start() //two workers, i.e. two go routines
workQueue.Enqueue(func(v ...interface{}) error {
    fmt.Printf("Work Item: %#v\n", v)
    return nil
}, "hello world")
// output: "Work Item: hello world"

Should be something like:

wq := workqueue.New()
wq.Start() //two workers, i.e. two go routines
wq.Enqueue(func(v ...interface{}) error {
    fmt.Printf("Work Item: %#v\n", v)
    return nil
}, "hello world")
// output: "Work Item: hello world"

v3: `r2.OptTLSRootCAs()` has "typed nil" issue when checking for a nil transport

When checking

		if r.Client.Transport == nil {
			r.Client.Transport = &http.Transport{}
		}

the check will fail if the transport is a typed nil. For example:

package main

import (
	"crypto/x509"
	"fmt"
	"net/http"

	"github.com/blend/go-sdk/r2"
)

func main() {
	var transport *http.Transport
	certPool, err := x509.SystemCertPool()
	if err != nil {
		panic(err)
	}

	req := r2.New(
		"https://web.invalid",
		// NOTE: Transport **must** come before the root CAs since the CAs get set
		//       **on** the transport.
		r2.OptTransport(transport),
		r2.OptTLSRootCAs(certPool),
	)
	fmt.Println(req)
}

results in

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x90 pc=0x126d2e7]

goroutine 1 [running]:
github.com/blend/go-sdk/r2.OptTLSRootCAs.func1(0xc0000dfb80, 0x0, 0x0)
        .../go/src/github.com/blend/go-sdk/r2/opt_tls_root_cas.go:26 +0x2b7
github.com/blend/go-sdk/r2.New(0x0, 0x0, 0xc000175f30, 0x4, 0x4, 0x1d)
        .../go/src/github.com/blend/go-sdk/r2/request.go:30 +0x1a2
main.main()
        .../main.go:26 +0x12a
exit status 2

r2.New() should set r.URL on the returned value

See: r2.New() puts the parsed URL into r.Request but OptHost() only touches r.URL

[assert] Function doesn't compare equal to itself

What happened:

Function doesn't compare equal to itself.

What you expected to happen:

it := assert.New(t)
it.Equal(f, f)
it.ReferenceEqual(f, f)

to pass. (What's worse, it.ReferenceEqual() panics)

How to reproduce it (as minimally and precisely as possible):

See: https://play.golang.org/p/FksPb7SsO6o

package main

import (
	"fmt"
	"net/http"
	"reflect"
)

func areEqual(expected, actual interface{}) bool {
	if expected == nil && actual == nil {
		return true
	}
	if (expected == nil && actual != nil) || (expected != nil && actual == nil) {
		return false
	}

	actualType := reflect.TypeOf(actual)
	if actualType == nil {
		return false
	}
	expectedValue := reflect.ValueOf(expected)
	if expectedValue.IsValid() && expectedValue.Type().ConvertibleTo(actualType) {
		return reflect.DeepEqual(expectedValue.Convert(actualType).Interface(), actual)
	}

	return reflect.DeepEqual(expected, actual)
}

func areReferenceEqual(expected, actual interface{}) bool {
	if expected == nil && actual == nil {
		return true
	}
	if (expected == nil && actual != nil) || (expected != nil && actual == nil) {
		return false
	}

	return expected == actual
}

func main() {
	tr := func(req *http.Request) error {
		return nil
	}
	fmt.Println(areEqual(tr, tr))
	fmt.Println(areReferenceEqual(tr, tr))
}

Environment:

• go-sdk version: master at 72508ca
• go version (e.g go version): go version go1.13 darwin/amd64
• OS (e.g: cat /etc/os-release): macOS

Port internal SDK changes to public `go-sdk`

In order of direct imports:

010: github.com/blend/go-sdk/web
008: github.com/blend/go-sdk/template
008: github.com/blend/go-sdk/envoyutil
008: github.com/blend/go-sdk/db
007: github.com/blend/go-sdk/secrets
007: github.com/blend/go-sdk/cron
006: github.com/blend/go-sdk/webutil
006: github.com/blend/go-sdk/oauth
006: github.com/blend/go-sdk/cmd/reverseproxy
005: github.com/blend/go-sdk/stats/metrics/dbmetrics
005: github.com/blend/go-sdk/logger
005: github.com/blend/go-sdk/grpcutil
005: github.com/blend/go-sdk/db/migration
005: github.com/blend/go-sdk/airbrake
004: github.com/blend/go-sdk/stats/metrics/httpmetrics
004: github.com/blend/go-sdk/stats/metrics/grpcmetrics
004: github.com/blend/go-sdk/stats
004: github.com/blend/go-sdk/sentry
004: github.com/blend/go-sdk/r2
004: github.com/blend/go-sdk/profanity
004: github.com/blend/go-sdk/configutil
004: github.com/blend/go-sdk/cmd/profanity
003: github.com/blend/go-sdk/validate
003: github.com/blend/go-sdk/statsd
003: github.com/blend/go-sdk/stats/tracing/webtrace
003: github.com/blend/go-sdk/slack
003: github.com/blend/go-sdk/reverseproxy
003: github.com/blend/go-sdk/env
003: github.com/blend/go-sdk/email
003: github.com/blend/go-sdk/datadog
003: github.com/blend/go-sdk/cmd/changelog
002: github.com/blend/go-sdk/stats/tracing/secretstrace
002: github.com/blend/go-sdk/stats/tracing/r2trace
002: github.com/blend/go-sdk/stats/tracing/oauthtrace
002: github.com/blend/go-sdk/stats/tracing/httptrace
002: github.com/blend/go-sdk/stats/tracing/grpctrace
002: github.com/blend/go-sdk/stats/tracing/dbtrace
002: github.com/blend/go-sdk/stats/tracing/crontrace
002: github.com/blend/go-sdk/proxyprotocol
002: github.com/blend/go-sdk/fileutil
002: github.com/blend/go-sdk/cmd/shamir
002: github.com/blend/go-sdk/cmd/bindata
002: github.com/blend/go-sdk/cmd/ask
002: github.com/blend/go-sdk/certutil
002: github.com/blend/go-sdk/async
001: github.com/blend/go-sdk/uuid
001: github.com/blend/go-sdk/stringutil
001: github.com/blend/go-sdk/stats/tracing
001: github.com/blend/go-sdk/spiffeutil
001: github.com/blend/go-sdk/shamir
001: github.com/blend/go-sdk/sh
001: github.com/blend/go-sdk/retry
001: github.com/blend/go-sdk/reflectutil
001: github.com/blend/go-sdk/jwt/test
001: github.com/blend/go-sdk/jwt
001: github.com/blend/go-sdk/grpcutil/calculator
001: github.com/blend/go-sdk/graceful
001: github.com/blend/go-sdk/crypto
001: github.com/blend/go-sdk/cmd/template
001: github.com/blend/go-sdk/cmd/semver
001: github.com/blend/go-sdk/cache
001: github.com/blend/go-sdk/bufferutil
001: github.com/blend/go-sdk/breaker
001: github.com/blend/go-sdk/bindata
001: github.com/blend/go-sdk/ansi
000: github.com/blend/go-sdk/yaml
000: github.com/blend/go-sdk/timeutil
000: github.com/blend/go-sdk/semver
000: github.com/blend/go-sdk/selector
000: github.com/blend/go-sdk/ref
000: github.com/blend/go-sdk/names
000: github.com/blend/go-sdk/mathutil
000: github.com/blend/go-sdk/jsonutil
000: github.com/blend/go-sdk/grpcutil/calculator/v1
000: github.com/blend/go-sdk/ex
000: github.com/blend/go-sdk/collections
000: github.com/blend/go-sdk/cmd/recover
000: github.com/blend/go-sdk/cmd/coverage
000: github.com/blend/go-sdk/bitflag
000: github.com/blend/go-sdk/assert
000: github.com/blend/go-sdk/ansi/slant

---

#!/bin/bash
set -e
OUTPUT_FILE=$(mktemp)
trap 'rm -f pkg.dot "'"${OUTPUT_FILE}"'"' EXIT
echo "Using ${OUTPUT_FILE}"
BASE_PATH=$(go list -e)
for PKG in $(go list ./...)
do
  WITHOUT_BASE=${PKG#"${BASE_PATH}"}
  RELATIVE_PATH=".${WITHOUT_BASE}"
  if [[ "${RELATIVE_PATH}" = ./examples* ]]; then
    continue
  fi
  godepgraph -nostdlib -novendor "${PKG}" > pkg.dot
  DEPENDENCY_COUNT=$(grep -c '"'"${PKG}"'" -> "github.com/blend/go-sdk' < pkg.dot | sed '/^$/d;s/[[:blank:]]//g')
  DEPENDENCY_COUNT_FORMATTED=$(printf "%03d" "${DEPENDENCY_COUNT}")
  echo "${DEPENDENCY_COUNT_FORMATTED}: ${PKG}" >> "${OUTPUT_FILE}"
done
cat "${OUTPUT_FILE}" | sort -r

Off-by-one error in line numbers in panic output from within a goroutine

May be related to: golang/go#34123 (comment)

I still need to reproduce this in a "minimal" and "open" fashion but we had an internal file where line 43 caused a panic in a goroutine but the stack referred to line 42.

`r2.Request` should follow 307 on non-GET

See #175 for context. To test

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"
	"net/url"

	"github.com/blend/go-sdk/r2"
)

func main() {
	path := "/v1/auth/token/lookup"
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if r.URL.Path == path {
			http.Redirect(w, r, "/more", http.StatusTemporaryRedirect)
			return
		}
		fmt.Fprintln(w, "Hello, client")
	}))
	defer ts.Close()
	u, err := url.Parse(ts.URL)
	if err != nil {
		panic(err)
	}

	// Do a GET.
	fmt.Println("--- Doing a GET ---")
	opts := []r2.Option{
		r2.OptScheme("http"),
		r2.OptHeaderValue("X-Vault-Token", "bar"),
		r2.OptHost(u.Host),
		r2.OptGet(),
		r2.OptPathf(path),
	}
	body, res, err := r2.New("", opts...).Bytes()
	if err != nil {
		panic(err)
	}
	fmt.Printf("body: %q\n\n%#v\n", string(body), res)

	data := map[string]string{"token": "foo"}
	// Do a POST.
	fmt.Println("--- Doing a POST ---")
	opts = []r2.Option{
		r2.OptScheme("http"),
		r2.OptHeaderValue("X-Vault-Token", "bar"),
		r2.OptHost(u.Host),
		r2.OptPost(),
		r2.OptPathf(path),
		r2.OptJSONBody(data),
	}
	r := r2.New("", opts...)
	body, res, err = r.Bytes()
	if err != nil {
		panic(err)
	}
	fmt.Printf("body: %q\n\n%#v\n", string(body), res)
}

which produces

--- Doing a GET ---
body: "Hello, client\n"

&http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Length":[]string{"14"}, "Content-Type":[]string{"text/plain; charset=utf-8"}, "Date":[]string{"Sat, 01 Feb 2020 00:01:16 GMT"}}, Body:(*http.bodyEOFSignal)(0xc0001bc100), ContentLength:14, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc0001c8000), TLS:(*tls.ConnectionState)(nil)}
--- Doing a POST ---
body: ""

&http.Response{Status:"307 Temporary Redirect", StatusCode:307, Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"Content-Length":[]string{"0"}, "Date":[]string{"Sat, 01 Feb 2020 00:01:16 GMT"}, "Location":[]string{"/more"}}, Body:http.noBody{}, ContentLength:0, TransferEncoding:[]string(nil), Close:false, Uncompressed:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc000176420), TLS:(*tls.ConnectionState)(nil)}

PostedFiles lost after trying to find nonexistent Param

What happened:

Calling web.Ctx.PostedFiles or webutil.PostedFiles(web.Ctx.Request) returns no files if web.Ctx.Param("param") is called beforehand with a missing parameter.

What you expected to happen:

PostedFiles should return the files from the request even if web.Ctx.Param is called with a missing parameter

How to reproduce it (as minimally and precisely as possible):

app := web.New(web.OptBindAddr(":8080"), web.OptLog(logger.All()))

app.POST("/file", func(r *web.Ctx) web.Result {
	r.Param("non-existent")
	files, err := webutil.PostedFiles(r.Request)
	if err != nil {
		return web.Text.BadRequest(err)
	}
	fmt.Println("num files", len(files))
	return web.Text.OK()
})```


**Anything else we need to know?**:

This happens because the request gets consumed by the `r.Param` searching for all of the possible parameters

**Environment**:
- go-sdk version: master
- go version (e.g `go version`): go1.11.4 darwin/amd64
- OS (e.g: `cat /etc/os-release`): macOS 10.13.4
- Others:

`db.Connection.BeginContext` throws away options

See source, in particular:

	if len(opts) > 0 {
		tx, err := dbc.Connection.BeginTx(context, opts[0])
		return tx, Error(err)
	}

only uses opts[0] but anything in opts[1:] is ignored. This is not surface to the caller in any way. Since the method already returns (*sql.Tx, error) it should be straightforward to add

	if len(opts) > 1 {
		return nil, ex.New("BeginContext accepts at most 1 option")
	}

(though having a runtime check for such a condition rather than a compile-time check is a bit unfortunate.)

selector/runes.go: isAlpha not robust

go-sdk/selector/runes.go

Lines 35 to 37 in 95d703f

	func isAlpha(ch rune) bool {
	return !isWhitespace(ch) && !unicode.IsControl(ch) && !isSymbol(ch)
	}

for example, this returns true:

fmt.Println(isAlpha(0x200E))

https://wikipedia.org/wiki/Left-to-right_mark

`make dev-deps` failing to install goreleaser

What happened:
make dev-deps fails to install goreleaser

$ go get github.com/goreleaser/goreleaser
package github.com/google/go-github/v28/github: cannot find package "github.com/google/go-github/v28/github" in any of:
	/usr/local/Cellar/go/1.14.3/libexec/src/github.com/google/go-github/v28/github (from $GOROOT)
	/Users/jpowers/go/src/github.com/google/go-github/v28/github (from $GOPATH)

Resolution:
Install goreleaser with go modules enabled

$ GO111MODULE=on go get -d github.com/goreleaser/goreleaser
go: github.com/goreleaser/goreleaser upgrade => v0.138.0

Reference: goreleaser/goreleaser#1013

Environment:

• fresh laptop, go installed using homebrew
• go-sdk version: master
• go version (e.g go version): go version go1.14.3 darwin/amd64
• OS (e.g: cat /etc/os-release): darwin

v2 bug: web.ViewCache() incorrectly claims to be cached after Parse failures

For example I recently encountered some code that emitted a.Views().InternalError(err) after a.Mock() failed parsing as part of the start-up tasks. Then when it came time to call InternalError(), the call to ViewCache.Lookup() resulted in a call to ViewCache.Templates() which in turn thinks it has already cached the result:

	if vc.cached {
		return vc.viewCache, nil
	}

and the default values of cached and viewCache from NewViewCache() are

		...
		viewCache:                 template.New(""), // an empty template tree.
		...
		cached:                    true,
		...

This was especially pernicious because the caching mechanism is just done via a nil check in v3. (I found this while upgrading a codebase from v2 to v3 and some unit tests started failing "for no reason".)

Sentry exceptions report the remote address as 127.0.0.1

What happened: The remote address reports as localhost for requests fronted by a sidecar proxy.

What you expected to happen: Expected the X-Forwarded-For address to be the value used, not the connection peer.

How to reproduce it (as minimally and precisely as possible): Report an error from a service fronted by a proxy.

Anything else we need to know?:

Environment:

• go-sdk version: master
• go version (e.g go version): 1.12.9
• OS (e.g: cat /etc/os-release): darwin
• Others:

Any r2/webutil option that modifies the body should also set content length

There is a large effort in http.NewRequestWithContext to set req.ContentLength and req.GetBody whenever possible (and req.Body is added in the Request{} struct literal). However, our webutil options just set req.Body directly.

Related to #141.

---

The implications of this (H/T to @PaulMH) are that

serialized, _ := json.Marshal(someStruct)
url := "http://localhost:5001/some/path"

payload := strings.NewReader(string(serialized))
req, _ := http.NewRequest("POST", url, payload)

req.Header.Add("Content-Type", "application/json")
res, err := http.DefaultClient.Do(req)

results in this being sent as

POST /some/path HTTP/1.1
Host: localhost:5001
User-Agent: Go-http-client/1.1
Content-Length: 19
Content-Type: application/json
Accept-Encoding: gzip

{"a":"b","c":101.5}

while

req := r2.New("http://localhost:5001",
	r2.OptPost(),
	r2.OptPath("/some/path"),
	r2.OptJSONBody(someStruct))

res, err := req.Do()

results in this being sent as

POST /some/path HTTP/1.1
Host: localhost:5001
User-Agent: Go-http-client/1.1
Transfer-Encoding: chunked
Content-Type: application/json; charset=UTF-8
Accept-Encoding: gzip

13
{"a":"b","c":101.5}
0

Span type and span operation name for `oauthtrace` are generic HTTP

See

go-sdk/stats/tracing/oauthtrace/tracer.go

Lines 30 to 33 in e7d08d5

	opentracing.Tag{Key: tracing.TagKeySpanType, Value: tracing.SpanTypeHTTP},
	opentracing.StartTime(time.Now().UTC()),
	}
	span, _ := tracing.StartSpanFromContext(ctx, t.tracer, tracing.OperationHTTPRequest, startOptions...)

The span.type=http tag and operationName=http.request in the StartSpanFromContext() don't seem specific enough to OAuth.

The OperationHTTPRequest is used in stats/tracing/oauthtrace, stats/tracing/r2trace and stats/tracing/webtrace (as of e7d08d5) and SpanTypeHTTP is used in stats/tracing/oauthtrace and stats/tracing/r2trace.

Add examples

Please add examples code at packages, mainly at big packages like 'logger'.
See examples at packages like:

https://godoc.org/go.uber.org/zap
https://godoc.org/github.com/rs/zerolog

That will help to get adoption/attraction.

`r2` sends invalid HTTP requests if `Path` does not have a leading `/`.

To reproduce (at 480ad70), first listen to port 5007 with netcat and then run this sample program

package main

import (
	"fmt"
	"time"

	"github.com/blend/go-sdk/r2"
)

func mustNil(err error) {
	if err != nil {
		panic(err)
	}
}

func main() {
	timeout := 5 * time.Second
	r := r2.New(
		"",
		r2.OptScheme("http"),
		r2.OptHost("localhost:5007"),
		r2.OptPath("v1/resource"),
		r2.OptTimeout(timeout),
	)

	res, err := r.Do()
	mustNil(err)
	fmt.Printf("%#v\n", res)

	// // Using r.Do() is equivalent to the following:
	// client := &http.Client{Timeout: timeout}
	// res, err := client.Do(&r.Request)
	// mustNil(err)
}

After running, we have

$ nc -l 5007
GET v1/resource HTTP/1.1
Host: localhost:5007
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

I will dig deeper into this at a later time, but just wanted to get the repro posted here.

`cmd/recover` doesn't work for programs that require a TTY / stdin

What happened:

$ go build -o main cmd/recover/main.go
$ ./main -verbose python
recover: "python" resolved to "/Users/dhermes/.pyenv/shims/python"
^Crecover: received SIGINT (interrupt) while sub process is running, killing sub process
recover: sub process exit: signal: interrupt

What you expected to happen:

$ ./main -verbose python
recover: "python" resolved to "/Users/dhermes/.pyenv/shims/python"
Python 3.7.3 (default, Mar 27 2019, 08:18:46) 
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
...

How to reproduce it (as minimally and precisely as possible):

See above

Anything else we need to know?:

This is not urgent or even necessarily a feature, but we could early exit if we detect TTY / stdin is needed.

Environment:

• go-sdk version: master at 9b8efca (I ran this as part of #138, so it's really at b55dead)
• go version (e.g go version): go version go1.13 darwin/amd64
• OS (e.g: cat /etc/os-release):

  $ sw_vers
  ProductName:    Mac OS X
  ProductVersion: 10.14.6
  BuildVersion:   18G103
  

crypto, only KeySize 32 is supported

What happened:

Can you explain why only key size = 32B is supported?
One would expect it will be at-least variable, thus can be override.

https://github.com/blend/go-sdk/blob/master/crypto/local_transit.go#L21

• the validation if len(decoded) != DefaultKeySize

// ParseKey parses a key from a string.
func ParseKey(key string) ([]byte, error) {
	decoded, err := hex.DecodeString(key)
	if err != nil {
		return nil, ex.New(err)
	}
	if len(decoded) != DefaultKeySize {
		return nil, ex.New("parse key; invalid key length")
	}
	return decoded, nil
}

`testutil` suite run silently aborts on panics in `OptBefore` hooks

What happened:
A panic in an OptBefore handler in a suite happened at runtime and the test run was aborted but reported as successful.

What you expected to happen:
The test run to be aborted but be reported as failed in the before hook.

How to reproduce it (as minimally and precisely as possible):
consider this illustrative example:

swallow.go

package swallow

func Add(a, b int) int {
	return a + b
}

With no panic in the before hook
swallow_test.go

func TestMain(m *testing.M) {
	suite := testutil.New(m,
		testutil.OptLog(logger.All()),
		testutil.OptBefore(func(c context.Context) error {
			return nil
		}))
	suite.Run()
}
func Test_Add_ShouldPass(t *testing.T) {
	result := swallow.Add(2, 2)
	if result != 4 {
		t.Fail()
	}
}

func Test_Add_ShouldFail(t *testing.T) {
	result := swallow.Add(2, 2)
	if result == 4 {
		t.Fail()
	}
}

 ~/workspace/go-test  go test -v ./pkg/swallow                                                                                   03:55:56 PM 
=== RUN   Test_Add_ShouldPass
--- PASS: Test_Add_ShouldPass (0.00s)
=== RUN   Test_Add_ShouldFail
--- FAIL: Test_Add_ShouldFail (0.00s)
FAIL
FAIL    github.com/softwarebygabe/go-test/pkg/swallow   0.126s
FAIL

Now with a panic
swallow_test.go

package swallow_test

import (
	"context"
	"testing"

	"github.com/blend/go-sdk/logger"
	"github.com/blend/go-sdk/testutil"
	"github.com/softwarebygabe/go-test/pkg/swallow"
)

func TestMain(m *testing.M) {
	suite := testutil.New(m,
		testutil.OptLog(logger.All()),
		testutil.OptBefore(func(c context.Context) error {
			panic("ahh")
			// return nil
		}))
	suite.Run()
}
func Test_Add_ShouldPass(t *testing.T) {
	result := swallow.Add(2, 2)
	if result != 4 {
		t.Fail()
	}
}

func Test_Add_ShouldFail(t *testing.T) {
	result := swallow.Add(2, 2)
	if result == 4 {
		t.Fail()
	}
}

 ~/workspace/go-test  go test -v ./pkg/swallow                                                                                   03:56:03 PM 
ok      github.com/softwarebygabe/go-test/pkg/swallow   0.127s
 ~/workspace/go-test  go test ./pkg/swallow                                                                                      03:58:12 PM 
ok      github.com/softwarebygabe/go-test/pkg/swallow   0.124s
 ~/workspace/go-test   

The only indication that the tests were not run is the lack of any tests being listed with the -v flag.

Anything else we need to know?:

Environment:

• go-sdk version: v1.20210826.18
• go version (e.g go version): go version go1.15.6 darwin/amd64
• OS (e.g: cat /etc/os-release): macOS 11.6
• Others:

When ever Particular key has empty value in XFCC Header, It is parsing wrongly

We have header XFCC something like this
dns=dnsvalue1;dns=dnsvalue2;URI=;DNS=dnsvalue3

But parsing is giving unexpected result

What happened:
DNS = {dnsvalue1,dnsvalue2}
URI = ;DNS=dnsvalue3

What you expected to happen:
DNS = {dnsvalue1,dnsvalue2,dnsvalue3}

How to reproduce it (as minimally and precisely as possible):
Just use this as xfcc header - dns=dnsvalue1;dns=dnsvalue2;URI=;DNS=dnsvalue3
and parse the function

Anything else we need to know?:

Environment:

• go-sdk version: v1.20220411.3
• go version (e.g go version): 1.19
• OS (e.g: cat /etc/os-release): Mac
• Others: