A Go library for parsing struct tags from environment variables.
License: Apache License 2.0
I'm an engineer at Google. Previously I worked at HashiCorp, Chef Software, CustomInk, and some Pittsburgh-based startups.
In some cases we may want to specify a list of values separated by spaces. That use case doesn't seem to be supported at the moment. This
SomeList []string `env:"SOME_LIST,delimiter= "`always produces a slice with a single value.
Any struct fields that come after a custom decodable field are ignored
If a default strategy for filling in env variables values into struct definition is to overwrite them, it would be nice to specify it once when calling envconfig.Process.
When processing structs that implement a marshaller, this library has typically left it up to the unmarshaling function to populate the object. For example, given this config.
type Config struct {
URL *url.URL `env:"URL"`
}
The result of processing this configuration through env-config would be equivalent to marshaling and unmarshaling a url.URL struct. For example:
func MarshalTest(t *testing.T) {
var cfg Config
if err := envconfig.Process(context.Background(), &cfg); err != nil {
t.Fatal(err)
}
u := &url.URL{}
b, _ := u.MarshalBinary()
_ = u.UnmarshalBinary(b)
want := Config{
URL: u,
}
if diff := cmp.Diff(want, cfg, cmp.AllowUnexported(Config{})); diff != "" {
t.Errorf("Configuration() mismatch (-want +got):\n%s", diff)
}
}
Since v0.8.0 this test would fail. It seems that the library is skipping the decoder of a struct field if there are no values set. This generally seems ok for fields that can be controlled with noinit, default tags but in this example, the initialization of fields internal to url.URL cannot be controlled with env tags.
I think introducing the concept of managed vs unmanaged fields in the configuration will help with this. If a configuration field has an env tag it should follow the rules laid out by the env-config library, which will only use the decoder if a conf value is set (default or otherwise). Any struct fields without the env tag should default to using their underlying decoder no matter what. This will prevent potentially initializing fields that were not intended to be initialized.
I've included a draft PR to demonstrate this.
Hey,
Nice library, thanks for adding it!
I noticed a small problem with an error messages for keys with prefixes. The error message will not show the prefix for missing keys.
Example from the tests:
Lets say I have a nested structure Remote with a key env="REMOTE_NAME,required" and I use it with a prefix VCR_ the error message will read
Remote: Name: missing required value: REMOTE_NAME
but it should actually read
Remote: Name: missing required value: VCR_REMOTE_NAME
I can open a PR for this
I want to implement:
noinline mode for slices and maps, for example:export SOME_SLICE_0=a
export SOME_SLICE_1=btype Config struct {
SomeSlice []string `env:"SOME_SLICE, noinline"`
}
func main() {
ctx := context.Background()
var cfg Config
if err := envconfig.Process(ctx, &cfg); err != nil {
log.Fatal(err)
}
// cfg.SomeSlice = ["a", "b"]
}noinline mode from the first step, for example:export SOME_SLICE_0_PORT=5555
export SOME_SLICE_0_USERNAME=a
export SOME_SLICE_1_PORT=6666
export SOME_SLICE_1_USERNAME=btype NestedStruct struct {
Port int `env:"PORT"`
Username string `env:"USERNAME"`
}
type Config struct {
SomeSlice []NestedStruct `env:"SOME_SLICE"`
}
func main() {
ctx := context.Background()
var cfg Config
if err := envconfig.Process(ctx, &cfg); err != nil {
log.Fatal(err)
}
// cfg.SomeSlice[0].Username = "a"
// cfg.SomeSlice[0].Port= 5555
// cfg.SomeSlice[1].Username = "b"
// cfg.SomeSlice[1].Port= 6666
}I think this will be useful in some cases. What do you think about it?
Is there any way to define a custom validator for an environment variable? Such as a custom regex test?
Custom decoders of structs are called even though the field is not annotated with env tag.
My use case is using go-envconfig to fill in values in a config struct that is already partially filled from other sources.
A minimal example results in an error Time.UnmarshalBinary: no data. This is unexpected behaviour as an error is returned and the original filled value is replaced by the zero value.
type Config struct {
Time time.Time
}
I have seen #64 and #68 and I understand the need for backward compatibility, though they seem to apply to fields with env tags. Would it be possible to skip processing of untagged fields that are structs with custom decoders?
Example:
type Base64Bytes []byte
func (b *Base64Bytes) EnvDecode(val string) error {
var err error
*b, err = base64.StdEncoding.DecodeString(val)
return err
}
type Base64ByteSlice []Base64Bytesenvconfig.go has this specific block of logic:
Lines 570 to 581 in 39c9bc7
I agree with the sentiment of the block logic: custom decoders may want to process empty env vars in a specific way, rather than return early.
However, if an env for is not set at all, it is currently loaded as an empty string (here:
Lines 468 to 491 in 39c9bc7
It seems as if env var overrides should not be applied if the env var does not exist. In my code, this is conflict with a type that uses a field like so,
type SomeType struct {
// ..other fields
LogLevel zapcore.Level `env:"LEVEL,omitempty"`
}I specifically define in my config file that the level is debug. The zapcore code handles empty input to UnmarshalText as ok and writes the log level to info (see here: https://github.com/uber-go/zap/blob/6f34060764b5ea1367eecda380ba8a9a0de3f0e6/zapcore/level.go#L142-L143), and I have no control to stop this override.
My only current option is to wrap this type in another type that specifically handles the empty string as a no-op. Unfortunately, this breaks the potentially desired behavior of when an empty string is actually specified changing the level to info.
ISTM that the better default would be to not process env vars if the env var does not exist.
Hi there,
I have a use case where a custom type (an enum) implements both the envconfig.Decoder and json.Unmarshaler interfaces (e.g. for use as a value in a JSON body in an HTTP call).
When an invalid value is provided, the EnvDecode method errors first, then the UnmarshalJSON errors too, and the last error returned by UnmarshalJSON is forwarded to the caller.
But that error is not human-friendly, because it's a generic JSON decoding failure, whereas the error I'm returning in EnvDecode would have been a better choice.
For reference, here's a bit of pseudo-code corresponding to the case described above:
EnvDecode: https://play.golang.org/p/4sDh1tnevVujson.Unmarshaler interface: https://play.golang.org/p/Z2GIhUNEMOsI would advocate that when the developer explicitly chooses to implement the envconfig.Decoder interface, the internal processAsDecoder method should only try to use EnvDecode and not go through other decoders on error.
In that case, the EnvDecode method is the most adequate logic available to decode the value, and the error returned, if any, is the most useful we can get and should be preferably returned to the caller.
I tried to use noinit with a []byte slice, but the parser complained that field must be a pointer to have noinit. It seems a bit unnecessary to insist on having a pointer to a slice, since a slice in itself can be nil. Consider the following code:
package main
import "fmt"
type Lala struct {
Blah []byte
}
func main() {
lala := Lala{}
if lala.Blah == nil {
fmt.Println("Blah is nil")
}
lala.Blah = []byte{}
if lala.Blah != nil {
fmt.Println("Blah is not nil")
}
}
The result is:
Blah is nil
Blah is not nil
Hey,
sorry to bother you again @sethvargo, but this did not work #30
You tagged the commit just one before the relevant one.
The correct commit to tag is aa9acea
Thanks
I'm finding that order matters when parsing a struct like:
type MyAppConfig struct {
DisableFoo `env:"APP_DISABLE_FOO"`
Http *HttpServerConfiguration `env:",prefix=APP_"`
}
type HttpServerConfiguration struct {
Addr string `env:"HTTP_ADDR,required`
}If the DisableFoo field is listed first, it works as expected. If it is listed after the field with ,prefix=APP_ then it can't be overridden (you must set APP_APP_DISABLE_FOO=true to set it to true). I think this is because the lookuper isn't reset after processing the struct here:
Line 299 in 04d32d3
Hi!
I've come across the project - we are thinking of using it in our company :)
So I decided to improve the tooling inside.
It's great that you decided to follow the practice from https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module 👍
But the current approach has a downside - we get the unnecessary tool dependencies when using your library, because the versions are committed to the root go.mod
With approach from this PR #5 you could track the tooling dependencies without any impact on the main project.
Will be glad if you'd like to adopt it :)
hey - i have a use case for map[string]struct but the struct section in the documentation is lacking examples!
type Config struct {
UserStatusTTL time.Duration `env:"USER_STATUS_TTL,default=7d"`
}Currently it is not possible to find out how someone can configure an application via environment variables without checking the source code or manually creating a help or usage (which is error prone).
It would be great if we had something like:
type MyConfig struct {
Port int `env:"PORT" usage:"The server port."`
Username string `env:"USERNAME" usage:"The login username."`
}and maybe a function like:
func Usage(w io.Writer) {
}that would print a usage like with variable names and type and default values. This could also be printed out, when a required field is missing.
What do you think? Good idea, bad idea?
Hi, one of the things I generally do is have different settings required only if the feature flag is enabled. So for example:
type AppConfig struct {
Port int `env:"PORT"`
KafkaConfig KafkaConfig `env:",prefix=KAFKA_"`
}
type KafkaConfig struct {
Enabled bool `env:"ENABLED,required,enabler"`
Brokers []string `env:"BROKERS,required"`
}
The enabler option would be used to designate a single bool on a struct which would be used to enable/disable the "required-ness" of the rest of the required fields on the struct. So in the example above, only if KAKFA_ENABLED == true, would go-envconfig require KAFKA_BROKERS to be set.
Thoughts?
By default : is used as key-value separator while parsing map.
Sometimes the keys or values might also contain : in them (like in case of IPv6 addresses), in which case it becomes difficult to directly take the input as map[string]string.
In order to incorporate such scenarios, the key-value separator can be made easily customizable.
if you have a struct where env is not the first tag such as the following, it does not get the value from the environment and does not error:
type DatabaseConfig struct {
Host string `yaml:"host" ,env:"MYSQL_HOST,required"`
Port int `yaml:"port" ,env:"MYSQL_PORT,required"`
}
It works if you have:
type DatabaseConfig struct {
Host string `env:"MYSQL_HOST,required" ,yaml:"host"`
Port int `env:"MYSQL_PORT,required" ,yaml:"port"`
}
{
name: "bool_pointer_overwrite_default",
input: &struct {
Field *bool `env:"FIELD,noinit,overwrite,default=true"`
}{
Field: new(bool),
},
exp: &struct {
Field *bool `env:"FIELD,noinit,overwrite,default=true"`
}{
Field: new(bool),
},
lookuper: MapLookuper(nil),
},
I wrote test cases as above and it failed.
It does not follow the logic listed in https://github.com/sethvargo/go-envconfig#overwrite
If the struct field has a non-zero value and a default is set and if no environment variable is specified, the struct field's existing value will be used (the default is ignored).
Hi!
import "github.com/sethvargo/go-envconfig/pkg/envconfig"
Such import contains almost duplicated parts - go-envconfig and envconfig at the end.
Have you thought about moving go code to the project root?
We will have more elegant and meaningful import path then.
As the minimum supported Go version is specified as 1.22 by #104, latest version of this package can't be built with Go 1.21 anymore.
go: github.com/sethvargo/[email protected] requires go >= 1.22 (running go 1.21.5; GOTOOLCHAIN=local)
Go 1.21 is still officially supported.
Each major Go release is supported until there are two newer major releases.
Similar to #46
By default , is used as element delimiter while parsing maps and slices.
In situations when , itself is present in map/slice's elements, user might not be able to take direct input to map or slice.
In order to incorporate such scenarios, the element delimiter can be made easily customizable.
Is there any way to dump a complete list of the ENV variables envcconfig is looking for, possibly with some additional information, like the data type, etc.? Asking the devops people to decode the config struct to get all the ENV vars they need to supply when deploying seems a bit much. I'd prefer to stay on friendly terms with them.
Not sure why, but URLs are not being decoded properly.
The url.URL type implements the BinaryUnmarshaler interface (https://golang.org/pkg/net/url/#URL.UnmarshalBinary)
However, when used in a config struct, the unmarshaling adds an @ see this (https://play.golang.org/p/VrZDD85bR6j)
I have a quick question. Is there a possibility to use it with map[string][]string type? I didn't find it in the description or the tests but maybe there is possible
If I pass a type that implements Decoder to Process, the EnvDecode function is not called. However, if a field within my input struct implements Decoder, then EnvDecode is called.
In https://go.dev/play/p/wforR2YscKr, I would expect CALLED to be printed after both 1 and 2. It is only printed after 2.
We're transitioning to using the Go 1.18 netip.AddrPort with go-envconfig parsing it from a string. We noticed a small corner case though, that some addresses don't fail on the binary unmarshaling
Line 521 in befaf9a
So e.g. "239.255.76.67:7667" would succeed (not error) when unmarshaled as binary (resulting in [3233:392e:3235:352e:3736:2e36:373a:3736%6]:13623 but e.g. 127.0.0.1:11311 would fail and fall back to using the TextUnmarshaler, resulting in a properly decoded IPv4 address.
Could we swap the order so the unmarshaling as text happens before the binary one?
First - thank you for writing and making this library available. It is very useful and works well integrating it into a few projects so far.
I'm wondering if you might consider a feature to enable for re-use of struct configuration. It might be easiest to explain with an example.
Today, to have an application configuration with two HTTP servers (app and admin), you might define it as follows:
type MyAppConfig struct {
Http *HttpConfiguration
Admin *AdminConfiguration
}
type HttpConfiguration struct {
Addr string `env:"APP_HTTP_ADDR,default=:8080"`
ReadTimeout time.Duration `env:"APP_HTTP_READ_TIMEOUT"`
ReadHeaderTimeout time.Duration `env:"APP_HTTP_READ_HEADER_TIMEOUT"`
WriteTimeout time.Duration `env:"APP_HTTP_WRITE_TIMEOUT"`
IdleTimeout time.Duration `env:"APP_HTTP_IDLE_TIMEOUT"`
MaxHeadersBytes int `env:"APP_HTTP_MAX_HEADERS_BYTES"`
}
type AdminConfiguration struct {
Addr string `env:"ADMIN_HTTP_ADDR,default=:8081"`
ReadTimeout time.Duration `env:"ADMIN_HTTP_READ_TIMEOUT"`
ReadHeaderTimeout time.Duration `env:"ADMIN_HTTP_READ_HEADER_TIMEOUT"`
WriteTimeout time.Duration `env:"ADMIN_HTTP_WRITE_TIMEOUT"`
IdleTimeout time.Duration `env:"ADMIN_HTTP_IDLE_TIMEOUT"`
MaxHeadersBytes int `env:"ADMIN_HTTP_MAX_HEADERS_BYTES"`
}With a struct tag like env:prefix, you might be able to simplify this to:
type MyAppConfig struct {
Http *HttpServerConfiguration `env:"prefix=APP_"`
Admin *HttpServerConfiguration `env:"prefix=ADMIN_"`
}
type HttpServerConfiguration struct {
Addr string `env:"HTTP_ADDR,required`
ReadTimeout time.Duration `env:"HTTP_READ_TIMEOUT"`
ReadHeaderTimeout time.Duration `env:"HTTP_READ_HEADER_TIMEOUT"`
WriteTimeout time.Duration `env:"HTTP_WRITE_TIMEOUT"`
IdleTimeout time.Duration `env:"HTTP_IDLE_TIMEOUT"`
MaxHeadersBytes int `env:"HTTP_MAX_HEADERS_BYTES"`
}Existing extension mechanisms don't seem to support this. The PrefixLookuper supports prefixes at the top-level. The Decoder extension isn't given the context available to support parsing a struct (it supports single values). A struct tag is just an example of how this functionality might be supported - welcome to discuss possible implementations.
I would love to have the ability to use noinit on fields that are pointers to non-struct types, such as *string, like this:
type Config struct {
Something *string `env:"SOMETHING,noinit"`
}
...and then have the Something field left as nil instead of an empty string.
To be honest, I assumed this was how it worked until the debugger proved me wrong. 😢
The Process functions run each variable key through the envvarNameRe regex, and throws an error if the key doesn't match the regex. The validation of keys in such a way is really only a requirement if you're looking up a variable in the OS environment, e.g. using an OsLookuper. Currently, I can't set an arbitrary key to be used in my environment struct, e.g. something with dashes, even if I'm using an additional Lookuper where this type of validation might not be a requirement. Moving the regex matching into the OsLookuper would solve this problem and would be a more natural place for this kind of validation.
When doing something like env:"OVERRIDE_PORT,default=$DEFAULT_PORT", you would expect the default value to be the environment variable you get when running echo $DEFAULT_PORT in your terminal. However, it uses the same prefix as the field it's on. So, if OVERRIDE_PORT has a prefix of MY_PREFIX, the default that would be used would actually be MY_PREFIX_DEFAULT_PORT.
My specific use case where this is a problem is when I'm trying to deprecate old environment variables. I have a deprecated variable called FEATURE_DISABLEBUS, and I want to replace this with BUS_CONTROLLER_ENABLED. The new variable lives in a BusControllerConfig struct with a prefix of BUS_CONTROLLER. However, if I try to default to the deprecated variable, it tries to use BUS_CONTROLLER_FEATURE_DISABLEBUS.
export MY_TEST='abc$what'
package main
import (
"context"
"fmt"
"log"
"github.com/sethvargo/go-envconfig"
)
type MyConfig struct {
Test string `env:"MY_TEST,required"`
}
func main() {
ctx := context.Background()
var c MyConfig
if err := envconfig.Process(ctx, &c); err != nil {
log.Fatal(err)
} // .if
fmt.Printf("test: %s \n", c.Test)
} // .main
Result -> test: abc
Expected -> test: abc$what
HI there 👋🏼
It would be great if we can have exported fields that are "skipped". This is because this library seems to instantiate deeply nested structs which may not be desirable.
Something like
type Config struct {
Val string `env:"-"`
}is common practice to skip a field.
Happy to submit a PR if that's something that is worth doing. Thanks!
In the latest commit there was a change to the way that the library initialized pointer structs by default.
I've opened a PR with a proposed fix to the regression:
#57
When using environment configuration as an override to a file-based configuration (e.g. viper w/yaml), the field needs to overwrite the existing when present in environment variables, use existing value if it is non-zero, and error if the value is both zero and not present in the configuration.
Currently overwrite makes required error when the field is non-zero, but not present in the environment variables.
Given:
var cfg struct {
Field1 string `env:"FIELD1,default=first"`
Field2 string `env:"FIELD1,default=second to $FIELD1"`
}
os.Setenv("FIELD1", "one")
err := envconfig.Process(context.Background(), &cfg)
I would expect any of:
cfg.Field2 == "second to one"cfg.Field2 == "second to $FIELD1"err != nilInstead, I observe: cfg.Field2 == "one"
func Process(ctx context.Context, i any, mus ...Mutator) error and func ProcessWith(ctx context.Context, c *Config) error have similar names and signaures, and it would be easy to accidentally call Process(ctx, *Config) which returns no error, but will not properly fill the target the user intended.
I can think of two paths to preventing such an error:
Process which will return an error or panic in the case that it is called with *envconfig.Config for i. There is no reason I can think of a user would intentionally call Process with config, so I think failing is a reasonable course of action.Process with a *envconfig.Config into an equivalent ProcessWith call. The main issue here is dealing with the mus ... which could be done by adding any Mutators in the vararg to the Config provided, but the behavior may be somewhat unexpected.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.