magiconair / properties Goto Github PK

would it be possible to rename the license file to LICENSE.md so that it can be picked up by https://github.com/mitchellh/golicense

Thanks,
Damian.

I use the code like below , when the programe resolve the properties file , printf the message and exit . I don't want to the program exit , just throw a error to me , I can catch is err and continue reslove the other file .
use code
file_content:=properties.MustLoadFile(path+"/"+name,properties.UTF8) keys := file_content.Keys();
message
2019/10/15 15:37:09 malformed expression

According to properties format documentation https://docs.oracle.com/javase/6/docs/api/java/util/Properties.html this should be loaded as one property:

 fruits                           apple, banana, pear, \
                                  cantaloupe, watermelon, \
                                  kiwi, mango

but instead it is loaded as

fruits = apple, banana, pear,
cantaloupe, = watermelon,
kiwi, = mango

Problem:

When the property=${foo} set in the file is expanded, the returned value is an empty String.

Is there a particular reason why we have this approach?

Personally, it would be much easier to debug if it was just returned as is when not found.

Expected:
properties.Get("property") = "${foo}"

Actual:
properties.Get("property") = ""

In case the properties file has a logical division of the configurations, I'd like to decode to several dedicated structs.

For example:
Given the following config.properties file

app.port=3000
db.host=127.0.0.1
db.port=8000
db.username=username123
db.password=password123
pglistener.min_reconn=10 * time.Second
pglistener.max_reconn=1 * time.Minute
...
...

I'd like to decode to the following structs:

type Config struct {
	AppPort string `properties:"app.port,default=3000"`
}

type DBConfig struct {
	Host          string  `properties:"db.host,default=127.0.0.1"`
	Port           int       `properties:"db.port,default=5432"`
	Username string `properties:"db.username"`
	Password  string `properties:"db.password"`
}


type PGListener struct {
	MinReconnectInterval time.Duration `properties:"pglistener.min_reconn"`
	MaxReconnectInterval time.Duration `properties:"pglistener.max_reconn"`
}

There should be a convenience function for loading a UTF8 string.

As reported in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=830727 magiconair/properties fails test on x86_64 as follows:

  === RUN   Test
  2016/07/10 22:26:08 circular reference
  exit status 1
  FAIL  github.com/magiconair/properties    0.004s

I'm not sure if I'm missing something, but it seems you can only set properties to strings. I know you can read other types, but what if I was to SetInt()?

In GetInt function should return default value instead of panic

// GetInt parses the expanded value as an int if the key exists.
// If key does not exist or the value cannot be parsed the default
// value is returned. If the value does not fit into an int the
// function panics with an out of range error.
func (p *Properties) GetInt(key string, def int) int {
	v, err := p.getInt64(key)
	if err != nil {
		return def
	}
	return intRangeCheck(key, v)
}

more:

Support loading properties from a URL to move configuration to the network.

On Fedora Rawhide with Go 1.16 beta 1, I get the following test error:

Testing    in: /builddir/build/BUILD/properties-1.8.4/_build/src
         PATH: /builddir/build/BUILD/properties-1.8.4/_build/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
       GOPATH: /builddir/build/BUILD/properties-1.8.4/_build:/usr/share/gocode
  GO111MODULE: off
      command: go test -buildmode pie -compiler gc -ldflags " -X github.com/magiconair/properties/version=1.8.4 -extldflags '-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld  '"
      testing: github.com/magiconair/properties
github.com/magiconair/properties
2021/01/16 17:04:51 properties: doesnotexist.properties not found. skipping
2021/01/16 17:04:51 properties: /tmp/properties477326312foo not found. skipping
2021/01/16 17:04:51 properties: /tmp/properties722766887foo not found. skipping
2021/01/16 17:04:51 properties: http://127.0.0.1:38403/c returned 404. skipping
	assert.go:62: got time: invalid duration "ghi" which does not match time: invalid duration ghi
--- FAIL: TestMustGetParsedDuration (0.00s)
FAIL
exit status 1
FAIL	github.com/magiconair/properties	0.025s

It seems the following code does not handle version over 1.15:

When reading comments \ are loaded correctly, but when writing they are then replaced by \\. This leads to wrong comments when writing and reading multiple times.

I assume that it is not necessary to encode the comments.

If no comment was set for a key,value pair then I expect the WriteComment to not write the extra lines.

#
eddy.aq.concurrent.consumers = 10

#
eddy.aq.max.concurrent.consumers = 10

vs

eddy.aq.concurrent.consumers = 10
eddy.aq.max.concurrent.consumers = 10

As part of our configuration service, we need to store configurations as is in key-value records without substituing the ${..} expressions. Application that retrieve properties from this service will do the expansions.

My suggestion is to provide another Load function that takes the extra boolean parameter expandExpressions

I'm doing the following test:

// Set a key to "a=b" and value "c"
p := properties.NewProperties()
p.Set("a=b", "c")
buf := new(bytes.Buffer)
p.Write(buf, properties.UTF8)
println(buf.String())

I'd espect the serialized version to be:

a\=b = c

Since the \ should be used to escape the first equal sign, but the program prints:

a=b = c

Which seems wrong. If saved in a file and read again, the key becomes "a" (while, when reading from the correct file, the key is correctly interpreted as "a=b").

Hello everyone.

I was testing some code that utilizes the MustGetString function, but was confused when the tests were failing when I was asserting a panic to happen.

I dug into the code and found that it doesn't actually panic, but calls log.Fatal through your ErrorHandler. It would have saved me a lot of time if I knew this from the function's documentation.

I can make the changes myself, but I wanted to see if I am incorrect in my thinking that the documentation is inaccurate first.

Current Documentation:
MustGetString returns the expanded value for the given key if exists or panics otherwise.

Proposed Change:
MustGetString returns the expanded value for the given key if exists or calls log.Fatal otherwise.

PR #21 added a Map() function to render a Properties object into a string map. LoadMap() should be the counterpart to this to create a Properties object from a string map.

func ScanProperties(content string) (map[string]string, error) {
    result := map[string]string{}
    props, err := properties.Load([]byte(content), properties.ISO_8859_1)
    if err != nil {
        return result, err
    }
    for _, each := range props.Keys() {
        result[each], _ = props.Get(each)
    }
    return result, nil
}

the result contains an empty string as key.
I did not expect this.

Hi all,

Thanks for sharing your work. I find your package very useful. My properties files must also be edited by administrators. In my program I remove program generated properties and rewrite the properties file. Maintaining the order of the keys and the comments enables a friendlier file for administrators. The comments are transferred using GetComments. I could not find another way to keep the unexpanded property values, except by filtering them.

Currently the order of the keys is not maintained when any of the Filter, FilterRegexp or FilterPrefix is called. A small set of changes in properties.go will enable this.

  // FilterRegexp returns a new properties object which contains all properties
  // for which the key matches the regular expression.
  func (p *Properties) FilterRegexp(re *regexp.Regexp) *Properties {
    pp := NewProperties()
  //for k, v := range p.m {
    for _, k := range p.k {
        if re.MatchString(k) {
  //        pp.Set(k, v)
            pp.Set(k, p.m[k])
        }
    }
    return pp
  }

  // FilterPrefix returns a new properties object which contains all properties
  // for which the key starts with the prefix.
  func (p *Properties) FilterPrefix(prefix string) *Properties {
    pp := NewProperties()
  //for k, v := range p.m {
    for _, k := range p.k {
        if strings.HasPrefix(k, prefix) {
  //        pp.Set(k, v)
            pp.Set(k, p.m[k])
        }
    }
    return pp
  }

Thanks,

Wilbert

This seems like the fundamental load operation.

We use viper on our project, and viper has a dependency with magiconair/properties.

When downloading the code for magiconair/properties, we also get magiconair/properties/assert.

This is an issue, because magiconair/properties/assert is easily confused with testify/assert. In fact, under IntelliJ/Goland, magiconair/properties/assert is generally what is imported by defaulted when typing assert. In fact, we regularly need to remove incorrect references to magiconair/properties/assert.

Is there something that can be done? I'm happy to work on a PR that might offer a fix (presumably by moving files under assert to the main folder, plus changing the name of the assert.go file to something ending in _test.go).

This behavior is not observed with Java Properties#load(InputStream). In that case, quotes are not preserved in the deserialized value.

Detected on armhf by a Debian reproducible build server here:

https://reproducible.debian.net/rb-pkg/unstable/armhf/golang-github-magiconair-properties.html

go test -v github.com/magiconair/properties
# github.com/magiconair/properties
src/github.com/magiconair/properties/properties_test.go:811: constant 4294967296 overflows int
FAIL    github.com/magiconair/properties [build failed]

This happens inside TestPanicOn32BitUintOverflow():

https://github.com/magiconair/properties/blob/master/properties_test.go#L811

This test fails on 32-bit platforms such as armhf, but works fine with 64-bit platforms such as amd64.

Thanks in advance!

Anthony

Hello, would be possible to add support to decode into map[string]interface{}. I need to decode a properties file into a generic interface{} but below code is returning an error. Is there a reason to only allow it to be a struct?

	if t.Kind() != reflect.Ptr || v.Elem().Type().Kind() != reflect.Struct {
		return fmt.Errorf("not a pointer to struct: %s", t)
	}

There are some bugs when we specify properties as part of the struct metadata.
e.g

config file

graphite.host=localhost
graphite.metric.name=gtest

cloud.tenant.name=xxx-qa
cloud.keystone.url=https://xxx.zzz.ccc.aaa.xx/v2.0
cloud.swift.username=yyy-bb-ccc
cloud.swift.passwd=changeme

kafka.topic=ttt
kafka.servers=localhost:9092,zzz.dev.yyy.ca:9092
kafka.consumergroup=test
kafka.consumeFromStart=true

bulk.delete.max_wait=10s
log.level=debug

// Configuration holds external configuration for the consumers
type Configuration struct {
	CloudTenant           string              `properties:"cloud.tenant.name"`
	KeystoneURL           string              `properties:"cloud.keystone.url"`
	Username              string              `properties:"cloud.swift.username"`
	Userpwd               string              `properties:"cloud.swift.passwd"`
	Servers               []string			  `properties:"kafka.servers"`
	Cgroupname            string              `properties:"kafka.consumergroup"`
	Topic                 string              `properties:"kafka.topic"`
	ConsumeFromStart      bool                `properties:"kafka.consumeFromStart,default=false"`
        KafkaVersion          sarama.KafkaVersion `properties:"kafka.version,default=0.11.0.2"`
	Port                  int                 `properties:"port,default=8383"`
	Graphite              string              `properties:"graphite.host, default=carbon.service.consul"`
	GraphiteMetricName    string              `properties:"graphite.metric.name"`
	LogLevel              string              `properties:"log.level,default=info"`
	BulkDeleteMaxItems    int                 `properties:"bulk.delete.max_items,default=100"`
	BulkDeleteMaxWaitTime time.Duration       `properties:"bulk.delete.max_wait,default=5s"`
	

}

It fails on the first field/property it has a issue within the struct. All fields after this filed (in struct order) are not processed. e.g if the field KafkaVersion could not be decoded then all fields below it will also not be decoded.

There are issues in following cases
1> The property is missing in property file and no default value is mentioned in the struct
2> if the field type is not supported by decoder eg (sarama.KafkaVersion)
3> if the fields in the struct do not have the property tag and a property with the field name is not present in the property file.

Is there any way we can exclude a field from decode logic?
Regards
Rohan

The README states:

Comments and the order of keys are preserved.

However, comments at the end of a properties file are not preserved. See #44 for a failing test.

Note that the example properties file shown in wikipedia has a trailing comment, so trailing comments are certainly valid.

A special, degenerate case of trailing comments is a properties file consisting of comments alone.

It is not necessary and makes Debian packaging more complex since they unbundle the vendored dependencies.

Is there a way to change values in a config struct/use the default values and write them back into a properties file?

Hi I ran into this issue from eBay fabio, i want to set property 'registry.consul.register.tags'. i.e. "key1=val1" as one of the tag, but it has this kind of error:
[FATAL] 1.1.5. circular reference

is that a way to escape equal symbol. thanks

Hello,

it seems v1.7.4 has been released in code but the tag has not been pushed to github.

It's pretty easy to leave a single tab on a line after editing a large properties file.
I think the parser should simply ignore such a line instead of failing.

example.conf

log_line_prefix='\1\n\t%p\t%r\t%u\t%d\t%t\t%e\t%T\t%S\t%U\t%E\t\t'

After I run this golang code:

import (
	"fmt"
	"github.com/magiconair/properties"
        "os"
)

func main() {
        confName := "example.conf"
        filePros, _ := properties.LoadFile(confName, properties.UTF8)
        stat, _ := os.Stat(confName)
        writer, _ := os.OpenFile(confName, os.O_WRONLY|os.O_TRUNC, stat.Mode())
        defer writer.Close()
        filePros.WriteSeparator = "="
        filePros.Write(writer, properties.UTF8)
}

The example.conf content is change to this:

log_line_prefix='1\n\t%p\t%r\t%u\t%d\t%t\t%e\t%T\t%S\t%U\t%E\t\t'

The value's first '\' character lost, I think this is a bug.

Currently the Decode function returns an error if the tag does not contain 'default' and the key does not exist in the properties file.

Sometimes it's preferable to do not fail in this case but use the original field value of the structure passed to the Decode function, e.g by passing some specific key ('predefined') to the properties tag:

type Config struct {
	Field string `properties:"somefield,predefined"`
}

conf := &Config{
    Field: "Default value"
}

p := properties.MustLoadFiles([]string{
    "config.properties",
}, properties.UTF8, true)

err := p.Decode(conf)

if err == nil {
    fmt.Println(conf.Field)
}

Given a properties file consisting of:

Test = foo,bar,baz

A config struct of

type Config struct {
    Test []string `properties:"Test"`
}

And the following decoding

    var cfg Config
    if err := p.Decode(&cfg); err != nil {
        log.Fatal(err)
    }
    fmt.Printf("got test values: %s\n", cfg.Test[1])
```

My program crashes as cfg.Test is of length 1.

After parsing some input there is still a goroutine with a lexer around which will never be cleaned up. Parsing a lot of inputs leaks a goroutine per input.

You'll notice it in the goroutine dump as something similar to this:

goroutine 21 [chan send]:
github.com/magiconair/properties.(*lexer).emit(0xc208004240, 0x2)
    /Users/frschroeder/gopath/src/github.com/magiconair/properties/lex.go:99 +0xb9
github.com/magiconair/properties.lexKey(0xc208004240, 0x414148)
    /Users/frschroeder/gopath/src/github.com/magiconair/properties/lex.go:250 +0x132
github.com/magiconair/properties.(*lexer).run(0xc208004240)
    /Users/frschroeder/gopath/src/github.com/magiconair/properties/lex.go:177 +0x4e
created by github.com/magiconair/properties.lex
    /Users/frschroeder/gopath/src/github.com/magiconair/properties/lex.go:170 +0xce