GithubHelp home page GithubHelp logo

kevinburke / go-bindata Goto Github PK

View Code? Open in Web Editor NEW
339.0 6.0 28.0 25.45 MB

A small utility which generates Go code from any file. Useful for embedding binary data in a Go program.

License: Other

Makefile 4.50% Go 95.50%
golang static-files

go-bindata's Introduction

bindata

This fork is maintained by Kevin Burke, and is the version trusted by Homebrew. Changes made include:

  • Atomic writes; generated file cannot be read while partially complete.

  • Better encoding of files that contain characters in the Unicode format range.

  • Generated file reports file sizes.

  • Generated code is run through go fmt.

  • SHA256 hashes are computed for all files and stored in the binary. You can use this to detect in-memory corruption and to provide easy cache-busting mechanisms.

  • Added AssetString and MustAssetString functions.

  • ByName is not public.

  • Some errors in file writes were unchecked, but are now checked.

  • File modes are stored in octal (0644) instead of nonsensical decimal (420)

This package converts any file into manageable Go source code. Useful for embedding binary data into a go program. The file data is optionally gzip compressed before being converted to a raw byte slice.

It comes with a command line tool in the go-bindata subdirectory. This tool offers a set of command line options, used to customize the output being generated.

Installation

On Macs, you can install the binary using Homebrew:

brew install go-bindata

You can also download a binary from the releases page. Switch in your GOOS for the word "linux" below, and the latest version for the version listed below:

curl --silent --location --output /usr/local/bin/go-bindata https://github.com/kevinburke/go-bindata/releases/download/v4.0.0/go-bindata-linux-amd64
chmod 755 /usr/local/bin/go-bindata

Alternatively, if you have a working Go installation, you can build the source and install the executable into $GOPATH/bin or $GOBIN:

go install github.com/kevinburke/go-bindata/v4/...@latest
# for versions of Go < 1.11, or without module support, use:
go install github.com/kevinburke/go-bindata/...

Usage

Conversion is done on one or more sets of files. They are all embedded in a new Go source file, along with a table of contents and an Asset function, which allows quick access to the asset, based on its name.

The simplest invocation generates a bindata.go file in the current working directory. It includes all assets from the data directory.

$ go-bindata data/

To include all input sub-directories recursively, use the ellipsis postfix as defined for Go import paths. Otherwise it will only consider assets in the input directory itself.

$ go-bindata data/...

To specify the name of the output file being generated, use the -o option:

$ go-bindata -o myfile.go data/

Multiple input directories can be specified if necessary.

$ go-bindata dir1/... /path/to/dir2/... dir3

The following paragraphs detail some of the command line options which can be supplied to go-bindata. Refer to the testdata/out directory for various output examples from the assets in testdata/in. Each example uses different command line options.

To ignore files, pass in regexes using -ignore, for example:

$ go-bindata -ignore=\\.gitignore data/...

Accessing an asset

To access asset data, we use the Asset(string) ([]byte, error) function which is included in the generated output.

data, err := Asset("pub/style/foo.css")
if err != nil {
	// Asset was not found.
}

// use asset data

Debug vs Release builds

When invoking the program with the -debug flag, the generated code does not actually include the asset data. Instead, it generates function stubs which load the data from the original file on disk. The asset API remains identical between debug and release builds, so your code will not have to change.

This is useful during development when you expect the assets to change often. The host application using these assets uses the same API in both cases and will not have to care where the actual data comes from.

An example is a Go webserver with some embedded, static web content like HTML, JS and CSS files. While developing it, you do not want to rebuild the whole server and restart it every time you make a change to a bit of javascript. You just want to build and launch the server once. Then just press refresh in the browser to see those changes. Embedding the assets with the debug flag allows you to do just that. When you are finished developing and ready for deployment, just re-invoke go-bindata without the -debug flag. It will now embed the latest version of the assets.

Lower memory footprint

Using the -nomemcopy flag, will alter the way the output file is generated. It will employ a hack that allows us to read the file data directly from the compiled program's .rodata section. This ensures that when we call our generated function, we omit unnecessary memcopies.

The downside of this, is that it requires dependencies on the reflect and unsafe packages. These may be restricted on platforms like AppEngine and thus prevent you from using this mode.

Another disadvantage is that the byte slice we create, is strictly read-only. For most use-cases this is not a problem, but if you ever try to alter the returned byte slice, a runtime panic is thrown. Use this mode only on target platforms where memory constraints are an issue.

The default behavior is to use the old code generation method. This prevents the two previously mentioned issues, but will employ at least one extra memcopy and thus increase memory requirements.

For instance, consider the following two examples:

This would be the default mode, using an extra memcopy but gives a safe implementation without dependencies on reflect and unsafe:

func myfile() []byte {
    return []byte{0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a}
}

Here is the same functionality, but uses the .rodata hack. The byte slice returned from this example can not be written to without generating a runtime error.

var _myfile = "\x89\x50\x4e\x47\x0d\x0a\x1a"

func myfile() []byte {
    var empty [0]byte
    sx := (*reflect.StringHeader)(unsafe.Pointer(&_myfile))
    b := empty[:]
    bx := (*reflect.SliceHeader)(unsafe.Pointer(&b))
    bx.Data = sx.Data
    bx.Len = len(_myfile)
    bx.Cap = bx.Len
    return b
}

Optional compression

When the -nocompress flag is given, the supplied resource is not GZIP compressed before being turned into Go code. The data should still be accessed through a function call, so nothing changes in the usage of the generated file.

This feature is useful if you do not care for compression, or the supplied resource is already compressed. Doing it again would not add any value and may even increase the size of the data.

The default behavior of the program is to use compression.

Path prefix stripping

The keys used in the _bindata map, are the same as the input file name passed to go-bindata. This includes the path. In most cases, this is not desireable, as it puts potentially sensitive information in your code base. For this purpose, the tool supplies another command line flag -prefix. This accepts a portion of a path name, which should be stripped off from the map keys and function names.

For example, running without the -prefix flag, we get:

$ go-bindata /path/to/templates/

_bindata["/path/to/templates/foo.html"] = path_to_templates_foo_html

Running with the -prefix flag, we get:

$ go-bindata -prefix "/path/to/" /path/to/templates/

_bindata["templates/foo.html"] = templates_foo_html

Build tags

With the optional -tags flag, you can specify any go build tags that must be fulfilled for the output file to be included in a build. This is useful when including binary data in multiple formats, where the desired format is specified at build time with the appropriate tags.

The tags are appended to a // +build line in the beginning of the output file and must follow the build tags syntax specified by the go tool.

Related projects

go-bindata-assetfs - implements http.FileSystem interface. Allows you to serve assets with net/http.

go-bindata's People

Contributors

abates avatar beyang avatar blazeeboy avatar byron avatar chenrui333 avatar craig-landry avatar dim13 avatar dmitshur avatar dnozay avatar dr2chase avatar elazarl avatar hut8 avatar ian-kent avatar imax9000 avatar inconshreveable avatar jbreitbart avatar johanbrandhorst avatar jwforres avatar keegancsmith avatar kevinburke avatar liggitt avatar mingrammer avatar octplane avatar riywo avatar rsc avatar rubenv avatar srhnsn avatar taichi avatar tamird avatar yosssi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

go-bindata's Issues

Error with linter due to obsolete use of ioutil.WriteFile

"io/ioutil" has been deprecated since Go 1.16: As of Go 1.16, the same functionality is now provided by package io or package os, and those implementations should be preferred in new code. See the specific function documentation for details. (SA1019)

Version number needs updating

It looks like the version number that's reported via go-bindata --version didn't get updated as part of the 3.18.0 release that resulted from #26. It still reports 3.17.0.

Remove error when fetching assets

We use gzip compression but we should be able to verify at compile time that the string in our file is valid, and remove the error from the method signature.

Error when using gosec.

Hi (great fork by the way) - our build uses go sec if we auto generate code then this causes a problem with the line io.Copy(&buf, gz) in release.go. Gosec flags this as I think an explosion issue.

I don't mind the act of doing the copy(unless it was easy to fix) as we are pretty controlled over what file we run this on but would it be possible to have the #nosec directive.

I am happy enough to raise a PR if needbe. It is perhaps something that would require a deeper look but I thought it worth raising as we incorporate on a pretty stringent CI lint/ sec etc etc setup and it works perfectly but for this one issue.

Thanks
Eoin

Remove fork attribution from weird repository

I forked this from the original project and now that project has been deleted so it appears like it's been forked from some random other fork.

I'd rather delete the fork but I think the only way to do that is to delete the project and readd it, which I'm not excited about - Homebrew installs releases from here for one thing.

ignore flag works differently with prefix flag set or not

Here is the code.
If my input path is .(aka pwd), and I set -ignore but not -prefix, it would not call filepath.Abs to make . into an abstract path. So I will get entries like a.go, b.go.
However, if I set -prefix, it would call filepath.Abs, so entries are turned into /abs/path/to/a.go, /abs/path/to/b.go.
This different handle may make -ignore not work as intended.
And I raise this issue to check whether this is working by designed. If not, I would be glad to contribute my PR to fix this.

Is it possible to disable the modified time portion, so that I will see the changes in generated files only when there is genuine changes in the asset files?

Currently, I am using this repo to generate files for swagger. I am tracking the generated files in git. When I re-run generate, even when there are no changes to swagger files, I can see the changes in git status, and the culprit is time in this line:

info := bindataFileInfo{name: "api.swagger.json", size: 20912, mode: os.FileMode(0664), modTime: time.Unix(1632899743, 0)}

Is it possible to disable the time portion, so that I will see the changes in generated files only when there is genuine changes in the asset files?

P.S. Passing -modtime 0 doesn't make any difference.

Edit: It seems passing integer other than 0 works fine. Thanks.

"go test -bench B ." writes files into testdata

This is anti-recommended, and causes spurious failures when tests and run using Go modules because all the source code is checked out read-only ( https://golang.org/ref/mod#module-cache also golang/go#28386).

Failure, observed with read-only testdata:

% go test -bench B .
--- FAIL: BenchmarkBindata
    benchmark_test.go:47: open testdata/assets/sf-fzlmnmzwlrm7hq2x.tmp: permission denied
FAIL
exit status 1
FAIL	github.com/kevinburke/go-bindata	0.793s
FAIL

Or, in a directory containing a go.mod:

go test github.com/kevinburke/go-bindata -bench B -run nope
--- FAIL: BenchmarkBindata
    benchmark_test.go:47: open testdata/assets/sf-jwvtvw26u2y72aov.tmp: permission denied
FAIL
exit status 1
FAIL	github.com/kevinburke/go-bindata	0.635s
FAIL

-run nope was specified above because of problems with other tests; this specific failure concerns running benchmarks, which is how I found this.

Update install from source instructions

If someone wants to install from source using 1.18, they get:

$ go get -u github.com/kevinburke/go-bindata/...
go: go.mod file not found in current directory or any parent directory.
        'go get' is no longer supported outside a module.
        To build and install a command, use 'go install' with a version,
        like 'go install example.com/cmd@latest'
        For more information, see https://golang.org/doc/go-get-install-deprecation
        or run 'go help get' or 'go help install'.

This is what's given in the README.

A fix will be given in a PR momentarily. Thanks for maintaining this awesome tool ๐Ÿ˜„

include version as a comments in the generated file

The output of go-bindata changes occasionally.

If a project using go-bindata run it on every build and checks in the output files, they have to ensure all contributors use the same version of go-bindata.

When a contributor finds that generated files on their machine come out different from what's checked in, they get confused on whether they did something wrong or the build is broken. Project maintainers could try to overcome the confusion by different means, of course, but it would be very convenient if go-bindata included some a version as part of the header comment in the generated file.

It's probably better to avoid using the release version of the tool, as generated files should have to change with every version, it could be the minor version (if files only change on minor version), or there could be a schema version of sort that has to increment every time a change to the generated code is inevitable.

So, if one has a simple git diff based test to find out whether generated files are up to date, right now they see something like:

diff --git a/pkg/addons/assets.go b/pkg/addons/assets.go
index 0cd5af1a..6aa21c66 100644
--- a/pkg/addons/assets.go
+++ b/pkg/addons/assets.go
@@ -25,7 +25,7 @@ import (
 func bindataRead(data []byte, name string) ([]byte, error) {
        gz, err := gzip.NewReader(bytes.NewBuffer(data))
        if err != nil {
-               return nil, fmt.Errorf("read %q: %v", name, err)
+               return nil, fmt.Errorf("Read %q: %v", name, err)
        }
 
        var buf bytes.Buffer
@@ -33,7 +33,7 @@ func bindataRead(data []byte, name string) ([]byte, error) {
        clErr := gz.Close()
 
        if err != nil {
-               return nil, fmt.Errorf("read %q: %v", name, err)
+               return nil, fmt.Errorf("Read %q: %v", name, err)
        }
        if clErr != nil {
                return nil, err

Instead, they could see this:

diff --git a/pkg/addons/assets.go b/pkg/addons/assets.go
index 0cd5af1a..6aa21c66 100644
--- a/pkg/addons/assets.go
+++ b/pkg/addons/assets.go
@@ -5,1 +5,1 @@ import (
- // go-bindata schema v1 (3.16.x)
+ // go-bindata schema v2 (3.17.x)
@@ -25,7 +25,7 @@ import (
 func bindataRead(data []byte, name string) ([]byte, error) {
        gz, err := gzip.NewReader(bytes.NewBuffer(data))
        if err != nil {
-               return nil, fmt.Errorf("read %q: %v", name, err)
+               return nil, fmt.Errorf("Read %q: %v", name, err)
        }
 
        var buf bytes.Buffer
@@ -33,7 +33,7 @@ func bindataRead(data []byte, name string) ([]byte, error) {
        clErr := gz.Close()
 
        if err != nil {
-               return nil, fmt.Errorf("read %q: %v", name, err)
+               return nil, fmt.Errorf("Read %q: %v", name, err)
        }
        if clErr != nil {
                return nil, err

That would make it much clearer that go-bindata version is key to the change in generated output.

Fewer allocations

The benchmark says a single run generates 16,000 allocations. That's going to be really slow.

Can we do anything to reduce the number of allocations?

Generated code is not gofmt simplified

Not strictly a necessity for generated code, but it doesn't hurt to fix. The offending line is

fmt.Fprintf(buf, "&bintree{%s, map[string]*bintree{", root.funcOrNil())
. When generated, this looks like this:

var _bintree = &bintree{nil, map[string]*bintree{
	"myfile": &bintree{myFile, map[string]*bintree{}},
}}

This can be simplified to

var _bintree = &bintree{nil, map[string]*bintree{
	"myfile": {myFile, map[string]*bintree{}},
}}

Which is what gofmt -s will do.

Include package names in errors

We ask the user to give us the package name, we can include it in the returned errors.

We could also add a IsNotExist(err) function for determining whether an error is because a file wasn't present.

Q: How to unpack a bindata.go file?

I have a bindata.go file - what is the easiest way to "unpack" all the assets back into regular files on disk?

I installed go-bindata with go install github.com/go-bindata/go-bindata/...@latest. Now I have go-bindata.exe in my global Go, but listing help it looks like there is no decompression option.
I guess I should write some simple Go program to uncompress (using RestoreAssets function in bindata.go?). But I've never written anything in Go, so any hints/directions are much appreciated.

expose whether build in debug mode

Use case: I have go templates as assets. In debug mode, I want to re-parse the template every time, in case it has changed. In non-debug mode, I want to parse only once. This decision and the code around it can't be done in go-bindata (unless you want to add template parsing helpers into the generated code). So I propose we add a AssetDebug() bool function or the like to the generated code.

How do I create bindata for linux/s390x?

Hi @kevinburke , very new to golang, so I might totally off-base here. A go application got onto my plate which compiles well for linux/amd64 & Darwin, but for linux/s390x, it is looking for a bindata file like go-bindata-linux-s390x

  • how do I create bindata for linux/s390x?
  • I downloaded the Source, but not sure what /data files are required.

staticcheck reports warning

src/assets/bindata.go:29746:53: string literal contains the Unicode format character U+200C, consider using the '\u200c' escape sequence (ST1018)

We should fix this to generate the right escape sequence.

cc @dominikh

bindata.go: List file sizes next to asset names

At the top of the file we do

// templates/index.html
// static/photos/style.css

I'm trying to figure out which files take up the most space and it might be helpful to append file sizes to that data.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.