minio / minio

The Object Store for AI Data Infrastructure

MinIO Quickstart Guide

MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Amazon S3 cloud storage service. Use MinIO to build high performance infrastructure for machine learning, analytics and application data workloads.

This README provides quickstart instructions on running MinIO on bare metal hardware, including container-based installations. For Kubernetes environments, use the MinIO Kubernetes Operator.

Container Installation

Use the following commands to run a standalone MinIO server as a container.

Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a minimum of 4 drives per MinIO server. See MinIO Erasure Code Overview for more complete documentation.

Stable

Run the following command to run the latest stable image of MinIO as a container using an ephemeral data volume:

podman run -p 9000:9000 -p 9001:9001 \
  quay.io/minio/minio server /data --console-address ":9001"

The MinIO deployment starts using default root credentials minioadmin:minioadmin. You can test the deployment using the MinIO Console, an embedded object browser built into MinIO Server. Point a web browser running on the host machine to http://127.0.0.1:9000 and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.

You can also connect using any S3-compatible tool, such as the MinIO Client mc commandline tool. See Test using MinIO Client mc for more information on using the mc commandline tool. For application developers, see https://min.io/docs/minio/linux/developers/minio-drivers.html to view MinIO SDKs for supported languages.

NOTE: To deploy MinIO on with persistent storage, you must map local persistent directories from the host OS to the container using the podman -v option. For example, -v /mnt/data:/data maps the host OS drive at /mnt/data to /data on the container.

macOS

Use the following commands to run a standalone MinIO server on macOS.

Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a minimum of 4 drives per MinIO server. See MinIO Erasure Code Overview for more complete documentation.

Homebrew (recommended)

Run the following command to install the latest stable MinIO package using Homebrew. Replace /data with the path to the drive or directory in which you want MinIO to store data.

brew install minio/stable/minio
minio server /data

NOTE: If you previously installed minio using brew install minio then it is recommended that you reinstall minio from minio/stable/minio official repo instead.

brew uninstall minio
brew install minio/stable/minio

The MinIO deployment starts using default root credentials minioadmin:minioadmin. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to http://127.0.0.1:9000 and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.

You can also connect using any S3-compatible tool, such as the MinIO Client mc commandline tool. See Test using MinIO Client mc for more information on using the mc commandline tool. For application developers, see https://min.io/docs/minio/linux/developers/minio-drivers.html/ to view MinIO SDKs for supported languages.

Binary Download

Use the following command to download and run a standalone MinIO server on macOS. Replace /data with the path to the drive or directory in which you want MinIO to store data.

wget https://dl.min.io/server/minio/release/darwin-amd64/minio
chmod +x minio
./minio server /data

The MinIO deployment starts using default root credentials minioadmin:minioadmin. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to http://127.0.0.1:9000 and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.

You can also connect using any S3-compatible tool, such as the MinIO Client mc commandline tool. See Test using MinIO Client mc for more information on using the mc commandline tool. For application developers, see https://min.io/docs/minio/linux/developers/minio-drivers.html to view MinIO SDKs for supported languages.

GNU/Linux

Use the following command to run a standalone MinIO server on Linux hosts running 64-bit Intel/AMD architectures. Replace /data with the path to the drive or directory in which you want MinIO to store data.

wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server /data

The following table lists supported architectures. Replace the wget URL with the architecture for your Linux host.

Architecture	URL
64-bit Intel/AMD	https://dl.min.io/server/minio/release/linux-amd64/minio
64-bit ARM	https://dl.min.io/server/minio/release/linux-arm64/minio
64-bit PowerPC LE (ppc64le)	https://dl.min.io/server/minio/release/linux-ppc64le/minio
IBM Z-Series (S390X)	https://dl.min.io/server/minio/release/linux-s390x/minio

The MinIO deployment starts using default root credentials minioadmin:minioadmin. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to http://127.0.0.1:9000 and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.

You can also connect using any S3-compatible tool, such as the MinIO Client mc commandline tool. See Test using MinIO Client mc for more information on using the mc commandline tool. For application developers, see https://min.io/docs/minio/linux/developers/minio-drivers.html to view MinIO SDKs for supported languages.

NOTE: Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a minimum of 4 drives per MinIO server. See MinIO Erasure Code Overview for more complete documentation.

Microsoft Windows

To run MinIO on 64-bit Windows hosts, download the MinIO executable from the following URL:

https://dl.min.io/server/minio/release/windows-amd64/minio.exe

Use the following command to run a standalone MinIO server on the Windows host. Replace D:\ with the path to the drive or directory in which you want MinIO to store data. You must change the terminal or powershell directory to the location of the minio.exe executable, or add the path to that directory to the system $PATH:

minio.exe server D:\

The MinIO deployment starts using default root credentials minioadmin:minioadmin. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to http://127.0.0.1:9000 and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.

You can also connect using any S3-compatible tool, such as the MinIO Client mc commandline tool. See Test using MinIO Client mc for more information on using the mc commandline tool. For application developers, see https://min.io/docs/minio/linux/developers/minio-drivers.html to view MinIO SDKs for supported languages.

NOTE: Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a minimum of 4 drives per MinIO server. See MinIO Erasure Code Overview for more complete documentation.

Install from Source

Use the following commands to compile and run a standalone MinIO server from source. Source installation is only intended for developers and advanced users. If you do not have a working Golang environment, please follow How to install Golang. Minimum version required is go1.21

go install github.com/minio/minio@latest

The MinIO deployment starts using default root credentials minioadmin:minioadmin. You can test the deployment using the MinIO Console, an embedded web-based object browser built into MinIO Server. Point a web browser running on the host machine to http://127.0.0.1:9000 and log in with the root credentials. You can use the Browser to create buckets, upload objects, and browse the contents of the MinIO server.

You can also connect using any S3-compatible tool, such as the MinIO Client mc commandline tool. See Test using MinIO Client mc for more information on using the mc commandline tool. For application developers, see https://min.io/docs/minio/linux/developers/minio-drivers.html to view MinIO SDKs for supported languages.

NOTE: Standalone MinIO servers are best suited for early development and evaluation. Certain features such as versioning, object locking, and bucket replication require distributed deploying MinIO with Erasure Coding. For extended development and production, deploy MinIO with Erasure Coding enabled - specifically, with a minimum of 4 drives per MinIO server. See MinIO Erasure Code Overview for more complete documentation.

MinIO strongly recommends against using compiled-from-source MinIO servers for production environments.

Deployment Recommendations

Allow port access for Firewalls

By default MinIO uses the port 9000 to listen for incoming connections. If your platform blocks the port by default, you may need to enable access to the port.

ufw

For hosts with ufw enabled (Debian based distros), you can use ufw command to allow traffic to specific ports. Use below command to allow access to port 9000

ufw allow 9000

Below command enables all incoming traffic to ports ranging from 9000 to 9010.

ufw allow 9000:9010/tcp

firewall-cmd

For hosts with firewall-cmd enabled (CentOS), you can use firewall-cmd command to allow traffic to specific ports. Use below commands to allow access to port 9000

firewall-cmd --get-active-zones

This command gets the active zone(s). Now, apply port rules to the relevant zones returned above. For example if the zone is public, use

firewall-cmd --zone=public --add-port=9000/tcp --permanent

Note that permanent makes sure the rules are persistent across firewall start, restart or reload. Finally reload the firewall for changes to take effect.

firewall-cmd --reload

iptables

For hosts with iptables enabled (RHEL, CentOS, etc), you can use iptables command to enable all traffic coming to specific ports. Use below command to allow access to port 9000

iptables -A INPUT -p tcp --dport 9000 -j ACCEPT
service iptables restart

Below command enables all incoming traffic to ports ranging from 9000 to 9010.

iptables -A INPUT -p tcp --dport 9000:9010 -j ACCEPT
service iptables restart

Test MinIO Connectivity

Test using MinIO Console

MinIO Server comes with an embedded web based object browser. Point your web browser to http://127.0.0.1:9000 to ensure your server has started successfully.

NOTE: MinIO runs console on random port by default, if you wish to choose a specific port use --console-address to pick a specific interface and port.

Things to consider

MinIO redirects browser access requests to the configured server port (i.e. 127.0.0.1:9000) to the configured Console port. MinIO uses the hostname or IP address specified in the request when building the redirect URL. The URL and port must be accessible by the client for the redirection to work.

For deployments behind a load balancer, proxy, or ingress rule where the MinIO host IP address or port is not public, use the MINIO_BROWSER_REDIRECT_URL environment variable to specify the external hostname for the redirect. The LB/Proxy must have rules for directing traffic to the Console port specifically.

For example, consider a MinIO deployment behind a proxy https://minio.example.net, https://console.minio.example.net with rules for forwarding traffic on port :9000 and :9001 to MinIO and the MinIO Console respectively on the internal network. Set MINIO_BROWSER_REDIRECT_URL to https://console.minio.example.net to ensure the browser receives a valid reachable URL.

Similarly, if your TLS certificates do not have the IP SAN for the MinIO server host, the MinIO Console may fail to validate the connection to the server. Use the MINIO_SERVER_URL environment variable and specify the proxy-accessible hostname of the MinIO server to allow the Console to use the MinIO server API using the TLS certificate.

For example: export MINIO_SERVER_URL="https://minio.example.net"

Dashboard	Creating a bucket

Test using MinIO Client mc

mc provides a modern alternative to UNIX commands like ls, cat, cp, mirror, diff etc. It supports filesystems and Amazon S3 compatible cloud storage services. Follow the MinIO Client Quickstart Guide for further instructions.

Upgrading MinIO

Upgrades require zero downtime in MinIO, all upgrades are non-disruptive, all transactions on MinIO are atomic. So upgrading all the servers simultaneously is the recommended way to upgrade MinIO.

NOTE: requires internet access to update directly from https://dl.min.io, optionally you can host any mirrors at https://my-artifactory.example.com/minio/

• For deployments that installed the MinIO server binary by hand, use mc admin update

mc admin update <minio alias, e.g., myminio>

• For deployments without external internet access (e.g. airgapped environments), download the binary from https://dl.min.io and replace the existing MinIO binary let's say for example /opt/bin/minio, apply executable permissions chmod +x /opt/bin/minio and proceed to perform mc admin service restart alias/.

• For installations using Systemd MinIO service, upgrade via RPM/DEB packages parallelly on all servers or replace the binary lets say /opt/bin/minio on all nodes, apply executable permissions chmod +x /opt/bin/minio and process to perform mc admin service restart alias/.

Upgrade Checklist

• Test all upgrades in a lower environment (DEV, QA, UAT) before applying to production. Performing blind upgrades in production environments carries significant risk.
• Read the release notes for MinIO before performing any upgrade, there is no forced requirement to upgrade to latest release upon every release. Some release may not be relevant to your setup, avoid upgrading production environments unnecessarily.
• If you plan to use mc admin update, MinIO process must have write access to the parent directory where the binary is present on the host system.
• mc admin update is not supported and should be avoided in kubernetes/container environments, please upgrade containers by upgrading relevant container images.
• We do not recommend upgrading one MinIO server at a time, the product is designed to support parallel upgrades please follow our recommended guidelines.

Explore Further

• MinIO Erasure Code Overview
• Use mc with MinIO Server
• Use minio-go SDK with MinIO Server
• The MinIO documentation website

Contribute to MinIO Project

Please follow MinIO Contributor's Guide

License

• MinIO source is licensed under the GNU AGPLv3.
• MinIO documentation is licensed under CC BY 4.0.
• License Compliance

minio's Issues

rpi2 UX

I ran go get -u github.com/minio/minio on my alarm rpi2 system and it finishes with:

./main.go:110: app.HideVersion undefined (type *cli.App has no field or method HideVersion)

Not sure where the binary is.

   Static hostname: alarmpi
         Icon name: computer
        Machine ID: 84dadb5e3d8a4370ae51d251350c4943
           Boot ID: a3c880f3dbc34dff8eac4228d44e8d09
  Operating System: Arch Linux ARM
            Kernel: Linux 4.1.13-1-ARCH
      Architecture: arm

Hardcoded 'mc' url in minio server command should be fixed.

We cannot really change this URL all the time, we should fix it by moving to 'mc/latest' URL which in-turn redirects to newest released binary.

Erasure decoding fails with new clang version

fatal error: unexpected signal during runtime execution
[signal 0xa code=0x2 addr=0xcb1e0f5520 pc=0x4004794]

runtime stack:
runtime.throw(0x459f9e0, 0x2a)
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/panic.go:527 +0x90
runtime.sigpanic()
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/sigpanic_unix.go:12 +0x5a

goroutine 67 [syscall, locked to thread]:
runtime.cgocall(0x4002d30, 0xc8202b1700, 0xc800000000)
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/cgocall.go:120 +0x11b fp=0xc8202b16b8 sp=0xc8202b1688
github.com/minio/minio/pkg/erasure._Cfunc_minio_get_source_target(0x200000000, 0x2, 0x0, 0x7fff5fbff670, 0xc8200fa000, 0xc82002c000, 0xc82002c020, 0x3e00000000)
        github.com/minio/minio/pkg/erasure/_obj/_cgo_gotypes.go:64 +0x3c fp=0xc8202b1700 sp=0xc8202b16b8
github.com/minio/minio/pkg/erasure.(*Erasure).Decode(0xc820193e30, 0xc82001a180, 0x4, 0x4, 0x130bf4, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/harsha/mygo/src/github.com/minio/minio/pkg/erasure/erasure_decode.go:103 +0x7d9 fp=0xc8202b18d0 sp=0xc8202b1700
github.com/minio/minio/pkg/donut.encoder.Decode(0xc820193e30, 0xc820010202, 0xc82001a180, 0x4, 0x4, 0x130bf4, 0x0, 0x0, 0x0, 0x3)
        /Users/harsha/mygo/src/github.com/minio/minio/pkg/donut/encoder.go:96 +0x76 fp=0xc8202b1950 sp=0xc8202b18d0
github.com/minio/minio/pkg/donut.bucket.decodeEncodedData(0xc82019342c, 0xa, 0x4510f78, 0x7, 0xecd96a0d1, 0x1018eb6d, 0x47487a0, 0xc820011a80, 0xe, 0xc820015f80, ...)
        /Users/harsha/mygo/src/github.com/minio/minio/pkg/donut/bucket.go:582 +0x41b fp=0xc8202b1ab0 sp=0xc8202b1950
github.com/minio/minio/pkg/donut.bucket.readObjectData(0xc82019342c, 0xa, 0x4510f78, 0x7, 0xecd96a0d1, 0x1018eb6d, 0x47487a0, 0xc820011a80, 0xe, 0xc820015f80, ...)
        /Users/harsha/mygo/src/github.com/minio/minio/pkg/donut/bucket.go:512 +0xc2f fp=0xc8202b1e90 sp=0xc8202b1ab0
runtime.goexit()
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8202b1e98 sp=0xc8202b1e90
created by github.com/minio/minio/pkg/donut.bucket.ReadObject
        /Users/harsha/mygo/src/github.com/minio/minio/pkg/donut/bucket.go:233 +0x5fe

Flat namespace problem with filesystem

Successful

$ mc cp main.go localhost/newbucket/test.go
main.go:                4.33 KB / 4.33 KB  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  100.00 % 959.03 KB/s 0

Not successful

 mc cp main.go localhost/newbucket/test.go/new1
mc: <ERROR> Failed to copy ‘main.go’. We encountered an internal error, please try again.
main.go:                4.33 KB / 4.33 KB  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  100.00 % 391.65 KB/s 0

Problem due to

{"Error":{"cause":"mkdir /Users/harsha/testdir/newbucket/test.go: not a directory","type":"*os.PathError","trace":[{"line":212,"file":"pkg/fs/fs-object.go","func":"fs.Filesystem.CreateObject"},{"line":180,"file":"object-handlers.go","func":"main.CloudStorageAPI.PutObjectHandler"}],"sysinfo":{"host.arch":"amd64","host.cpus":"4","host.lang":"go1.5.1","host.name":"command.local","host.os":"darwin","mem.heap.total":"4.8MB","mem.heap.used":"3.2MB","mem.total":"8.0MB","mem.used":"3.2MB"}},"level":"error","msg":"CreateObject failed.","time":"2015-11-02T19:07:51-08:00"}

Implement S3 compatible blob server

• blob server
  • s3 compatible
  • simple get/put (no on disk format)
  • aws compatible auth

Implement channel based FTW

Limit number of requests per ip

WriteHeader and Write(ErrorResponse) seems to have a race where we are not able to send proper XML replies back

WriteHeader and Write(ErrorResponse) - in writeErrorResponse() helper doesn't seem to work.

client receives a short read EOF without any body even with Content-Length being non zero, for some reason tests seem to work fine.

I tested this by disabling all handlers but still seems like setting 'Connection: 'close' in setCommonHeaders along with WriteHeader with error code leads to connection being closed abruptly before even reading header..

Enable bucket ACL for dl.minio.io, but an owner can upload

usage of path instead of filepath is not portable

You have many mistakes for using path and path/filepath.
When the argument should be treated as filepath, you must use filepath package. When the argument is the pseudo path something like a part of URL, you must use path instead of filepath.
For example, using path.Join for file-system deosn't work correctly on windows.

a := `C:\TEMP\`
b := `foo`
fmt.Println(path.Join(a, b))

You want this become C:\TEMP\foo but it become C:\TEMP\/foo.

I tried to fix but it's too many. So I worry about your mc will work correctly. Probably, other codes have same problems.

diff --git a/commands.go b/commands.go
index da70627..22c0d11 100644
--- a/commands.go
+++ b/commands.go
@@ -2,7 +2,7 @@ package main

 import (
    "os/user"
-   "path"
+   "path/filepath"
    "strings"
    "time"

@@ -164,7 +164,7 @@ func runDonut(c *cli.Context) {
    // supporting multiple paths
    var paths []string
    if strings.TrimSpace(c.Args().First()) == "" {
-       p := path.Join(u.HomeDir, "minio-storage", "donut")
+       p := filepath.Join(u.HomeDir, "minio-storage", "donut")
        paths = append(paths, p)
    } else {
        for _, arg := range c.Args() {
diff --git a/pkg/api/config/config.go b/pkg/api/config/config.go
index 9aa27b8..96f9e9e 100644
--- a/pkg/api/config/config.go
+++ b/pkg/api/config/config.go
@@ -21,7 +21,7 @@ import (
    "io"
    "os"
    "os/user"
-   "path"
+   "path/filepath"
    "sync"

    "github.com/minio/minio/pkg/iodine"
@@ -49,13 +49,13 @@ func (c *Config) SetupConfig() error {
        return iodine.New(err, nil)
    }

-   confPath := path.Join(u.HomeDir, ".minio")
+   confPath := filepath.Join(u.HomeDir, ".minio")
    if err := os.MkdirAll(confPath, 0700); err != nil {
        return iodine.New(err, nil)
    }

    c.ConfigPath = confPath
-   c.ConfigFile = path.Join(c.ConfigPath, "config.json")
+   c.ConfigFile = filepath.Join(c.ConfigPath, "config.json")
    if _, err := os.Stat(c.ConfigFile); os.IsNotExist(err) {
        _, err = os.Create(c.ConfigFile)
        if err != nil {
diff --git a/pkg/api/config/config_test.go b/pkg/api/config/config_test.go
index 4126fea..f2ca5bd 100644
--- a/pkg/api/config/config_test.go
+++ b/pkg/api/config/config_test.go
@@ -19,7 +19,7 @@ package config
 import (
    "io/ioutil"
    "os"
-   "path"
+   "path/filepath"
    "sync"
    "testing"

@@ -38,7 +38,7 @@ func (s *MySuite) TestConfig(c *C) {
    conf.ConfigLock = new(sync.RWMutex)
    conf.ConfigPath, _ = ioutil.TempDir("/tmp", "minio-test-")
    defer os.RemoveAll(conf.ConfigPath)
-   conf.ConfigFile = path.Join(conf.ConfigPath, "config.json")
+   conf.ConfigFile = filepath.Join(conf.ConfigPath, "config.json")
    if _, err := os.Stat(conf.ConfigFile); os.IsNotExist(err) {
        _, err = os.Create(conf.ConfigFile)
        if err != nil {
diff --git a/pkg/iodine/iodine.go b/pkg/iodine/iodine.go
index 481a793..b297c71 100644
--- a/pkg/iodine/iodine.go
+++ b/pkg/iodine/iodine.go
@@ -22,7 +22,7 @@ import (
    "errors"
    "fmt"
    "os"
-   "path"
+   "path/filepath"
    "reflect"
    "runtime"
    "strconv"
@@ -151,7 +151,7 @@ func createStackEntry() StackEntry {
    host, _ := os.Hostname()
    pc, file, line, _ := runtime.Caller(2)
    function := runtime.FuncForPC(pc).Name()
-   _, function = path.Split(function)
+   _, function = filepath.Split(function)
    file = strings.TrimPrefix(file, gopath) // trim gopath from file

    data := GetGlobalState()
@@ -222,8 +222,8 @@ func (err Error) Error() string {

 func init() {
    _, iodineFile, _, _ := runtime.Caller(0)
-   iodineFile = path.Dir(iodineFile)   // trim iodine.go
-   iodineFile = path.Dir(iodineFile)   // trim iodine
-   iodineFile = path.Dir(iodineFile)   // trim minio
-   gopath = path.Dir(iodineFile) + "/" // trim github.com
+   iodineFile = filepath.Dir(iodineFile)   // trim iodine.go
+   iodineFile = filepath.Dir(iodineFile)   // trim iodine
+   iodineFile = filepath.Dir(iodineFile)   // trim minio
+   gopath = filepath.Dir(iodineFile) + "/" // trim github.com
 }
diff --git a/pkg/storage/donut/donut_disk_darwin.go b/pkg/storage/donut/donut_disk_darwin.go
index e912ea1..b6eba7e 100644
--- a/pkg/storage/donut/donut_disk_darwin.go
+++ b/pkg/storage/donut/donut_disk_darwin.go
@@ -19,7 +19,7 @@ package donut
 import (
    "errors"
    "os"
-   "path"
+   "path/filepath"
    "syscall"

    "io/ioutil"
@@ -88,12 +88,12 @@ func (d disk) GetFSInfo() map[string]string {

 // MakeDir - make a directory inside disk root path
 func (d disk) MakeDir(dirname string) error {
-   return os.MkdirAll(path.Join(d.root, dirname), 0700)
+   return os.MkdirAll(filepath.Join(d.root, dirname), 0700)
 }

 // ListDir - list a directory inside disk root path, get only directories
 func (d disk) ListDir(dirname string) ([]os.FileInfo, error) {
-   contents, err := ioutil.ReadDir(path.Join(d.root, dirname))
+   contents, err := ioutil.ReadDir(filepath.Join(d.root, dirname))
    if err != nil {
        return nil, iodine.New(err, nil)
    }
@@ -109,7 +109,7 @@ func (d disk) ListDir(dirname string) ([]os.FileInfo, error) {

 // ListFiles - list a directory inside disk root path, get only files
 func (d disk) ListFiles(dirname string) ([]os.FileInfo, error) {
-   contents, err := ioutil.ReadDir(path.Join(d.root, dirname))
+   contents, err := ioutil.ReadDir(filepath.Join(d.root, dirname))
    if err != nil {
        return nil, iodine.New(err, nil)
    }
@@ -128,9 +128,9 @@ func (d disk) MakeFile(filename string) (*os.File, error) {
    if filename == "" {
        return nil, iodine.New(errors.New("Invalid argument"), nil)
    }
-   filePath := path.Join(d.root, filename)
+   filePath := filepath.Join(d.root, filename)
    // Create directories if they don't exist
-   if err := os.MkdirAll(path.Dir(filePath), 0700); err != nil {
+   if err := os.MkdirAll(filepath.Dir(filePath), 0700); err != nil {
        return nil, iodine.New(err, nil)
    }
    dataFile, err := os.OpenFile(filePath, os.O_WRONLY|os.O_CREATE, 0600)
@@ -145,7 +145,7 @@ func (d disk) OpenFile(filename string) (*os.File, error) {
    if filename == "" {
        return nil, iodine.New(errors.New("Invalid argument"), nil)
    }
-   dataFile, err := os.Open(path.Join(d.root, filename))
+   dataFile, err := os.Open(filepath.Join(d.root, filename))
    if err != nil {
        return nil, iodine.New(err, nil)
    }
diff --git a/pkg/storage/donut/donut_disk_linux.go b/pkg/storage/donut/donut_disk_linux.go
index da18689..daa5484 100644
--- a/pkg/storage/donut/donut_disk_linux.go
+++ b/pkg/storage/donut/donut_disk_linux.go
@@ -19,7 +19,7 @@ package donut
 import (
    "errors"
    "os"
-   "path"
+   "path/filepath"
    "syscall"

    "io/ioutil"
@@ -88,12 +88,12 @@ func (d disk) GetFSInfo() map[string]string {

 // MakeDir - make a directory inside disk root path
 func (d disk) MakeDir(dirname string) error {
-   return os.MkdirAll(path.Join(d.root, dirname), 0700)
+   return os.MkdirAll(filepath.Join(d.root, dirname), 0700)
 }

 // ListDir - list a directory inside disk root path, get only directories
 func (d disk) ListDir(dirname string) ([]os.FileInfo, error) {
-   contents, err := ioutil.ReadDir(path.Join(d.root, dirname))
+   contents, err := ioutil.ReadDir(filepath.Join(d.root, dirname))
    if err != nil {
        return nil, iodine.New(err, nil)
    }
@@ -109,7 +109,7 @@ func (d disk) ListDir(dirname string) ([]os.FileInfo, error) {

 // ListFiles - list a directory inside disk root path, get only files
 func (d disk) ListFiles(dirname string) ([]os.FileInfo, error) {
-   contents, err := ioutil.ReadDir(path.Join(d.root, dirname))
+   contents, err := ioutil.ReadDir(filepath.Join(d.root, dirname))
    if err != nil {
        return nil, iodine.New(err, nil)
    }
@@ -128,9 +128,9 @@ func (d disk) MakeFile(filename string) (*os.File, error) {
    if filename == "" {
        return nil, iodine.New(errors.New("Invalid argument"), nil)
    }
-   filePath := path.Join(d.root, filename)
+   filePath := filepath.Join(d.root, filename)
    // Create directories if they don't exist
-   if err := os.MkdirAll(path.Dir(filePath), 0700); err != nil {
+   if err := os.MkdirAll(filepath.Dir(filePath), 0700); err != nil {
        return nil, iodine.New(err, nil)
    }
    dataFile, err := os.OpenFile(filePath, os.O_WRONLY|os.O_CREATE, 0600)
@@ -145,7 +145,7 @@ func (d disk) OpenFile(filename string) (*os.File, error) {
    if filename == "" {
        return nil, iodine.New(errors.New("Invalid argument"), nil)
    }
-   dataFile, err := os.Open(path.Join(d.root, filename))
+   dataFile, err := os.Open(filepath.Join(d.root, filename))
    if err != nil {
        return nil, iodine.New(err, nil)
    }
diff --git a/pkg/storage/donut/donut_object.go b/pkg/storage/donut/donut_object.go
index a392bbe..794fce3 100644
--- a/pkg/storage/donut/donut_object.go
+++ b/pkg/storage/donut/donut_object.go
@@ -18,7 +18,7 @@ package donut

 import (
    "errors"
-   "path"
+   "path/filepath"

    "encoding/json"
    "io/ioutil"
@@ -39,13 +39,13 @@ func NewObject(objectName, p string) (Object, error) {
    }
    o := object{}
    o.name = objectName
-   o.objectPath = path.Join(p, objectName)
+   o.objectPath = filepath.Join(p, objectName)
    return o, nil
 }

 func (o object) GetObjectMetadata() (map[string]string, error) {
    objectMetadata := make(map[string]string)
-   objectMetadataBytes, err := ioutil.ReadFile(path.Join(o.objectPath, objectMetadataConfig))
+   objectMetadataBytes, err := ioutil.ReadFile(filepath.Join(o.objectPath, objectMetadataConfig))
    if err != nil {
        return nil, err
    }
@@ -58,7 +58,7 @@ func (o object) GetObjectMetadata() (map[string]string, error) {

 func (o object) GetDonutObjectMetadata() (map[string]string, error) {
    donutObjectMetadata := make(map[string]string)
-   donutObjectMetadataBytes, err := ioutil.ReadFile(path.Join(o.objectPath, donutObjectMetadataConfig))
+   donutObjectMetadataBytes, err := ioutil.ReadFile(filepath.Join(o.objectPath, donutObjectMetadataConfig))
    if err != nil {
        return nil, err
    }
diff --git a/pkg/storage/donut/donut_test.go b/pkg/storage/donut/donut_test.go
index 6277385..23b6976 100644
--- a/pkg/storage/donut/donut_test.go
+++ b/pkg/storage/donut/donut_test.go
@@ -23,7 +23,7 @@ import (
    "io"
    "io/ioutil"
    "os"
-   "path"
+   "path/filepath"
    "strconv"
    "testing"
    "time"
@@ -42,7 +42,7 @@ func createTestNodeDiskMap(p string) map[string][]string {
    nodes := make(map[string][]string)
    nodes["localhost"] = make([]string, 16)
    for i := 0; i < len(nodes["localhost"]); i++ {
-       diskPath := path.Join(p, strconv.Itoa(i))
+       diskPath := filepath.Join(p, strconv.Itoa(i))
        if _, err := os.Stat(diskPath); err != nil {
            if os.IsNotExist(err) {
                os.MkdirAll(diskPath, 0700)
diff --git a/pkg/storage/drivers/donut/donut.go b/pkg/storage/drivers/donut/donut.go
index c64072e..ccf23a4 100644
--- a/pkg/storage/drivers/donut/donut.go
+++ b/pkg/storage/drivers/donut/donut.go
@@ -21,7 +21,7 @@ import (
    "encoding/hex"
    "io"
    "os"
-   "path"
+   "path/filepath"
    "sort"
    "strconv"
    "strings"
@@ -56,7 +56,7 @@ func createNodeDiskMap(p string) map[string][]string {
    nodes := make(map[string][]string)
    nodes["localhost"] = make([]string, 16)
    for i := 0; i < len(nodes["localhost"]); i++ {
-       diskPath := path.Join(p, strconv.Itoa(i))
+       diskPath := filepath.Join(p, strconv.Itoa(i))
        if _, err := os.Stat(diskPath); err != nil {
            if os.IsNotExist(err) {
                os.MkdirAll(diskPath, 0700)
@@ -74,7 +74,7 @@ func createNodeDiskMapFromSlice(paths []string) map[string][]string {
    diskPaths := make([]string, len(paths))
    nodes := make(map[string][]string)
    for i, p := range paths {
-       diskPath := path.Join(p, strconv.Itoa(i))
+       diskPath := filepath.Join(p, strconv.Itoa(i))
        if _, err := os.Stat(diskPath); err != nil {
            if os.IsNotExist(err) {
                os.MkdirAll(diskPath, 0700)
diff --git a/pkg/storage/drivers/fs/fs_bucket.go b/pkg/storage/drivers/fs/fs_bucket.go
index c57a904..75dba02 100644
--- a/pkg/storage/drivers/fs/fs_bucket.go
+++ b/pkg/storage/drivers/fs/fs_bucket.go
@@ -95,7 +95,7 @@ func (fs *fsDriver) GetBucketMetadata(bucket string) (drivers.BucketMetadata, er
        return drivers.BucketMetadata{}, iodine.New(drivers.BucketNameInvalid{Bucket: bucket}, nil)
    }
    // get bucket path
-   bucketDir := path.Join(fs.root, bucket)
+   bucketDir := filepath.Join(fs.root, bucket)
    bucketMetadata := drivers.BucketMetadata{}
    fi, err := os.Stat(bucketDir)
    // check if bucket exists
@@ -140,7 +140,7 @@ func (fs *fsDriver) SetBucketMetadata(bucket, acl string) error {
        return iodine.New(drivers.InvalidACL{ACL: acl}, nil)
    }
    // get bucket path
-   bucketDir := path.Join(fs.root, bucket)
+   bucketDir := filepath.Join(fs.root, bucket)
    err := os.Chmod(bucketDir, aclToPerm(acl))
    if err != nil {
        return iodine.New(err, nil)
@@ -160,7 +160,7 @@ func (fs *fsDriver) ListObjects(bucket string, resources drivers.BucketResources
        return []drivers.ObjectMetadata{}, resources, iodine.New(drivers.ObjectNameInvalid{Bucket: bucket, Object: resources.Prefix}, nil)
    }

-   rootPrefix := path.Join(fs.root, bucket)
+   rootPrefix := filepath.Join(fs.root, bucket)
    // check bucket exists
    if _, err := os.Stat(rootPrefix); os.IsNotExist(err) {
        return []drivers.ObjectMetadata{}, resources, iodine.New(drivers.BucketNotFound{Bucket: bucket}, nil)
diff --git a/pkg/storage/drivers/fs/fs_multipart.go b/pkg/storage/drivers/fs/fs_multipart.go
index e2f8987..ab10f08 100644
--- a/pkg/storage/drivers/fs/fs_multipart.go
+++ b/pkg/storage/drivers/fs/fs_multipart.go
@@ -13,7 +13,7 @@ import (
    "io/ioutil"
    "math/rand"
    "os"
-   "path"
+   "path/filepath"
    "sort"
    "strconv"
    "strings"
@@ -37,7 +37,7 @@ type Multiparts struct {
 }

 func (fs *fsDriver) loadActiveSessions(bucket string) {
-   bucketPath := path.Join(fs.root, bucket)
+   bucketPath := filepath.Join(fs.root, bucket)
    _, err := os.Stat(bucketPath)
    if err != nil {
        return
@@ -112,7 +112,7 @@ func (fs *fsDriver) ListMultipartUploads(bucket string, resources drivers.Bucket
    if !drivers.IsValidBucket(bucket) {
        return drivers.BucketMultipartResourcesMetadata{}, iodine.New(drivers.BucketNameInvalid{Bucket: bucket}, nil)
    }
-   bucketPath := path.Join(fs.root, bucket)
+   bucketPath := filepath.Join(fs.root, bucket)
    _, err := os.Stat(bucketPath)
    // check bucket exists
    if os.IsNotExist(err) {
@@ -207,7 +207,7 @@ func (fs *fsDriver) NewMultipartUpload(bucket, key, contentType string) (string,
        return "", iodine.New(drivers.ObjectNameInvalid{Object: key}, nil)
    }

-   bucketPath := path.Join(fs.root, bucket)
+   bucketPath := filepath.Join(fs.root, bucket)
    _, err := os.Stat(bucketPath)
    // check bucket exists
    if os.IsNotExist(err) {
@@ -216,8 +216,8 @@ func (fs *fsDriver) NewMultipartUpload(bucket, key, contentType string) (string,
    if err != nil {
        return "", iodine.New(drivers.InternalError{}, nil)
    }
-   objectPath := path.Join(bucketPath, key)
-   objectDir := path.Dir(objectPath)
+   objectPath := filepath.Join(bucketPath, key)
+   objectDir := filepath.Dir(objectPath)
    if _, err := os.Stat(objectDir); os.IsNotExist(err) {
        err = os.MkdirAll(objectDir, 0700)
        if err != nil {
@@ -318,7 +318,7 @@ func (fs *fsDriver) CreateObjectPart(bucket, key, uploadID string, partID int, c
        expectedMD5Sum = hex.EncodeToString(expectedMD5SumBytes)
    }

-   bucketPath := path.Join(fs.root, bucket)
+   bucketPath := filepath.Join(fs.root, bucket)
    _, err := os.Stat(bucketPath)

    // check bucket exists
@@ -329,7 +329,7 @@ func (fs *fsDriver) CreateObjectPart(bucket, key, uploadID string, partID int, c
        return "", iodine.New(drivers.InternalError{}, nil)
    }

-   objectPath := path.Join(bucketPath, key)
+   objectPath := filepath.Join(bucketPath, key)
    objectDir := path.Dir(objectPath)
    if _, err := os.Stat(objectDir); os.IsNotExist(err) {
        err = os.MkdirAll(objectDir, 0700)
@@ -400,7 +400,7 @@ func (fs *fsDriver) CompleteMultipartUpload(bucket, key, uploadID string, parts
        return "", iodine.New(drivers.InvalidUploadID{UploadID: uploadID}, nil)
    }

-   bucketPath := path.Join(fs.root, bucket)
+   bucketPath := filepath.Join(fs.root, bucket)
    _, err := os.Stat(bucketPath)
    // check bucket exists
    if os.IsNotExist(err) {
@@ -410,7 +410,7 @@ func (fs *fsDriver) CompleteMultipartUpload(bucket, key, uploadID string, parts
        return "", iodine.New(drivers.InternalError{}, nil)
    }

-   objectPath := path.Join(bucketPath, key)
+   objectPath := filepath.Join(bucketPath, key)
    // check if object exists
    if _, err := os.Stat(objectPath); !os.IsNotExist(err) {
        return "", iodine.New(drivers.ObjectExists{
@@ -506,7 +506,7 @@ func (fs *fsDriver) ListObjectParts(bucket, key string, resources drivers.Object
        startPartNumber = objectResourcesMetadata.PartNumberMarker
    }

-   bucketPath := path.Join(fs.root, bucket)
+   bucketPath := filepath.Join(fs.root, bucket)
    _, err := os.Stat(bucketPath)
    // check bucket exists
    if os.IsNotExist(err) {
@@ -516,7 +516,7 @@ func (fs *fsDriver) ListObjectParts(bucket, key string, resources drivers.Object
        return drivers.ObjectResourcesMetadata{}, iodine.New(drivers.InternalError{}, nil)
    }

-   objectPath := path.Join(bucketPath, key)
+   objectPath := filepath.Join(bucketPath, key)
    multiPartfile, err := os.OpenFile(objectPath+"$multiparts", os.O_RDONLY, 0600)
    if err != nil {
        return drivers.ObjectResourcesMetadata{}, iodine.New(err, nil)
@@ -563,7 +563,7 @@ func (fs *fsDriver) AbortMultipartUpload(bucket, key, uploadID string) error {
        return iodine.New(drivers.InvalidUploadID{UploadID: uploadID}, nil)
    }

-   bucketPath := path.Join(fs.root, bucket)
+   bucketPath := filepath.Join(fs.root, bucket)
    _, err := os.Stat(bucketPath)
    // check bucket exists
    if os.IsNotExist(err) {
@@ -573,7 +573,7 @@ func (fs *fsDriver) AbortMultipartUpload(bucket, key, uploadID string) error {
        return iodine.New(drivers.InternalError{}, nil)
    }

-   objectPath := path.Join(bucketPath, key)
+   objectPath := filepath.Join(bucketPath, key)
    multiPartfile, err := os.OpenFile(objectPath+"$multiparts", os.O_RDWR, 0600)
    if err != nil {
        return iodine.New(err, nil)
diff --git a/pkg/storage/drivers/fs/fs_object.go b/pkg/storage/drivers/fs/fs_object.go
index 5437e3d..67f2691 100644
--- a/pkg/storage/drivers/fs/fs_object.go
+++ b/pkg/storage/drivers/fs/fs_object.go
@@ -20,7 +20,7 @@ import (
    "bytes"
    "io"
    "os"
-   "path"
+   "path/filepath"
    "strings"

    "crypto/md5"
@@ -47,7 +47,7 @@ func (fs *fsDriver) GetPartialObject(w io.Writer, bucket, object string, start,
        return 0, iodine.New(drivers.ObjectNameInvalid{Bucket: bucket, Object: object}, nil)
    }

-   objectPath := path.Join(fs.root, bucket, object)
+   objectPath := filepath.Join(fs.root, bucket, object)
    filestat, err := os.Stat(objectPath)
    switch err := err.(type) {
    case nil:
@@ -94,7 +94,7 @@ func (fs *fsDriver) GetObject(w io.Writer, bucket string, object string) (int64,
    if drivers.IsValidObjectName(object) == false {
        return 0, iodine.New(drivers.ObjectNameInvalid{Bucket: bucket, Object: object}, nil)
    }
-   objectPath := path.Join(fs.root, bucket, object)
+   objectPath := filepath.Join(fs.root, bucket, object)
    filestat, err := os.Stat(objectPath)
    switch err := err.(type) {
    case nil:
@@ -213,7 +213,7 @@ func (fs *fsDriver) CreateObject(bucket, key, contentType, expectedMD5Sum string
    }

    // check bucket exists
-   if _, err := os.Stat(path.Join(fs.root, bucket)); os.IsNotExist(err) {
+   if _, err := os.Stat(filepath.Join(fs.root, bucket)); os.IsNotExist(err) {
        return "", iodine.New(drivers.BucketNotFound{Bucket: bucket}, nil)
    }

@@ -229,8 +229,8 @@ func (fs *fsDriver) CreateObject(bucket, key, contentType, expectedMD5Sum string
    contentType = strings.TrimSpace(contentType)

    // get object path
-   objectPath := path.Join(fs.root, bucket, key)
-   objectDir := path.Dir(objectPath)
+   objectPath := filepath.Join(fs.root, bucket, key)
+   objectDir := filepath.Dir(objectPath)
    if _, err := os.Stat(objectDir); os.IsNotExist(err) {
        err = os.MkdirAll(objectDir, 0700)
        if err != nil {

Integrate database into signers for AccessKey, SecretKey addition

Implement min free disk precondition and handle gracefully upon such a condition

Add lru expire objects over time

Relook at SSL implementation along with ``mc`` client - needs more clarity for CA Authority and Client self signed certificates

Evaluate if 'milkyway' is the right region for minio server.

I think we should make it 'us-east-1' or 'minio' instead to signify something more meaningful. 'milkyway' is not easy to grasp and doesn't really make much sense.

s3cmd 1.6.0 fails to authenticate with 500 error

Hi.
I can't access minio with s3cmd:

 s3cmd -c ~/.s3cmd.minio la # list all buckets, using plain http with localhost:9000 as proxy and host

->

ERRO[0002] Initializing signature v4 failed. Error={Cause:Missing fields in auth header Type:*errors.errorString CallTrace:[{Line:46 Filename:api-signature.go Function:main.getCredentialsFromAuth Env:map[]} {Line:77 Filename:api-signature.go Function:main.isValidRegion Env:map[]} {Line:89 Filename:api-signature.go Function:main.stripAccessKeyID Env:map[]} {Line:108 Filename:api-signature.go Function:main.initSignatureV4 Env:map[]} {Line:92 Filename:signature-handler.go Function:main.signatureHandler.ServeHTTP Env:map[]}] SysInfo:map[host.os:linux host.lang:go1.5.1 host.cpus:4 mem.used:639kB mem.total:5.0MB mem.heap.used:639kB mem.heap.total:1.7MB host.arch:amd64 host.name:aves]}

Such access results in 500 response, which, i believe, should be 200 or 400/403 depending on client's authentication data. Also, if s3cmd is incompatible with minio at all, this should be mentioned in the docs, i guess.

Donut has bad hashmap state - map races

$ ./minio mode donut ""
Starting minio server on: http://127.0.0.1:9000
Starting minio server on: http://192.168.122.1:9000
Starting minio server on: http://192.168.200.142:9000
fatal error: bad map state

goroutine 228 [running]:
runtime.gothrow(0x87cb10, 0xd)
/usr/local/go/src/runtime/panic.go:503 +0x8e fp=0xc2080dc878 sp=0xc2080dc860
runtime.evacuate(0x72f9a0, 0xc208114210, 0x4)
/usr/local/go/src/runtime/hashmap.go:791 +0x370 fp=0xc2080dc938 sp=0xc2080dc878
runtime.growWork(0x72f9a0, 0xc208114210, 0xa)
/usr/local/go/src/runtime/hashmap.go:761 +0x8a fp=0xc2080dc958 sp=0xc2080dc938
runtime.mapassign1(0x72f9a0, 0xc208114210, 0xc2080dcb48, 0xc2080dcc80)
/usr/local/go/src/runtime/hashmap.go:401 +0x190 fp=0xc2080dc9f8 sp=0xc2080dc958
github.com/minio/minio/pkg/storage/donut.bucket.ListObjects(0xc20800bfab, 0xa, 0x8680b0, 0x7, 0xecd1e3510, 0xa74265b, 0xc5a020, 0x862870, 0x7, 0xc2080c2e70, ...)
/home/harsha/work/minio/gopath/src/github.com/minio/minio/pkg/storage/donut/bucket.go:99 +0x992 fp=0xc2080dcd88 sp=0xc2080dc9f8
github.com/minio/minio/pkg/storage/donut.donut.GetObjectMetadata(0x862870, 0x7, 0xc2080c2ea0, 0xc2080c2e70, 0xc20801ea20, 0xc2084726f6, 0xa, 0xc208472701, 0xb, 0x0, ...)
/home/harsha/work/minio/gopath/src/github.com/minio/minio/pkg/storage/donut/donut.go:327 +0x3bf fp=0xc2080dcf00 sp=0xc2080dcd88
github.com/minio/minio/pkg/storage/donut.(_donut).GetObjectMetadata(0xc2080c3b90, 0xc2084726f6, 0xa, 0xc208472701, 0xb, 0x7, 0x0, 0x0)
:41 +0xe6 fp=0xc2080dcf68 sp=0xc2080dcf00
github.com/minio/minio/pkg/storage/drivers/donut.donutDriver.GetObjectMetadata(0x7f365c21c4c0, 0xc2080c3b90, 0xc20802b000, 0x1, 0x1, 0xc2084726f6, 0xa, 0xc208472701, 0xb, 0x0, ...)
/home/harsha/work/minio/gopath/src/github.com/minio/minio/pkg/storage/drivers/donut/donut.go:311 +0x323 fp=0xc2080dd168 sp=0xc2080dcf68
github.com/minio/minio/pkg/storage/drivers/donut.(_donutDriver).GetObjectMetadata(0xc2080c3bc0, 0xc2084726f6, 0xa, 0xc208472701, 0xb, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
:7 +0x113 fp=0xc2080dd288 sp=0xc2080dd168
github.com/minio/minio/pkg/api.(_minioAPI).headObjectHandler(0xc20802abb0, 0x7f365c220b80, 0xc208768ea0, 0xc208035040)
/home/harsha/work/minio/gopath/src/github.com/minio/minio/pkg/api/api_object_handlers.go:109 +0x1fd fp=0xc2080dd558 sp=0xc2080dd288
github.com/minio/minio/pkg/api._minioAPI.(github.com/minio/minio/pkg/api.headObjectHandler)·fm(0x7f365c220b80, 0xc208768ea0, 0xc208035040)
/home/harsha/work/minio/gopath/src/github.com/minio/minio/pkg/api/api_router.go:59 +0x45 fp=0xc2080dd580 sp=0xc2080dd558
net/http.HandlerFunc.ServeHTTP(0xc20802b630, 0x7f365c220b80, 0xc208768ea0, 0xc208035040)
/usr/local/go/src/net/http/server.go:1265 +0x41 fp=0xc2080dd5a0 sp=0xc2080dd580
github.com/gorilla/mux.(_Router).ServeHTTP(0xc208046f00, 0x7f365c220b80, 0xc208768ea0, 0xc208035040)
/home/harsha/work/minio/gopath/src/github.com/gorilla/mux/mux.go:98 +0x297 fp=0xc2080dd6a8 sp=0xc2080dd5a0
github.com/minio/minio/pkg/api.contentTypeHandler.ServeHTTP(0x7f365c21f6c0, 0xc208046f00, 0x7f365c220b80, 0xc208768ea0, 0xc208035040)
/home/harsha/work/minio/gopath/src/github.com/minio/minio/pkg/api/api_generic_handlers.go:141 +0xb3 fp=0xc2080dd6f8 sp=0xc2080dd6a8
github.com/minio/minio/pkg/api.(_contentTypeHandler).ServeHTTP(0xc208087d80, 0x7f365c220b80, 0xc208768ea0, 0xc208035040)
:4 +0xbd fp=0xc2080dd730 sp=0xc2080dd6f8
github.com/minio/minio/pkg/api.timeHandler.ServeHTTP(0x7f365c21f6e8, 0xc208087d80, 0x7f365c220b80, 0xc208768ea0, 0xc208035040)
/home/harsha/work/minio/gopath/src/github.com/minio/minio/pkg/api/api_generic_handlers.go:170 +0x355 fp=0xc2080dd7b0 sp=0xc2080dd730
github.com/minio/minio/pkg/api.(_timeHandler).ServeHTTP(0xc208087d90, 0x7f365c220b80, 0xc208768ea0, 0xc208035040)
:5 +0xbd fp=0xc2080dd7e8 sp=0xc2080dd7b0
github.com/minio/minio/pkg/api.resourceHandler.ServeHTTP(0x7f365c21f710, 0xc208087d90, 0x7f365c220b80, 0xc208768ea0, 0xc208035040)
/home/harsha/work/minio/gopath/src/github.com/minio/minio/pkg/api/api_generic_handlers.go:228 +0xca fp=0xc2080ddaa0 sp=0xc2080dd7e8
github.com/minio/minio/pkg/api.(_resourceHandler).ServeHTTP(0xc208087da0, 0x7f365c220b80, 0xc208768ea0, 0xc208035040)
:7 +0xbd fp=0xc2080ddad8 sp=0xc2080ddaa0
github.com/minio/minio/pkg/api.validateAuthHandler.ServeHTTP(0x7f365c21f738, 0xc208087da0, 0x7f365c220b80, 0xc208768ea0, 0xc208035040)
/home/harsha/work/minio/gopath/src/github.com/minio/minio/pkg/api/api_generic_handlers.go:205 +0x250 fp=0xc2080ddb78 sp=0xc2080ddad8
github.com/minio/minio/pkg/api.(_validateAuthHandler).ServeHTTP(0xc208087db0, 0x7f365c220b80, 0xc208768ea0, 0xc208035040)
:6 +0xbd fp=0xc2080ddbb0 sp=0xc2080ddb78
github.com/minio/minio/pkg/api/quota.(_rateLimit).ServeHTTP(0xc20801ff60, 0x7f365c220b80, 0xc208768ea0, 0xc208035040)
/home/harsha/work/minio/gopath/src/github.com/minio/minio/pkg/api/quota/rate_limiter.go:40 +0x8b fp=0xc2080ddbf0 sp=0xc2080ddbb0
github.com/minio/minio/pkg/api/logging.(_logHandler).ServeHTTP(0xc20801ffa0, 0x7f365c220b48, 0xc20805ec80, 0xc208035040)
/home/harsha/work/minio/gopath/src/github.com/minio/minio/pkg/api/logging/logging.go:79 +0x1b6 fp=0xc2080ddcb0 sp=0xc2080ddbf0
net/http.serverHandler.ServeHTTP(0xc20805d2c0, 0x7f365c220b48, 0xc20805ec80, 0xc208035040)
/usr/local/go/src/net/http/server.go:1703 +0x19a fp=0xc2080ddd08 sp=0xc2080ddcb0
net/http.(_conn).serve(0xc20808c0a0)
/usr/local/go/src/net/http/server.go:1204 +0xb57 fp=0xc2080ddfd8 sp=0xc2080ddd08
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080ddfe0 sp=0xc2080ddfd8
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1751 +0x35e

Permissive policy

How do I configure a permissive listing policy of my object store?

<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Resource>/</Resource><RequestId>3L137</RequestId><HostId>3L137</HostId></Error>

Similar to aws s3api put-bucket-acl --bucket example.com --acl public-read.

I want to be able to address objects on my server as URLs. Bonus: SSL.

ON windows erasure compiles fine but throws error upon go test -race

C:\mygo\src\github.com\minio\minio\pkg\erasure>go test -race
fatal error: unexpected signal during runtime execution
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x609a1d]

runtime stack:
invalid spdelta 0x6099e7 0x609a1d 0xa4551 -1
gf_4vect_dot_prod_avx.next_vect()
?:0 +0x36

goroutine 8 [syscall, locked to thread]:
runtime.cgocall_errno(0x605eb0, 0xc082027788, 0x0)
c:/go/src/runtime/cgocall.go:130 +0xeb fp=0xc082027768 sp=0xc082027740
github.com/minio/minio/pkg/erasure._Cfunc_ec_encode_data(0xa00000040, 0xc000000005, 0x23e880, 0x23f
github.com/minio/minio/pkg/erasure/_test/_obj_test/_cgo_gotypes.go:32 +0x7a fp=0xc082027788
github.com/minio/minio/pkg/erasure.(_Erasure).Decode(0xc08205e3f0, 0xc08206e180, 0xf, 0xf, 0x23e, 0
C:/mygo/src/github.com/minio/minio/pkg/erasure/erasure_decode.go:113 +0xcc8 fp=0xc082027998
github.com/minio/minio/pkg/erasure.(_MySuite).TestCauchyEncodeDecodeSuccess(0x119c270, 0xc0820660f0
C:/mygo/src/github.com/minio/minio/pkg/erasure/cauchy_test.go:65 +0x309 fp=0xc082027aa0 sp=
runtime.call16(0x689928, 0xc082008740, 0x1000000010)
c:/go/src/runtime/asm_amd64.s:401 +0x4c fp=0xc082027ab8 sp=0xc082027aa0
reflect.Value.call(0x689860, 0x119c270, 0x313, 0x6af7b0, 0x4, 0xc082027f78, 0x1, 0x1, 0x0, 0x0, ...
c:/go/src/reflect/value.go:419 +0x1339 fp=0xc082027dd0 sp=0xc082027ab8
reflect.Value.Call(0x689860, 0x119c270, 0x313, 0xc082027f78, 0x1, 0x1, 0x0, 0x0, 0x0)
c:/go/src/reflect/value.go:296 +0xe0 fp=0xc082027e48 sp=0xc082027dd0
github.com/minio/check.func┬╖003(0xc0820660f0)
C:/mygo/src/github.com/minio/check/check.go:763 +0x569 fp=0xc082027fb0 sp=0xc082027e48
github.com/minio/check.func┬╖001()
C:/mygo/src/github.com/minio/check/check.go:657 +0xff fp=0xc082027fe0 sp=0xc082027fb0
runtime.goexit()
c:/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc082027fe8 sp=0xc082027fe0
created by github.com/minio/check.(*suiteRunner).forkCall
C:/mygo/src/github.com/minio/check/check.go:658 +0x687

goroutine 1 [chan receive]:
testing.RunTests(0x71e4b8, 0x118ac70, 0x1, 0x1, 0x1)
c:/go/src/testing/testing.go:556 +0xdca
testing.(*M).Run(0xc08200c190, 0x119bd00)
c:/go/src/testing/testing.go:485 +0xe8
main.main()
github.com/minio/minio/pkg/erasure/_test/_testmain.go:52 +0x294

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
c:/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 5 [chan receive]:
github.com/minio/check.(_suiteRunner).runTest(0xc08200e100, 0xc082012150, 0xc082066000)
C:/mygo/src/github.com/minio/check/check.go:801 +0x86
github.com/minio/check.(_suiteRunner).run(0xc08200e100, 0x119c270)
C:/mygo/src/github.com/minio/check/check.go:606 +0x4f8
github.com/minio/check.Run(0x689860, 0x119c270, 0xc082002ac0, 0x500000000000000)
C:/mygo/src/github.com/minio/check/run.go:92 +0x5f
github.com/minio/check.RunAll(0xc082002ac0, 0x0)
C:/mygo/src/github.com/minio/check/run.go:84 +0x13a
github.com/minio/check.TestingT(0xc082064000)
C:/mygo/src/github.com/minio/check/run.go:72 +0x4d3
github.com/minio/minio/pkg/erasure.Test(0xc082064000)
C:/mygo/src/github.com/minio/minio/pkg/erasure/cauchy_test.go:30 +0x3d
testing.tRunner(0xc082064000, 0x118ac70)
c:/go/src/testing/testing.go:447 +0x13b
created by testing.RunTests
c:/go/src/testing/testing.go:555 +0xd55

goroutine 6 [select]:
github.com/minio/check.(_resultTracker)._loopRoutine(0xc082064090)
C:/mygo/src/github.com/minio/check/check.go:452 +0x515
created by github.com/minio/check.(_resultTracker).start
C:/mygo/src/github.com/minio/check/check.go:432 +0x47
exit status 2
FAIL github.com/minio/minio/pkg/erasure 0.362s

ListBuckets doesn't work when there's no buckets

mc ls http://127.0.0.1:9000
mc: <ERROR> Unable to list target ‘http://127.0.0.1:9000/’. The specified bucket is not valid.

$ mc mb http://127.0.0.1:9000/foo
Bucket created successfully  ‘http://127.0.0.1:9000/foo’

$ mc ls http://127.0.0.1:9000
[2015-09-17 14:30:43 PDT]     0B foo/

modifying existing blobs

Does it possible to modify exiting blob that already stored in server?

Implement (DOA) Operations Console (Basic health monitoring)

Bad hash map state while using contentdb

fatal error: bad map state

goroutine 51 [running]:
runtime.throw(0x510870, 0xd)
    /usr/local/opt/go/libexec/src/runtime/panic.go:527 +0x90 fp=0xc8200a9840 sp=0xc8200a9828
runtime.evacuate(0x3be0c0, 0xc82041aba0, 0x74)
    /usr/local/opt/go/libexec/src/runtime/hashmap.go:825 +0x3b1 fp=0xc8200a9900 sp=0xc8200a9840
runtime.growWork(0x3be0c0, 0xc82041aba0, 0xbe)
    /usr/local/opt/go/libexec/src/runtime/hashmap.go:795 +0x83 fp=0xc8200a9920 sp=0xc8200a9900
runtime.mapassign1(0x3be0c0, 0xc82041aba0, 0xc8200a9ab0, 0xc8200a9aa0)
    /usr/local/opt/go/libexec/src/runtime/hashmap.go:433 +0x176 fp=0xc8200a99c8 sp=0xc8200a9920
github.com/minio/mc/vendor/github.com/minio/minio/pkg/contentdb.loadDB(0x0, 0x0)
    /Users/harsha/mygo/src/github.com/minio/mc/vendor/github.com/minio/minio/pkg/contentdb/contentdb.go:71 +0x429 fp=0xc8200a9bb8 sp=0xc8200a99c8
github.com/minio/mc/vendor/github.com/minio/minio/pkg/contentdb.Init(0x0, 0x0)
    /Users/harsha/mygo/src/github.com/minio/mc/vendor/github.com/minio/minio/pkg/contentdb/contentdb.go:85 +0x7d fp=0xc8200a9bf8 sp=0xc8200a9bb8
github.com/minio/mc/vendor/github.com/minio/minio/pkg/contentdb.Lookup(0xc82035e0f6, 0x5, 0x0, 0x0, 0x0, 0x0)
    /Users/harsha/mygo/src/github.com/minio/mc/vendor/github.com/minio/minio/pkg/contentdb/contentdb.go:94 +0x41 fp=0xc8200a9c38 sp=0xc8200a9bf8
github.com/minio/mc/pkg/client/s3.(*s3Client).Put(0xc8203ef620, 0x89f878, 0xc82044c420, 0x4d1d50, 0x0)
    /Users/harsha/mygo/src/github.com/minio/mc/pkg/client/s3/s3.go:165 +0x91 fp=0xc8200a9d38 sp=0xc8200a9c38
main.putTarget(0xc8203fc4b0, 0x41, 0x89f878, 0xc82044c420, 0x4d1d50, 0xc820525c80)
    /Users/harsha/mygo/src/github.com/minio/mc/common-methods.go:65 +0x124 fp=0xc8200a9d98 sp=0xc8200a9d38
main.doCopy(0xc8201f1b00, 0xc8201f1b80, 0x0, 0xc820452b80, 0xc820471020, 0xc820468850, 0xc820452bb0, 0xc820471140)
    /Users/harsha/mygo/src/github.com/minio/mc/cp-main.go:177 +0x68c fp=0xc8200a9f60 sp=0xc8200a9d98
runtime.goexit()
    /usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc8200a9f68 sp=0xc8200a9f60
created by main.doCopySession.func2
    /Users/harsha/mygo/src/github.com/minio/mc/cp-main.go:373 +0x404

Implement update command before release

server: Support multiple api keys

We'd like have each key be limited to specific buckets. There are many use cases where this would be useful. Out use case is a different buffer for development servers and production. We'd like to limit access to production buckets to only team members with access to production servers. The rest of the team would just have access to the development buckets

Handling out of memory conditions

Start minio server with memory mode

$ ./minio mode memory 1G

Upload a 2GB file from mc

$ ./mc cp bigfile localhost:testbucket/test
521.28 MB / 2.00 GB [==========================>                                                                               ] 25.45 % 53.16 MB/s 27smc: <FATAL> Failed to copy from source [bigfile] to target [http://localhost:9000/testbucket/test]. Reason: [io: read/write on closed pipe].

Minio server output

rting HTTP Server on: :9000
2015/04/24 13:56:55 Starting HTTP Server on: :9001
2015/04/24 14:04:35 evicting: testbucket/newfile
2015/04/24 14:04:35 evicting: testbucket/test.md
2015/04/24 14:04:35 evicting: testbucket/2014/test.md
2015/04/24 14:04:35 evicting: testbucket/test
fatal error: runtime: out of memory

runtime stack:
runtime.SysMap(0xc32c100000, 0x40000000, 0x0, 0xc3b3d8)
    /usr/local/go/src/runtime/mem_linux.c:149 +0x98
runtime.MHeap_SysAlloc(0xc40ac0, 0x40000000, 0x43a252)
    /usr/local/go/src/runtime/malloc.c:284 +0x124
runtime.MHeap_Alloc(0xc40ac0, 0x20000, 0x10100000000, 0x0)
    /usr/local/go/src/runtime/mheap.c:240 +0x66

goroutine 37 [running]:
runtime.switchtoM()
    /usr/local/go/src/runtime/asm_amd64.s:198 fp=0xc2080ccfc8 sp=0xc2080ccfc0
runtime.mallocgc(0x3ffffe00, 0x70dfc0, 0xc200000001, 0x1)
    /usr/local/go/src/runtime/malloc.go:199 +0x9f3 fp=0xc2080cd078 sp=0xc2080ccfc8
runtime.newarray(0x70dfc0, 0x3ffffe00, 0xc31500)
    /usr/local/go/src/runtime/malloc.go:365 +0xc1 fp=0xc2080cd0b0 sp=0xc2080cd078
runtime.makeslice(0x6ffdc0, 0x3ffffe00, 0x3ffffe00, 0x0, 0x0, 0x0)
    /usr/local/go/src/runtime/slice.go:32 +0x15c fp=0xc2080cd0f8 sp=0xc2080cd0b0
bytes.makeSlice(0x3ffffe00, 0x0, 0x0, 0x0)
    /usr/local/go/src/bytes/buffer.go:191 +0x6a fp=0xc2080cd148 sp=0xc2080cd0f8
bytes.(*Buffer).ReadFrom(0xc2080973b0, 0x7f98f9022a28, 0xc20800b7c0, 0x1ffffe00, 0x0, 0x0)
    /usr/local/go/src/bytes/buffer.go:163 +0xda fp=0xc2080cd1e0 sp=0xc2080cd148
io.Copy(0x7f98f90214d0, 0xc2080973b0, 0x7f98f9022a28, 0xc20800b7c0, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:358 +0x13d fp=0xc2080cd298 sp=0xc2080cd1e0
github.com/minio-io/minio/pkg/storage/drivers/memory.(*memoryDriver).CreateObject(0xc20803d560, 0xc208094665, 0xa, 0xc208094670, 0x4, 0x8831d0, 0x18, 0x0, 0x0, 0x7f98f9022a28, ...)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/minio/pkg/storage/drivers/memory/memory.go:180 +0x5f9 fp=0xc2080cd580 sp=0xc2080cd298
github.com/minio-io/minio/pkg/api.(*minioAPI).putObjectHandler(0xc20801eb20, 0x7f98f9022818, 0xc2080d1900, 0xc208035380)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/minio/pkg/api/api_object_handlers.go:158 +0x38b fp=0xc2080cd7c0 sp=0xc2080cd580
github.com/minio-io/minio/pkg/api.*minioAPI.(github.com/minio-io/minio/pkg/api.putObjectHandler)·fm(0x7f98f9022818, 0xc2080d1900, 0xc208035380)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/minio/pkg/api/api_router.go:43 +0x45 fp=0xc2080cd7e8 sp=0xc2080cd7c0
net/http.HandlerFunc.ServeHTTP(0xc20802b7d0, 0x7f98f9022818, 0xc2080d1900, 0xc208035380)
    /usr/local/go/src/net/http/server.go:1265 +0x41 fp=0xc2080cd808 sp=0xc2080cd7e8
github.com/gorilla/mux.(*Router).ServeHTTP(0xc208046730, 0x7f98f9022818, 0xc2080d1900, 0xc208035380)
    /home/harsha/work/minio/gopath/src/github.com/gorilla/mux/mux.go:98 +0x297 fp=0xc2080cd910 sp=0xc2080cd808
github.com/minio-io/minio/pkg/api.rHandler.ServeHTTP(0x7f98f90214f8, 0xc208046730, 0x7f98f9022818, 0xc2080d1900, 0xc208035380)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/minio/pkg/api/api_generic_handlers.go:121 +0xca fp=0xc2080cdbc8 sp=0xc2080cd910
github.com/minio-io/minio/pkg/api.(*rHandler).ServeHTTP(0xc20802bff0, 0x7f98f9022818, 0xc2080d1900, 0xc208035380)
    <autogenerated>:2 +0xbd fp=0xc2080cdc00 sp=0xc2080cdbc8
github.com/minio-io/minio/pkg/api.vHandler.ServeHTTP(0xc20801f020, 0x13, 0xc20801f080, 0x1f, 0xc20801f0c0, 0x0, 0x7f98f9021520, 0xc20802bff0, 0x7f98f9022818, 0xc2080d1900, ...)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/minio/pkg/api/api_generic_handlers.go:65 +0xc5 fp=0xc2080cdc50 sp=0xc2080cdc00
github.com/minio-io/minio/pkg/api.(*vHandler).ServeHTTP(0xc20800b880, 0x7f98f9022818, 0xc2080d1900, 0xc208035380)
    <autogenerated>:1 +0xbe fp=0xc2080cdcb0 sp=0xc2080cdc50
net/http.serverHandler.ServeHTTP(0xc208030c60, 0x7f98f9022818, 0xc2080d1900, 0xc208035380)
    /usr/local/go/src/net/http/server.go:1703 +0x19a fp=0xc2080cdd08 sp=0xc2080cdcb0
net/http.(*conn).serve(0xc2080d1860)
    /usr/local/go/src/net/http/server.go:1204 +0xb57 fp=0xc2080cdfd8 sp=0xc2080cdd08
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc2080cdfe0 sp=0xc2080cdfd8
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1751 +0x35e

goroutine 1 [select, 8 minutes]:
reflect.Select(0xc2080109a0, 0x2, 0x2, 0xc2080109a0, 0x0, 0x0, 0x0, 0x2)
    /usr/local/go/src/reflect/value.go:1965 +0x218
main.startMinio(0xc2080c0ef0, 0x2, 0x2)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/minio/main.go:291 +0x321
main.runMemory(0xc2080ba1e0)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/minio/main.go:218 +0x356
github.com/minio-io/cli.Command.Run(0x836ed0, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8b8470, ...)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/cli/command.go:123 +0x106d
github.com/minio-io/cli.(*App).RunAsSubcommand(0xc208086300, 0xc2080ba000, 0x0, 0x0)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/cli/app.go:274 +0xc53
github.com/minio-io/cli.Command.startApp(0x837150, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x864f10, ...)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/cli/command.go:186 +0x37f
github.com/minio-io/cli.Command.Run(0x837150, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x864f10, ...)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/cli/command.go:50 +0x1517
github.com/minio-io/cli.(*App).Run(0xc208086200, 0xc20800a000, 0x4, 0x4, 0x0, 0x0)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/cli/app.go:181 +0x856
github.com/minio-io/cli.(*App).RunAndExitOnError(0xc208086200)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/cli/app.go:192 +0x57
main.main()
    /home/harsha/work/minio/gopath/src/github.com/minio-io/minio/main.go:376 +0x283

goroutine 17 [syscall, 9 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 6 [IO wait]:
net.(*pollDesc).Wait(0xc208010a70, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010a70, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208010a10, 0x0, 0x7f98f9021118, 0xc2100d0578)
    /usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc208044160, 0x55a23e, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net/http.tcpKeepAliveListener.Accept(0xc208044160, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1976 +0x4c
net/http.(*Server).Serve(0xc208030c60, 0x7f98f9022660, 0xc208044160, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1728 +0x92
net/http.(*Server).ListenAndServe(0xc208030c60, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1718 +0x154
github.com/minio-io/minio/pkg/server/httpserver.start(0xc208030900, 0xc208030960, 0x7f98f9021548, 0xc20800b880, 0xc20802b195, 0x5, 0x0, 0x0, 0x0, 0x0, ...)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/minio/pkg/server/httpserver/httpserver.go:65 +0x2eb
created by github.com/minio-io/minio/pkg/server/httpserver.Start
    /home/harsha/work/minio/gopath/src/github.com/minio-io/minio/pkg/server/httpserver/httpserver.go:42 +0xd3

goroutine 7 [IO wait, 8 minutes]:
net.(*pollDesc).Wait(0xc208010ae0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010ae0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208010a80, 0x0, 0x7f98f9021118, 0xc2080a85a0)
    /usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc208044168, 0x559f44, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net/http.tcpKeepAliveListener.Accept(0xc208044168, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1976 +0x4c
net/http.(*Server).Serve(0xc208030cc0, 0x7f98f9022660, 0xc208044168, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1728 +0x92
net/http.(*Server).ListenAndServe(0xc208030cc0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1718 +0x154
github.com/minio-io/minio/pkg/server/httpserver.start(0xc208030ba0, 0xc208030c00, 0x7f98f90214f8, 0xc2080469b0, 0xc20802b1f0, 0x5, 0x0, 0x827e70, 0x0, 0x827e70, ...)
    /home/harsha/work/minio/gopath/src/github.com/minio-io/minio/pkg/server/httpserver/httpserver.go:65 +0x2eb
created by github.com/minio-io/minio/pkg/server/httpserver.Start
    /home/harsha/work/minio/gopath/src/github.com/minio-io/minio/pkg/server/httpserver/httpserver.go:42 +0xd3

Keeping this issue here so that we fix this eventually.

Clang lacks GAS extended format support

<instantiation>:1:1: error: unknown directive
.altmacro
^
pkg/utils/checksum/crc32c/crc32c_amd64.S:461:1: note: while in macro instantiation
.rept 128-1
^
<instantiation>:1:5: error: invalid register name
crc_%i:
    ^~
<instantiation>:2:1: note: while in macro instantiation
LABEL crc_ %i
^
pkg/utils/checksum/crc32c/crc32c_amd64.S:461:1: note: while in macro instantiation
.rept 128-1
^
<instantiation>:3:1: error: unknown directive
.noaltmacro
^
pkg/utils/checksum/crc32c/crc32c_amd64.S:461:1: note: while in macro instantiation
.rept 128-1
^
<instantiation>:8:1: error: unknown directive
.altmacro
^
pkg/utils/checksum/crc32c/crc32c_amd64.S:461:1: note: while in macro instantiation
.rept 128-1
^
<instantiation>:1:5: error: invalid register name
crc_%i:
    ^~
<instantiation>:9:1: note: while in macro instantiation
LABEL crc_ %i
^
pkg/utils/checksum/crc32c/crc32c_amd64.S:461:1: note: while in macro instantiation
.rept 128-1
^
<instantiation>:10:1: error: unknown directive

[RFE]: Missing altmacro support in integrated assembler - http://llvm.org/bugs/show_bug.cgi?id=18918

multipart: Unstable on rpi2

Tried to copy a 1.4GB fedora 23 ISO to my Rpi2 minio bucket and it just stalled the whole system. Nothing in journalctl.

http://ix.io/mpi

On windows handle not just delimiter also prefixes ending with "/" delimiter

With delimiter "/" on windows for the time being we are manually converting it to "" to make sense on windows.

We need to also translate all the paths for incoming prefixes as well to be translated into windows understandable format.

iodine: potential data race

https://github.com/minio/minio/blob/master/pkg/iodine/iodine.go#L89

Accesses the globalState map without taking a lock.

`make` fails with "... recipe for target 'gomake-all' failed"

Tested on commit - cdeadae
Tested on machine - 'Linux trantor 4.0.5-1-ARCH #1 SMP PREEMPT'

Golang environment

Golang version - go version go1.4.2 linux/amd64
GOROOT=/home/user/go
GOPATH=/home/user/work/go
MINIO_BASE=/home/user/work/go/src/github.com/minio/minio

Sample backtrace

---

PANIC: api_test.go:957: MySuite.TestXMLNameNotInObjectListJson

... Panic: runtime error: invalid memory address or nil pointer dereference (PC=0x470435)

/usr/local/go/src/runtime/panic.go:387
in gopanic
/usr/local/go/src/runtime/panic.go:42
in panicmem
/usr/local/go/src/runtime/sigpanic_unix.go:26
in sigpanic
/home/kp/work/go/src/github.com/minio/minio/pkg/storage/drivers/donut/donut.go:161
in donutDriver.CreateBucket
:2
in donutDriver.CreateBucket
api_test.go:971
in MySuite.TestXMLNameNotInObjectListJson
/usr/local/go/src/reflect/value.go:296
in Value.Call
/home/kp/work/go/src/github.com/minio/minio/Godeps/_workspace/src/github.com/minio/check/check.go:763
in func.003
/home/kp/work/go/src/github.com/minio/minio/Godeps/_workspace/src/github.com/minio/check/check.go:657
in func.001
/usr/local/go/src/runtime/asm_amd64.s:2232
in goexit
2015/06/20 11:48:40 Running API Suite: *filesystem.fsDriver
OOPS: 95 passed, 5 FAILED, 12 PANICKED
--- FAIL: Test (1.87s)

Minor typo with go install instructions on OSX?

In the OSX installation section of INSTALLGO.md, it mentions adding those environment variables to .bashrc.

Shouldn't that be .profile on OSX? Asking because on my OSX 10.9.5 desktop, .bashrc isn't sourced when terminals are started, though .profile is. 😄

Limit total transfer size by ip

Minio returns wrong error message

when bucket/1 already exists:

$ mc cp file localhost:bucket/1

mc: <DEBUG> HTTP/1.1 501 Not Implemented
Connection: close
Content-Length: 207
Accept-Ranges: bytes
Content-Type: application/xml
Date: Mon, 27 Apr 2015 03:13:07 GMT
Server: Minio

Implement auto expiry for objects

Add examples for aws-sdk-go

Implement authorization handler layer with for validating HMAC authorization requests

Minio cli odd behavior after using cobra

Doesn't work

> ./minio --tls --key key.pem --cert cert.pem
flag needs an argument: --key

Usage: 
  minio [flags]

 Available Flags:
  -c, --cert="": cert file path
  -h, --help=false: help for minio
  -a, --http-address=":8080": http address
  -k, --key="": key file path
  -s, --storage-type="file": file,inmemory
  -t, --tls=false: enable tls

Works with following styles

./minio --tls -k key.pem -c cert.pem
2015/01/28 17:29:27 Starting HTTP Server on: :8080

 ./minio --tls --key=key.pem --cert=cert.pem
2015/01/28 17:30:07 Starting HTTP Server on: :8080

This is a odd behavior which didn't exist with Codegangsta, i guess the issue is cobra implementation which doesn't honor certain inputs under long form args

example in README doesn't work

README says we should launch minio as:

./minio mode memory limit 12GB expire 2h

but after I build minio from source, it says mode is not a valid command:

minio: <ERROR> Old Golang runtime version ‘150’ detected., ‘mc’ requires minimum go1.5 or later.
minio: <FATAL> Command not found: ‘mode’

(note the complain about go version, go version gives me go version go1.5 linux/amd64.

From the help strings, looks like one just starts minio with minio server and options are not accepted. Additionally --debug doesn't work, which makes it hard to check why mc doesn't work:

$ minio --debug --address "127.0.0.1:9000" server 
minio: <ERROR> Old Golang runtime version ‘150’ detected., ‘mc’ requires minimum go1.5 or later.
Starting minio server on: http://127.0.0.1:9000, PID: 11154

mc ls http://127.0.0.1:9000
mc: <ERROR> Old Golang runtime version ‘150’ detected., ‘mc’ requires minimum go1.5.1 or later. 
mc: <ERROR> Unable to list target ‘http://127.0.0.1:9000/’. The specified bucket is not valid.

Through trial and error, eventually I figured out that when there's no buckets, list buckets doesn't work:

$ mc mb http://127.0.0.1:9000/foo
mc: <ERROR> Old Golang runtime version ‘150’ detected., ‘mc’ requires minimum go1.5.1 or later. 
Bucket created successfully  ‘http://127.0.0.1:9000/foo’
$ mc ls http://127.0.0.1:9000
mc: <ERROR> Old Golang runtime version ‘150’ detected., ‘mc’ requires minimum go1.5.1 or later. 
[2015-09-17 14:30:43 PDT]     0B foo/

Running minio controller first time generates an error

After a fresh install, if I run minio controller, I get this error because the .minio directory does not yet exist.

$ minio controller
FATA[0000] Failed to generate keys for minio.            error=open /Users/mattbutcher/.minio/users.json: no such file or directory probe={"cause":{"Op":"open","Path":"/Users/mattbutcher/.minio/users.json","Err":2},"trace":[{"line":198,"file":".../src/github.com/minio/minio/pkg/quick/quick.go","func":"quick.config.Save"},{"line":114,"file":".../src/github.com/minio/minio/server-auth-config.go","func":"main.SaveConfig"},{"line":134,"file":".../src/github.com/minio/minio/controller-main.go","func":"main.genAuthFirstTime"},{"line":166,"file":".../src/github.com/minio/minio/controller-main.go","func":"main.firstTimeAuth"},{"line":222,"file":".../src/github.com/minio/minio/controller-main.go","func":"main.controllerMain"}],"sysinfo":{"host.arch":"amd64","host.cpus":"8","host.lang":"go1.5.1","host.name":"ENG001590.local","host.os":"darwin","mem.heap.total":"786kB","mem.heap.used":"334kB","mem.total":"3.1MB","mem.used":"334kB"}}

Implement logging connectors support to mongodb, syslog and regular log file

Implement secondary HTTP server with websocket handlers for a testbed client UI

deleteObject: Implement bulk delete API for deleting multiple objects

http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html

Add request logging

Allow setting a config directory directly

It is not possible to specify a config directory, on Linux minio insists on using .minio in the current user's home directory. This fails when minio is run as user nobody on my system, because that user's home directory is / and minio is not allowed to create /.minio. Apart from abusing the workaround for docker images and setting DOCKERIMAGE=1 so that minio uses the current working directory, I did not find a way to set the dir.

I think it would be very nice to be able to configure the configuration directory, e.g. by setting an environment variable (e.g. MINIO_CONFIG_DIR), and falling back to the current user's home directory if the environment variable is not set.

I do not understand why you're collecting lots of information on the current user in userCurrent(), but only ever access the HomeDir string of the returned struct. This could be nicely cleaned up and even replace the DOCKERIMAGE workaround.

In addition, the function userCurrent even exists twice: https://github.com/minio/minio/blob/master/pkg/fs/config.go#L33 and https://github.com/minio/minio/blob/master/utils.go#L63

Are you interested in a PR that cleans this up a bit?

Implement getBucketACL handler

Is pip a real dependency of the project ?

As I understood, pip is only needed to build mkdocs, so I think it is useless to check it in checkdeps.sh

``boom`` integration for self testing HTTP object server

Compile and test erasure coding for windows

[erasurecode] Clang compiler optimization -O2 leads to decoding failure

To reproduce the issue

master!minio/pkg/storage/erasure *> CC=clang go test -race

----------------------------------------------------------------------
FAIL: cauchy_test.go:31: MySuite.TestCauchyDecode

cauchy_test.go:50:
    c.Fatalf("Recovered data mismatches with original data")
... Error: Recovered data mismatches with original data


----------------------------------------------------------------------
FAIL: vandermonde_test.go:24: MySuite.TestVanderMondeDecode

chunks length: 15
length: 574
vandermonde_test.go:45:
    c.Fatalf("Recovered data mismatches with original data")
... Error: Recovered data mismatches with original data

OOPS: 0 passed, 2 FAILED
--- FAIL: Test (0.01s)
FAIL
exit status 1
FAIL    github.com/minio-io/minio/pkg/storage/erasure   0.024s

Change CC to gcc

master!minio/pkg/storage/erasure *> CC=gcc go test -race
OK: 2 passed
PASS
ok      github.com/minio-io/minio/pkg/storage/erasure   0.013s

A simple modification with CFLAGS: -O0

master!minio/pkg/storage/erasure *> grep CFLAGS *.go
decode.go:// #cgo CFLAGS: -O0
encode.go:// #cgo CFLAGS: -O0
master!minio/pkg/storage/erasure *>

Now with clang tests work fine

master!minio/pkg/storage/erasure *> CC=clang go test -race
OK: 2 passed
PASS
ok      github.com/minio-io/minio/pkg/storage/erasure   0.015s
master!minio/pkg/storage/erasure *>