jacobsa / gcloud Goto Github PK

bucket_tests.go currently contains a workaround for Google-internal bug 19717210 in the GCS XML API. We switched to the XML API because of Google-internal bug 19700548 in URL handling in the JSON API.

So:

• If bug 19700548 in the JSON API is fixed, pull the json-api branch (which switches back to the JSON API) and remove the workaround.
• If bug 19717210 in the XML API is fixed, just remove the workaround.

In order to allow for custom user agents (cf. GoogleCloudPlatform/gcsfuse#26), we need to be able to generate our own HTTP requests. (We do control the http.RoundTripper, but that interface is not supposed to modify the request. Anyway, it's probably better to just continue the trend of cutting dependencies.)

So, we want to stop relying on other people's code for all operations. I believe these are left to do:

• List
• Stat
• Delete
• Update

This is blocked by #3.

google.golang.org/cloud/storage is used for less and less with every change like e309733, motivated by bugs and missing features. It would be a weird wart to have some of its types left over if we use it for nothing in implementation anymore. Plus having our own version of storage.Object etc. let's us document better and narrow to just the fields that we have tested.

So ensure there are no more mentions of that package in the public interface. Also take this opportunity to update the GitHub project page to be a bit less glum about how it is "just a wrapper", which is not really true anymore.

As discovered in GoogleCloudPlatform/gcsfuse#52 and as documented here, composite objects in GCS do not advertise MD5 hash fields, but this package assumes they are always there. Oops.

Plan:

• Turn Object.MD5 into a pointer to an array. (Breaking change, oops.)
• Document when it may be missing.
• Tolerate the field being missing from the JSON in conversions.go.

This library was created before Go 1.13 where error wrapping was introduced. So, it did not wrap errors, which made error handling at the caller side a bit tricky.

For instance, when opening an unknown bucket, the HTTP 404 error was replaced by a custom "Unknown bucket" error.
https://github.com/jacobsa/gcloud/blob/master/gcs/conn.go#L191-L192

		case http.StatusNotFound:
			err = fmt.Errorf("Unknown bucket %q", b.Name())

This error is hard to be handled by the caller. However, if we wrap the internal http error like this:

		case http.StatusNotFound:
			err = fmt.Errorf("Unknown bucket %q: %w", b.Name(), err)

On the user side, such error can be casted back to an 404, and properly handled:

	if errors.As(err, &apiErr) {
		switch apiErr.Code {
		case http.StatusNotFound:
			return syscall.EINVAL
		}
	}

The Go JSON package maps invalid UTF-8 to the Unicode replacement rune (cf. json.Marshal). Other similar packages do the same. So it's not insane to think that somehow we may wind up with an object name containing the rune.

So, make sure the "interesting names" test case in the integration test covers this character. It may or may not already be part of the control character generation code.

Allow the user to set a custom User-Agent header in calls to GCS (as desired in e.g. GoogleCloudPlatform/gcsfuse#26).

Do this by adding a ConnConfig struct with fields for name and project ID, and making NewConn take that. Then add a user agent field.