palantir / conjure-go Goto Github PK

The generated client incorrectly errors when an endpoint returns an absent optional repsonse.

The Conjure specification stipulates that clients should be able to round-trip unknown variants of enums and unions: https://github.com/palantir/conjure/blob/master/docs/spec/wire.md#44-round-trip-of-unknown-variants

This means that, even if a union unmarshals into an "unknown" type, it should be able to re-marshal the unknown union into its original form.

Currently, the generated code does not do this -- although it will unmarshal unknown variants, the serializer code returns an error when an unmarshaled unknown is re-marshaled.

In order to fix this, the generated unions will likely need to store the raw JSON and re-write that when it is marshaled. Not sure if there's a straightforward way to properly support this for YAML, but at the very least we should match the required JSON behavior.

What happened?

Defining an error with a parameter that has the same name as the error cause a break in generated code, example below:

InvalidRids:
  namespace: Example
  code: INVALID_ARGUMENT
  docs: |
    Thrown when an operation is attempted on invalid RIDs
  safe-args:
    invalidRids: set<rid>

This generates code that looks like this:

// NewInvalidRids returns new instance of InvalidRids error.
func NewInvalidRids(invalidRids []rid.ResourceIdentifier) *InvalidRids {
	return &InvalidRids{errorInstanceID: uuid.NewUUID(), invalidRids: invalidRids{InvalidRids: foo}}
}

// InvalidRids is an error type.
//
// Thrown when an operation is attempted on invalid RIDs
type InvalidRids struct {
	errorInstanceID uuid.UUID
	invalidRids
}

What did you want to happen?

Would prefer if the generated code renamed the parameter to avoid the compile break.

Per the conjure spec, an endpoint that returns an optional<binary> type should return an empty stream with a status code of 204. The distinguishes an empty binary from a present binary of zero length.

I attempted to take advantage of this by specifying an endpoint with an optional<binary> type and returning nil in the resource layer if the binary was not found. Rather than return a 204 status code I got a 500 status code. One key issue is that this special casing treats optional<binary> and binary identically so the conjure go infra fails on a nil return type.

The solution is to improve the special casing to convert optional<binary> -> *ReadCloser rather than the standard ReadCloser object and providing handling for the nil -> 204 case.

What happened?

• Define a Conjure object with a map, list and set field
• Marshal the empty Conjure object
• The JSON representation contains the key for each of the fields and the values are null

What did you want to happen?

• The JSON representation should contain a key for each of the field, but the value should be an empty map/list rather than null

Although the spec notes that Conjure clients/servers should coerce null to an empty value, in practice older Java implementations did not do this, and there are implementations like Typescript that do not have a serde layer and cannot properly handle this. There have also been conversations about updating the spec to require this behavior.

Updating the Go implementation to produce empty values rather than null will improve compatibility with other implementations.

Boolean map keys do not work

The current CircleCI config includes a publish step to Bintray for all tagged builds which is no longer supported. In addition, the conjure-go os-arch-bin distribution is not used given godel-conjure-plugin vendors this library directly to invoke the code generation.

All recent tagged builds end up failing due to this publish step:

• https://app.circleci.com/pipelines/github/palantir/conjure-go/911/workflows/c8628742-ad2e-4fb9-a384-35d25f3021ea
• https://app.circleci.com/pipelines/github/palantir/conjure-go/945/workflows/86cc3197-db2e-4eb2-9dd9-c8e3a85dc377

Rather than try to publish to Bintray, we should instead switch this project over to use Github actions to publish to Github releases as done by other OSS projects. The resolvers have already been updated to support this, but the publishing code has not yet been updated.

What happened?

• Define an alias type that resolves to another alias type that resolves to a primitive that defines PLAIN serialization (for example, rid)
• Generate Go code
• The generated code for the alias of the alias does not define a String() method, which means that PLAIN serialization fails (it outputs the struct form of rid instead of the string representation)

What did you want to happen?

• The generated code for the alias of the alias should generate a String() method that converts the receiver to the type it aliases and call String() on that type

When the errorName collides with the fieldName (either safe or unsafe), the fieldName ends up masking the private struct type, and ultimately will fail to compile.

$ go build ./...
    # github.com/tdabboud/testing/pkg/testservice/errors
    pkg/testservice/errors/errors.conjure.go:657:94: unknownSortOrder is not a type

The following conjure-IR will repro the issue

{
  "version": 1,
  "errors": [
    {
      "errorName": {
        "name": "UnknownSortOrder",
        "package": "com.palantir.testservice.errors"
      },
      "docs": "Sort order was unknown.\n",
      "namespace": "TestService",
      "code": "INVALID_ARGUMENT",
      "safeArgs": [
        {
          "fieldName": "unknownSortOrder",
          "type": {
            "type": "primitive",
            "primitive": "STRING"
           }
        }
      ],
      "unsafeArgs": []
    }
  ]
}

The resulting code

// shortened for brevity...

type unknownSortOrder struct {
	UnknownSortOrder string `json:"unknownSortOrder" yaml:"unknownSortOrder,omitempty"`
}

// NewUnknownSortOrder returns new instance of UnknownSortOrder error.
func NewUnknownSortOrder(unknownSortOrder string) *UnknownSortOrder {
	return &UnknownSortOrder{errorInstanceID: uuid.NewUUID(), unknownSortOrder: unknownSortOrder{UnknownSortOrder: unknownSortOrder}}
}

// UnknownSortOrder is an error type.
//
// Sort order was unknown.
type UnknownSortOrder struct {
	errorInstanceID uuid.UUID
	unknownSortOrder
}

As you can see above the fieldName is unknownSortOrder which collides with the private struct unknownSortOrder

Java has a @Safe annotation for marking params as safe which is attached via markers, we should figure out a way to do the same. This requirement will make it tough since this seems to be overfit to java annotations:

List of types that serve as additional metadata for the argument. If the value of the field is a string it MUST be a type name that exists within the Conjure definition.

When processing a valid Conjure definition that contains:

              "type" : "external",
              "external" : {
                "externalReference" : {
                  "name" : "ExampleExternalType",
                  "package" : "com.palantir.types"
                },
                "fallback" : {
                  "type" : "primitive",
                  "primitive" : "STRING"
                }

conjure-go panics with the following output:

panic: runtime error: index out of range

goroutine 1 [running]:
.../vendor/github.com/palantir/conjure-go/conjure/types.NewGoTypeFromExternalType(0xc0005d6140, 0x13, 0xc0005d6180, 0x19, 0xc0005347f0, 0x9, 0xc00057a9a0, 0x0, 0x0, 0x0, ...)
        .../vendor/github.com/palantir/conjure-go/conjure/types/types.go:226 +0x158

Because external types are on a path towards deprecation I don't think we need to add support for specifying external Go types, but at the very least we should try to parse and use the fallback type (and use the any type if a fallback doesn't exist).

If that's too much work, then a stopgap could be to just use the any type for now -- even that would be strictly better (while technically still fulfilling the spec) than panicking on a valid definition as we do now.

What happened?

When I generated Conjure output files for a definition that defined a type in a package that ended in "v2" (com.palantir.test.v2) and an object in a different package that referenced that type, the generated code did not initially compile.

Running the generator again (after the initial non-compiling code was generated) added the proper import statement, after which the code compiled as expected.

The initial behavior is incorrect (running the generator for a valid definition should create valid code), and the fact that running the same Conjure generation operation again generates different code that fixes the issue is unintuitive (the generator is expected to be idempotent).

Example Conjure definition:

types:
  definitions:
    objects:
      ObjectOne:
        package: com.palantir.test.common
        fields:
          objectTwo:
            type: ObjectTwo
      ObjectTwo:
        package: com.palantir.test.v2
        fields:
          name:
            type: string

IR for the definition:

{
  "version" : 1,
  "errors" : [ ],
  "types" : [ {
    "type" : "object",
    "object" : {
      "typeName" : {
        "name" : "ObjectOne",
        "package" : "com.palantir.test.common"
      },
      "fields" : [ {
        "fieldName" : "objectTwo",
        "type" : {
          "type" : "reference",
          "reference" : {
            "name" : "ObjectTwo",
            "package" : "com.palantir.test.v2"
          }
        }
      } ]
    }
  }, {
    "type" : "object",
    "object" : {
      "typeName" : {
        "name" : "ObjectTwo",
        "package" : "com.palantir.test.v2"
      },
      "fields" : [ {
        "fieldName" : "name",
        "type" : {
          "type" : "primitive",
          "primitive" : "STRING"
        }
      } ]
    }
  } ],
  "services" : [ ],
  "extensions" : {}
}

Initial lines of output of first run (at common/structs.conjure.go):

// This file was generated by Conjure and should not be manually edited.

package common

import (
        "github.com/palantir/pkg/safejson"
        "github.com/palantir/pkg/safeyaml"
)

type ObjectOne struct {
        ObjectTwo v2.ObjectTwo `json:"objectTwo"`
}

Initial lines of output of second run (at common/structs.conjure.go):

// This file was generated by Conjure and should not be manually edited.

package common

import (
        v2 "github.com/palantir/conjure-go/v6/cmd/outdir/test/v2"
        "github.com/palantir/pkg/safejson"
        "github.com/palantir/pkg/safeyaml"
)

type ObjectOne struct {
        ObjectTwo v2.ObjectTwo `json:"objectTwo"`
}

What did you want to happen?

The output of the first run should produce compiling code (have the proper imports) and running the command multiple times should not change the generated code.

What happened?

A list or set somefield as URL parameter is translated into a URL string somefield=%5Bvalue1,value2%5D. This format is not supported by all servers, and in our specific case we get an HTTP 400 response.

Example for such a field:

{
    "fieldName": "somefield",
    "type": {
        "type": "optional",
        "optional": {
            "itemType": {
                "type": "set",
                "set": {
                    "itemType": {
                        "type": "external",
                        "external": {
                            "externalReference": {
                                "name": "Somefield",
                                "package": "com.somepackage"
                            },
                            "fallback": {
                                "type": "primitive",
                                "primitive": "STRING"
                            }
                        }
                    }
                }
            }
        }
    },
    "docs": null
}

What did you want to happen?

It seems that a translation like somefield=value1&somefield=value2 would be supported more widely (in our case this is the only format that our server accepts). This seems not to be supported by URL translation via Encode in the net/url package (as used in github.com/palantir/conjure-go-runtime/[email protected]/conjure-go-client/httpclient/client.go) though.

There is bug is in the generation code, specifically the way we inline the private error type rather than naming it. Take the example of the following definition:

types:
  definitions:
    errors:
      MyError:
        code: CUSTOM_SERVER
        namespace: MyNamespace
        safe-args:
          code: integer
          message: string

The above definition would generate roughly the following:

type myError {
  Code int
  Message string
}

type MyError struct {
  errorInstanceID uuid.UUID
  myError
  cause error
  stack werror.StackTrace
}

The collision occurs on the generated SafeParams method.

// safeParams returns a set of named safe parameters detailing this particular error instance.
func (e *MyError) safeParams() map[string]interface{} {
	return map[string]interface{}{"code": e.Code, "message": e.Message, "errorInstanceId": e.errorInstanceID}
}

Because Code and Message are also methods on MyError, this actually returns functions rather than the values stored on myError.

Think the fix is just to name the internal error type on the exported error type - I don't see any reason this needs to be inlined.

Problem

Today, Conjure-generated union types in Go are harder to handle than their counterparts, primarily due to the lack of generics. If I want to return a value with type T determined by visiting a union type, I currently have to either (1) define a stateful struct which implements the existing Visitor or VisitorWithContext interface, or (2) leverage closures by instantiating a variable outside of the AcceptFuncs method and setting it in the Visit methods provided to AcceptFuncs. These both have disadvantages; the former approach makes reuse of the stateful struct a recipe for concurrency bugs, and the latter forces the use of closures, which can degrade readability.

Proposed Solution

The advent of generics in go1.18 lets us generate type-parametrized code to make usage much cleaner. Given the following union type definition:

MyUnion:
  union:
    stringVal: string
    boolVal: bool

I propose generating this code:

type MyUnionWithT[T any] MyUnion

func (u myUnionWithT) Accept(ctx context.Context, v myUnionVisitorWithT[T]) (T, error) {
  switch u.typ {
    case default:
      if u.typ == "" { return nil, fmt.Errorf("unknown")}
      return v.VisitUnknown(ctx, u.typ)
    case "stringVal":
      return v.VisitStringVal(ctx, *u.stringVal)
    case "boolVal":
      return v.VisitBoolVal(ctx, *u.boolVal)
  }
}

type MyUnionVisitorWithT[T any] interface {
	VisitStringVal(ctx context.Context, v string) (T, error)
	VisitBoolVal(ctx context.Context, v bool) (T, error)
	VisitUnknown(ctx context.Context, typ string) (T, error)
}

Where example usage looks like:

func Stringify(ctx, context.Context, myUnion MyUnion) (string, error) {
  var stringifyingVisitor MyUnionVisitorWithT[string] = NewStringifyingVisitor()
  return MyUnionWithT[string](myUnion).Accept(ctx, stringifyingVisitor)
}

See: palantir/conjure-go-runtime#33

There are currently a number of places we incorrectly handle alias types of optionals.

• Encoding methods: Today we would attempt to create encoding methods on whose receiver is a pointer type: this is not legal in go.
• Client responses: If an endpoint returns an optional, we incorrectly require the response to be non-nil.

To do this correctly, I think we need to generate a struct with a pointer field instead of an aliased pointer type. Example:

// old
type OptionalDateAlias *datetime.DateTime
// new
type OptionalDateAlias struct {
	Value *datetime.DateTime
}
// unchanged
type ConcreteDateAlias datetime.DateTime

We will need to generate Marshal/Unmarshal methods which read/write from the value field. Clients will receive this container type and have to check resp.Value != nil instead of resp != nil. This has the added benefit that generated clients will return a concrete struct like they do with non-optional aliases, so we do not need a special case in the servicewriter.

Because this is more a result of the optional than the alias, I think we should leave non-optional aliases as-is for ease of use. The asymmetry should be palatable or even expected given the expectation that the response can be empty.

We currently generate client interfaces with and without auth token arguments. In cases where we use short-lived tokens, we currently must provide a token to each request.

Instead, we should generate a second implementation for the *WithAuth interface which can be constructed with a httpclient.TokenProvider.

#45 illustrates a test case that should error but currently panics because the generated code dereferences a pointer it assumes is present but may not be at line 97:

What happened?

When running conjure generation on a definition that produces services and structs in a project that is set up as a Go module, the imports in the generated Go code used import paths that started with _/Volumes/..., which are invalid.

What did you want to happen?

The import paths should be correct (should be the module path + relative path from module root to the package)

What did you do?

1. Publish the following Conjure definition (which will be referred to as common-service.conjure.json):

types:
  definitions:
    default-package: com.palantir.test
    errors:
      EntityNotFound:
        namespace: V6
        code: NOT_FOUND
        docs: Thrown when the provided entity ID is not found
        safe-args:
          entityId: string

2. Create module github.com/palantir/test-library-module that generates code using the common-service.conjure.json definition and references the generated code
3. Create module github.com/palantir/test-service-module that generates code using the common-service.conjure.json definition and references the generated code AND that has github.com/palantir/test-library-module as a module dependency in go.mod
4. Run the github.com/palantir/test-service-module

What happened?

The program crashes on startup with the following error:

panic: ErrorName V6:EntityNotFound already registered as v6.EntityNotFound

goroutine 1 [running]:
github.com/palantir/conjure-go-runtime/v2/conjure-go-contract/errors.RegisterErrorType({0x10237d60e, 0x11}, {0x10272f050?, 0x1021e2960})

What did you want to happen?

Program should start normally

What happened?

An alias definition for an external type that uses any as the fallback will generate serde methods for the type. Since the type is any, which maps to an interface{}, methods cannot be added to the type and thus results in a compile time error.

Here is a short repro of the issue:

Conjure definition

types:
  imports:
    ExternalLong:
      base-type: any
      external:
        java: java.lang.Long
  definitions:
    default-package: api
    objects:
      LongAlias:
        alias: ExternalLong

When compiled with conjure-go >= v6.6.0 the following will be produced:

// This file was generated by Conjure and should not be manually edited.

package api

import (
	safejson "github.com/palantir/pkg/safejson"
	safeyaml "github.com/palantir/pkg/safeyaml"
)

type LongAlias interface{}

func (a LongAlias) MarshalJSON() ([]byte, error) {
	return safejson.Marshal(interface{}(a))
}

func (a *LongAlias) UnmarshalJSON(data []byte) error {
	var rawLongAlias interface{}
	if err := safejson.Unmarshal(data, &rawLongAlias); err != nil {
		return err
	}
	*a = LongAlias(rawLongAlias)
	return nil
}

func (a LongAlias) MarshalYAML() (interface{}, error) {
	jsonBytes, err := safejson.Marshal(a)
	if err != nil {
		return nil, err
	}
	return safeyaml.JSONtoYAMLMapSlice(jsonBytes)
}

func (a *LongAlias) UnmarshalYAML(unmarshal func(interface{}) error) error {
	jsonBytes, err := safeyaml.UnmarshalerToJSONBytes(unmarshal)
	if err != nil {
		return err
	}
	return safejson.Unmarshal(jsonBytes, *&a)
}

What did you want to happen?

Since the fallback type is an interface, no methods should be serde methods should be generated, so we would result in just the type definition.

// This file was generated by Conjure and should not be manually edited.

package api

type LongAlias interface{}

The behavior above matches what was generated prior to conjure-go v6.6.0

What happened?

golint enforces initialisms, while conjure does not. That results in generated code which does not comply with golint.

Generated code

GetServiceBackupDirectoryV2(req.Context(), backupId, nodeId, serviceId, mount)

Here backupId should be backupID, nodeId should be nodeID, and serviceId should be serviceID

What did you want to happen?

Generated go code should comply to go's style guidelines by adjusting capitalization as required.

The Go version for this library is managed by Excavator via the .palantir/go-version file, but the CircleCI config does not respect this version and must be manually updated. As a result, the versions are lagging behind what the expected managed version is and any follow on excavators that update dependencies and manage the Go version end up failing due to the misaligned Go version.

What happened?

The integer conjure type refers to a 32-bit signed integer. Go generates the int type, which is a platform/arch dependent keyword. When run on a 64 bit system, it's usually equivalent to int64 (see https://tour.golang.org/basics/11).

See the example yaml and the generated go code

What did you want to happen?

Generate an int32 instead.

@ungureanuvladvictor had an issue that was hard to debug where a generated client was receiving 404s because a path parameter was provided as the empty string to a path like /service/api/path/{param} so it rendered as /service/api/path/

Possible solution: Add a precondition before sending a request that all path parameters are non-empty.

@iamdanfox @ferozco does Java deal with this? Any edge cases where empty would be correct?

cc @nmiyake

We have been writing repetitive code for sending protobuf across the wire:

func() io.ReadCloser { return ioutil.NopCloser(bytes.NewReader(body)) 

API recommended by @bmoylan:

BodyProviderFromBytes(body []byte) func() io.ReadCloser

What happened?

I added a java external import to my conjure definitions, including base-type: string (https://github.com/palantir/conjure/blob/master/docs/spec/conjure_definitions.md#externaltypedefinition). The generated go server files attempt to cast to string, but don't actually compile.

Generated code:

	backupIdInternal := req.URL.Query().Get("backupId")
	backupId = string(backupIdInternal)

The backupId hasn't been defined before in this context, so it needs an assignment operator :=.

What did you want to happen?

Have external types with a base-type defined actually generate compiling code

What happened?

Server code generation breaks with the following error if you use a list or set argument for a query param-type:

Error: failed to generate Conjure: failed to generate AST for service FooService: can not assign string expression to list type

To reproduce, use this conjure yaml:

types:
  definitions:
    objects:
services:
  FooService:
    name: "Foo Service"
    package: com.palantir.foo
    base-path: /foo
    endpoints:
      bar:
        http: GET /bar
        args:
          listOfStrings:
            param-type: query
            type: list<string> # this breaks

Which produces the following conjure IR:

{
  "version" : 1,
  "errors" : [ ],
  "types" : [ ],
  "services" : [ {
    "serviceName" : {
      "name" : "FooService",
      "package" : "com.palantir.foo"
    },
    "endpoints" : [ {
      "endpointName" : "bar",
      "httpMethod" : "GET",
      "httpPath" : "/foo/bar",
      "args" : [ {
        "argName" : "listOfStrings",
        "type" : {
          "type" : "list",
          "list" : {
            "itemType" : {
              "type" : "primitive",
              "primitive" : "STRING"
            }
          }
        },
        "paramType" : {
          "type" : "query",
          "query" : {
            "paramId" : "listOfStrings"
          }
        },
        "markers" : [ ]
      } ],
      "markers" : [ ]
    } ]
  } ]
}

Now reproduce the failure by running conjure-go directly with and without server code generation:

# works fine without generating the server code
$ go run main.go --output /tmp/ /tmp/conjure.json

# breaks if you generate the server code
$ go run main.go --output /tmp/ --server /tmp/conjure.json
Error: failed to generate Conjure: failed to generate AST for service FooService: can not assign string expression to list type
exit status 1

What did you want to happen?

Server code generation should work in this case.

Go version: 1.11.5

Conjure definition:

getImport:
    http: GET /{importId}
    args:
      importId:
        type: imports.ImportId
        param-type: path
    returns: imports.Import

Generated code (strconv.Quote returns a single argument):

var importId imports.ImportId
if importIdQuote, err := strconv.Quote(importIdStr); err != nil {
	return werror.Wrap(err, "failed to quote argument", werror.SafeParam("argName", "importId"), werror.SafeParam("argType", "imports.ImportId"))
} else {
	if err := safejson.Unmarshal([]byte(importIdQuote), &importId); err != nil {
		return werror.Wrap(err, "failed to unmarshal argument", werror.SafeParam("argName", "importId"), werror.SafeParam("argType", "imports.ImportId"))
	}
}

What happened?

• Create a Conjure definition that sends a double value using PLAIN serialization (for example, specify an endpoint with a query parameter of type double) and send a double with the value of negative or positive infinity
• The value is sent as +Inf or -Inf

What did you want to happen?

Per https://github.com/palantir/conjure/blob/master/docs/spec/wire.md#6-plain-format, the value should be sent as +Infinity/-Infinity

What happened?

A panic was encountered after upgrading from v6.5.0 to v.6.6.0, which includes the code-gen change to jennifer from goastwriter.
Of note, this appears when an API uses an external type in either path or query parameters.

panic: unrecognized type BackupId (string)

goroutine 1 [running]:
github.com/palantir/conjure-go/v6/conjure.astForDecodeHTTPParamInternal(0xc00038fe60, {0xc00023a588, 0x8}, {0x1465920, 0xc00041e080}, {0xc00023a588, 0x8}, {0x145a6f0, 0xc000389a58}, 0x0)
        /Volumes/git/go/src/github.com/palantir/conjure-go/conjure/serverwriter.go:387 +0x980d
github.com/palantir/conjure-go/v6/conjure.astForDecodeHTTPParam(...)
        /Volumes/git/go/src/github.com/palantir/conjure-go/conjure/serverwriter.go:289
github.com/palantir/conjure-go/v6/conjure.astForHandlerMethodQueryParam(0x0, 0xc000028d80)
        /Volumes/git/go/src/github.com/palantir/conjure-go/conjure/serverwriter.go:246 +0x1247
github.com/palantir/conjure-go/v6/conjure.astForHandlerMethodQueryParams(0x0, {0xc00034e378, 0x1, 0x0})
        /Volumes/git/go/src/github.com/palantir/conjure-go/conjure/serverwriter.go:234 +0x38
github.com/palantir/conjure-go/v6/conjure.astForHandlerMethodBody(0x100e367, {0xc0004e9020, 0xb}, 0xc0003a4bb0)
        /Volumes/git/go/src/github.com/palantir/conjure-go/conjure/serverwriter.go:149 +0x115
github.com/palantir/conjure-go/v6/conjure.astForHandlerMethods.func1(0xc00005ad80)
        /Volumes/git/go/src/github.com/palantir/conjure-go/conjure/serverwriter.go:137 +0x2c
github.com/dave/jennifer/jen.(*Statement).BlockFunc(0xc0002beee8, 0xc0005cb6a8)
        /Volumes/git/go/src/github.com/palantir/conjure-go/vendor/github.com/dave/jennifer/jen/generated.go:234 +0x89
github.com/palantir/conjure-go/v6/conjure.astForHandlerMethods(0xc00033e1e0)
        /Volumes/git/go/src/github.com/palantir/conjure-go/conjure/serverwriter.go:136 +0x1265
github.com/palantir/conjure-go/v6/conjure.writeServerType(0xc000285980, 0xc00033e1e0)
        /Volumes/git/go/src/github.com/palantir/conjure-go/conjure/serverwriter.go:54 +0x597
github.com/palantir/conjure-go/v6/conjure.GenerateOutputFiles({0x1, {0x1edea50, 0x0, 0x0}, {0xc00019a540, 0x4, 0x4}, {0xc0001f4400, 0x1, 0x4}, ...}, ...)
        /Volumes/git/go/src/github.com/palantir/conjure-go/conjure/conjure.go:95 +0x1005
github.com/palantir/conjure-go/v6/conjure.Generate({0x1, {0x1edea50, 0x0, 0x0}, {0xc00019a540, 0x4, 0x4}, {0xc0001f4400, 0x1, 0x4}, ...}, ...)
        /Volumes/git/go/src/github.com/palantir/conjure-go/conjure/conjure.go:29 +0x65
main.run({0x13d4413, 0x480}, {0x13b2b14, 0x6})
        /Volumes/git/go/src/github.com/palantir/conjure-go/integration_test/testgenerated/generate.go:61 +0x14f
main.main()
        /Volumes/git/go/src/github.com/palantir/conjure-go/integration_test/testgenerated/generate.go:33 +0xa5
exit status 2

What did you want to happen?

Code should be generated as it was prior to v6.6.0

In Java, they set a Deprecation header and clients log a warning when they execute a deprecated endpoint: palantir/conjure#617 (comment)

We should do the same for go

What happened?

Phonograph2 defines an Conjure endpoint with an optional header arg:

args:
      onBehalfOf:
        type: optional<uuid>
        param-type: header
        param-id: On-Behalf-Of
        markers:
          - Safe
        docs: |
          The Multipass user ID of the user on whose behalf the event should be posted.
          The `phonograph2:write-table-edits-on-behalf-of` operation is required on the writeback dataset RID
          associated with the given table if this field is specified.

conjure-go ends up creating clients that just pass nil values in the header if a nil value is passed in the optional parameter:

func (c *tableStorageServiceClient) PostEvent(ctx context.Context, authHeader bearertoken.Token, onBehalfOfArg *uuid.UUID, tableRidArg TableRid, tableEditedEventPostRequestArg TableEditedEventPostRequest) error {
var requestParams []httpclient.RequestParam
requestParams = append(requestParams, httpclient.WithRPCMethodName("PostEvent"))
requestParams = append(requestParams, httpclient.WithRequestMethod("POST"))
requestParams = append(requestParams, httpclient.WithHeader("Authorization", fmt.Sprint("Bearer ", authHeader)))
requestParams = append(requestParams, httpclient.WithPathf("/storage/edits/tables/%s/events", url.PathEscape(fmt.Sprint(tableRidArg))))
requestParams = append(requestParams, httpclient.WithJSONRequest(tableEditedEventPostRequestArg))
requestParams = append(requestParams, httpclient.WithHeader("On-Behalf-Of", fmt.Sprint(onBehalfOfArg)))

What did you want to happen?

A nil should ignore the header field instead of passing along a nil value.

Unlike with other return types, conjure-go special cases binary endpoints to use the ReadCloser stream type rather than the default []byte. Specifically, endpoints that return binary use ReadCloser while optional<binary> endpoints will return *ReadCloser after the resolution of #192. In contrast endpoints with either a set<binary> or list<binary> type use an unstreamable [][]byte go type. Ultimately this might be the right choice but we should compare the implementation against conjure-java, validate that this type works, and add unit/integration testing to guarantee this behavior.

To send a slice of bytes across the wire we need to convert the bytes to a func () io.ReadCloser, which is pretty cumbersome. I would like to propose we expose an API where we can simply provide []byte and conjure-go will do the rest.
Before:

c.PutEndpoint(ctx, func() io.ReadCloser { return ioutil.NopCloser(bytes.NewReader(bodyBytes)) })

After:

c.PutEndpoint(ctx, bodyBytes)

Double values NaN, Infinity, and -Infinity do not work.

What happened?

Generate Conjure using the following definition:

types:
  definitions:
    default-package: com.palantir.apollo.catalog.api.module
    objects:
      VariableSpec:
        union:
          string: StringVariable

      StringVariable:
        fields:
          defaultValue:
            type: optional<string>
            safety: unsafe
          description:
            type: string
            safety: unsafe

(from https://github.pb/foundry/apollo-catalog/blob/develop/apollo-catalog-api/src/main/conjure/ModuleService.yml#L107-L118)

This generates the following Go:

type VariableSpec struct {
	typ    string
	string *StringVariable
}

type variableSpecDeserializer struct {
	Type   string          `json:"type"`
	String *StringVariable `json:"string"`
}

func (u *variableSpecDeserializer) toStruct() VariableSpec {
	return VariableSpec{typ: u.Type, string: u.String}
}

func (u *VariableSpec) toSerializer() (interface{}, error) {
	switch u.typ {
	default:
		return nil, fmt.Errorf("unknown type %s", u.typ)
	case "string":
		return struct {
			Type   string         `json:"type"`
			String StringVariable `json:"string"`
		}{Type: "string", String: *u.string}, nil
	}
}

Note the String: *u.string at line 267 (https://github.pb/deployability/apollo-cli/blob/c66d938a2da8d1ec054f9d8a84ab21c0f08f5795/internal/generated/conjure/apollo_catalog_api/apollo/catalog/api/module/unions.conjure.go#L267) -- because this is a dereference of u.string, if StringVariable is nil, this will panic.

Although it's not possible to construct a nil value via constructors, it is via JSON unmarshal. Thus, the following code panics:

	var spec module.VariableSpec

	err := json.Unmarshal([]byte(`{"type": "string"}`), &spec)
	require.NoError(t, err)

	_, err = spec.MarshalJSON()
	require.NoError(t, err)

What did you want to happen?

Either:

1. If StringValue cannot be nil in the Union type, then the json.Unmarshal call should return an error
2. Otherwise, the MarshalJSON code should produce valid JSON or produce an error (but not panic)

From my reading of the code, it seems like (1) is more correct -- in the union type definition, the StringVariable definition is not optional, so it being nil should not be allowed (the expected unmarshal should produce a VariableSpec with a non-nil StringVariable that has nil/empty content

What happened?

A common use case is to look at a returned error and determine whether it's a known type and handle accordingly. E.g.:

err := doSomething()
if err != nil {
    if e, ok := werror.RootCause(err).(errors.Error); ok {
        if e.Name() == "Namespace:FooBarConjureError" {
            // do something
        }
    }
}

That's not great because (a) it's a lot of boilerplate and (b) it requires copy-pasting the error name from the generated conjure since it's not exported as a const.

What did you want to happen?

Instead, we should generate a small helper function:

func IsFooBarConjureError(err error) bool {
    if err == nil {
        return false
    }
    if e, ok := werror.RootCause(err).(errors.Error); ok {
        if e.Name() == "Namespace:FooBarConjureError" {
           return true
        }
    }
    return false
}

This would turn the above code into:

err := doSomething()
if err != nil && api.IsFooBarConjureError(err) {
    // do something
}

What happened?

An internal product defines a ResourceIdentifier type alias like so:

// The RID for a Table.
type TableRid rid.ResourceIdentifier

This TableRid type is used as a service query parameter:

postEvent:
    http: POST /edits/tables/{tableRid}/events
    args:
        tableRid:
            docs: The identifier for the Table being edited.
            type: TableRid
            markers:
                - Safe

which generates this interface function:

PostEvent(ctx context.Context, authHeader bearertoken.Token, onBehalfOfArg *uuid.UUID, tableRidArg TableRid, tableEditedEventPostRequestArg TableEditedEventPostRequest) error

The client code generated uses fmt.Sprint on the TableRid object to path escape the parameter:

requestParams = append(requestParams, httpclient.WithPathf("/storage/edits/tables/%s/events", url.PathEscape(fmt.Sprint(tableRidArg))))

fmt.Sprint does not call TableRid.String() because this does not exist on the alias. A generic string is used and the rid string ends up space delimited instead of what ResourceIdentifier.String() does.

Once the parameter is passed incorrectly the conjure-java based server returns errors because TableRid does not convert to a properly formatted rid string.

What did you want to happen?

Aliased ResourceIdentifiers should carry forward the ResourceIdentifier.String() func.

What happened?

If you have aliases in your conjure definition and generated go code, then remove all aliases and regenerate the go code into the same directory, the old aliases.conjure.go file and alias definitions still exist when they should not.

$ cat /tmp/defs.yml
types:
  definitions:
    default-package: com.palantir
    objects:
      SomeObject:
        fields:
          someField: string
      SomeAlias:
  1 types:
        alias: string
$ ~/Downloads/conjure-4.6.1/bin/conjure compile /tmp/defs.yml /tmp/defs.json
$ go run main.go /tmp/defs.json --output /tmp/repro
$ cat /tmp/repro/com/palantir/aliases.conjure.go
// This file was generated by Conjure and should not be manually edited.

package palantir

type SomeAlias string
$ vi /tmp/defs.yml # remove alias
$ cat /tmp/defs.yml
types:
  definitions:
    default-package: com.palantir
    objects:
      SomeObject:
        fields:
          someField: string
$ ~/Downloads/conjure-4.6.1/bin/conjure compile /tmp/defs.yml /tmp/defs.json
$ go run main.go /tmp/defs.json --output /tmp/repro
$ cat /tmp/repro/com/palantir/aliases.conjure.go # THIS IS THE BUG -- these defs should no longer exist
// This file was generated by Conjure and should not be manually edited.

package palantir

type SomeAlias string

What did you want to happen?

conjure-go should delete the aliases.conjure.go if there are no aliases.

NOTE: This may apply to other files that conjure-go writes as well.

What happened?

If an API definition has cyclic package references, the code generated by Conjure Go won't build:

• MyService uses InfoType in package my.api
• InfoType references InfoSubType in package my.api.subpackage
• InfoSubType references InfoEnum in package my.api

Please note that the API definition is not under our control. It is corresponding to an existing system written in a language that supports cyclic references (Java, I assume).

Compiling the generated code fails with:

import cycle not allowed
package ./src/myprogram
	imports sample/my/api/service
	imports sample/my/api
	imports sample/my/api/subpackage
	imports sample/my/api

Full API definition:

{
    "version": 1,
    "errors": [],
    "types": [
        {
            "type": "object",
            "object": {
                "typeName": {
                    "name": "InfoType",
                    "package": "my.api"
                },
                "fields": [
                    {
                        "fieldName": "someField",
                        "type": {
                            "type": "reference",
                            "reference": {
                                "name": "InfoSubType",
                                "package": "my.api.subpackage"
                            }
                        },
                        "docs": "Some detailed information.\n"
                    }
                ],
                "docs": "Some information.\n"
            }
        },
        {
            "type": "object",
            "object": {
                "typeName": {
                    "name": "InfoSubType",
                    "package": "my.api.subpackage"
                },
                "fields": [
                    {
                        "fieldName": "someSubField",
                        "type": {
                            "type": "reference",
                            "reference": {
                                "name": "InfoEnum",
                                "package": "my.api"
                            }
                        },
                        "docs": "Make your choice.\n"
                    }
                ],
                "docs": "Some detailed information.\n"
            }
        },
        {
            "type": "enum",
            "enum": {
                "typeName": {
                    "name": "InfoEnum",
                    "package": "my.api"
                },
                "values": [
                    {
                        "value": "GOOD",
                        "docs": null
                    },
                    {
                        "value": "BAD",
                        "docs": null
                    }
                ],
                "docs": "Either good or bad.\n"
            }
        }
    ],
    "services": [
        {
            "serviceName": {
                "name": "MyService",
                "package": "my.api.service"
            },
            "endpoints": [
                {
                    "endpointName": "getInfo",
                    "httpMethod": "GET",
                    "httpPath": "/info",
                    "auth": {
                        "type": "header",
                        "header": {}
                    },
                    "args": [],
                    "returns": {
                        "type": "reference",
                        "reference": {
                            "name": "InfoType",
                            "package": "my.api"
                        }
                    },
                    "docs": "Some information.\n",
                    "deprecated": null,
                    "markers": []
                }
            ],
            "docs": "Returns information.\n"
        }
    ]
}

What did you want to happen?

Conjure Go should at least detect such an issue and fail. But what we would actually need is that it generates extra packages that transform the import cycles into a linear structure (in the above example generate separate packages for InfoType and InfoEnum).

What happened?

Define a conjure errors type like this:

types:
  definitions:
    default-package: com.foo.bar
    errors:
      CollisionError:
        namespace: MyNameSpace
        code: PERMISSION_DENIED
        safe-args:
          uuid: optional<uuid>

This generates code like this:

// NewCollisionError returns new instance of CollisionError error.
func NewCollisionError(uuid *uuid.UUID) *CollisionError {
	return &CollisionError{errorInstanceID: uuid.NewUUID(), collisionError: collisionError{Uuid: uuid}}
}

Parameter name uuid collides with the package uuid and results in the parameter's function being called instead of the uuid package's NewUUID function.

Compile errors result.

What did you want to happen?

The parameter name should be munged to something like uuidArg or something (to avoid collision).

What happened?

• Right now clients will happily accept responses that will missing required fields
• This is in violation of https://github.com/palantir/conjure/blob/master/docs/spec/wire.md in which we should error if we are missing a required field and allow unknown json keys to exist

What did you want to happen?

• Clients error if receiving a response missing required keys