kubernetes / gengo

gengo library for code generation.

Kubernetes (K8s)

---

Kubernetes, also known as K8s, is an open source system for managing containerized applications across multiple hosts. It provides basic mechanisms for the deployment, maintenance, and scaling of applications.

Kubernetes builds upon a decade and a half of experience at Google running production workloads at scale using a system called Borg, combined with best-of-breed ideas and practices from the community.

Kubernetes is hosted by the Cloud Native Computing Foundation (CNCF). If your company wants to help shape the evolution of technologies that are container-packaged, dynamically scheduled, and microservices-oriented, consider joining the CNCF. For details about who's involved and how Kubernetes plays a role, read the CNCF announcement.

---

To start using K8s

See our documentation on kubernetes.io.

Take a free course on Scalable Microservices with Kubernetes.

To use Kubernetes code as a library in other applications, see the list of published components. Use of the k8s.io/kubernetes module or k8s.io/kubernetes/... packages as libraries is not supported.

To start developing K8s

The community repository hosts all information about building Kubernetes from source, how to contribute code and documentation, who to contact about what, etc.

If you want to build Kubernetes right away there are two options:

You have a working Go environment.

git clone https://github.com/kubernetes/kubernetes
cd kubernetes
make

You have a working Docker environment.

git clone https://github.com/kubernetes/kubernetes
cd kubernetes
make quick-release

For the full story, head over to the developer's documentation.

Support

If you need support, start with the troubleshooting guide, and work your way through the process that we've outlined.

That said, if you have questions, reach out to us one way or another.

Community Meetings

The Calendar has the list of all the meetings in the Kubernetes community in a single location.

Adopters

The User Case Studies website has real-world use cases of organizations across industries that are deploying/migrating to Kubernetes.

Governance

Kubernetes project is governed by a framework of principles, values, policies and processes to help our community and constituents towards our shared goals.

The Kubernetes Community is the launching point for learning about how we organize ourselves.

The Kubernetes Steering community repo is used by the Kubernetes Steering Committee, which oversees governance of the Kubernetes project.

Roadmap

The Kubernetes Enhancements repo provides information about Kubernetes releases, as well as feature tracking and backlogs.

gengo's Issues

Unclear when strings are package names vs paths

I started some hacking this weekend in prep for other work, and I realized that it was unclear when string fields are package names ("foo") vs paths ("github.com/me/foo"). So I started making a strong type for them. I an not done with the work yet, but one thing that's clear is that at least some code (protobuf, I am looking at you) abuses types like Name to mean whatever it wants.

I am filing this for discussion. #13 is WIP

deepcopy-gen generates invalid code if package name is a reserved golang word

I have a CRD which I want to make use of the protobuf struct, which I build from types.proto: (pay attention to import "google/protobuf/struct.proto"; and google.protobuf.Struct settings)

syntax = 'proto3';

// ...
import "gogoproto/gogo.proto";
import "google/protobuf/struct.proto";
import "k8s.io/apimachinery/pkg/apis/meta/v1/generated.proto";
// ...

option go_package = "v1";

// SomeSpec ...
message SomeSpec {
  // Somefield ...
  // +optional
  string somefield = 1;
  // Settings ...
  // +optional
  google.protobuf.Struct settings = 2;
}

// ...

which generation with protoc results to types.pb.go: (pay attention to _struct "github.com/golang/protobuf/ptypes/struct")

// Code generated by protoc-gen-gogo. DO NOT EDIT.
// source: types.proto

package v1

import (
	// ...
	_ "github.com/gogo/protobuf/gogoproto"
	proto "github.com/gogo/protobuf/proto"
	// ...
	_struct "github.com/golang/protobuf/ptypes/struct"
	// ...
	v11 "k8s.io/api/core/v1"
	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	// ...
)

// ...

which generation with generate-groups.sh all ... results to zz_generated.deepcopy.go: (pay attention to struct "github.com/golang/protobuf/ptypes/struct"

// +build !ignore_autogenerated

// Code generated by deepcopy-gen. DO NOT EDIT.

package v1

import (
	struct "github.com/golang/protobuf/ptypes/struct"
	corev1 "k8s.io/api/core/v1"
	runtime "k8s.io/apimachinery/pkg/runtime"
	net "net"
)

// ...

// and some usage of struct.Struct in the generated code
// ...
in, out := &in.Settings, &out.Settings
*out = new(struct.Struct)
(*in).DeepCopyInto(*out)
// ...

which is resulting into following error:

unable to format file ".../v1/zz_generated.deepcopy.go" (8:2: expected 'STRING', found 'struct' (and 10 more errors)).

And is obviously also not usable afterwards

Can you also add an underline _ to reserved golang words like gogoproto does with _struct?

Tracking Issue for improvements for built-in declarative defaults generator

Originally PR linked here: #192

Future Improvements:

• Avoid reflect.ValueOf and precompute the zero values instead. #192 (comment)
• Unmarshal once for structs/arrays/maps and use shallow/deep copy instead of unmarshalling every time #192 (comment)
• Create a defaulting function to call instead of generating the code every time a defaulter is needed #192 (comment)
• Decompose writeDefaulter into separate functions #192 (comment)
• Remove PruneDefaults from openapi test https://github.com/kubernetes/kubernetes/blob/dee67230af5ee72f616c46856bf5c700d0c9b6b9/pkg/generated/openapi/openapi_test.go#L34

We should target these for the beginning of 1.21

Use x/tools/go/packages

To better support modules, we should plan to switch to the x/tools/go/package library to scan Go packages.

This issue is a follow up from kubernetes/code-generator#69.

master doesn't work with release-1.8 kubernetes/code-generator

kubernetes/code-generator/cmd/client-go is missing the --go-header and -o flags. @stts told me the issue was with the version of gengo I was using.

We should have a branch for kubernetes/gengo that matches the code vendored by kubernetes/kubernetes at the same branch.

To get the right version of gengo I need to pin it to a commit in my glide.yaml. I should be able to pin all kubernetes/* repos to a release-1.* and they should work together.

Deepcopy broken for deeper nested pointers of builtins as slice+map value type

For []**string, []***int, map[string]**bool etc. we generate deepcopy code which calls DeepCopyInto, which is not defined for those types.

xref #107 (comment)

Codegen attempts to deepcopy time.Time

I have a type that contains times (time.Time).

type KayTime struct {
	// Type is the type of host repair action.
	Type KayTime `json:"type"`

	StartTime time.Time `json:"startTime"`

	ExpireTime time.Time `json:"expireTime"`
}

When I put the follow flag:
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

the code generator tries to do the following which is undefined for time.Time since it's underlying type is an integer:

in.StartTime.DeepCopyInto(&out.StartTime)
in.EndTime.DeepCopyInto(&out.EndTime)

Generating non-go code

Hi, I'm looking into using gengo for generating ".adoc" files from go code.

Is it a valid use case? I see everything is built around code.

Support for JSONMap in CRDs

I have a CRD that I'm working on that is a wrapper around other objects, and so has the form

type UnstructuredJSON map[string]interface{}

type MyMessage struct {
  Objects []UnstructuredJSON
}

map[string]interface{} doesn't translate well with DeepCopy:

*out = make([]UnstructuredJSON, len(*in))
for i := range *in {
	if (*in)[i] != nil {
		in, out := &(*in)[i], &(*out)[i]
		*out = make(UnstructuredJSON, len(*in))
		for key, val := range *in {
			if val == nil {
				(*out)[key] = nil
			} else {
				(*out)[key] = val.DeepCopyinterface{}()
			}
		}
	}
}

(*out)[key] = val.DeepCopyinterface{}() is an error

I think some sort of UnstructuredJSON (called JSONMap in kubernetes Runtime) could be supported -- it's quite constrained on what can be in those interface{} values, but it would probably require custom copy logic.

What do you think?

Context: http://github.com/GoogleCloudPlatform/k8s-cluster-bundle

Deep copy gen generates invalid code if struct has an `_` field

Running deepcopy-gen against structs that have _ fields in them generates invalid go code.

Source struct:

type CreateBucketConfiguration struct {
	_ struct{} `type:"structure"`

	LocationConstraint BucketLocationConstraint `type:"string" enum:"true"`
}
type BucketLocationConstraint string

doc.go directives to deepcopy:

// +k8s:deepcopy-gen=package,register

deepcopy directives above structs that the above struct is a child of (not sure if this makes a difference or not - this is autogenerated code from the operator sdk)

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

Output by deepcopy-gen:

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *CreateBucketConfiguration) DeepCopyInto(out *CreateBucketConfiguration) {
	*out = *in
	out._ = in._
	return
}

Setting or accessing _ fields is invalid.

I applied the following patch locally and it outputs correct code (ie does not attempt to copy the _ field). No idea if this is the correct way to solve the problem though.

diff --git a/vendor/k8s.io/gengo/examples/deepcopy-gen/generators/deepcopy.go b/vendor/k8s.io/gengo/examples/deepcopy-gen/generators/deepcopy.go
index 4548108..1d90578 100644
--- a/vendor/k8s.io/gengo/examples/deepcopy-gen/generators/deepcopy.go
+++ b/vendor/k8s.io/gengo/examples/deepcopy-gen/generators/deepcopy.go
@@ -823,6 +823,9 @@ func (g *genDeepCopy) doStruct(t *types.Type, sw *generator.SnippetWriter) {

 	// Now fix-up fields as needed.
 	for _, m := range ut.Members {
+		if m.Name == "_" {
+			continue
+		}
 		ft := m.Type
 		uft := underlyingType(ft)

After applying this patch, output is as expected:

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *CreateBucketConfiguration) DeepCopyInto(out *CreateBucketConfiguration) {
	*out = *in
	return
}

Add plural rules to namer for words ending in 'o'

The namer package, specifically the method https://github.com/font/kubernetes/blob/91e7b4fd31fcd3d5f436da26c980becec37ceefe/vendor/k8s.io/gengo/namer/plural_namer.go#L54, does not contain any rules for words ending in o. See https://howtospell.co.uk/making-O-words-plural for more details.

Copyright boilerplate changes YEAR, but shouldn't

The copyrighted code is either copyright the first year it was generated or else it is copyright the year we wrote gengo. It obviously doesn't change copyright by nature of being regenerated.

Clean up examples

Examples should contain only 100% general things.

Deep copy, defaulter, proto need to move to the kube-generator repo.

Import boss can probably be deleted, since we enforce that w/ bazel now.

gengo change of behavior with generated import statements

When using gengo in service-catalog, I found that the import statements aren't generated in the same way any more. The change of behavior started after the commit of 7ca94dd, which was on Jan 3. I filed a bug kubernetes-retired/service-catalog#223 showing the difference in behavior.

CI is silently failing

Running the current first step of the build shows the issue:

$ find . -name vendor -prune -o -name Makefile -execdir make test \;
unknown flag: --logtostderr
Usage of /tmp/defaulter-gen:
...
make: *** [test] Error 2
...

This started happening 5 months when the repo switched over from glog to klog (#129). The build log for it is https://travis-ci.org/kubernetes/gengo/jobs/450604661.

Since find wraps the error and returns zero, the CI has been silently failing since.

are `--input-dirs` directories or packages

Using lister-gen from https://github.com/kubernetes/kubernetes/tree/master/cmd/libs/go2idl/lister-gen

--input-dirs only accepts packages and not directories. Is lister-gen using the input-dirs argument incorrectly, or is the argument supposed to represent directories and not packages?

Update for go modules

Let's begin publishing the module file.

Let's version this thing and start respecting the semver guarantees.

Moving to go mod has caused the generators to become about 40% slower.

The package solvers seem to search the filesystem too much. Vendoring deps should be leveraged when the go tool has a vendor dir (1.14 defaults to exclusive vendor searches if vendoring is in use).

travis-ci.org is going to be shutdown the end of this month

👋 just saw the project is still using travis-ci.org for CI, and I have to call out that Travis is going to shut it down on this May 31.

deepcopy-gen doesn't work on Windows

There are a couple of bugs which stop deepcopy-gen from working on Windows:

• golangTrackerLocalName uses the path separator to split package names, but package names can contain forward-slashes as well (I'm not actually sure whether the package names which reach this function can actually contain back-slashes). I worked around this locally by using a split on either forward- or back-slashes.
• In parser/parse.go: strings.HasSuffix(f.name, "/doc.go"). I just hacked this locally to have an || strings.HasSuffix(f.name, "\doc.go"), it seems like this one probably should be using path separator.

I can put together a PR if anyone has any rough suggestions about how to fix these more properly (I am not a Go expert).

Time to version!

Gengo is slow enough moving that I think it's time to call it 1.0.0 and start paying attention to our interface changes.

Method name in comment does not match generated code by deepcopy-gen

pkg/apis/apps/zz_generated.deepcopy.go:

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, creating a new ControllerRevision.
func (x *ControllerRevision) DeepCopy() *ControllerRevision {
	if x == nil {
		return nil
	}
	out := new(ControllerRevision)
	x.DeepCopyInto(out)
	return out
}

The method name in comment does not match the real method name.

No comments are collected for string alias constants.

In service-apis, we use the following pattern:

type ListenerConditionType string

const (
	// ListenerConditionConflicted indicates that the controller
	// was unable to resolve conflicting specification requirements
	// for this Listener. If a Listener is conflicted, its network
	// port should not be configured on any network elements.
	//
	// Possible reasons for this condition to be true are:
	//
	// * "HostnameConflict"
	// * "ProtocolConflict"
	// * "RouteConflict"
	//
	// Controllers may raise this condition with other reasons,
	// but should prefer to use the reasons listed above to improve
	// interoperability.
	ListenerConditionConflicted ListenerConditionType = "Conflicted"

	// ListenerReasonHostnameConflict is used when the Listener
	// violates the Hostname match constraints that allow collapsing
	// Listeners. For example, this reason would be used when multiple
	// Listeners on the same port use the "Any" hostname match type.
	ListenerReasonHostnameConflict ListenerConditionReason = "HostnameConflict"
)

We also use a variation of this with eum validation:

// +kubebuilder:validation:Enum=Domain;Exact;Any
type HostnameMatchType string

const (
	// HostnameMatchExact specifies that the hostname provided
	// by the client must exactly match the specified value.
	//
	// This match type MUST be case-insensitive.
	HostnameMatchExact HostnameMatchType = "Exact"

	// HostnameMatchDomain specifies that the hostname provided
	// by the client should be matched against a DNS domain value.
	// The domain match removes the leftmost DNS label from the
	// hostname provided by the client and compares the resulting
	// value.
	//
	// For example, "example.com" is a "Domain" match for the host
	// name "foo.example.com", but not for "foo.bar.example.com"
	// or for "example.foo.com".
	//
	// This match type MUST be case-insensitive.
	HostnameMatchDomain HostnameMatchType = "Domain"

	// HostnameMatchAny specifies that this Listener accepts
	// all client traffic regardless of the presence or value of
	// any hostname supplied by the client.
	HostnameMatchAny HostnameMatchType = "Any"
)

What I would like to do is bubble the documentation from the constants up to the generated HTML documentation for the CRD (we are using https://github.com/ahmetb/gen-crd-api-reference-docs), but it looks like comments aren't collected for DeclarationOf types.

For example, here's the DeclarationOf type:

I1012 08:38:00.636089   32798 main.go:467] type -> &types.Type{Name:types.Name{Package:"sigs.k8s.io/service-apis/apis/v1alpha1", Name:"HostnameMatchExact", Path:""}, Kind:"DeclarationOf", CommentLines:[]string(nil), SecondClosestCommentLines:[]string(nil), Members:[]types.Member(nil), Elem:(*types.Type)(nil), Key:(*types.Type)(nil), Underlying:(*types.Type)(0xc009e70d10), Methods:map[string]*types.Type(nil), Signature:(*types.Signature)(nil)}

And here's its underlying type:

I1012 08:38:00.636102   32798 main.go:468] underlying -> &types.Type{Name:types.Name{Package:"sigs.k8s.io/service-apis/apis/v1alpha1", Name:"HostnameMatchType", Path:""}, Kind:"Alias", CommentLines:[]string{"HostnameMatchType specifies the types of matches that are valid", "for hostnames.", "Valid match types are:", "", "* \"Domain\"", "* \"Exact\"", "* \"Any\"", "", "+kubebuilder:validation:Enum=Domain;Exact;Any"}, SecondClosestCommentLines:[]string{""}, Members:[]types.Member(nil), Elem:(*types.Type)(nil), Key:(*types.Type)(nil), Underlying:(*types.Type)(0x17ef600), Methods:map[string]*types.Type(nil), Signature:(*types.Signature)(nil)}

The alias has comments, but the constant doesn't.

Adding Proto3 support

We are exploring the possibility of adding proto3 support while keeping support for proto2. This is proving to be quite difficult, quite likely for a lack of knowledge on my side.

The implications of switching the syntax affect several parts of the codebase that have no parameter (or at least, not an obvious one) in common.

Can you provide some help/guidance on this?

If this is not the appropriate channel for asking this kinds of questions, please, let us know.

Thank you!

Capture the underlying value of constants.

When using gengo to generate API documentation, it would be pretty convenient to be able to get the underlying value of the constant. This could be used to generate documentation for the acceptable values of an enum.

Useless error report when hitting an unknown struct

Some type I had in my source code had a problem. Sadly the output of the generator is not very useful to pinpoint what type exactly:

go generate -v ./pkg/... ./cmd/...                                            
pkg/apis/apis.go                                          
F0305 17:48:56.504879  139794 deepcopy.go:873] Hit an unsupported type invalid type                              
Makefile:69: recipe for target 'generate' failed           

feature request: cli tools

I am looking for the generator of proto file from golang struct,
the go-to-protobuf looks like related about this,
I have tried proteus, but does not works well,
this project will offer some cli tools for code gen?
thx!

Clarify ownership and merge policy

After a couple of times with PRs not being merged here for weeks, can we clarify who can approve, lgtm and merge PRs in here? Looking at older PRs neither of these groups match the OWNERS file.

Any way to get anonymous variables? (blank identifier)

I like generating some basic implementation for types. And I link these using a compile time check:

var _ DeepCopier = (*SomeTypeOfMine)(nil)

Using the parser builder, I was not able to get such variables.

Need examples and documentation for generating deep-copy functions for structs that reference ptypes

I am trying to generate deep-copy functions for my struct that contains structpb.Struct.

// +genclient
// +genclient:noStatus
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// My CRD
type MyStruct struct {
metav1.TypeMeta
metav1.ObjectMeta
Spec MySpec
Status MyStatus
ListOfStructs structpb.Struct
}
DeepCopyInto function is not generated for ListOfStructs. How do I either generate the functions or specify my own.

proposal: deepequal code generation

Hi,
While working on a project I had a need for a "DeepEqual" method that handled some usecases specific to our application that reflect.DeepEqual does not handle. I was wondering if there is any interest in me contributing this as a code generator along side the deepcopy-gen example that is provided in this repo or if there is any objection to splitting this off into a separate non-Kubernetes repo?

The two special cases that we needed handled are as follows. Both are off by default unless a special struct comment tag is supplied by the user:

1. We needed slices to be compared such that order is unimportant. That is, two slices containing the same elements but in different order would report as equal. This is contrary to how reflect.DeepEqual works. I can see this being useful to other people as it is not always convenient to keep a slice in sorted order when adding/removing from it, or to want to sort it before comparing.

The type would be tagged like this to enable this special this handling:

// +k8s:deepequal-gen:unordered-array=true
type MyList []string

2. We needed to handle a special usecase for struct comparisons whereby nil fields on the left hand side (the receiver) of the comparison would be ignored and assumed to be equal regardless of the value on the right hand side (the input). The reason for this is that we have a need to express certain fields as optional features and if the struct on the right has it set then we just treat that an acceptable value. For example, consider this struct.

// +k8s:deepequal-gen:ignore-nil-fields=true
type MyStruct struct {
    Name string `json:"name"`
    Address string `json:"address"`
    Age *int `json:"age,omitempty"`
}

a := MyStruct{Name: "John", Address: "Somewhere"}
ageB := 30
b := MyStruct{Name: "John", Address: "Somewhere", Age: &ageB}
ageC := 31
c := MyStruct{Name: "John", Address: "Somewhere", Age: &ageC}

a.DeepEqual(&b) == true
b.DeepEqual(&c) == false

I admit that this particular behavior may not be applicable to many users, but having this available as part of the code generator was a time saver as we have many structures and many of those have these optional fields that we needed to handle in this way.

If there is interest I can augment the unit tests and prepare a PR.

add conversion-gen into example

hi:
could be add conversion-gen from k8s into example dir?

AddDirRecursive panics when GOROOT is not set correctly

It seems that when GOROOT is misconfigured, Builder.AddDirRecursive panics:

W0611 10:07:38.859903   39295 parse.go:224] Ignoring directory github.com/k8spatterns/hello-operator/pkg/apis: unable to import "github.com/k8spatterns/hello-operator/pkg/apis": go/build: importGo github.com/k8spatterns/hello-operator/pkg/apis: exit status 2
go: cannot find GOROOT directory: /usr/local/go

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1760595]

goroutine 1 [running]:
github.com/operator-framework/operator-sdk/vendor/k8s.io/gengo/parser.(*Builder).AddDirRecursive(0xc000962230, 0xc0006b71d0, 0x2e, 0x2, 0x2)
	src/github.com/operator-framework/operator-sdk/vendor/k8s.io/gengo/parser/parse.go:229 +0xb5

See operator-framework/operator-sdk#1545 for more details.

Issue with Installing gengo on docker image without GO111MODULE

Im trying to install gengo on a docker image. This is Dockerfile that doesnt work

FROM golang:1.14.2-stretch
RUN go get k8s.io/gengo

Error produced:

Sending build context to Docker daemon   1.47GB
Step 1/2 : FROM golang:1.14.2-stretch
 ---> a58e31ece45e
Step 2/2 : RUN go get k8s.io/gengo
 ---> Running in 61735ad7dec3
package k8s.io/gengo: no Go files in /go/src/k8s.io/gengo
The command '/bin/sh -c go get k8s.io/gengo' returned a non-zero code: 1

The issue happens with 1.13.x base image of go also. I believe this is due to the latest klog v2 change that was made here: 565683f

There seems to be a fix for this. If i have GO111MODULE=on, the build succeeds.
This is Dockerfile that works:

FROM golang:1.14.2-stretch
ENV GO111MODULE=on
RUN go get k8s.io/gengo

Not sure if this is expected. It breaks some downstream users. If this is not a bug, does it warrant at least a documentation change?

We need continuous testing on windows

...to make sure there's no path / filepath regressions.

Create a SECURITY_CONTACTS file.

As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS
file.

The template for the file can be found in the kubernetes-template repository[2].
A description for the file is in the steering-committee docs[3], you might need
to search that page for "Security Contacts".

Please feel free to ping me on the PR when you make it, otherwise I will see when
you close this issue. :)

Thanks so much, let me know if you have any questions.

(This issue was generated from a tool, apologies for any weirdness.)

[1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE
[2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS
[3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md

Numerous pathname bugs on Windows

This code appears to have quite a number of pathname bugs on Windows, as a result of assumptions about the path separator character (e.g. it looks for "/vendor/" in path strings in several places).

parser errors from external packages should be ignored

There's lots of false positives like:

W0516 14:25:25.242] 	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/golang_org/x/net/lex/httplex (from $GOPATH))
W0516 14:25:25.243] I0516 14:25:25.243437    1949 parse.go:394] type checker error: /usr/local/go/src/net/http/h2_bundle.go:850:19: undeclared name: hpack
W0516 14:25:25.244] I0516 14:25:25.243796    1949 parse.go:394] type checker error: /usr/local/go/src/net/http/h2_bundle.go:1834:11: undeclared name: hpack
W0516 14:25:25.244] I0516 14:25:25.243833    1949 parse.go:394] type checker error: /usr/local/go/src/net/http/h2_bundle.go:1858:52: undeclared name: hpack
W0516 14:25:25.244] I0516 14:25:25.243854    1949 parse.go:394] type checker error: /usr/local/go/src/net/http/h2_bundle.go:1869:51: undeclared name: hpack
W0516 14:25:25.244] I0516 14:25:25.243880    1949 parse.go:394] type checker error: /usr/local/go/src/net/http/h2_bundle.go:5530:11: undeclared name: hpack
W0516 14:25:25.245] I0516 14:25:25.244088    1949 parse.go:394] type checker error: /usr/local/go/src/net/http/h2_bundle.go:7400:20: undeclared name: hpack
W0516 14:25:25.245] I0516 14:25:25.244903    1949 parse.go:394] type checker error: /usr/local/go/src/net/http/h2_bundle.go:3252:18: undeclared name: hpack
W0516 14:25:25.245] I0516 14:25:25.244985    1949 parse.go:394] type checker error: /usr/local/go/src/net/http/h2_bundle.go:3312:46: undeclared name: hpack
W0516 14:25:25.248] I0516 14:25:25.247794    1949 parse.go:394] type checker error: /usr/local/go/src/net/http/h2_bundle.go:7551:22: undeclared name: hpack
W0516 14:25:25.248] I0516 14:25:25.247900    1949 parse.go:394] type checker error: /usr/local/go/src/net/http/h2_bundle.go:7687:30: undeclared name: hpack
W0516 14:25:25.251] I0516 14:25:25.251452    1949 parse.go:346] type checking encountered some errors in "net/http", but ignoring.
W0516 14:25:26.515] I0516 14:25:26.514660    1949 execute.go:214] Processing package "v1", disk location "/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/example/v1"
W0516 14:25:26.524] I0516 14:25:26.524049    1949 execute.go:67] Assembling file "/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/apis/example/v1/zz_generated.conversion.go"

That make it harder to find errors in kubernetes-verify jobs. Can we suppress type checker errors for stdlib packages?

deepcopy-gen doesn't handle pointer-to-slice correctly

If you have a struct with a pointer to a slice, eg:

Ports *[]NetworkPolicyPort `json:"ports,omitempty"`

(because you need to distinguish "field is not set" from "field is set to []") then deepcopy-gen generates incorrect code to copy it, resulting in:

../../apis/extensions/zz_generated.deepcopy.go:587: invalid argument *in (type *[]NetworkPolicyPort) for len

because it outputs code as though the field was a slice, not a pointer-to-slice. (Presumably pointer-to-map fields have the same problem.)

(kubernetes/kubernetes#25835 has more discussion of the use case.)

I don't have a simple reproducer, but, eg, changing NetworkPolicyIngressRule.Ports in kubernetes/pkg/apis/extensions/types.go from []NetworkPolicyPort to *[]NetworkPolicyPort and regenerating stuff will trigger the error above.

package k8s.io/gengo: no Go files in /Users/like/gomod/src/k8s.io/gengo

when i use go get k8s.io/gengo to install gengo

go get k8s.io/gengo
package k8s.io/gengo: no Go files in /Users/like/gomod/src/k8s.io/gengo

deepcopy-gen generates invalid code for interface{}

Declaring a map[string]interface{} member in a struct handled by deepcopy-gen generates invalid code.
It seems there could be two options available:

• for map[string]interface{}, generate code that leverages runtime.DeepCopyJSON
• for any interface{}, error out instead of generating code that does not compile.

Related issues: #128, kubernetes-sigs/kubebuilder#528 and kubernetes-sigs/controller-tools#126

Example

• foo.go

// +k8s:deepcopy-gen=true
type Foo struct {
  Data map[string]interface{}
}

• zz_generated.deepcopy.go (DeepCopy function omitted for clarity as it basically calls DeepCopyInto)

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *Foo) DeepCopyInto(out *Foo) {
	*out = *in
	if in.Data != nil {
		in, out := &in.Data, &out.Data
		*out = make(map[string]interface{}, len(*in))
		for key, val := range *in {
			if val == nil {
				(*out)[key] = nil
			} else {
				// INVALID FUNCTION NAME
				(*out)[key] = val.DeepCopyinterface{}()
			}
		}
	}
	return
}

pkg/parser: Parses too much stuff, eventually goes into stdlib reflect, rune etc.

I'm author of https://github.com/ahmetb/gen-crd-api-reference-docs/ which is a tool to generate API reference docs for most kubernetes CRD projects (e.g. usually built with kubebuilder). I use gengo/parser package.

Recently it started to break because its parsing goes waaaay too deep.

It's parsing this k8s.io/apimachinery/pkg/runtime.Scheme type: https://sourcegraph.com/github.com/knative/serving@07437021afd54ecd391f84e66616751108bc6644/-/blob/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go#L47:6

This Scheme has unexported fields like gvkToType map[schema.GroupVersionKind]reflect.Type which refer to reflect.Type.

So gengo/parser goes and parses reflect.Type, which ends up going muuuuch deeper and ends up parsing types like rune builtin as Unsupported kind.

Eventually my program bursts out with Type{Name: "invalid type", Kind: Unsupported, Package: ""} which isn't great.

Would it be possible to change gengo/parser so that

• it doesn't parse lowercase (unexported) fields
• it doesn't parse stdlib runtime.* package

I think these two improvements would help massively.

Consider decoupling package->directory lookups from $PWD

Branching discussion from kubernetes/repo-infra#19

Gengo currently assumes that a package "k8s.io/foo/bar" can always be found in ${PWD}/k8s.io/foo/bar/, and becomes very unhappy if the generator program is called from a different directory. This makes its use from Bazel rules complicated, and requires Kubernetes BUILD files to use custom logic such as go_genrule() instead of Bazel's built-in genrule().

Int8 should not be byte type

Current builtin types in gengo

		Types: map[string]*Type{
			"bool":    Bool,
			"string":  String,
			"int":     Int,
			"int64":   Int64,
			"int32":   Int32,
			"int16":   Int16,
			"int8":    Byte,
			"uint":    Uint,
			"uint64":  Uint64,
			"uint32":  Uint32,
			"uint16":  Uint16,
			"uint8":   Byte,
			"uintptr": Uintptr,
			"byte":    Byte,
			"float":   Float,
			"float64": Float64,
			"float32": Float32,
		}

Description

int8 and uint8 are basic kind in go and defined as reflect.Kind (See https://golang.org/pkg/reflect/#Kind)

However byte which is uint8 kind actually is not basic kind

Expected

int8 is Int8 type, uint8 is Uint8 type and byte is Uint8 type

Actual

Now int8, uint8 and byte are Byte type

Missing CONTRIBUTING.md file

All K8s subrepositories should have a CONTRIBUTING.md file, which at the minimum should point to https://github.com/kubernetes/community/blob/master/contributors/guide/README.md. Care should be taken that all information is in sync with the contributor guide.

Subrepositories may also have contributing guidelines specific to that repository. They should be explicitly documented and explained in the CONTRIBUTING.md

Ref: kubernetes/community#1832

Golint errors in generated files by `deepcopy-gen`

When I want to enable golint with pkg/scheduler in k8s kubernetes/kubernetes#58234, there are files generated by deepcopy-gen having lint errors. e.g.:

pkg/scheduler/api/zz_generated.deepcopy.go:265:9: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary)

The code:

// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *Policy) DeepCopyObject() runtime.Object {
	if c := in.DeepCopy(); c != nil {
		return c
	} else {
		return nil
	}
}

I think these errors should be fixed in deepcopy-gen so these files can pass the golint check.

segfault when running make, seems to come from gengo

I get a segfault when I run make. I am rebased on latest upstream/master for k8s. This is not good.

Substituted private paths w/ PATHYMCPATHFACE:

$ make
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x45dcf2]

goroutine 1 [running]:
panic(0x692840, 0xc4200102e0)
	/PATHYMCPATHFACE/go/src/runtime/panic.go:500 +0x1a1
k8s.io/kubernetes/vendor/k8s.io/gengo/examples/defaulter-gen/generators.getManualDefaultingFunctions(0xc430007a40, 0x0, 0xc4302989c0)
	/PATHYMCPATHFACE/go-tools/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/gengo/examples/defaulter-gen/generators/defaulter.go:113 +0x102
k8s.io/kubernetes/vendor/k8s.io/gengo/examples/defaulter-gen/generators.Packages(0xc430007a40, 0xc42001c680, 0x6db1d5, 0x6, 0xc430007a40)
	/PATHYMCPATHFACE/go-tools/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/gengo/examples/defaulter-gen/generators/defaulter.go:236 +0x6d2
k8s.io/kubernetes/vendor/k8s.io/gengo/args.(*GeneratorArgs).Execute(0xc42001c680, 0xc420016fc0, 0x6db1d5, 0x6, 0x700988, 0x0, 0x0)
	/PATHYMCPATHFACE/go-tools/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/gengo/args/args.go:165 +0x170
main.main()
	/PATHYMCPATHFACE/go-tools/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/cmd/libs/go2idl/defaulter-gen/main.go:74 +0x1fb
make[1]: *** [gen_defaulter] Error 1
make: *** [generated_files] Error 2

Gengo passes file path as package path to build.Context.Import().

In parse.go, the function Builder.importWithMode() calls build.Context.Import() with dir as the first argument. But dir is a filesystem path, as opposed to a package path, and Import expects a package path as its first argument.

On UNIX systems, this doesn't matter because the two turn out to be the same thing. On Windows, it is a huge problem because package paths are always slash separated while filesystem paths use backslashes.

`k8s:deepcopy-gen` does not work when a struct has comments

The deepcopy generator cannot consider more than one line of comments
before the actual structure.

A part of my code was this:

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// +k8s:deepcopy-gen=true

// SkinnyPod is a stripped down version of k8s' Pod structure.
// By keeping only the parts we use from the pod in memory,
// we reduce the memory footprint of the application.
type SkinnyPod struct {
	metav1.ObjectMeta
	Spec   corev1.PodSpec
	Status corev1.PodStatus
}

Running the kubernetes generator, I had this error:

F0204 10:23:39.567087   91737 deepcopy.go:183] Type <redacted>/internal/types.SkinnyObject requests deepcopy generation but is not copyable

The error's stack pointed out to this line of code:

gengo/examples/deepcopy-gen/generators/deepcopy.go

Line 183 in 1e9b17e

klog.Fatalf("Type %v requests deepcopy generation but is not copyable", t)

.

After reading the code, It made sense just to remove the comments from the struct and try again. It did work like this:

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// +k8s:deepcopy-gen=true

type SkinnyPod struct {
	metav1.ObjectMeta
	Spec   corev1.PodSpec
	Status corev1.PodStatus
}

Type aliases are flattened to the underlying type.

Over in service-apis , I used https://github.com/ahmetb/gen-crd-api-reference-docs to generate the CRD API documentation. This project uses gengo to parse the Go types for the CRD, and generates HTML documentation.

In the service-apis CRD, we use type aliases like this:

type ConfigMapsDefaultLocalObjectReference struct {
...
}

type GatewayClassParametersObjectReference = ConfigMapsDefaultLocalObjectReference
type RouteHostExtensionObjectReference = ConfigMapsDefaultLocalObjectReference
type RouteActionExtensionObjectReference = ConfigMapsDefaultLocalObjectReference

The end result of this is that all the aliases get flattened to the underlying type ConfigMapsDefaultLocalObjectReference.

I'm not at all familiar with any of the machinery underlying this, but the only way I could see to detect the alias was using IsAlias on the type name before walking the type. Once you do this, it might be reasonable to emit the type alias as a type of Kind "Alias", but I'm not sure whether that's a good idea or not.

Here's a bit of a patch that I was messing with that is probably very naive about different type alias possibilities:

@@ -534,7 +535,22 @@ func (b *Builder) findTypesIn(pkgPath importPathString, u *types.Universe) error
                obj := s.Lookup(n)
                tn, ok := obj.(*tc.TypeName)
                if ok {
-                       t := b.walkType(*u, nil, tn.Type())
+                       var typeName *types.Name
+
+                       if tn.IsAlias() {
+                               log.Printf("found TypeName alias name=%q type.Name=%q pkg.Name=%q id=%q",
+                                       tn.Name(), tn.Type().String(), tn.Pkg().Path(), tn.Id())
+                               // typeName = tcNameToName(tn.Name())
+                               typeName = &types.Name{
+                                       Name: tn.Name(),
+                               }
+
+                               if tn.Pkg() != nil {
+                                       typeName.Package = tn.Pkg().Name()
+                               }
+                       }
+
+                       t := b.walkType(*u, typeName, tn.Type())
                        c1 := b.priorCommentLines(obj.Pos(), 1)
                        // c1.Text() is safe if c1 is nil
                        t.CommentLines = splitLines(c1.Text())