openshift / library-go

Helpers for going from apis and clients to useful runtime constructs

library-go

Helpers for going from apis and clients to useful runtime constructs. config.ServingInfo to useful serving constructs is the canonical example. Anything introduced here must have concrete use-cases in at least two separate openshift repos and be of some reasonable complexity. The bar here is high. We'll start with openshift/api-review as the approvers.

This repo must not depend on k8s.io/kubernetes or openshift/origin.

library-go's Issues

Controller factory issuing warning about FastControllerResync even when resyncInterval set to 0 (the default)

Please let me know if this isn't the proper place for this issue - thanks in advance!

Using the controller factory to create a controller, I noticed I keep getting a warning about FastControllerResync.

I believe it's coming from here
https://github.com/openshift/library-go/blame/master/pkg/controller/factory/factory.go#L241-L243

In my case I am not setting a resync interval at all, and relying on informers to resync at specific intervals. If I understand correctly if ResyncEvery() is not called, then it should default to 0 and not perform periodic resyncs.

Switch staticpod unit tests to use validateActions instead of messing with reactors

A follow-up for #70

https://github.com/openshift/service-serving-cert-signer/blob/master/pkg/controller/apiservicecabundle/apiservice_cabundle_controller_test.go#L24

ApplyValidatingWebhookConfiguration does not work

There are couple of issues in ApplyValidatingWebhookConfiguration introduced in #902

1. When used in the StaticController, ApplyDirectly fails with:

   "webhook_configuration.yaml" (string): cannot decode "webhook_configuration.yaml": no kind "ValidatingWebhookConfiguration" 
   is registered for version "admissionregistration.k8s.io/v1" in scheme "pkg/runtime/scheme.go:101"
   

   This can be fixed by registering admission/v1 API around these parts:

   library-go/pkg/operator/resource/resourceapply/generic.go

   Lines 32 to 33 in bd449d1

   	utilruntime.Must(api.InstallKube(genericScheme))
   	utilruntime.Must(apiextensionsv1beta1.AddToScheme(genericScheme))

---

2. ApplyValidatingWebhookConfiguration itself fails on updating a webhook config with:

   validatingwebhookconfigurations.admissionregistration.k8s.io "snapshot.storage.k8s.io" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update
   

   It fails because the function tries to call Update() with required:
   

   library-go/pkg/operator/resource/resourceapply/admissions.go

   Line 43 in bd449d1

   actual, err := client.ValidatingWebhookConfigurations().Update(context.TODO(), required, metav1.UpdateOptions{})

   
   All the other Apply* functions merge required and existingCopy to make sure all the metadata is there...

---

3. Change detection in ApplyValidatingWebhookConfiguration does not work at all. In OpenShift, it's common that an admission plugin injects the actual caBundle into webhooks.clientConfig. Therefore, the required.webhooks and existingCopy.webhooks will never be the same and the function will always update the ValidatingWebhookConfiguration object.
   

   library-go/pkg/operator/resource/resourceapply/admissions.go

   Lines 34 to 37 in bd449d1

   	contentSame := equality.Semantic.DeepEqual(existingCopy.Webhooks, required.Webhooks)
   	if contentSame && !*modified {
   	return existingCopy, false, nil
   	}

@qiujian16, is the ApplyValidatingWebhookConfiguration used anywhere? How did you test it?

Refactor common.NewMultiLineAggregate

#150 (comment)

Could we refactor this actually just be a common.ErrorToConditionMessage() that handles the "normal" aggregate?

/assign @sanchezl

Static pod installer pods are being OOM killed

On pretty poor machine, in libvirt I saw two out of four installer pods OutOfMemory. These pods seems to be never retried.
We already set resource requests/limits (100mb), but I guess when the master node is really under fire, we can still get scheduled and fail.

I think we should set the podPriorityClass for the installer pods as minimum, but we should also discuss additional ways how to make sure the installer pod can be retried. (can we re-create the pod under some circumstances? should we automatically start a new revision? etc)

Please add more details around role and usage of this project in OpenShift

The current Readme does not give much information around this code. Can we add more details to the Readme with respect to OpenShift 4.0 architecture and where this project fits in.

Use ImagePullPolicy: IfNotPresent

My cluster died this morning when I rebooted it; I believe because the release payload was GC'd, and we were using Always for image pull policy. I think this is the code that ends up in the /etc/kubernetes/static-pods bits which are used for the API server...if not please redirect me to the right place.

diff --git a/pkg/operator/render/options/manifest.go b/pkg/operator/render/options/manifest.go
index 279ea38..e893edf 100644
--- a/pkg/operator/render/options/manifest.go
+++ b/pkg/operator/render/options/manifest.go
@@ -23,7 +23,7 @@ func NewManifestOptions(componentName, image string) *ManifestOptions {
 	return &ManifestOptions{
 		Namespace:             fmt.Sprintf("openshift-%s", componentName),
 		Image:                 image,
-		ImagePullPolicy:       "Always",
+		ImagePullPolicy:       "IfNotPresent",
 		ConfigHostPath:        "/etc/kubernetes/bootstrap-configs",
 		ConfigFileName:        fmt.Sprintf("%s-config.yaml", componentName),
 		CloudProviderHostPath: "/etc/kubernetes/cloud",

The 24 hour time frame for the first certificate rotation should be customizable

The current certificate rotation to start we needs to wait for 24 hours, which is a blocker for CodeReady Containers team's build process. Also for CI process also I think the initial cert rotation time interval should be customized to minutes and not 24 hours.

Check #328 (comment) for details about the build process we follow.

There must be someway to reduce the initial cert rotation to couple of minutes or less.

https://github.com/openshift/library-go/blob/master/pkg/operator/certrotation/config.go#L13-L40

// GetCertRotationScale  The normal scale is based on a day.  The value returned by this function
// is used to scale rotation durations instead of a day, so you can set it shorter.
func GetCertRotationScale(client kubernetes.Interface, namespace string) (time.Duration, error) {
	certRotationScale := time.Duration(0)
	err := wait.PollImmediate(time.Second, 1*time.Minute, func() (bool, error) {
		certRotationConfig, err := client.CoreV1().ConfigMaps(namespace).Get("unsupported-cert-rotation-config", metav1.GetOptions{})
		if err != nil {
			if errors.IsNotFound(err) {
				return true, nil
			}
			return false, err
		}
		if value, ok := certRotationConfig.Data["base"]; ok {
			certRotationScale, err = time.ParseDuration(value)
			if err != nil {
				return false, err
			}
		}
		return true, nil
	})
	if err != nil {
		return 0, err
	}
	if certRotationScale > 24*time.Hour {
		return 0, fmt.Errorf("scale longer than 24h is not allowed: %v", certRotationScale)
	}
	return certRotationScale, nil
}

Add WaitForHealthyAPIServer()

waitForHealthyAPIServer from origin/pkg/cmd/openshift-controller-manager/controller_manager.go is useful for external controllers, so @deads2k says it should be here.

@soltysh says it should take the wait time as a parameter

Should ApplyDeployment be testing Spec equality?

https://github.com/openshift/library-go/blob/master/pkg/operator/resource/resourceapply/apps.go#L34

Should this not be comparing spec to see if something changed? My operator has an asset via go-bindata but doesn't know it's expected generation as a result. Because we never compare spec, most changes that I expect the operator to deploy are not taking effect. Would it be reasonable to test equality of spec here?

event recorder exposes involvedObjectRef for callers

Currently, we use the event recorder to send our events, in some cases, we need to send an event in a Reconcile, and the event may have different involvedObjectRef, so we need to create new recorder in every Reconcile, but we have some concern to Shutdown the recorder when exiting the Reconcile, since the events are sent asynchronously, it seems hard to know whether the events are actually sent before calling the Shutdown.

So if the event recorder can provide some new methods like following that will be very helpful:

// Event emits the normal type event and allows passing the involved object.
func (r *recorder) EventWithInvolvedObjectRef(involvedObjectRef *corev1.ObjectReference, reason, message string)

// Warning emits the warning type event and allows passing the involved object..
func (r *recorder) WarningWithInvolvedObjectRef(involvedObjectRef *corev1.ObjectReference, reason, message string)

RFE: ability to dynamically set ClusterOperator.Status.RelatedObjects

In the CNO, the set of related objects is dynamic - the details aren't important. However, the standard ClusterOperatorStatusSyncer doesn't support dynamic RelatedObjects.

We need a way to dynamically update RelatedObjects.

In general, the StatusSyncer distils the OperatorStatus to a ClusterOperatorStatus. One solution, therefore, would be to add a RelatedObjects field to the OperatorStatus struct.

Another option is to add a RelatedObjects() function to the OperatorClient.

/cc @deads2k

Add general helper in resourceapply that evaluate multiple resources created.

Follow up for #70

CVE-2021-3121 (gogo/protobuf) remains in dependencies

I've been trying to squash CVE-2021-3121 in a library which imports library-go. My test is:

$ git grep -l "if skippy < 0 {"
vendor/go.etcd.io/etcd/auth/authpb/auth.pb.go
vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/etcdserver.pb.go
vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/raft_internal.pb.go
vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go
vendor/go.etcd.io/etcd/mvcc/mvccpb/kv.pb.go
vendor/go.etcd.io/etcd/raft/raftpb/raft.pb.go
vendor/k8s.io/apimachinery/pkg/api/resource/quantity_proto.go

The hit in apimachinery was missed when they bumped their gogo/protobuf dep, but there's now a PR to fix that: kubernetes/kubernetes#101306

library-go currently pulls in

go.etcd.io/etcd v0.5.0-alpha.5.0.20200910180754-dd1b699fc489

which is included in etcd v3.4.14. etcd merged a gogo/protobuf fix in commit 1b300cd9e269c99b2045bff89e12318146a897d3 which is:

• On the v3.5 branch
• Not included in any tag

I'm not personally confident in trying to bump library-go's etcd dep between branches, especially to a non-stable branch, as I don't understand the implications of this.

The etcd commit which bumps gogo/protobuf makes it clear they don't believe there is an actual vulnerability there: etcd-io/etcd@19f7c6e . However, it would make verification across the product much simpler if we were able to eliminate all potentially vulnerable code.

I'm not giving this github issue security handling because everything in it is already public, and etcd specifically don't think it's an issue. It should be a useful tracker for operators trying to squash this CVE in their own codebases.

resourcesynccontroller miss key configmaps

1 problem description
when resourcesynccontroller miss key configmap in CombineCABundleConfigMaps, and err is notFound, it will continue and override this failure, this will cause the caller func generate wrong configgurations

library-go/pkg/operator/resourcesynccontroller/core.go

Line 17 in 81bb04e

func CombineCABundleConfigMaps(destinationConfigMap ResourceLocation, lister corev1listers.ConfigMapLister, inputConfigMaps ...ResourceLocation) (*corev1.ConfigMap, error) {

2 bug fix
this problem is related to the kube-controller-manager-operator bug openshift/cluster-kube-controller-manager-operator kube-controller-manager rootCA miss key conf

tokenrequest.callbackServer does not implement CORS / PNA

The callbackServer in pkg/oauth/tokenrequest/callback_server.go does not implement CORS / Private-Network-Access, which might result in browser based authentication flows becoming unusable in future versions of Chrome: https://developer.chrome.com/blog/private-network-access-preflight

Future Release Branches Frozen For Merging | branch:release-4.17 branch:release-4.18

The following branches are being fast-forwarded from the current development branch (master) as placeholders for future releases. No merging is allowed into these release branches until they are unfrozen for production release.

• release-4.17
• release-4.18

For more information, see the branching documentation.

CVE-2020-26160 on dependency library

Facing CVE issue for current version of github.com/dgrijalva/jwt-go

details

package: github.com/dgrijalva/jwt-go

high severity
Vulnerable versions: <= 3.2.0
Patched version: No fix
jwt-go allows attackers to bypass intended access restrictions in situations with []string{} for m["aud"] (which is allowed by the specification). Because the type assertion fails, "" is the value of aud. This is a security problem if the JWT token is presented to a service that lacks its own audience check. There is no patch available and users of jwt-go are advised to migrate to golang-jwt at version 3.2.1

IBM/ibm-vpc-block-csi-driver-operator#12

--domain says it is optional, but it is not

https://github.com/openshift/library-go/blob/master/cmd/crd-schema-gen/generator/generator.go

https://github.com/openshift/library-go/blob/master/cmd/crd-schema-gen/generator/generator.go#L38 lists the domain flag as optional, but https://github.com/openshift/library-go/blob/master/cmd/crd-schema-gen/generator/generator.go#L73-L75 throws an error if --domain is not specified.

So, is it optional, or not?

delete and recreate ClusterRoleBinding and RoleBinding if roleref not match

When applying a new ClusterRoleBinding and RoleBinding with roleRef changed, resourceapply.ApplyDirectly will fail to update the resource, as "cannot change roleRef". As a workaround, need to delete that resource and recreate.

I see the same requirement is listed as a todo in ApplyClusterRoleBinding and ApplyRoleBinding . Can we enhance the ApplyClusterRoleBinding and ApplyRoleBinding to delete and recreate clusterrolebinding and rolebinding on non-matching roleref? I can help create a PR for that.

Some events are lost

We see a lot of events reported in the operator logs, but they won't make it to API server (as we don't see them in events.json file in CI).

We should check the sink/broadcaster wiring and make sure we don't rate limit the event client.

Support multiplatform builds

needed for cli-artifacts image producing Mac and Windows binaries

✨ Static pod CustomPruningController

I am utilizing the static pod package (library-go/pkg/operator/staticpod) for a use case in which all pods require an init container for authz and authn.

Due to this requirement, I need a CustomPruningController that, similar to the CustomInstallerController, accepts a function to changing the pruner pod before it is created. I have implemented a POC here.

This simple approach passes the function to the NewPruningController function.

An alternative approach would be to refactor the PruningController function to implement the factory.controller similar to InstallerController. This approach would allow the function to be passed using the builder pattern similar to InstallerController but would require significant refactoring.

Do the owners of this repo have a preference on the approach?

find the most recent `LastTransitionTime`

Originally posted by @deads2k in #108

APISerivce apply failed

Hello, I'm using resourceapply.ApplyAPIService, but I found changes of Spec.CABundle won't trigger the updates.

library-go/pkg/operator/resource/resourceapply/apiregistration.go

Line 39 in a53ff0c

if !*modified && serviceSame && prioritySame && insecureSame {

Should there be a comparison of CABundle?

Migrate LeaderElection to Lease API

Upstream has done it for the controller manager and scheduler
kubernetes/kubernetes#80289

should we do it here too?

library-go/pkg/config/leaderelection/leaderelection.go

Line 53 in 965974e

resourcelock.ConfigMapsResourceLock,

it comes with some upgrade problems thought

pin mergo

@sanchezl did you pin mergo in glide.yaml to the level used in kube? if not, please do so.

Originally posted by @deads2k in #150