Comments (16)
Another thing that bit me and caused a hot loop was not having a stable serialization of my parent spec.
My spec contained a field of typemap[string]string
, which has no order when being marshaled to JSON. As a result, the metacontroller.k8s.io/last-applied-configuration
would change frequently because the order of the map items would get shuffled by the encoder. This was fixed by converting my parent CRD to a list of structs rather than a map. Not sure if there is any way around this, or if metacontroller had some way of creating a more stable encoding of the last-applied-configuration.
from metacontroller.
Is there any workaround for this? Iām using the kubernetes go library in my golang hook and seeing this same issue.
from metacontroller.
To unblock people who prefer to write hooks in Go, I've proposed #94 which enforces the invariant that Metacontroller should never try to update read-only, system-populated metadata fields. That should fix the immediate issue here with creationTimestamp
, but it remains to be seen if we'll encounter other problems outside ObjectMeta where the Kubernetes Go API failed to use pointers for omitempty
struct fields.
from metacontroller.
As mentioned above, the fix in #94 wouldn't work if there are other fields outside ObjectMeta that don't get omitted when left on their Go zero values.
For Pod, it seems like we are safe, but @danisla has found a new example in ReplicaSet (status.replicas is intentionally not omitempty
), which shows that there will likely be other examples scattered throughout the Go API structs.
Given that, I'm leaning towards the other alternative proposed in the first post of this issue, which is to make Metacontroller's apply semantics match kubectl apply
more closely in this scenario:
- You apply something like
replicas: 0
. - Something else (e.g. Horizontal Pod Autoscaler) edits the object to set
replicas: 5
. - You apply again, but you still say
replicas: 0
.
In the case of kubectl apply
, the HPA "wins" and your second apply leaves replicas
at 5. That's because when you apply the same value as last time for a given field, kubectl assumes you mean that you don't care to change that field right now, so it remains at whatever value the live object on the server has.
By contrast, currently in that scenario Metacontroller assumes the controller is trying to maintain the state replicas: 0
even in the face of outside interference. I thought it would be surprising to a controller author if, for example, the user directly alters a field (e.g. with kubectl edit
), and the controller doesn't reset it back to the "right" value. After all, the job of a controller is to continually push towards the desired state.
If we instead make Metacontroller match the kubectl apply
behavior, it should prevent the whole class of endless updates discussed in this issue, but it might cause surprises in the other direction where people expect updates and they don't happen.
If any users want to weigh in on which behavior would be least surprising, that would help.
from metacontroller.
I think most of my use cases would prefer to match the behavior of kubectl apply
. Can we have both and let the author pick the appropriate behavior via the controller config?
from metacontroller.
The example is a little weird, because HPA isn't going to be updating the status.replicas
it will be updating the spec.replicas
and spec.replicas
does have omitempty
set properly.
I think it depends on the update strategy for me. All of the custom controllers I've written use the Update
method and want to reset the values on every write. If two controllers are in contention it's usually a bug and one of them needs to use a different way to update the field. e.g update the parent resources and let it's controller own things instead of trying to update the child directly.
I tend to think kubectl apply semantics are more useful when a human is on the other side of the client.
from metacontroller.
The example is a little weird
Sorry for using a confusing example. I actually did mean spec.replicas
in the HPA example, but I was using it as a stand-in for a hypothetical spec field that is not omitempty
. Given that we found status.replicas
is not omitempty
, I'm worried there are also spec fields lurking in the API that are not omitempty
.
I tend to think kubectl apply semantics are more useful when a human is on the other side of the client.
Thanks for weighing in. This seems to confirm my suspicion that there are people out there who would reasonably be surprised by Metacontroller giving up so easily on enforcing what you ask it to.
@danisla wrote:
Can we have both and let the author pick the appropriate behavior via the controller config?
We could do that, but I'm worried that it will be difficult to know when you should use that setting, especially if the failure mode for using the wrong setting is endless updates. It would also force you to choose "give up semantics" instead of "fight semantics" just because you wrote your hook in Go. Even if we support both semantics, the language you use really ought to be orthogonal to your choice of apply semantics.
Maybe we can do a grep survey of existing Kubernetes APIs and see if my fear that there are non-omitempty spec fields is founded. If it's only status fields, we can introduce special behavior for that section of the object. Of course, it's always possible that new API fields will be introduced that break the rule. Also, people can easily make that mistake while writing their own Go structs for custom APIs.
from metacontroller.
So my temporary workaround is to copy the child type structs from the k8s go API to my controller and just omit their Status
field so that when it's marshaled to JSON, the status field is omitted. This feels like a hack, but it emphasizes the importance of knowing how your data gets serialized with Go.
Maybe we can communicate some of these golang and k8s API caveats in the metacontroller docs?
from metacontroller.
@danisla Although you may observe the order of map fields changing in the last-applied-configuration
annotation over time, the way we check diffs to see if an update is required should ignore map ordering. So if your hook response is the same except for map ordering, Metacontroller should decline to do an update.
What you might be seeing is that something else (perhaps one of the issues above) is causing endless updates, and the map order changing is a side effect but not actually causing the updates. If you've really eliminated all the known causes of update hot loops, and Metacontroller still is choosing to do an update solely because you returned map items in a different order, that would be a bug in Metacontroller. Please let me know if you have a reproduction for that.
from metacontroller.
Thanks @enisoc, I'll keep debugging now that I know the map order shouldn't affect it.
from metacontroller.
Hi, I'm running into this hot loop as well could it be because the status of my resources have conditions that include a lastProbeTime
? Something like this:
status:
conditions:
- lastProbeTime: "2019-05-22T14:15:49Z"
status: "True"
type: Foo
- lastProbeTime: "2019-05-22T14:15:49Z"
status: "False"
type: bar
from metacontroller.
BTW, I'm using https://github.com/tidwall/gjson and https://github.com/tidwall/sjson so the problem is not the same as originally reported.
I have a const
with an empty template of the child resource I want to generate and then I use sjson.Set()
to populate it.
something like this:
const cnameCRD = `{"apiVersion": "example.com/v1","kind": "Cname","metadata": {"name": "","namespace": ""},"spec": {"record": "","target": ""}}`
// newCname Returns a new CNAME CRD in json format
func newCname(name, cname string) string {
nameParts := strings.Split(name, ".")
crd, _ := sjson.Set(cnameCRD, "metadata.name", nameParts[0])
crd, _ = sjson.Set(crd, "metadata.namespace", nameParts[1])
crd, _ = sjson.Set(crd, "spec.record", cname)
crd, _ = sjson.Set(crd, "spec.target", name)
return crd
}
from metacontroller.
And this is my response to metacontroller:
{
"status": {
"cnameRef": "",
"hostsStatus": [],
"conditions": [
{
"lastProbeTime": "2019-05-22T14:48:06Z",
"status": "False",
"type": "GSLBResolves"
},
{
"lastProbeTime": "2019-05-22T14:48:06Z",
"status": "False",
"type": "CNAMEResolves"
}
]
},
"children": [
{
"apiVersion": "example.com/v1",
"kind": "Cname",
"metadata": {
"name": "service-1",
"namespace": "new-crd"
},
"spec": {
"record": "new-crd.dev.example.com",
"target": "service-1.new-crd.dev.example.com"
}
}
]
}
from metacontroller.
@luisdavim I have been using k8s.io/apimachinery's unstructured.Unstructured to send the desired response. So far I dont think I have encountered above issues. In fact I am avoiding use of k8s APIs totally. I am once again relying on unstructured's helper functions to extract the values when needed versus going typed go way.
from metacontroller.
@AmitKumarDas cool, do you have an example?
from metacontroller.
https://github.com/mayadata-io/cstorpoolauto cc @luisdavim
Let me know if you need exact snippets
from metacontroller.
Related Issues (20)
- Support for custom sub-resources
- --client-config-path option does not work HOT 1
- Feature Request: Support intentional orphaning of CompositeController children HOT 1
- Propose Metacontroller to be included in Kubernetes core HOT 1
- CRD updates require restart of metacontroller
- CompositeController - Infinite loop reconciling when having multiple volumemounts with same named configmap HOT 2
- Is there a way to trigger a resync? HOT 2
- Is this repo maintained? HOT 34
- New rabbitMQ operator - attaching my project
- [feature request] multiple metacontroller with object sharding strategy HOT 1
- [bug] Metacontroller crash when one of the CompositeControllers misbehave HOT 6
- Converting/Migrating from one Custom Resource to another HOT 3
- Make it easier to take over existing resources HOT 2
- Have a separate resync-after-error period flag HOT 1
- failed to sync CompositeController using apiextensions.k8s.io/v1
- System rapidly making calls to sync hook HOT 1
- Redis Operator Example
- Metacontroller omitting updated status in subsequent sync HOT 1
- Image metacontroller:v0.4.0 is vulnerable, CVEs' found in clair scan HOT 1
- Service-per-pod example: no matches for kind "Deployment" in version "apps/v1beta1"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from metacontroller.