Potential Data Loss from lvm_by_vg.sh use of vgremove --removemissing about resource-agents

[lhh]

@davidvossel - I think Jon Brassow wrote that, but I don't recall the specifics of the implementation.

[phedders]

Does "vgreduce --removemissing" not seem to be quite a brutal way to work with a volume group? I would normally only ever expect to see that as an emergency recover from a broken disk type of "get me out of trouble somewhat". I would never expect to see that automated and part of routine cluster management! Am I missing something?

[davidvossel]

Yeah, I completely agree with you. This is not something I'd expect the agent to be doing behind the scenes on me at all.

[phedders]

When I first saw the Cleanup messages in my cluster logs I was "intruiged", especially since it took 7 minutes - 7 minutes during which my lvm/archives directory got multiple writes... so I found a variation (RHEL5) of the code above and my first thought was "how EVIL!". I was clinging on to the hope that it wasn't doing what I thought it does, even though I have evidence that it did what I thought it would do... and that there had to be a really good explanation and reason for why it is there.

Thanks. So what are the steps to getting this reviewed and replaced with code that would be less disruptive?

[davidvossel]

Well, I'm not sure there is code that is less disruptive that handles attempting to "clean up" that sort of failure. We need to have that discussion with the person who introduced the logic (Jon Brassow) to make sure though. Maybe you have some suggestions?

perhaps the vgchange --partial option could be of use here.

[phedders]

I'm not sure I understand enough of the purpose of this code to give meaningful suggestions yet...

[phedders]

Does anyone know if Jon Brassow is still involved and likely to reading these issue logs? If not I'll try and give him a prod.

Thanks

[davidvossel]

I'll point him to this discussion.

[jbrassow]

The following commit removes the 'vgreduce --removemissing --force' command in 'vg_start_single' and replaces it with an LV-by-LV approach to activating the logical volumes. RAID LVs are handled differently than 'mirror' LVs and non-redundant LVs that have failed devices cause the service not to start. I think this is exactly the behavior you are looking for.
518b65f

The 'CLVM' method (i.e. 'vg_start_clustered') should change to either:

1. attempt repair 1 LV at a time, as above
2. use 'vgreduce --removemissing --force --mirrorsonly '. This will only repair redundant LVs and will leave non-redundant LVs alone - failing to activate the service (rather than removing PVs under non-redundant LVs) if LVs cannot be activated.
   Both of these options will give you the behavior you are looking for and are more correct.

[phedders]

Thanks Jon - I'll test that commit as soon as I get the chance. Thanks for your time on it!

[nrwahl2]

@phedders all good I presume?