sapcc / openstack-nannies

nannies to keep nova, cinder and the vcenter healthy

Openstack Nannies

This repository contains so called nannies, which take care of keeping nova, cinder and the vcenter clean and healthy. They contain a growing number of jobs, which find and cleanup inconsistencies in the nova, manila and cinder db, sync quota values, spot orphaned resources and so on. They are still in their early stages and some of the more disruptive functionalities are still in some kind of "reporting only" mode so far.

the provided Dockerfiles to build the corresponding containers contain a placeholder for the image they are based on - the placeholders will have to be replaced by a corresponding image. the below list is what we are using:

• CINDER_NANNY_BASE_IMAGE - a cinder-api image built from here https://github.com/sapcc/kolla/tree/master/docker/cinder/cinder-api
• MANILA_NANNY_BASE_IMAGE - a manila-api image built from here https://github.com/sapcc/kolla/tree/master/docker/manila/manila-api
• NOVA_NANNY_BASE_IMAGE - a nova image build from here https://github.com/sapcc/kolla/blob/master/docker/nova/nova-api/Dockerfile.j2 - it is important, that it contains this fix: https://github.com/sapcc/nova/commit/21c1f4ddf3132af4c83c738c7a7d4a840eed4b17
• VCENTER_NANNY_BASE_IMAGE - we use some internal image, which will be published soon, but some image with python support with the openstack-client and pyvmomi python packages added should do

openstack-nannies's Issues

[manila] manila-share-sync share instances should respect replicas

openstack-nannies/scripts/manila-share-sync.py

Line 473 in cc6033d

shares.join(instances, shares.c.id == instances.c.share_id))\

joins share instances on share_id and implicitly assumes this is a 1-1 relation, but a share can have multiple share instances, e.g. in a replication setup one instance exists with replica_state active and one or more instances may exist with replica_state not active [in_sync, out_of_sync ..] which could lead to surprises on the returned share_instance_id

[manila] api limit 1000

The api returns a max of 1000 entities. We have places where we list objects for all_projects and may hit that limit.

e.g.

• openstack-nannies/nannies/manila-missing-snapshot.py

  Line 69 in 5b3d963

  for _snapshot in manila.share_snapshots.list(search_opts={'all_tenants': True}):

I suggest to solve by pagination (sort + offset params) or by directly querying the database instead of API.

[manila] share-snapshot improvements

apply the same fixes from #69 to sync_share_snapshot_state()

e.g. log when a snap is skipped, fix for stuck in creating, add --debug

[manila] inconsistency cleanup should avoid working on recent items

One example:

The nanny is cleaning up inconsistencies.. But apparently a bit too eager 😄 We got a race condition, chances are quite low, but we got (un-)lucky.
What happened:
Customer deletes share network ->manila does cascading delete of share servers -> manila deletes all related network allocations aka neutron ports.
The last step fails, the network allocation cannot be found in manila, because it had been deleted before.
Reason: At the same time the hourly manila db consistency check runs and sees network allocations where the corresponding share server has been (soft-)deleted and (soft-)deletes those.

Proposed fix:
Only touch inconsistencies of items that have been deleted for at least 2 hours.

[manila] snapshots: renew client

copy the snippet to the snapshot nanny:

openstack-nannies/scripts/manila-share-sync.py

Lines 117 to 119 in 72b20d8

	# TODO: implement proper re-auth after token lifetime ended
	# Need to recreate manila client each run, because of session timeout
	self.renew_manila_client()

otherwise if the token times out, the nanny stops working until the next container restart

and: check if other scripts needs the same workaround

[manila] fix snapshot instance inconsistencies

share snapshot instances can get into the situation that they have a null value in share_instance_id column, which is inconsistent and leads to internal server errors

current workaround is below sql

update share_snapshot_instances as ssi \
join share_snapshots as ss on ssi.snapshot_id=ss.id \
join share_instances as si on si.share_id=ss.share_id \
set share_instance_id=si.id \
where ssi.share_instance_id is NULL and ssi.deleted='False';

add this to the inconsistency nanny

bonus points: find the cause - it occurred quite recently, so it may be related to sapcc/manila#114

Apache header in each script

each script should contain an apache header

[manila] share sync TASK_SHARE_STATE does not respect some 'creating' shares

Looking at the following example of a share that is pending in 'creating':

$ manila share-instance-show <share_instance_id>
+------------------------+--------------------------------------+
| Property               | Value                                |
+------------------------+--------------------------------------+
| availability_zone      | None                                 |
| created_at             | 2022-11-24T07:57:53.591416           |
| updated_at             | None                                 |
| host                   |                                      |
| status                 | creating                             |
| share_server_id        | None                                 |
| access_rules_status    | active                               |
| replica_state          | None                                 |
| progress               | None                                 |
| export_locations       | []                                   |
...
+------------------------+--------------------------------------+

• Note that the updated_at field is None, the nanny method should fallback to looking at created_at if the instance has been touched recently or if it should be handled by the nanny. see

  openstack-nannies/scripts/manila-share-sync.py

  Lines 218 to 222 in 977592d

  	if instance_updated_at is not None:
  	if is_utcts_recent(instance_updated_at, 900):
  	continue
  	else:
  	continue

• Note that the replica_state field is None, the nanny should not only work on replica_state active. see

  openstack-nannies/scripts/manila-share-sync.py

  Line 153 in 977592d

  _shares = self._query_shares(only_active_instance=True)

  and

  openstack-nannies/scripts/manila-share-sync.py

  Lines 544 to 545 in 977592d

  	if (only_active_instance == False or
  	(only_active_instance == True and replica_state == 'active')):

[manila] share sync - sync_share_size: spam on shares in state != 'available'

Rework 04029b4
changed _query_shares to include all shares instead of shares with state 'available'.

Therefore all shares are watched for size changes, but the fixing method set_share_size is only operating on available shares.
Such shares having size differences are logged forever, but not corrected. Log spam would only end if externally the size/state would be fixed.

vm_balance_nanny* metrics not initialized

Hi, most of the metrics on

openstack-nannies/nannies/vm_load_balance.py

Lines 237 to 256 in e9fd084

	mymetrics.set_metrics('vm_balance_nanny_host_size_bytes', 'des:vm_balance_nanny_host_size_bytes', ['nodename'])
	mymetrics.set_metrics('vm_balance_nanny_host_size_consume_all_vm_bytes',
	'des:vm_balance_nanny_host_size_consume_all_vm_bytes', ['nodename'])
	mymetrics.set_metrics('vm_balance_nanny_host_size_consume_big_vm_bytes',
	'des:vm_balance_nanny_host_size_consume_big_vm_bytes', ['nodename'])
	mymetrics.set_metrics('vm_balance_nanny_suggestion_bytes', 'des:vm_balance_nanny_suggestion_bytes',
	['source_node', 'target_node', 'big_vm_name', 'big_vm_size'])
	mymetrics.set_metrics('vm_balance_nanny_manual_suggestion_bytes', 'des:vm_balance_nanny_manual_suggestion_bytes',
	['source_node', 'target_node', 'big_vm_name', 'big_vm_size'])
	mymetrics.set_metrics('vm_balance_building_block_consume_all_vm_bytes',
	'des:vm_balance_building_block_consume_all_vm_bytes',
	['Building_block'])
	mymetrics.set_metrics('vm_balance_building_block_consume_big_vm_bytes',
	'des:vm_balance_building_block_consume_big_vm_bytes',
	['Building_block'])
	mymetrics.set_metrics('vm_balance_building_block_total_size_bytes',
	'des:vm_balance_building_block_total_siz_bytes',
	['Building_block'])
	mymetrics.set_metrics('vm_balance_error_count','des:vm_balance_error_count',['error_type'])
	mymetrics.set_metrics('vm_balance_too_full_building_block','des:vm_balance_too_full_building_block', ['consolidated_needed'])

only are present if the nanny was working on something or if an error occurred.

E.g. if big_vm_to_move_list is empty

openstack-nannies/nannies/vm_load_balance.py

Line 162 in e9fd084

if len(big_vm_to_move_list) > 0:

, vm_balance_error_count will not be present, but I would expect it to be there with value '0'

[manila] reset reserved quota

Reserved quota is set when share is being created or deleted. It's value should be reset when the process finishes. If the process were broken, the values may stay non-zero. The nanny should reset the values. Check the -ing shares while reset the values.

[nova] nanny fails reading the nova config

After Nova was upgraded to Xena release the nova-nanny fails when reading the nova configuration. Either with error 2023-05-23 06:56:54,498 ERROR: Check Nova configuration file. or with sed: can't read /etc/nova/nova.conf.d/db.conf: No such file or directory

[manila] implement db_cleanup

Find objects that belong to projects that already have been deleted.

This is already done for cinder + nova.

[manila] snapshot reset state error

2023-06-21 07:15:26,027 share_snapshot_instance_reset_state(snapshot_instance_id=<ID>, state=error): API version '2.7' is not supported on 'manilaclient.v2.share_snapshot_instances.ShareSnapshotInstanceManager.reset_state' method.
Traceback (most recent call last):
  File "/scripts/manilananny.py", line 112, in share_snapshot_instance_reset_state
    self.manilaclient.share_snapshot_instances.reset_state(snapshot_instance_id, state)
  File "/var/lib/openstack/lib/python3.8/site-packages/manilaclient/api_versions.py", line 390, in substitution
    raise exceptions.UnsupportedVersion(
manilaclient.common.apiclient.exceptions.UnsupportedVersion: API version '2.7' is not supported on 'manilaclient.v2.share_snapshot_instances.ShareSnapshotInstanceManager.reset_state' method.

[manila] share sync does not get all prom results in big regions

https://github.com/sapcc/openstack-nannies/blob/ece24002e23711ba669a2144fa4186fedb1170bc/scripts/manila-share-sync.py#L464C6-L497 may have a very large result up to the point, where not everything is in anymore - e.g. leading to skips like 994e3f6#diff-851aa4a582500825154e8a29ea39eba4d3dc4a920b446bb0aece22dc821c813aR164-R167

IDEA: look at the pattern of castellum with the netapp-scout