sapcc / octavia-f5-provider-driver Goto Github PK

License: Apache License 2.0

Python 98.49% Mako 0.07% Smarty 0.06% Shell 0.55% HCL 0.49% Mustache 0.34%

octavia-f5-provider-driver's Introduction

Octavia Provider Driver for F5 BigIP devices

This is the Octavia provider driver for F5 BigIP appliances. It communicates with BigIP devices via the declarative AS3 API. The worker uses the driver-agent API, but it hooks more deeply into Octavia (similar to the Octavia Amphora driver) than the Provider Agents concept permits, e. g. by accessing the database directly.

Modules

octavia_f5/api: Driver, running in Octavia main process (extends AmphoraProviderDriver)
octavia_f5/cmd: Entry points for house-keeping and status manager.
- house_keeping: DB cleanup. Uses Octavia class DatabaseCleanup
octavia_f5/controller: Communication with BigIP device
- status_manager: Manages table entries representing BigIP devices
- controller_worker: REST endpoints for Octavia, synchronization loop
- sync_manager: Builds AS3 declarations and sends them to the BigIP device.
- status: Methods for setting status in database. Used by controller_worker.
db: Repository classes (CRUD abstractions over sqlalchemy ORM objects)
network: Layer 2 network drivers (Neutron hierarchical port binding driver, no-op driver)
restclient: Classes for building AS3 declarations. Used by sync_manager and status_manager.

Special database handling

This provider driver uses Octavias mariadb database to store some data, but doesn't define any new tables. Instead, otherwise unused tables are used in a specific way:

The amphora table is used in two ways:
- For each load balancer an amphora entry is created. This is done to prevent problems with Octavias health manager, which makes assumptions about amphora entries.
  - compute_flavor holds the name of the device the load balancer is scheduled to. This can be used to query the device via openstack loadbalancer amphora show $LB_ID.
  - Since an amphora table entry is never updated as long as its respective load balancer lives, the updated_at field will always be null until the load balancer is being deleted, which will update the amphora entry status to DELETED as well.
- For each F5 device that is managed by a provider driver worker a special entry is created in the amphora table.
  - compute_flavor holds the name of the managed F5 device
  - cached_zone holds the hostname
  - load_balancer_id will always be null
  - role (must contain one of the values defined in the amphora_roles table) holds information about whether the device is in active status (MASTER) or standby status (BACKUP)
  - status (must contain one of the values defined in the provisioning_status table) holds device state.
    - ALLOCATED means the the device is offline (no entry in device status response)
    - READY means the device is online
    - BOOTING if it was offline and is now back online. In this case the device receives a full sync and the status is set to READY.
  - If vrrp_interface is set to 'disabled' for a given F5 amphora entry, the scheduler will not take that device into account when scheduling new load balancers.
  - vrrp_priority holds the amount of listeners on that device

F5-specific configuration options

There are lots of F5-specific configuration options. They can be found in octavia_f5/common/config.py.

If agent_scheduler in the [networking] section of the configuration is set to loadbalancer, new load balancers are scheduled to the device with the least load balancers. This is the default. If it is set to listener, new load balancers are scheduled to the device with the least listeners.

Listener type to AS3 service class mapping

Mapping happens in octavia_f5/restclient/as3objects/service.py.

Openstack listener type	AS3 service class	Notes
TCP	Service_L4	Uses L4 acceleration
UDP	Service_UDP
HTTP	Service_HTTP
HTTPS	Service_L4	Uses L4 acceleration, since HTTPS simply gets passed through without decryption
PROXY	Service_TCP	Does not use L4 acceleration, since it's incompatible with the Proxy Protocol iRule
TERMINATED_HTTPS	Service_HTTPS

octavia-f5-provider-driver's People

Contributors

Stargazers

Watchers

Forkers

notandy sungwon-ahn

octavia-f5-provider-driver's Issues

Scenario: Disappearing/reappearing device

Scenario: Device disappears (maintenance, ...), comes back online after 1h or so.
Full sync needed. Automate that.

Assign networking-f5 selfip ports to service project

f5_networking: associate selfip device_id with subnet, not listener port

Integrate ML2 component

as soon as the ML2 component is finished

member_batch_update: No-Op if all members deleted

Due to the implementation of the default endpoints, the member_batch_update doesn't do anything if all members are deleted via member batch update:

Batch updating members: old='[]', new='[]', updated='[]'...
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server [-] Exception during message handling: IndexError: list index out of range
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/local/lib/python2.7/site-packages/octavia/controller/queue/endpoint.py", line 122, in batch_update_members
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server     old_member_ids, new_member_ids, updated_members)
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server   File "/var/lib/openstack/src/octavia-f5-provider-driver/octavia_f5/controller/worker/controller_worker.py", line 356, in batch_update_members
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server     pool = updated_members[0][0].pool
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server IndexError: list index out of range
2020-04-07 09:21:51,252.252 33 ERROR oslo_messaging.rpc.server

The index out of bounds error is fixed by d79e88a, but the operation is now no op. This is later fixed by the reconilliation loop.

Proposal:
Diverge from the default endpoints and implement own worker process (since endpoints are hard-coded by upstream worker implementation)

Removal/Deletion of Healthmonitors doesn't work

Our AS3 implementation fails to remove healthmonitors even though they are not longer referenced in the declaration. This looks like a Bug in AS3. I will test this with newest AS3 code, else we need a workraround that first detaches the healthmonitor, and then deletes it.

report object status via octavia-lib Driver Support Library

https://docs.openstack.org/octavia/latest/contributor/guides/providers.html

Octavia API functionality

This issue is for tracking and verifying features of Octavia API working with the F5 driver

Check if HTTP_COOKIE is correctly inserting headers

And is compatible to old f5 agent, see here:
https://docs.openstack.org/api-ref/load-balancer/v2/?expanded=create-member-detail,create-pool-detail#create-pool

Properly support pending_delete in the _refresh periodic sync

Currently, the periodic refresh sync picks up all loadbalancers in Pending and iterates with ACTIVE status for every resource even though it's still in pending_delete. Refresh needs to pick up pending_delete items and retry to delete them.

LoadBalancer object not hashable

Found at revision: 05bae2a

Steps to reproduce:

Clear Octavia database
Start Octavia API and octavia-f5-provider-driver
Create a load balancer via the API

Expected behavior:
Load balancer is created without errors

Observed behavior:
Exception at octavia_f5/controller/worker/controller_worker.py:102.
Instance of LoadBalancer cannot be added to set due to not being hashable.

Stack trace:

[-] Failed to call immediate 'octavia_f5.controller.worker.controller_worker.ControllerWorker.pending_sync' (it runs every 120.00 seconds): TypeError: unhashable type: 'LoadBalancer'
Traceback (most recent call last):
  File "/home/laerling/.virtualenv/octavia/lib/python3.7/site-packages/futurist/periodics.py", line 290, in run
    work()
  File "/home/laerling/.virtualenv/octavia/lib/python3.7/site-packages/futurist/periodics.py", line 64, in __call__
    return self.callback(*self.args, **self.kwargs)
  File "/home/laerling/.virtualenv/octavia/lib/python3.7/site-packages/futurist/periodics.py", line 178, in decorator
    return f(*args, **kwargs)
  File "/home/laerling/.virtualenv/octavia/lib/python3.7/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
    return f(*args, **kwargs)
  File "/home/laerling/workspace/octavia-f5-provider-driver/octavia_f5/controller/worker/controller_worker.py", line 102, in pending_sync
    lbs.add(lb)
TypeError: unhashable type: 'LoadBalancer'

Implement driver-agent inside of octavia-worker status-manager

Currently, driver-agent runs in parallel, listens to json status updates via domain socket and updates status updates to the database - although every json-call blocks.

Integrate status-updates into inbuilt status_manager.

Don't set loadbalancer to error for unknown error condition

Due to the often diverse response pattern of AS3, remove error-setting of loadbalancers and therefor enable the reconcilation loop to retry to update an AS3 schema indefinitely.

Following reasons observed in production clusters:

AS3 often returns temporarly strange errors due to readiness / business (device in configuration or AS3 endpoint not reachable)
AS3 error messages proxied from BigIP tmsh are often hard to parse correctly
Only set entity to error for known errors like broken TLS certificate.

Failover loadbalancer to different hosts

Enable ability to failover loadbalancer to different hosts:

reuse loadbalancer failover api
invalidate vip_port host mapping caching
update listener_ip

get_all_from_host doesn't work as expected

the sqlalchemy filter for amphora.compute_flavor doesn't work as expected, use server_group_id of loadbalancer for identifying the scheduled host.

Mark listener as active

Listener are for now not marked as active even though they've been created.

Write unit tests for AS3 functionality

Write regression tests for at least all AS3 functionality.

LB remains OFFLINE as long as other LB is erroneous

Steps to reproduce

Create a loadbalancer that is erroneous (before Octavia PR #2 is merged, this can be achieved by creating a member with the same address as the LB).
Create a second LB in the same project that is not erroneous

Expected behavior

The second loadbalancer is created (and comes ONLINE) without respect to the first erroneous LB.

Observed behavior

Both loadbalancers are being provisioned within the same AS3 declaration. Due to the first one being erroneous, the declaration fails. Thus, the second LB remains OFFLINE.

Proposed fix

Separate AS3 declarations somehow. For example, first create a declaration for everything that's PENDING_CREATE and then (and only then) for everything that's PENDING_UPDATE.

X-Forwarded-For header: Differentiate between replacing/appending

Customer request for both, an iRule that removes/inserts client IP and one that appends client IP if there is existing XFF with some IPs already in it?

Something like:

X_FORWARDED_FOR_INSERT = """when HTTP_REQUEST {
    HTTP::header remove "X-Forwarded-For"
    HTTP::header insert "X-Forwarded-For" [getfield [IP::remote_addr] "%" 1]
}"""
 
X_FORWARDED_FOR_APPEND = """when HTTP_REQUEST {
  if {[HTTP::header exists X-Forwarded-For]}{
    HTTP::header replace X-Forwarded-For "[HTTP::header X-Forwarded-For], [getfield [IP::client_addr] "%" 1]"
  } else {
    HTTP::header insert X-Forwarded-For [getfield [IP::client_addr] "%" 1]
  }
}"""

iRules would conflict though. If we have to choose one, replace the default XFF behaviour with the append edition.

Support of Default profiles for listener

Support configuration of default profiles for listeners:

HTTP
HTTP Compression
Fast L4
TCP
UDP
Multiplex (OneConnect)

Make status_manager aware which bigIP is active

Currently status_manager scrapes statistics from a single bigip, but since they can failover or even run in an active-active scenario, the status manager needs to scrape both device pairs and figure out what to submit

Related to #4

Use Apache WSGI for octavia API

Fail if no ESD config file found

Currently, octavia-f5-provider-driver does not indicate a missing ESD[1] configuration.

I'm not sure but I think it also showed erroneous behavior due to a missing ESD config.
I wasn't able to reproduce it up to now, so for now this issue is labeled as enhancement only.

In octavia_f5.utils.esd_repo.EsdJSONValidation.__init__ warn (fail?) when no config files have been found.

[1] https://clouddocs.f5.com/training/community/private-cloud/html/class1/module3/lab1.html

Prometheus Metric for AS3 Version

Trigger ESDs via something else than specially named L7policies

We use F5 ESDs (Enhanced Services Definitions) to extend LB functionality beyond L7 policies.
In F5 neutron-lbaas plugin those ESDs are triggered by creating a L7policy with the same name as the ESD.
This leads to L7rules within the L7policy being ignored, thus abusing the whole idea of L7 policies.

That's why we want to trigger ESDs with something else than L7policies and get rid of ESDs, if necessary.

We tried using Octavia Flavors instead, but every LB can only use one flavor.

A better idea is to use tags.
Tags can be used on the following Octavia objects:

loadbalancer
listener
pool
member
L7rule
L7policy
healthmonitor

Implement housekeeping

'Fork' octavia/cmd/house_keeping.py for F5 provider driver to delete DB entries representing deleted LBs. Use prometheus for counting deletions.
Basically only thing that's needed is octavia.controller.housekeeping.house_keeping.DatabaseCleanup.

Python3 Support

Cleanup orphaned partition

Detect and delete partition that shouldn't be exist on the device anymore.

HTTP 422 when creating listener with insert_header

F5 is responding with HTTP status 422 when trying to create listener with insert_header via Octavia API.

Steps to reproduce:

Create load balancer via Octavia API
Create listener on load balancer with insert_header field set, e. g.:

{
    "listener": {
        "loadbalancer_id": "a67bf7f7-4750-4ca3-9544-ae74cfeefbec",
        "protocol": "HTTP",
        "protocol_port": "1234",
        "insert_headers": {
        	"X-Forwarded-For": "true"
        }
    }
}

Observations:

Octavia responds with an error if the boolean literal is unquoted (true instead of "true"), even though unquoted booleans are idiomatic JSON. In this case the provider driver does not receive anything.
After creating the listener with the above JSON, the database table listener has the following content in its column insert_header: <80>^D<95>^] }<94><8C>^OX-Forwarded-For<94><8C>^Dtrue<94>s. (escape sequences are an artifact of displaying the output in less)
The F5 responds with HTTP status 422 (supported media type, good request, but unable to process semantically)

If Octavia not accepting true is the cause of all this, I fear we'll have to file an issue upstream...

decrement_quota must be implemented in the backend

amphora driver uses decrement_quota after deletion of objects, this also needs to be incorporated into octavia-f5-provider

nlbaas2octavia: Migration from f5-openstack-agent to octavia-f5-provider

We need a working migration path for existing loadbalancers in using the neutron lbaas f5-openstack-agent.

I backported the nlbaas2octavia driver from stein to our queens release:
https://github.com/sapcc/neutron-lbaas/tree/stable/queens-m3/tools/nlbaas2octavia

Features needed:

Migration of all lbaas objects
Migration of neutron listener ports
Migration of selfip ports and discovering of used / unused selfips
~~Migration of ESD policies (if applicable)~~

Redirect to custom sorry page when no pool member available

Extended policy for the ELB which will redirect traffic to the custom sorry page in case all nodes in lb's pool are dead (response timeout or status code check).

Make worker fallback between F5 pairs

In case connection to one F5 breaks, worker should fallback to other one.

Set default loadbalancer status to "active"

Since this seems to be the expected default behaviour of lbaasv2

Deletion of loadbalancer doesn't work

Caching of segmentation ids in agent

Currently f5 agent needs to fetch segmentation_id from neutron for defaultRouteDomain on every refresh

Implement timeout for all REST related calls

It looks like the agent can hang in rest calls if traffic is interrupted, e.g. due to device reboot.

Migration mode: only sync passive device

we need an option to only sync the passive device as a special case of migration.

Remove ESD-L7policy functionality once all customers are using tags

For backwards compatibility, Enhanced Service Definitions can still be triggered via specially named L7policies. Remove this once all customers have migrated to using tags instead.
The corresponding code is here

I labelled this migration, since it technically is part of it.

B1 seems to be building some tools for notifying customers who match some pattern. So after we start ESD migration and wrote a blog post, we can notify them.

Implement support for dockerized AS3 wrapper

https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/latest/userguide/as3-container.html

Implement prometheus/client_python or generic metrics plugin

We use Prometheus for alerting and metrics collection, export useful metrics via prometheus python library: https://github.com/prometheus/client_python

Customized TLS Settings

Resolve ESDs to Octavia flavors at migration time

Fork the official F5-LBaaS-to-Octavia migration tool and add functionality to resolve ESDs at migration time instead of maintaining ESD code in F5 provider driver for an unforseeable time.
Maybe the ESD interpretation code currently in use can be reused in the fork.

Error creating policy endpoints

2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server [-] Exception during message handling: TypeError: __init__() missing 1 required positional argument: 'type'
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server   File "/Users/d072895/.virtualenvs/stein_python3/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server   File "/Users/d072895/.virtualenvs/stein_python3/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server   File "/Users/d072895/.virtualenvs/stein_python3/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server   File "/Users/d072895/Workspace/stein/octavia-f5-provider-driver/octavia_f5/controller/worker/controller_worker.py", line 109, in refresh
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server     tenant_update(network_id, loadbalancers, self.bigip)
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server   File "/Users/d072895/Workspace/stein/octavia-f5-provider-driver/octavia_f5/controller/worker/f5agent_driver.py", line 62, in tenant_update
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server     m_policy.get_endpoint_policy(l7policy)
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server   File "/Users/d072895/Workspace/stein/octavia-f5-provider-driver/octavia_f5/restclient/as3objects/policy_endpoint.py", line 82, in get_endpoint_policy
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server     args['conditions'] = [_get_condition(l7rule) for l7rule in l7policy.l7rules]
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server   File "/Users/d072895/Workspace/stein/octavia-f5-provider-driver/octavia_f5/restclient/as3objects/policy_endpoint.py", line 82, in <listcomp>
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server     args['conditions'] = [_get_condition(l7rule) for l7rule in l7policy.l7rules]
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server   File "/Users/d072895/Workspace/stein/octavia-f5-provider-driver/octavia_f5/restclient/as3objects/policy_endpoint.py", line 50, in _get_condition
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server     compare_string = Policy_Compare_String(operand=operand, values=[l7rule.value])
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server TypeError: __init__() missing 1 required positional argument: 'type'
2019-12-17 12:00:50.786 61626 ERROR oslo_messaging.rpc.server

Make worker fallback between F5 pairs

In case connection to one F5 breaks, worker should fallback to other one.

Use dedicated BigipAS3RestClient instance for each BigIP

I don't like the handling of self.active_bigip inside the as3restclient. As I said, this seems very hacky and I would like to see two instances of the as3restclient for each device itself.

Use dedicated BigipAS3RestClient instance for every BigIP device instead.
ControllerWorker would have to create and maintain all the instances.
The failover mechanism would therefore have to be moved into ControllerWorker as well (?)

Write docs: README, configuration, setup

Hi, I'm pleased to find this software. However it is a little cumbersome, to read all the python code, to get an idea of how it is supposed to be configured, where do I find some documentation?

Thx

Implement unit tests

Pretty much what it says in the title.

Multiple agent support and Agent scheduler

Currently there is just one f5 agent for octavia. We need a scheduler that schedules new loadbalancer objects to a specific F5 agent depending on load. We maybe could exploit the amphorae table to create the necissary mappings.

Fix hardcoded strings like this

Forbid pool members with same address as any VIP in the network

From F5 log: warning: [appsvcs] {"status":422,"message":"declaration is invalid","errors":"declaration has duplicate values in members","level":"warning"}
This is caused by customers using invalid member addresses.
Build a check in the octavia-api to disallow members with the same address as any VIP in the same network.

Check at member creation time. This is possible because a member must be tied to a VIP at creation time: member --> pool --> (listener -->) loadbalancer --> vip_network_id/vip_subnet_id/vip_port_id
- What about changes to member after creation?
- What about changes to pool/listener/loadbalancer after creation?
- Do vip_subnet_id and vip_port_id have to be cared about as well?
Set provisioning_status accordingly.
- Does the member have to be deleted to prevent octavia-f5-provider-driver from syncing it after a restart?
How to report back the error? Speak to our frontend developer. Possibilities (simplest first):
- display a hint about invalid IP addresses every time a member is in an unexpected provisioning_status
- abuse some field of DB table members? E. g. ip_address (could that interfere with backend?), monitor_*, name, protocol_port, subnet_id, updated_at, weight
- abuse some provisioning_status predefined value not usually used for members?
- custom provisioning_status?
- ...