Fix Octavia LB provisioning_status from PENDING_UPDATE or ERROR to ACTIVE

Find broken Octavia Loadbalancer / Amphora VMs
The "loadbalancer amphora failover" command are avaiable since OSC Stein release: https://docs.openstack.org/releasenotes/python-octaviaclient/stein.html but the python-octaviaclient 1.7 OSC also works with OS Rocky realase. (pip install -U "python-octaviaclient==1.7.0")

DB_PASS=$(grep octavia_database_password /etc/kolla/passwords.yml | cut -d " " -f2)
 
LB_IDS=$(openstack loadbalancer amphora list --status ERROR -c loadbalancer_id -f value)
for LB_ID in ${LB_IDS}; do
    # update LB status to ACTIVE
    mysql --host=db.service.example.com --port=3006 --database=octavia --user=octavia --password=${DB_PASS} \
        --execute="update load_balancer set provisioning_status = 'ACTIVE' where id = '${LB_ID}'"
 
    # get amphora ID
    AMPHORA_ID=$(openstack loadbalancer amphora list --status ERROR --loadbalancer ${LB_ID} -c id -f value)
 
    # failover broken amphora VM
    openstack loadbalancer amphora failover ${AMPHORA_ID}
 
    sleep 60
done

Manual debug Amphora VMs

# List broken LoadBalancer instances
openstack loadbalancer list --provisioning-status ERROR
openstack loadbalancer list --provisioning-status PENDING_UPDATE
 
# List broken Loadbalancer VMs
openstack loadbalancer amphora list --provisioning-status ERROR
openstack loadbalancer amphora list --role STANDALONE

Show Loadbalancer state

LB_ID=0ce30f0e-1d75-486c-a09f-79125abf44b8
 
# List LoadBalancer details
openstack loadbalancer show ${LB_ID}
 
# List LoadBalancer VMs details
openstack loadbalancer amphora list --loadbalancer ${LB_ID}
 
# show project
PROJECT_ID=$(openstack loadbalancer show -c project_id -f value ${LB_ID})
openstack project show -c name -f value ${PROJECT_ID}
 
# show domain
DOMAIN_ID=$(openstack project show -c domain_id -f value ${PROJECT_ID})
openstack domain show -c name -f value ${DOMAIN_ID}

Manual update provisioning_status from PENDING_UPDATE / ERROR state to ACTIVE in Octavia Database
Currently there is no OpenStack CLI command to change the provisioning_status with the CLI

DB_PASS=$(grep octavia_database_password /etc/kolla/passwords.yml | cut -d " " -f2)
 
mysql --host=db.service.example.com --port=3306 --database=octavia --user=octavia --password=${DB_PASS} \
    --execute="update load_balancer set provisioning_status = 'ACTIVE' where id = '${LB_ID}'"

Trigger manual LoadBalancer failover
This will cause a short interruption of the LB service since both VMs will by recreated

# manually failover loadbalancer
openstack loadbalancer failover ${LB_ID}
 
# manually failover amphora vm
AMPHORA_ID=$(openstack loadbalancer amphora list --loadbalancer ${LB_ID} --status ERROR -c id -f value)
openstack loadbalancer amphora failover ${AMPHORA_ID}

Check LoadBalancer state

# check LoadBalancer state
openstack loadbalancer show ${LB_ID}
 
# check LoadBalancer VMs state
openstack loadbalancer amphora list --loadbalancer ${LB_ID}

Debug

# list all loadbalancer instances
openstack loadbalancer list
 
# list all Amphora instances
openstack loadbalancer amphora list
 
# List all Amphora VMs
openstack server list --all --long --name amphora
 
# get LoadBalancer count
openstack loadbalancer list -c id -f value | wc -l
 
# get Amphora VMs count
openstack loadbalancer amphora list -c loadbalancer_id -f value | wc -l
openstack server list --all --long --name amphora -c Name -f value | wc -l
 
# restart LB VM
openstack server reboot --soft amphora-${AMPHORA_ID}
 
# show LB VM console
openstack console log show amphora-${AMPHORA_ID}