network | panticz.de

Firewall

NAT
https://tailscale.com/blog/how-nat-traversal-works/

opnsense
https://opnsense.org/download/

OpenStack: Recreate neutron port

Get broken / INACTIVE ports

DB_NAME=neutron
DB_USER=neutron
DB_PASS=$(grep neutron_database_password /etc/kolla/passwords.yml | cut -d " " -f2)
mysql --host=${DB_HOST} --password=${DB_PASS} --port=6033 --user=${DB_USER} --database=${DB_NAME}

select port_id, host from ml2_port_bindings where status = 'INACTIVE';
select port_id, host from ml2_port_bindings where status = 'INACTIVE' and host = 'com3-dev';
select * from ml2_port_bindings where port_id = "65db0c73-1533-4c36-9fd4-79b5e4e9704c";
#select * from ml2_port_binding_levels where port_id = "65db0c73-1533-4c36-9fd4-79b5e4e9704c" \G

# debug
PORT_IDS=$(mysql --host=${DB_HOST} --password=${DB_PASS} --port=6033 --user=${DB_USER} --database=${DB_NAME} \
  -se "select port_id from ml2_port_bindings where status = 'INACTIVE'")
echo ${PORT_IDS}

# get VMs
for PORT_ID in ${PORT_IDS}; do
    echo ${PORT_ID}
    openstack server show -c name -c "OS-EXT-SRV-ATTR:host" $(openstack port show -c device_id -f value ${PORT_ID})
done

Recreate port

SERVER_ID=f4b8828d-a717-49f6-8ad3-7b0bb1921123

nova interface-list ${SERVER_ID}

PORT_ID=57944199-014c-4907-9d6e-1fb233151234

# debug
openstack port show ${PORT_ID} -c name -c network_id -c mac_address -c fixed_ips -c security_group_ids -c project_id -c device_id

get port data

OpenStack: Floating IP port forward (in development / experimental)

List floatin IPs

openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Floating Network                     | Project                          |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| dc049c28-6562-4c37-834b-d3a612d4b580 | 1.2.3.4        | None             | None                                 | 39583230-154f-4b56-a56e-2fd83c9986ce | 1eede1bdc28344f3acf6b48b232e406f |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+

List VMs

openstack server list
+--------------------------------------+---------------------------------------------------+--------+---------------------------------+-----------------------------------+-----------+
| ID                                   | Name                                              | Status | Networks                        | Image                             | Flavor    |
+--------------------------------------+---------------------------------------------------+--------+---------------------------------+-----------------------------------+-----------+
| d278cb52-d258-491b-8787-25aaab75f84c | dev-u1910                                         | ACTIVE | dev-net=10.0.1.9                | Ubuntu 19.10 minimal              | m1.micro  |
| 1da2e275-e680-4c69-be00-93871274ceb3 | dev-u1804                                         | ACTIVE | dev-net=10.0.1.12               | Ubuntu 18.04                      | m1.micro  |
+--------------------------------------+---------------------------------------------------+--------+---------------------------------+-----------------------------------+-----------+

Get port IDs for server

openstack port list --server ${SERVER_ID} -c id -f value
4f20d5a9-bdc6-4046-b1bc-291c4d4bf86d

openstack port list --server ${SERVER_ID} -c id -f value
9e583adb-618f-42ba-96ee-1a7b89a377b3

Create port forward to HTTP

LXD: Network

Configure default profile

lxc network create lxdbr0
lxc network create lxdbr0 ipv4.address=10.0.33.1/24 ipv4.nat=true ipv6.address=none
lxc profile device add default eth0 nic nictype=bridged parent=lxdbr0 name=eth0

# disable IPv6 inside of container
lxc network set lxdbr0 ipv6.address none

Configure static IP address

lxc stop c1
lxc network attach lxdbr0 c1 eth0 eth0
lxc config device set c1 eth0 ipv4.address 10.0.0.12
lxc start c1

# add NICs
lxc config device add vm-dhcp1-dev dev-mgmt-new nic name=dev-mgmt nictype=macvlan parent=dev-mgmt

ipv6

lxc network set lxdbr0 ipv6.dhcp.stateful true

ovs network

lxc profile create disk-only
lxc storage create pool1 dir
lxc profile device add disk-only root disk path=/ pool=pool1
lxc profile show disk-only
lxc launch ubuntu:18.04 ovs1 -p disk-only
lxc config device add ovs1 eth0 nic nictype=bridged parent=ovsbridge host_name=vport11
lxc network list

# test static ip
lxc launch redis r
lxc config device override r
lxc config device set r eth0 ipv4.address 10.100.0.100

Links
https://stgraber.org/2016/03/15/lxd-2-0-installing-and-configuring-lxd-212/
https://thomas-leister.de/en/container-overlay-network-openvswitch-linux/
https://stgraber.org/2016/10/27/network-management-with-lxd-2-3/

Mellanox: automatic firmware update

Auto / Online firmware update
# Search latest mlxup version
http://www.mellanox.com/page/mlxup_firmware_tool

URL=https://www.mellanox.com/downloads/firmware/mlxup/4.26.0/SFX/linux_x64/mlxup

# Download mlxup
wget ${URL} -qO /tmp/mlxup
chmod +x /tmp/mlxup

# auto flash all Mellanox network cards
/tmp/mlxup -y

Manual firware update

Download firmware
http://www.mellanox.com/supportdownloader/

mst start

# get card description
flint -d /dev/mst/mt4103_pci_cr0 dc | grep Desc

for MST in $(mst status | grep "mst" | cut -d" " -f1); do
    flint -d ${MST} query #| egrep "FW Version|UEFI|PSID"
    echo
done

wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_34_5000-MCX311A-XCA_Ax-FlexBoot-3.4.521.bin.zip -P /tmp
apt-get install -y unzip
unzip /tmp/fw-ConnectX3-rel-*-MCX312A-XCB_A2-A6-FlexBoot-*.bin.zip -d /tmp/

Update firmware

for MST in $(ls /dev/mst/mt*_pci_cr*); do
    echo ${MST}
    flint -y -d ${MST} -i /tmp/fw-ConnectX3Pro-rel-2_42_5000-MCX312B-XCC_Ax-FlexBoot-3.4.752-UEFI-14.11.46.bin b
done

Hot reload firmware (without reboot)
https://docs.mellanox.com/pages/viewpage.action?pageId=25137354

#mlxfwreset -d /dev/mst/mt4117_pciconf0 query
mlxfwreset -d /dev/mst/mt4117_pciconf0 reset

# show firmware version
for MST in $(mst status | grep "mst" | cut -d" " -f1); do     flint -d ${MST} query | egrep "FW Version"; done

Links
http://www.mellanox.com/supportdownloader/
http://www.mellanox.com/page/firmware_table_ConnectX3EN

HAProxy

Install
https://haproxy.debian.net/#?distribution=Ubuntu&release=focal&version=2.1

sudo apt install -y haproxy

Check status

systemctl status haproxy

Include configurations from /etc/haproxy/haproxy.cfg directory

sed -i 's|[#]*CONFIG=.*|CONFIG=/etc/haproxy/haproxy.cfg.d|g' /etc/default/haproxy
service haproxy restart

Check configuration / Debug

haproxy -c -f /etc/haproxy/haproxy.cfg
service haproxy reload
tail -f /var/log/haproxy.log

Loadbalancer
http://www.loadbalancer.org/blog/category/haproxy/

Prometheus
https://www.haproxy.com/blog/haproxy-exposes-a-prometheus-metrics-endpoint/

ACL

acl valid-ua hdr(user-agent) -f exact-ua.lst -i -f generic-ua.lst test
http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7
https://www.haproxy.com/de/blog/introduction-to-haproxy-acls/

Configuration

/etc/haproxy/haproxy.cfg

...
frontend www1
    bind :80
    mode http
    acl whitelist src -f /etc/haproxy/whitelist.lst
    acl all src 0.0.0.0
    use_backend backend1 if whitelist

backend backend1
    mode http
    balance roundrobin
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1rnHost:localhost
    server web1.example.com 10.0.1.10:80
    server web2.example.com 10.0.1.11:80
    server web3.example.com 10.0.1.12:80

listen stats 
    bind :9000
    mode http
    stats enable
    stats hide-version
    stats realm Haproxy\ Statistics
    stats refresh 30s
    stats show-node
    stats auth admin:password
    stats uri /

ACL whiterlist

unbound

/etc/unbound/unbound.conf.d/forward.conf
# unbound  forward-zone output
for IP in $(consul catalog nodes | grep ctl | cut -d " " -f6); do
    echo "  forward-addr: ${IP}@53"
done
systemctl restart unbound

# disable IPv6
# /etc/unbound/unbound.conf.d/server.conf 
server:
...
   do-ip6: no

# flush zone
unbound-control flush_zone dev.example.com

Links
https://wiki.alpinelinux.org/wiki/Setting_up_unbound_DNS_server
https://calomel.org/unbound_dns.html

consul

echo '["abcdef123458"]' /var/consul/serf/local.keyring
service consul restart

# config
/etc/consul/config.json
/etc/consul/consul.d/*

# log
/var/log/syslog

CLI

consul catalog datacenters
consul catalog nodes
consul catalog services

consul monitor
consul validate /etc/consul/config.json 
consul operator raft list-peers

UI listen on external
https://stackoverflow.com/questions/35132687/how-to-access-externally-to-consul-ui

# Redirect UI to localhost
ssh -L 8500:localhost:8500 root@node1.example.com -N

# cat /etc/consul/config.json 
{
...
    "addresses": {
        "dns": "127.0.0.1",
        "grpc": "127.0.0.1",
        "http": "0.0.0.0",
        "https": "127.0.0.1"
    },
...

Remove service from specific node

ssh node1
consul services deregister -id=prometheus

Remove node
https://www.consul.io/commands/force-leave

consul force-leave com52-dev

checks
https://www.consul.io/api/agent/check.html

Get service state
https://consul.example.com/v1/agent/service/neutron

Icinga / Nagios checks

for SERVICE in $(consul catalog services); do
    echo ${SERVICE}
    /usr/lib/nagios/plugins/check_consul_service.py ${SERVICE}
    echo
done

Ansible module
https://docs.ansible.com/ansible/latest/modules/consul_module.html

Consul data directory

/var/consul/*

ESP
# remove service

Enable UEFI / PXE boot on Mellanox ConnectX NIC

Install Mellanox CLI tools (MFT)
http://www.mellanox.com/page/management_tools

#URL=https://www.mellanox.com/downloads/MFT/mft-4.15.0-104-x86_64-deb.tgz
URL=https://www.mellanox.com/downloads/MFT/mft-4.16.3-12-x86_64-deb.tgz

# install requirements
apt update
apt install -y gcc make dkms linux-headers-$(uname -r)

# download mft
wget -qO- ${URL} | tar xvz -C /tmp

# install mft
/tmp/mft-*-deb/install.sh

Enable UEFI and PXE boot

mst start
#for MST in $(ls /dev/mst/* | egrep -v '\.1'); do
for MST in $(ls /dev/mst/*); do
    echo ${MST}
    mlxconfig -d ${MST} -y set EXP_ROM_UEFI_x86_ENABLE=1
    mlxconfig -d ${MST} -y set EXP_ROM_PXE_ENABLE=1
    mlxconfig -d ${MST} q | egrep "EXP_ROM"
done

Debug: Show device state

mst status

for MST in $(ls /dev/mst/*); do
    mlxconfig -d ${MST} q
    echo
done

Ubuntu: Instlal lldpd (Link Layer Discovery Protocol)

sudo apt-add-repository universe
sudo apt install -y lldpd

# optional: enable Cisco CDP protocol
cat < /etc/default/lldpd 
DAEMON_ARGS="-c"
EOF
service lldpd restart

# get info
lldpctl

# get interface / switch / port
lldpctl | egrep "Interface|SysName|PortID"

lldpctl | egrep "Interface|SysName|PortID" | awk -F  ' *: *' '{print $2}' | awk -F  ',' '{print $1}' | paste - - - | \
  egrep "^e" | awk '{print "'"$NODE"'",$1,$2,$4}' | column -t | sort -u -k 1,2

network