mellanox

Mellanox: Install pre-build driver

https://www.mellanox.com/products/ethernet-drivers/linux/mlnx_en

VERSION=5.7-1.0.2.0
URL=http://www.mellanox.com/downloads/ofed/MLNX_EN-${VERSION}/mlnx-en-${VERSION}-ubuntu$(lsb_release -rs)-$(uname -p).tgz
 
wget ${URL} -q -O /tmp/${URL##*/}
tar -C /tmp -xzf /tmp/mlnx-en-*-ubuntu*.tgz
cd /tmp/mlnx-en-${VERSION}-ubuntu*/
 
# prebuild driver only
./install --add-kernel-support-build-only
 
# install
./install --force
 
# show packages
ll /tmp/mlnx-en-*-generic/mlnx-en-*-ext.tgz

Links
https://developer.nvidia.com/networking/ethernet-software

Mellanox: SR-IOV (Single Root IO Virtualization)

Install Mellanox Driver
http://www.panticz.de/mellanox/install-dirver

lspci | grep Mellanox
mstconfig -y -d  18:00.1 set SRIOV_EN=1 NUM_OF_VFS=16
 
#cat /etc/modprobe.d/mlnx.conf 
#options mlx4_core num_vfs=5 probe_vf=5
 
apt install -y sysfsutils
 
cat <<EOF> /etc/sysfs.d/mlnx-sriov_numvfs.conf
class/net/ens6f0/device/sriov_numvfs = 8
class/net/ens6f1/device/sriov_numvfs = 8
class/net/ens7f0/device/sriov_numvfs = 8
class/net/ens7f1/device/sriov_numvfs = 8
EOF
 
# /boot/grub/grub.cf
intel_iommu=on
 
ll /sys/class/net/en{p,s}*
echo 8 > /sys/class/net/ens6f0/device/sriov_numvfs

Configure VLAN

Mellanox ConnectX-3 Pro UEFI iPXE boot

  Device Type:      ConnectX3Pro
  Part Number:      MCX312B-XCC_Ax
  Description:      ConnectX-3 Pro EN network interface card; 10GigE; dual-port SFP+; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1200111023
  PCI Device Name:  /dev/mst/mt4103_pci_cr0
  Port1 MAC:        ec0d9a00aab1
  Port2 MAC:        ec0d9a00aab2
  Versions:         Current        Available    
     FW             2.42.5000      N/A          
     PXE            3.4.0752       N/A          

Flash UEFI firmware
Request UEFI firmware from support@mellanox.com
http://www.panticz.de/mellanox/firmware-update

Install MFT
http://www.panticz.de/install-mellanox-mft

Flash firmware

flint -y -d /dev/mst/mt4103_pci_cr0 -i firmware fw-ConnectX3Pro-rel-2_42_5000-MCX312B-XCC_Ax-FlexBoot-3.4.752-UEFI-14.11.46.bin b
  Device Type:      ConnectX3Pro
  Part Number:      MCX312B-XCC_Ax
  Description:      ConnectX-3 Pro EN network interface card; 10GigE; dual-port SFP+; PCIe3.0 x8 8GT/s; RoHS R6
  PSID:             MT_1200111023
  PCI Device Name:  /dev/mst/mt4103_pci_cr0
  Port1 MAC:        ec0d9a00aab1
  Port2 MAC:        ec0d9a00aab2
  Versions:         Current        Available    
     FW             2.42.5000      N/A          
     PXE            3.4.0752       N/A          
     UEFI           14.11.0046     N/A     

Fix iPXE boot issue (recompile ipxe.efi) when connected to LACP swith port
http://www.panticz.de/ipxe/compile