16.4.3. 为 Intel FPGA PAC N3000 配置 SR-IOV-FEC Operator
本节论述了如何为 Intel FPGA PAC N3000 编程 SR-IOV-FEC Operator。SR-IOV-FEC Operator 处理用于在 vRAN L1 应用程序中加快 FEC 进程的正向错误修正 (FEC) 设备的管理。
配置 SR-IOV-FEC Operator 涉及以下任务:
- 为 FEC 设备创建所需的虚拟功能(VF)
- 将 VF 绑定到适当的驱动程序
- 为 4G 或 5G 部署中所需的功能配置 VF 队列
转发错误更正 (FEC) 的角色是更正传输错误,其中消息中的某些位可以丢失或垃圾化。由于传输媒体中的批注、干扰或信号强度较低,消息可能会丢失或被破坏。如果没有 FEC,则必须重新发送已垃圾化的消息,以添加到网络负载并影响吞吐量和延迟。
先决条件
- Intel FPGA PAC N3000 卡
- 使用 OpenNESS Operator 为 Intel FPGA PAC N3000 安装的节点(编程)
- 使用 OpenNESS Operator 为 Wireless FEC 加速器安装的节点
- 使用 Performance Addon Operator 配置 RT 内核
流程
进入
vran-acceleration-operators
项目:$ oc project vran-acceleration-operators
验证 SR-IOV-FEC Operator 是否已安装:
$ oc get csv -o custom-columns=Name:.metadata.name,Phase:.status.phase
输出示例
Name Phase sriov-fec.v1.1.0 Succeeded n3000.v1.1.0 Succeeded
验证
N3000
和sriov-fec
pod 是否正在运行:$ oc get pods
输出示例
NAME READY STATUS RESTARTS AGE fpga-driver-daemonset-8xz4c 1/1 Running 0 15d fpgainfo-exporter-vhvdq 1/1 Running 1 15d N3000-controller-manager-b68475c76-gcc6v 2/2 Running 1 15d N3000-daemonset-5k55l 1/1 Running 1 15d N3000-discovery-blmjl 1/1 Running 1 15d N3000-discovery-lblh7 1/1 Running 1 15d sriov-device-plugin-j5jlv 1/1 Running 1 15d sriov-fec-controller-manager-85b6b8f4d4-gd2qg 1/1 Running 1 15d sriov-fec-daemonset-kqqs6 1/1 Running 1 15d
以下部分提供有关已安装 pod 的信息:
-
fpga-driver-daemonset
提供和加载所需的开放编程加速器引擎 (OPAE) 驱动程序 -
fpgainfo-exporter
为 Prometheus 提供 N3000 遥测数据 -
N3000-controller-manager
将 N3000Node CR 应用到集群,并管理所有操作对象容器 -
N3000-daemonset
是主 worker 应用程序 -
N3000-discovery
发现安装 N3000 加速器设备,并在设备存在时标记 worker 节点 -
sriov-device-plugin
将 FEC 虚拟功能公开为节点下的资源 -
sriov-fec-controller-manager
将 CR 应用到节点,并维护操作对象容器 sriov-fec-daemonset
的责任包括:- 发现每个节点上的 SRIOV NIC。
- 同步第 6 步中定义的自定义资源 (CR) 的状态。
- 使用 CR 的 spec 作为输入并配置发现的 NIC。
-
检索包含其中一个支持的 vRAN FEC 加速器设备的所有节点:
$ oc get sriovfecnodeconfig
输出示例
NAME CONFIGURED node1 Succeeded
查找要配置的 SR-IOV FEC 加速器设备的物理功能 (PF):
$ oc get sriovfecnodeconfig node1 -o yaml
输出示例
status: conditions: - lastTransitionTime: "2021-03-19T17:19:37Z" message: Configured successfully observedGeneration: 1 reason: ConfigurationSucceeded status: "True" type: Configured inventory: sriovAccelerators: - deviceID: 0d5c driver: "" maxVirtualFunctions: 16 pciAddress: 0000.1d.00.0 1 vendorID: "8086" virtualFunctions: [] 2
使用所需的设置配置 FEC 设备。
创建以下自定义资源 (CR),并将 YAML 保存到
sriovfec_n3000_cr.yaml
文件中:apiVersion: sriovfec.intel.com/v1 kind: SriovFecClusterConfig metadata: name: config namespace: vran-acceleration-operators spec: nodes: - nodeName: node1 1 physicalFunctions: - pciAddress: 0000:1d:00.0 2 pfDriver: pci-pf-stub vfDriver: vfio-pci vfAmount: 2 3 bbDevConfig: n3000: # Network Type: either "FPGA_5GNR" or "FPGA_LTE" networkType: "FPGA_5GNR" pfMode: false flrTimeout: 610 downlink: bandwidth: 3 loadBalance: 128 queues: 4 vf0: 16 vf1: 16 vf2: 0 vf3: 0 vf4: 0 vf5: 0 vf6: 0 vf7: 0 uplink: bandwidth: 3 loadBalance: 128 queues: 5 vf0: 16 vf1: 16 vf2: 0 vf3: 0 vf4: 0 vf5: 0 vf6: 0 vf7: 0
注意对于 Intel PAC N3000 for vRAN Acceleration,用户可以最多创建 8 个 VF 设备。每个 FEC PF 设备都提供要配置的 64 个队列,32 个队列用于 uplink,32 个队列用于 downlink。队列通常在 VF 之间均匀分布。
应用 CR:
$ oc apply -f sriovfec_n3000_cr.yaml
应用 CR 后,SR-IOV FEC 守护进程将开始配置 FEC 设备。
验证
检查状态:
$ oc get sriovfecclusterconfig config -o yaml
输出示例
status: conditions: - lastTransitionTime: "2020-12-15T17:19:37Z" message: Configured successfully observedGeneration: 1 reason: ConfigurationSucceeded status: "True" type: Configured inventory: sriovAccelerators: - deviceID: 0d8f driver: pci-pf-stub maxVirtualFunctions: 8 pciAddress: 0000:1d:00.0 vendorID: "8086" virtualFunctions: - deviceID: 0d90 driver: vfio-pci pciAddress: 0000:1d:00.1 - deviceID: 0d90 driver: vfio-pci pciAddress: 0000:1d:00.2
检查日志:
确定 SR-IOV 守护进程 pod 的名称:
$ oc get pod | grep sriov-fec-daemonset
输出示例
sriov-fec-daemonset-kqqs6 1/1 Running 0 19h
查看日志:
$ oc logs sriov-fec-daemonset-kqqs6
输出示例
2020-12-16T12:46:47.720Z INFO daemon.NodeConfigurator.applyConfig configuring PF {"requestedConfig": {"pciAddress":"0000:1d:00.0","pfDriver":"pci-pf-stub","vfDriver":"vfio-pci","vfAmount":2,"bbDevConfig":{"n3000":{ "networkType":"FPGA_5GNR","pfMode":false,"flrTimeout":610,"downlink":{"bandwidth":3,"loadBalance":128,"queues":{"vf0":16,"vf1":16}},"uplink":{"bandwidth":3,"loadBalance":128,"queues":{"vf0":16,"vf1":16}}}}}} 2020-12-16T12:46:47.720Z INFO daemon.NodeConfigurator.loadModule executing command {"cmd": "/usr/sbin/chroot /host/ modprobe pci-pf-stub"} 2020-12-16T12:46:47.724Z INFO daemon.NodeConfigurator.loadModule commands output {"output": ""} 2020-12-16T12:46:47.724Z INFO daemon.NodeConfigurator.loadModule executing command {"cmd": "/usr/sbin/chroot /host/ modprobe vfio-pci"} 2020-12-16T12:46:47.727Z INFO daemon.NodeConfigurator.loadModule commands output {"output": ""} 2020-12-16T12:46:47.727Z INFO daemon.NodeConfigurator device's driver_override path {"path": "/sys/bus/pci/devices/0000:1d:00.0/driver_override"} 2020-12-16T12:46:47.727Z INFO daemon.NodeConfigurator driver bind path {"path": "/sys/bus/pci/drivers/pci-pf-stub/bind"} 2020-12-16T12:46:47.998Z INFO daemon.NodeConfigurator device's driver_override path {"path": "/sys/bus/pci/devices/0000:1d:00.1/driver_override"} 2020-12-16T12:46:47.998Z INFO daemon.NodeConfigurator driver bind path {"path": "/sys/bus/pci/drivers/vfio-pci/bind"} 2020-12-16T12:46:47.998Z INFO daemon.NodeConfigurator device's driver_override path {"path": "/sys/bus/pci/devices/0000:1d:00.2/driver_override"} 2020-12-16T12:46:47.998Z INFO daemon.NodeConfigurator driver bind path {"path": "/sys/bus/pci/drivers/vfio-pci/bind"} 2020-12-16T12:46:47.999Z INFO daemon.NodeConfigurator.applyConfig executing command {"cmd": "/sriov_workdir/pf_bb_config FPGA_5GNR -c /sriov_artifacts/0000:1d:00.0.ini -p 0000:1d:00.0"} 2020-12-16T12:46:48.017Z INFO daemon.NodeConfigurator.applyConfig commands output {"output": "ERROR: Section (FLR) or name (flr_time_out) is not valid. FEC FPGA RTL v3.0 UL.DL Weights = 3.3 UL.DL Load Balance = 1 28.128 Queue-PF/VF Mapping Table = READY Ring Descriptor Size = 256 bytes --------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ | PF | VF0 | VF1 | VF2 | VF3 | VF4 | VF5 | VF6 | VF7 | --------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ UL-Q'00 | | X | | | | | | | | UL-Q'01 | | X | | | | | | | | UL-Q'02 | | X | | | | | | | | UL-Q'03 | | X | | | | | | | | UL-Q'04 | | X | | | | | | | | UL-Q'05 | | X | | | | | | | | UL-Q'06 | | X | | | | | | | | UL-Q'07 | | X | | | | | | | | UL-Q'08 | | X | | | | | | | | UL-Q'09 | | X | | | | | | | | UL-Q'10 | | X | | | | | | | | UL-Q'11 | | X | | | | | | | | UL-Q'12 | | X | | | | | | | | UL-Q'13 | | X | | | | | | | | UL-Q'14 | | X | | | | | | | | UL-Q'15 | | X | | | | | | | | UL-Q'16 | | | X | | | | | | | UL-Q'17 | | | X | | | | | | | UL-Q'18 | | | X | | | | | | | UL-Q'19 | | | X | | | | | | | UL-Q'20 | | | X | | | | | | | UL-Q'21 | | | X | | | | | | | UL-Q'22 | | | X | | | | | | | UL-Q'23 | | | X | | | | | | | UL-Q'24 | | | X | | | | | | | UL-Q'25 | | | X | | | | | | | UL-Q'26 | | | X | | | | | | | UL-Q'27 | | | X | | | | | | | UL-Q'28 | | | X | | | | | | | UL-Q'29 | | | X | | | | | | | UL-Q'30 | | | X | | | | | | | UL-Q'31 | | | X | | | | | | | DL-Q'32 | | X | | | | | | | | DL-Q'33 | | X | | | | | | | | DL-Q'34 | | X | | | | | | | | DL-Q'35 | | X | | | | | | | | DL-Q'36 | | X | | | | | | | | DL-Q'37 | | X | | | | | | | | DL-Q'38 | | X | | | | | | | | DL-Q'39 | | X | | | | | | | | DL-Q'40 | | X | | | | | | | | DL-Q'41 | | X | | | | | | | | DL-Q'42 | | X | | | | | | | | DL-Q'43 | | X | | | | | | | | DL-Q'44 | | X | | | | | | | | DL-Q'45 | | X | | | | | | | | DL-Q'46 | | X | | | | | | | | DL-Q'47 | | X | | | | | | | | DL-Q'48 | | | X | | | | | | | DL-Q'49 | | | X | | | | | | | DL-Q'50 | | | X | | | | | | | DL-Q'51 | | | X | | | | | | | DL-Q'52 | | | X | | | | | | | DL-Q'53 | | | X | | | | | | | DL-Q'54 | | | X | | | | | | | DL-Q'55 | | | X | | | | | | | DL-Q'56 | | | X | | | | | | | DL-Q'57 | | | X | | | | | | | DL-Q'58 | | | X | | | | | | | DL-Q'59 | | | X | | | | | | | DL-Q'60 | | | X | | | | | | | DL-Q'61 | | | X | | | | | | | DL-Q'62 | | | X | | | | | | | DL-Q'63 | | | X | | | | | | | --------+-----+-----+-----+-----+-----+-----+-----+-----+-----+ Mode of operation = VF-mode FPGA_5GNR PF [0000:1d:00.0] configuration complete!"} 2020-12-16T12:46:48.017Z INFO daemon.NodeConfigurator.enableMasterBus executing command {"cmd": "/usr/sbin/chroot /host/ setpci -v -s 0000:1d:00.0 COMMAND"} 2020-12-16T12:46:48.037Z INFO daemon.NodeConfigurator.enableMasterBus commands output {"output": "0000:1d:00.0 @04 = 0102\n"} 2020-12-16T12:46:48.037Z INFO daemon.NodeConfigurator.enableMasterBus executing command {"cmd": "/usr/sbin/chroot /host/ setpci -v -s 0000:1d:00.0 COMMAND=0106"} 2020-12-16T12:46:48.054Z INFO daemon.NodeConfigurator.enableMasterBus commands output {"output": "0000:1d:00.0 @04 0106\n"} 2020-12-16T12:46:48.054Z INFO daemon.NodeConfigurator.enableMasterBus MasterBus set {"pci": "0000:1d:00.0", "output": "0000:1d:00.0 @04 0106\n"} 2020-12-16T12:46:48.160Z INFO daemon.drainhelper.Run() worker function - end {"performUncordon": true}
检查卡的 FEC 配置:
$ oc get sriovfecnodeconfig node1 -o yaml
输出示例
status: conditions: - lastTransitionTime: "2020-12-15T17:19:37Z" message: Configured successfully observedGeneration: 1 reason: ConfigurationSucceeded status: "True" type: Configured inventory: sriovAccelerators: - deviceID: 0d8f 1 driver: pci-pf-stub maxVirtualFunctions: 8 pciAddress: 0000:1d:00.0 vendorID: "8086" virtualFunctions: - deviceID: 0d90 2 driver: vfio-pci pciAddress: 0000:1d:00.1 - deviceID: 0d90 driver: vfio-pci pciAddress: 0000:1d:00.2