潘伟涛 · 2022年05月23日

开源100 Gbps NIC Corundum环境搭建介绍(一)

25803f0ccc96c1fd7bc1679dc8f0395c.jpg

公众号文章《业界第一个真正意义上开源100 Gbps NIC Corundum介绍》和《揭秘:普通电脑换上Xilinx Alveo U50 100G网卡传文件会有多快?》发出后,得到了很多粉丝的关注,大家纷纷留言询问重现开源工程的详细过程。团队李钊同学详细写了一下具体的实现步骤,具体如下。

目标:在Linux环境下,基于VCU118板卡恢复出100G corundum NIC。

一、Corundum简介

GitHub repository: https://github.com/corundum/corundum

GitHub wiki: https://github.com/corundum/corundum/wiki

  Corundum是一个基于FPGA的开源原型平台,用于高达100Gbps及更高的网络接口开发。Corundum平台包括一些用于实现实时,高线速操作的核心功能,包括:高性能数据路径,10G/ 25G / 100G以太网MAC,PCIExpress第3代,自定义PCIeDMA引擎以及本机高精确的IEEE 1588 PTP时间戳。一个关键功能是可扩展队列管理,它可以支持超过10,000个队列以及可扩展的传输调度程序,从而可以对包传输进行细粒度的硬件控制。结合多个网络接口,每个接口多个端口以及每个端口事件驱动的传输调度,这些功能可实现高级网络接口,体系结构和协议的开发。这些硬件功能的软件接口是Linux网络协议栈的高性能驱动程序。该平台还支持分散/聚集DMA,校验和卸载,接收流散列和接收端缩放。一个全面的,基于Python的开放源代码仿真框架促进了开发和调试,该框架包括整个系统,从驱动程序和PCIExpress接口的仿真模型到以太网接口。通过实现微秒级时分多址(TDMA)硬件调度程序,以100Gbps的线速执行TDMA调度,而没有CPU开销,证明了Corundum的强大功能和灵活性。

二、环境准备

2.1:Linux系统

  本机为win10专业版,由于环境需要

  • 在VMware虚拟机安装Ubuntu 20.04 LTS系统供日常学习测试使用
  • 安装python库
  • 使用tox尝试仿真
  • 安装双系统Ubuntu 20.04 LTS
  • 在Ubuntu系统上安装vivado 2019.2
  • 注意vivado 2019.2的license,基本组件的license会编译出错,需要添加IP license。

2.2:Obtaining the source code

  • 直接从git地址下载zip压缩包使用;
  • $ git clone [https://github.com/corundum/corundum.git](https://github.com/corundum/corundum.git)

2.3:FPGA仿真环境搭建

2.3.1 system packages

  • Python3.9;当前ubuntu 20.04自带python3版本为python3.8,需要下载python3.9并配置系统python3为python3.9。其中遇到很多问题需要百度自行解决。。。
$ sudo apt-get install python3.9$ sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1$ sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 2$ sudo update-alternatives --config python3  
There are 2 choices for the alternative python3 (providing /usr/bin/python3).  
   
 Selection Path  Priority Status  
\------------------------------------------------------------  
\* 0  /usr/bin/python3.8 2 auto mode 1  /usr/bin/python3.8 2 manual mode 2  /usr/bin/python3.9 1 manual mode  
   
Press <enter> to keep the current choice\[\*\], or type selection number: 2

测试python版本:

$ python3 -V  
Python 3.9
  • 安装/升级pip
$ sudo apt-get install python3-pip  
$ sudo pip3 install --upgrade pip
  • 安装python-dev

$ sudo apt install python3-dev

  • 安装Icarus Verilog 

$ sudo apt-get install iverilog

  • 安装GTKWave

$ sudo apt-get install gtkwave

2.3.2 required python packages 

  • cocotb
  • cocotb-bus
  • cocotb-test
  • cocotbext-axi
  • cocotbext-eth
  • cocotbext-pcie
  • pytest
  • scapy
  • Recommended additional python packages:
  • tox (to run pytest inside a python virtual environment)
  • pytest-xdist (to run tests in parallel with pytest -n auto)
  • pytest-sugar (makes pytest output a bit nicer)
$ pip3 install cocotb  
$ pip3 install cocotb-bus  
$ pip3 install cocotb-test  
$ pip3 install cocotbext-axi  
$ pip3 install cocotbext-eth  
$ pip3 install cocotbext-pcie  
$ pip3 install pytest  
$ pip3 install scapy  
$ pip3 install tox  
$ pip3 install pytest-xdist  
$ pip3 install pytest-sugar

注意:安装完package可能会报warning提示package不在PATH内

014551ca0742f2835768cf509d268256.png

此时需要将package添加到系统PATH中

$ sudo gedit ~/.bashrc

在bashrc中添加

export PATH="$PATH:/home/yourname/.local/bin"

然后在终端source

$ sudo source ~/.bashrc

2.4 安装双系统Ubuntu 20.04

在win10下的机械盘上分了一块200G大小的空间用于安装新系统,事实证明200G有点小。

安装过程见:

https://www.jianshu.com/p/54d9a3a695cc
https://zhuanlan.zhihu.com/p/135953477
https://www.zhihu.com/question/34611974
注意分区时:/boot在UEFI上是非必要的,系统直接安在efi上即可。

将分区/:的空间设置足够大,这是总目录,以后安装的软件默认在此目录下,vivado2019占用90多G,因此此块空间或者/home的空间要足够大。

安装完系统配置环境,同时安装vivado2019.2,装在/opt/xilinx下即可。

三、Running test

单核的PC不建议(太慢了),建议上服务器跑。

确保Linux下所需要的package都安装完毕。

在corundum根目录下运行仿真。

$ cd /path/to/corundum-master  
$ tox

py39 installed: apipkg==1.5,attrs==21.2.0,cocotb==1.5.2,cocotb-bus==0.1.1,cocotb-test==0.2.0,cocotbext-axi==0.1.12,cocotbext-eth==0.1.14,cocotbext-pcie==0.1.10,execnet==1.8.0,iniconfig==1.1.1,packaging==20.9,pluggy==0.13.1,py==1.10.0,pyparsing==2.4.7,pytest==6.2.4,pytest-forked==1.3.0,pytest-split==0.1.5,pytest-xdist==2.2.1,scapy==2.4.5,toml==0.10.2  
py39 run-test-pre: PYTHONHASHSEED='438068548'  
py39 run-test: commands\[0\] | pytest -n auto  
\============================= test session starts ==============================  
platform linux -- Python 3.9.2, pytest-6.2.4, py-1.10.0, pluggy-0.13.1  
cachedir: .tox/py39/.pytest_cache  
rootdir: /path/to/corundum, configfile: tox.ini, testpaths: fpga/common/tb, fpga/mqnic, fpga/mqnic_tdma  
plugins: xdist-2.2.1, forked-1.3.0, cocotb-test-0.2.0, split-0.1.5  
gw0 [38] / gw1 [38] / gw2 [38] / gw3 [38] / gw4 [38] / gw5 [38] / gw6 [38] / gw7 [38] / gw8 [38] / gw9 [38] / gw10 [38] / gw11 [38] / gw12 [38] / gw13 [38] / gw14 [38] / gw15 [38] / gw16 [38] / gw17 [38] / gw18 [38] / gw19 [38] / gw20 [38] / gw21 [38] / gw22 [38] / gw23 [38] / gw24 [38] / gw25 [38] / gw26 [38] / gw27 [38] / gw28 [38] / gw29 [38] / gw30 [38] / gw31 [38] / gw32 [38] / gw33 [38] / gw34 [38] / gw35 [38] / gw36 [38] / gw37 [38] / gw38 [38] / gw39 [38] / gw40 [38] / gw41 [38] / gw42 [38] / gw43 [38] / gw44 [38] / gw45 [38] / gw46 [38] / gw47 [38] / gw48 [38] / gw49 [38] / gw50 [38] / gw51 [38] / gw52 [38] / gw53 [38] / gw54 [38] / gw55 [38] / gw56 [38] / gw57 [38] / gw58 [38] / gw59 [38] / gw60 [38] gw0 [38] / gw1 [38] / gw2 [38] / gw3 [38] / gw4 [38] / gw5 [38] / gw6 [38] / gw7 [38] / gw8 [38] / gw9 [38] / gw10 [38] / gw11 [38] / gw12 [38] / gw13 [38] / gw14 [38] / gw15 [38] / gw16 [38] / gw17 [38] / gw18 [38] / gw19 [38] / gw20 [38] / gw21 [38] / gw22 [38] / gw23 [38] / gw24 [38] / gw25 [38] / gw26 [38] / gw27 [38] / gw28 [38] / gw29 [38] / gw30 [38] / gw31 [38] / gw32 [38] / gw33 [38] / gw34 [38] / gw35 [38] / gw36 [38] / gw37 [38] / gw38 [38] / gw39 [38] / gw40 [38] / gw41 [38] / gw42 [38] / gw43 [38] / gw44 [38] / gw45 [38] / gw46 [38] / gw47 [38] / gw48 [38] / gw49 [38] / gw50 [38] / gw51 [38] / gw52 [38] / gw53 [38] / gw54 [38] / gw55 [38] / gw56 [38] / gw57 [38] / gw58 [38] / gw59 [38] / gw60 [38] / gw61 [38] / gw62 [38] / gw63 [38] 
...................................... \[100%\]  
======================== 38 passed in 792.90s (0:13:12) ========================  
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ summary \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
py39: commands succeeded  
congratulations :)

6a31086a2e6e5e621f42235832efdfb2.png

 我在虚拟机里面跑的大概时间37分钟。。。

$ cd /path/to/corundum-master  
$ pytest -n auto (多核)  
$ pytest (单核)



$ cd /path/to/corundum/
$ pytest -n auto
Test session starts (platform: linux, Python 3.9.2, pytest 6.2.2, pytest-sugar 0.9.4)
rootdir: /path/to/corundum, configfile: tox.ini, testpaths: fpga/common/tb, fpga/mqnic, fpga/mqnic_tdma
plugins: parallel-0.1.0, cocotb-test-0.2.0, split-0.1.6.dev1+g97d96c2, sugar-0.9.4, xdist-2.2.1, forked-1.3.0, metadata-1.11.0, html-3.1.1, flake8-1.0.7, cov-2.11.1
gw0 [38] / gw1 [38] / gw2 [38] / gw3 [38] / gw4 [38] / gw5 [38] / gw6 [38] / gw7 [38] / gw8 [38] / gw9 [38] / gw10 [38] / gw11 [38] / gw12 [38] / gw13 [38] / gw14 [38] / gw15 [38] / gw16 [38] / gw17 [38] / gw18 [38] / gw19 [38] / gw20 [38] / gw21 [38] / gw22 [38] / gw23 [38] / gw24 [38] / gw25 [38] / gw26 [38] / gw27 [38] / gw28 [38] / gw29 [38] / gw30 [38] / gw31 [38] / gw32 [38] / gw33 [38] / gw34 [38] / gw35 [38] / gw36 [38] / gw37 [38] / gw38 [38] / gw39 [38] / gw40 [38] / gw41 [38] / gw42 [38] / gw43 [38] / gw44 [38] / gw45 [38] / gw46 [38] / gw47 [38] / gw48 [38] / gw49 [38] / gw50 [38] / gw51 [38] / gw52 [38] / gw53 [38] / gw54 [38] / gw55 [38] / gw56 [38] / gw57 [38] / gw58 [38] / gw59 [38] / gw60 [38] gw0 [38] / gw1 [38] / gw2 [38] / gw3 [38] / gw4 [38] / gw5 [38] / gw6 [38] / gw7 [38] / gw8 [38] / gw9 [38] / gw10 [38] / gw11 [38] / gw12 [38] / gw13 [38] / gw14 [38] / gw15 [38] / gw16 [38] / gw17 [38] / gw18 [38] / gw19 [38] / gw20 [38] / gw21 [38] / gw22 [38] / gw23 [38] / gw24 [38] / gw25 [38] / gw26 [38] / gw27 [38] / gw28 [38] / gw29 [38] / gw30 [38] / gw31 [38] / gw32 [38] / gw33 [38] / gw34 [38] / gw35 [38] / gw36 [38] / gw37 [38] / gw38 [38] / gw39 [38] / gw40 [38] / gw41 [38] / gw42 [38] / gw43 [38] / gw44 [38] / gw45 [38] / gw46 [38] / gw47 [38] / gw48 [38] / gw49 [38] / gw50 [38] / gw51 [38] / gw52 [38] / gw53 [38] / gw54 [38] / gw55 [38] / gw56 [38] / gw57 [38] / gw58 [38] / gw59 [38] / gw60 [38] / gw61 [38] / gw62 [38] / gw63 [38]

 fpga/common/tb/tdma_scheduler/test_tdma_scheduler.py ✓            3% ▍         
 fpga/common/tb/tdma_ber/test_tdma_ber.py ✓                        5% ▌         
 fpga/common/tb/queue_manager/test_queue_manager.py ✓              8% ▊         
 fpga/common/tb/cpl_queue_manager/test_cpl_queue_manager.py ✓     11% █▏        
 fpga/common/tb/rx_checksum/test_rx_checksum.py ✓✓                18% █▉        
 fpga/common/tb/rx_hash/test_rx_hash.py ✓✓                        24% ██▍       
 fpga/common/tb/tdma_ber_ch/test_tdma_ber_ch.py ✓                 21% ██▏       
 fpga/common/tb/tx_checksum/test_tx_checksum.py ✓✓                29% ██▉       
 fpga/mqnic/VCU108/fpga_10g/tb/fpga_core/test_fpga_core.py ✓      32% ███▎      
 fpga/mqnic/ExaNIC_X10/fpga/tb/fpga_core/test_fpga_core.py ✓      34% ███▌      
 fpga/mqnic/AU50/fpga_100g/tb/fpga_core/test_fpga_core.py ✓       37% ███▊      

######## SNIP ########

 fpga/mqnic/fb2CG/fpga_25g/tb/fpga_core/test_fpga_core.py ✓       92% █████████▎
 ...ic/ADM_PCIE_9V3/fpga_25g/tb/fpga_core/test_fpga_core.py ✓     95% █████████▌
 ...nic_tdma/VCU118/fpga_10g/tb/fpga_core/test_fpga_core.py ✓     97% █████████▊
 ...ma/ADM_PCIE_9V3/fpga_10g/tb/fpga_core/test_fpga_core.py ✓    100% ██████████

Results (797.62s):
      38 passed

电脑还在跑。。。
c1d2cfc2717c935678a7d26f96bcbce4.png

27ae7f42fa115247c66d927710409639.png

 跑了快两个小时终于跑完了,虚拟机还是不行,速度太慢。

未完待续

文章作者丨李钊
图文校对丨潘伟涛
责任编辑丨潘伟涛
原文链接:网络交换FPGA

推荐阅读

更多IC设计技术干货请关注IC设计技术专栏。欢迎添加极术小姐姐微信(id:aijishu20)加入技术交流群,请备注研究方向。
推荐阅读
关注数
19607
内容数
1303
主要交流IC以及SoC设计流程相关的技术和知识
目录
极术微信服务号
关注极术微信号
实时接收点赞提醒和评论通知
安谋科技学堂公众号
关注安谋科技学堂
实时获取安谋科技及 Arm 教学资源
安谋科技招聘公众号
关注安谋科技招聘
实时获取安谋科技中国职位信息