Site Navigation
Savannah: Efficient mmWave Baseband Processing with Minimal and Heterogeneous Resources
Description
In this tutorial, we demonstrate Savannah, a framework for efficient mmWave baseband processing with minimal and heterogeneous resources.
- The codebase, README and installation instructions for Savannah software can be found in attachments called
Savannah.zip
under RENEW license. - The COSMOS team contributes to:
- Adding the UHD support for the Agora software under the UHD license;
- Updating DSP pipeline (Equalization; Demodulation);
- Integrating ACC100 BBDEV acceleration for Decoding.
We thank the RENEW team for their support and help throughout the process.
The following papers describes the integration of the IBM 28 GHz PAAMs with Savannah in the COSMOS testbed. We would appreciate it if you cite this paper when publishing results obtained using the PAAMs deployed in COSMOS.
- Z. Qi, Z. Gao, C. Tung, and T. Chen, "Programmable Millimeter-Wave MIMO Radios with Real-Time Baseband Processing". in Proc. ACM Mobi Com'23 Workshop on Wireless Network Testbeds, Experimental evaluation & Characterization (WiNTECH '23), 2023
- Z. Qi*, C. Tung*, A. Kalia, and T. Chen, "Savannah: Efficient mmWave Baseband Processing with Minimal and Heterogeneous Resources". in Proc. 30th Annual International Conference on Mobile Computing and Networking (Mobi Com'24), 2024
Authors:
Zhenzhou (Tom) Qi, Duke University <zhenzhou.qi[at]duke[dot]edu>
Chung-Hsuan Tung, Duke University <chunghsuan.tung[at]duke[dot]edu>
Zhihui Gao, Duke University <zhihui.gao[at]duke[dot]edu>
Tingjun Chen, Duke University <tingjun.chen[at]duke[dot]edu>
Last updated: Sep. 15, 2024
Prerequisites
In order to access COSMOS-SB2, create a reservation in COSMOS testbed and have it approved by the reservation service. Access to the resources is granted after the reservation is confirmed. Please follow the process shown on the COSMOS getting started page to get started.
Resources Required
- 2 USRP N310 SDRs (
sdr1-s1-lg1
andsdr1-md1
in SB2) - 2 IBM 28GHz PAAMs (
rfdev2-1
andrfdev2-2
in SB2 ) - 1 Server (
srv1-lg1
)
The current hardware connection in SB2 as shown below
sdr1-s1-lg1
RF2 TX/RX —rfdev2-1
IC0/TX/H,sdr1-s1-lg1
RF2 RX2 —rfdev2-1
IC1/RX/Hsdr1-s1-lg1
RF3 TX/RX —rfdev2-1
IC0/TX/V,sdr1-s1-lg1
RF3 RX2 —rfdev2-1
IC1/RX/Vsdr1-md1
RF2 TX/RX —rfdev2-2
IC0/TX/H,sdr1-md1
RF2 RX2 —rfdev2-2
IC1/RX/Hsdr1-md1
RF3 TX/RX —rfdev2-2
IC0/TX/V,sdr1-md1
RF2 RX2 —rfdev2-2
IC1/RX/V
The required software components used in this demo are already loaded to the Savannah.ndz.ndz
node image, the node image includes:
- Ubuntu 20.04, UHD 4.1.0
- PAAM Control to initialize the PAAM boards to integrate with USRPs.
- PAAM CFO calibration.
- Savannah: a framework for efficient mmWave baseband processing with minimal and heterogeneous resources. Please refer to
Savannah.zip
under attachments for more information.
Tutorial Setup
Follow the steps below to gain access to the sandbox console and set up nodes with appropriate images.
- If you don't have one already, sign up for a COSMOS account
- Create a resource reservation on COSMOS SB2
- Login into sandbox console
console.sb2.cosmos-lab.org
) with four SSH sessions. - In terminal 1, make sure all the nodes and devices used in the experiment are turned off. Use the following command for SB2
omf tell -a offh -t sdr1-s1-lg1,sdr1-md1,rfdev2-1,rfdev2-2,srv1-lg1
- Load
Savannah.ndz
on the server.omf load -i Savannah.ndz -t srv1-lg1 -r 0
- Turn all the required resources on and check the status of all the resources. Use the following commands for SB2.
omf tell -a on -t sdr1-s1-lg1,sdr1-md1,rfdev2-1,rfdev2-2,srv1-lg1
omf stat -t all
ssh
to the server with option -Y for using GUI for our live demo.ssh -Y root@srv1-lg1
- In all the ssh sessions for
srv1-lg1
, use the following to source the correct environment.source /opt/intel/oneapi/setvars.sh --force --config="/opt/intel/oneapi/renew-config.txt"
Find and prepare USRPs
- Upon logging into the server, set up the 10G data interfaces DATA1, DATA2 in terminal 1.
ifconfig DATA1 10.117.1.1 netmask 255.255.0.0 mtu 9000 up ifconfig DATA2 10.118.1.1 netmask 255.255.0.0 mtu 9000 up
sudo sysctl -w net.core.rmem_max=536870912 sudo sysctl -w net.core.wmem_max=536870912
After running the above commands, you should see that the data interfaces have the appropriate IP addresses assigned.
root@srv1-lg1:~# ifconfig DATA1 DATA1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::1e34:daff:fe42:d4c prefixlen 64 scopeid 0x20<link> ether 1c:34:da:42:0d:4c txqueuelen 1000 (Ethernet) RX packets 21092 bytes 1881634 (1.8 MB) RX errors 0 dropped 19183 overruns 0 frame 0 TX packets 686 bytes 204975 (204.9 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 root@srv1-lg1:~# ifconfig DATA2 DATA2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::1e34:daff:fe42:d4d prefixlen 64 scopeid 0x20<link> ether 1c:34:da:42:0d:4d txqueuelen 1000 (Ethernet) RX packets 21091 bytes 1881530 (1.8 MB) RX errors 0 dropped 19184 overruns 0 frame 0 TX packets 690 bytes 226549 (226.5 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
- Run
uhd_find_devices
to make sure that both USRP N310s can be reached:[INFO] [UHD] linux; GNU C++ version 9.4.0; Boost_107100; UHD_4.1.0.HEAD-0-g25d617ca -------------------------------------------------- -- UHD Device 3 -------------------------------------------------- Device Address: serial: 315A35A addr: 10.117.2.1 claimed: False fpga: XG mgmt_addr: 10.116.2.1 mgmt_addr: 10.117.2.1 product: n310 type: n3xx -------------------------------------------------- -- UHD Device 4 -------------------------------------------------- Device Address: serial: 3176DF7 addr: 10.118.3.1 claimed: False fpga: XG mgmt_addr: 10.116.3.1 mgmt_addr: 10.118.3.1 product: n310 type: n3xx
Configure IBM 28GHz PAAM
COSMOS uses the IBM 28GHz PAAM API to configure and initialize the PAAM boards. Under the console in terminal 2, please run the following 4 commands:
<user_name>@console:~$ curl "http://am1.orbit-lab.org:5054/array_mgmt/configure?dev_name=rfdev2-1.sb2.cosmos-lab.org&ics=0&num_elements=16&txrx=tx&pol=h&theta=0&phi=0" <user_name>@console:~$ curl "http://am1.orbit-lab.org:5054/array_mgmt/configure?dev_name=rfdev2-1.sb2.cosmos-lab.org&ics=0&num_elements=16&txrx=tx&pol=v&theta=0&phi=0" <user_name>@console:~$ curl "http://am1.orbit-lab.org:5054/array_mgmt/configure?dev_name=rfdev2-2.sb2.cosmos-lab.org&ics=1&num_elements=16&txrx=rx&pol=h&theta=0&phi=0" <user_name>@console:~$ curl "http://am1.orbit-lab.org:5054/array_mgmt/configure?dev_name=rfdev2-2.sb2.cosmos-lab.org&ics=1&num_elements=16&txrx=rx&pol=v&theta=0&phi=0"
An example message will print out to after each command to indicate that the PAAM board is being successfully configured:
<?xml version="1.0" encoding="UTF-8"?> <response status="OK"> <action service="array_mgmt" name="configure" ipaddr="10.116.5.2"> <step name="open" duration="3.780022"/> <step name="initializaition" duration="0.125465"/> <step name="enabling" duration="0.022732"/> <step name="steering" duration="0.006349" theta="0" phi = "0" ipaddr = "10.116.5.2" /> <state PAAM_ID="0x23" LO_switch="PLL" if_sw1="0xF" if_sw2="0xF" if_sw3="0xF" if_sw4="0xF" txrx="rx" polarization="v" /> <adc> <conv index="0" name="1v2" tADC="114" tVolt="0.279" tCurr="0.139"/> <conv index="1" name="1v5" tADC="214" tVolt="0.523" tCurr="1.046"/> <conv index="2" name="1v8" tADC="4" tVolt="0.010" tCurr="0.005"/> <conv index="3" name="2v7_0" tADC="13" tVolt="0.032" tCurr="0.064"/> <conv index="4" name="2v7_1" tADC="183" tVolt="0.447" tCurr="0.894"/> <conv index="5" name="2v7_2" tADC="18" tVolt="0.044" tCurr="0.088"/> <conv index="6" name="2v7_3" tADC="40" tVolt="0.098" tCurr="0.196"/> <conv index="7" name="3v3_pll" tADC="303" tVolt="0.740" tCurr="0.370"/> <conv index="8" name="5v_uzed" tADC="276" tVolt="0.674" tCurr="0.674"/> <conv index="9" name="12v" tADC="140" tVolt="0.342" tCurr="1.140"/> <conv index="10" name="0V" tADC="0" tVolt="0.000"/> <conv index="11" name="1V8" tADC="735" tVolt="1.796"/> </adc> <step name="status" duration="0.069065"/> <step name="close" duration="0.100179"/> </action> </response>
Experiment Execution
General information
Notice For every terminal you opened up, please run:
source /opt/intel/oneapi/setvars.sh --force --config="/opt/intel/oneapi/renew-config.txt"
We have already built and installed all the necessary packages for running Savannah, please direct to the ~/Savannah/agora-mmwave/
folder. There are different modes we are able to run and the corresponding configurations are shown in the following. To configure to different modes, please direct to <savannah folder>/build
, and use cmake .. -D<VAR>=<OPTION>
.
TIME_EXCLUSIVE
should be always true to ensure the best performance by avoiding unnecessary recording.LDPC_TYPE
allows users to select the LDPC decoder: FlexRAN (software) vs. ACC100 (hardware).LDPC_ENQ_BULK
determines whether OFDM symbols are delayed to push into the ACC100 accelerator at the last uplink symbol for a frame.MAT_OP_TYPE
selects the compute scheme for vectorized matrix operations. Only effective whensmall_mimo_acc
is enabled in.json
config.AVX512
is always recommended for performance if supported.ARMA_VEC
is the vectorized option wrapped by Armadillo, and thus is recommended when avx512 is unavailable.ARMA_CUBE
is for research/experiment evaluation only and does not have good performance as expected.SINGLE_THREAD
defines if Savannah merges the only worker thread and the main scheduling thread. When it is set to True, the worker thread count is limited to 1 and is merged with the main thread. When it is set to False, Savannah acts as a multi-thread model as Agora, and has a total thread count is woker threads + 1 (scheduling, main thread).
We have also provided executables that can help you to do experiments with Savannah called savannah.sh
. We set the default executable configuration to be tddconfig-sim-ul-fr2-mu3-100Mhz.json
. You are also able to choose different configurations to run in Savannah/agora-mmwave/files/config/ci
.
Simulation Mode: Savannah-mc
In this section, we are going to cover to how to Savannah-mc in simulation mode.
Please double-check the CMake flags in ~/Savannah/agora-mmwave/build
directory.
# you are expect to see the following results: root@srv1-lg1:~/Savannah/agora-mmwave/build# cmake ../ -- Using GNU compiler, compiler ID GNU -- Debugging is disabled -- CMAKE_CXX_FLAGS: -std=c++17 -Wall -g -march=native -m64 -O3 -Ofast -DNDEBUG -- CURRENT DIRECTORY: /root/Savannah/agora-mmwave -- CMAKE_CURRENT_SOURCE_DIR: /root/Savannah/agora-mmwave -- Processor supports AVX-512 -- Using FlexRAN's (i.e., not Agora's) AVX512 encoder -- -- ----- Configuration values ----- -- DEBUG: OFF -- RADIO_TYPE: SIMULATION -- LOG_LEVEL: info -- USE_SPDLOG: True -- ENABLE_MAC: False -- ENABLE_CSV_LOG: False -- ENABLE_MAT_LOG: False -- USE_AVX2_ENCODER: False -- ENABLE_HDF5: False -- TIME_EXCLUSIVE: TRUE -- LDPC_TYPE: FlexRAN -- LDPC_ENQ_BULK: False -- MAT_OP_TYPE: AVX512 -- SINGLE_THREAD: False -- -------------------------------- -- -- Enabled SIMULATION radio type -- Enabled SW Intel FlexRAN LDPC Decoding -- LDPC: Sequential Enqueue Mode -- Time-exlusive: Report only timing but not other characteristics -- MAT_OP_TYPE: Enable AVX512 matrix operation -- SINGLE_THREAD: Enable worker thread (use multi-thread model) -- Using spdlog async logger as the default logger -- Build spdlog: 1.11.0 -- Build type: Release -- Disabled Csv/Mat Logger -- Logging level = info. -- Configuring done -- Generating done -- Build files have been written to: /root/Savannah/agora-mmwave/build
In terminal 1, please first generate the binary files by running:
root@srv1-lg1:~/Savannah/agora-mmwave/# ./savannah.sh -g -s
In terminal 1, please execute the following to bring up the UE. UE will first be in standby mode to wait for gNodeB to be brought up.
root@srv1-lg1:~/Savannah/agora-mmwave/# ./savannah.sh -u -s
In terminal 2, please execute the following to bring up the gNodeB.
root@srv1-lg1:~/Savannah/agora-mmwave/# ./savannah.sh -x -s -r
You are expected to see that both the gNodeB and UE start to run. Specifically on the gNodeB side, you are able to see the following example results during the running process:
[14:539488][I] Main [frame 5878 + 0.3376 ms]: Completed LDPC decoding (16 UL symbols) [14:539508][I] Frame 5878 Summary: FFT (32 tasks): 0.0206114 ms (~0.00478991 + 0.00756673 + 0.00548272 ms), CSI (4 tasks): 0.0030098 ms (~0.000714161 + 0.00095141 + 0.00116174 ms), Beamweights (1 tasks): 0.00130209 ms (~4.04698e-05 + 0.0012592 + 0 ms), Equal (16 tasks): 0.0268793 ms (~0.00325447 + 0.0104146 + 0.0101594 ms), Demul (32 tasks): 0.0224507 ms (~5.13671e+12 + 0 + 0 ms), Decode (32 tasks): 0.228612 ms (~0.000104702 + 0.219754 + 0 ms), Total: 0.302866 ms
where it contains the total elpased time for processing a frame and a breakdown of the CPU time for each DSP stage. Taking the above frame as an example, the total elapsed time for processing frame 5878 is 0.3376ms, and the CPU time spent on doing equalization is 0.0268793ms. The general takeaway here is that the framework is able to process the frame within the deadline of 0.375ms.
You can either stop the program by enter ctrl+c
or wait till the program to stop (The default frames to run are set to be 200000 frames).
Once you finish the process, you are expected to see the following results:
Uplink totals (tasks, frames): CSI (23524, 5881.00), [14:540519][Beamweights (5881, 5881.00), FFT (188186, 5880.81), Demul (188172, 15.00), Decode (188166, 5880.19) Thread 0 performed (tasks, fraction of tasks): FFT (47136, 25.05%), CSI (6067, 25.79%), Beamweights (1584, 26.93%), Equal (23283, 24.75%), Demul (46566, 24.75%), Decode (47216, 25.09%), FFT (47136, 25.05%), Thread 1 performed (tasks, fraction of tasks): FFT (46389, 24.65%), CSI (5707, 24.26%), Beamweights (1480, 25.17%), Equal (23770, 25.26%), I] Agora: printing stats and saving to file Demul (47540, 25.26%), [14:540523][Decode (46632, 24.78%), FFT (46389, 24.65%), Thread 2 performed (tasks, fraction of tasks): FFT (48401, 25.72%), CSI (5826, 24.77%), Beamweights (1257, 21.37%), Equal (23214, 24.67%), Demul (46428, 24.67%), Decode (46919, 24.93%), FFT (48401, 25.72%), I] Stats: total processed frames 5880 Thread 3 performed (tasks, fraction of tasks): FFT (46260, 24.58%), CSI (5924, 25.18%), Beamweights (1560, 26.53%), Equal (23822, 25.32%), Demul (47644, 25.32%), Decode (47405, 25.19%), FFT (46260, 24.58%), [14:540621][I] Stats: Saving master timestamps to ./files/experiment/timeresult.txt [14:563553][I] Stats: Printing detailed results to ./files/experiment/timeresult_detail.txt [14:578927][I] UE 0: Uplink bit errors (BER) 0/79689632 (0.000000), block errors (BLER) 0/82324 (0.000000) [14:578935][I] UE 1: Uplink bit errors (BER) 0/79689632 (0.000000), block errors (BLER) 0/82324 (0.000000) [14:578936][I] Agora: terminating ================================= CORE LIST SUMMARY ================================= Total Number of Cores: 48 : 48 || Core ID: 3 || Requested: 24 || ThreadType: Master || ThreadId: 140397163316672 || Core ID: 5 || Requested: 25 || ThreadType: TXRX || ThreadId: 140396821804800 || Core ID: 7 || Requested: 26 || ThreadType: TXRX || ThreadId: 140396799424256 || Core ID: 9 || Requested: 27 || ThreadType: Worker || ThreadId: 140396855375616 || Core ID: 11 || Requested: 28 || ThreadType: Worker || ThreadId: 140396846982912 || Core ID: 13 || Requested: 29 || ThreadType: Worker || ThreadId: 140396838590208 || Core ID: 15 || Requested: 30 || ThreadType: Worker || ThreadId: 140396830197504 =================================
It summarizes the final results of the previous experiments and the core maps. Please notice the BER and BLER you are expected to see should all be 0%, as we did not apply channel noise in the simulation mode.
RRU Mode: Savannah-mc
In this section, we are going to cover to how to Savannah-mc in RRU mode.
Notice Please make sure you run the "Configure IBM 28GHz PAAM" section in tutorial setup.
Please double-check the CMake flags in ~/Savannah/agora-mmwave/build
directory. Specifically for the CMake flag: RADIO_TYPE
, you should update from SIMULATION
to PURE_UHD
.
# you are expect to see the following results: root@srv1-lg1:~/Savannah/agora-mmwave/build# cmake ../ -DRADIO_TYPE=PURE_UHD -- Using GNU compiler, compiler ID GNU -- Debugging is disabled -- CMAKE_CXX_FLAGS: -std=c++17 -Wall -g -march=native -m64 -O3 -Ofast -DNDEBUG -- CURRENT DIRECTORY: /root/Savannah/agora-mmwave -- CMAKE_CURRENT_SOURCE_DIR: /root/Savannah/agora-mmwave -- Processor supports AVX-512 -- Using FlexRAN's (i.e., not Agora's) AVX512 encoder -- -- ----- Configuration values ----- -- DEBUG: OFF -- RADIO_TYPE: PURE_UHD -- LOG_LEVEL: info -- USE_SPDLOG: True -- ENABLE_MAC: False -- ENABLE_CSV_LOG: False -- ENABLE_MAT_LOG: False -- USE_AVX2_ENCODER: False -- ENABLE_HDF5: False -- TIME_EXCLUSIVE: TRUE -- LDPC_TYPE: FlexRAN -- LDPC_ENQ_BULK: False -- MAT_OP_TYPE: AVX512 -- SINGLE_THREAD: False -- -------------------------------- -- -- Enabled Pure UHD radio type -- Enabled SW Intel FlexRAN LDPC Decoding -- LDPC: Sequential Enqueue Mode -- Time-exlusive: Report only timing but not other characteristics -- MAT_OP_TYPE: Enable AVX512 matrix operation -- SINGLE_THREAD: Enable worker thread (use multi-thread model) -- Using spdlog async logger as the default logger -- Build spdlog: 1.11.0 -- Build type: Release -- Disabled Csv/Mat Logger -- Logging level = info. -- Enabled PURE_UHD radio -- Found UHD: /usr/local/lib/libuhd.so (Required is at least version "3.15.0") -- Using UHD Version: 4.1.0.HEAD-0-g25d617ca -- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found suitable version "1.71.0", minimum required is "1.65") found components: program_options system thread -- Using Boost Version: 1.71.0 -- pure_uhd_lib library initialized -- Configuring done -- Generating done -- Build files have been written to: /root/Savannah/agora-mmwave/build
You are expected to see UHD is being linked and using UHD Version: 4.1.0.HEAD-0-g25d617ca.
Please then recompile the program by running:
root@srv1-lg1:~/Savannah/agora-mmwave/build# make -j
Similar to previous simulation experiments: In terminal 1, please first generate the binary files by running:
root@srv1-lg1:~/Savannah/agora-mmwave/# ./savannah.sh -g -r
In terminal 1, please execute the following to bring up the gNodeB. gNodeB will first be in standby mode to wait for UE to send out beacons to notify that UE is ready:
root@srv1-lg1:~/Savannah/agora-mmwave/# ./savannah.sh -x -r -r
In terminal 2, please execute the following to bring up the UE.
root@srv1-lg1:~/Savannah/agora-mmwave/# ./savannah.sh -u -r
You are expected to see that both the gNodeB and UE start to run. Specifically on the gNodeB side, you are able to see a very similar results during the running process as of simulation mode, but with a jitter between the host and the USRP which will cause the total elapsed processing to be delayed by a bit.
Once you finish the process, you are expected to see the following results similar as simulation mode
[53:029073][I] UE 0: Uplink bit errors (BER) 32491/2639736 (0.012308), block errors (BLER) 67/2727 (0.024569)
but notice, the BER and BLER will no longer be perfectly 0%, as the initial synchronization between gNodeB and UE will take some time and cause incorrect decoding result for the first few frames.
The default USRP and PAAM gain settings will result in a SNR around 26dB which will produce very low BER and BLER. You can also change the gain settings in Savannah/agora-mmwave/files/experiments/ul-usrp.json
to vary the SNR values.
EVM, SNR Printouts
We also provide an alternative to enable users to check the EVM values and other metrics during transmission, but as those calculations introduce overheads in computation, the real-time performance will be influenced. But if you wish to check the correctness for each DSP stage, you can disable the CMake flag TIME_EXCLUSIVE
by setting it to false. After recompiling, you should be able to see EVM and SNR values printed out after each frame as the following:
[15:462812][I] Main [frame 5309 + 0.63 ms]: Completed demodulation [15:462817][I] Frame 5309 Constellation: EVM 7.1707 , SNR 11.4444 True SNR is: 27.0316
Experiment Execution - ACC100 DPDK BBDev
ACC100 Unittests
Notice: COSMOS testbed currently does not have ACC100 card installed in the servers, but we provide unit test code and testcases for your reference if you have installed the card on your own system.
In Savannah.zip
, you will find a README file called BBDev_testcase.md
. In this file, you will find detailed instructions on the ACC100 system requirements, initialization, configuration, how to run default validation test and how to build your own testcase.
We have provided example testcase w.r.t different MCS in the mcs_enc_dec_pair
folder. Inside this folder, you will find encoding and decoding testcases for different MCSs specifically for 5G mmWave FR2 Numeorology-3 100MHz; A program called test_bbdev_perf.c
based on DPDK's bbdev program; and expected encoding and decoding test results. Please feel free to contact the author if you have any questions related to how to use ACC100 card.
Savannah-sc mode
To run Savannah-sc, please update the CMake flag: LDPC_TYPE
to be ACC100
and update the CMake flag: SINGLE_THREAD
to be True
.
After compiling, the new framework will be linked to a new decoder DSP function to perform the decoding task using the ACC100 acceleration card. The experimental steps are similar to Savannah-mc mode.
Attachments (3)
- cosmos_paam_202406.png (9.6 MB ) - added by 3 months ago.
- CMake_flags.png (35.0 KB ) - added by 3 months ago.
- Savannah.zip (1.6 MB ) - added by 3 months ago.