[[Include(WikiToC)]] = Getting Started with !InterDigital 5G NR Performance System = [[TOC()]] == Description == In this tutorial we'll demonstrate how to use [wiki:/Hardware/SubSystems/IDPerformance InterDigital 5G-NR Performance platform]. The setup for the experiment is shown in Figure 1. [[Image(InterDigital-5G-NR-Tut.png, 800px)]] As can be seen from the figure, the setup consists of one gNodeB and one UE unit and the two sandbox 1 servers (in this case srv1-in1.sb1.cosmos-lab.org and srv1-in2.sb1.cosmos-lab.org). The first server (srv1-in1) acts as a gateway while the second one is acting as a UE computing platform (to which a UE modem is attached). The 5G Platform devices and the nodes are connected to the sandbox 1 data 1 network and rely on IP routing to ensure that the traffic flow from the client follows the dashed red line in the figure. The intended route is divided into three segments (IP subnets): * gNodeB Subnet (192.168.1.X) - connecting gateway to the gNodeB device * UE Subnet (192.168.2.X) - connects client node to the UE device * 5G Subnet (192.168.3.X) - connects UE to the gNodeB over the 5G link This setup requires configuring the routing tables on all 4 devices to ensure appropriate traffic flow (in addition to a common '''Data 1''' Ethernet that is shared between all 4 units, the server nodes also have a second direct Ethernet connection through the '''Control''' network that is used for node management and metric collection). == Prerequisites == In order to access the test bed, create a reservation and have it approved by the reservation service. Access to the resources is granted only after the reservation is confirmed. Please follow the process shown on [wiki:CosmosOverview/Workflow the COSMOS work flow page] to get started. == Tutorial Setup == Follow the steps below to gain access to the [wiki:Architecture/Domains/cosmos_sb1 sandbox 1] console and set up nodes with appropriate images. 1. If you don't have one already, sign up for a [wiki:GettingStarted#MakeanAccount COSMOS account] 1. [wiki:GettingStarted#MakeaReservation Create a resource reservation] on sandbox 1 1. Prepare your computer (i.e. laptop) for 5G-NR status page access: either prepare [wiki:UserGuide/RemoteAccess/SshTunnel ssh tunnel] thought the sandbox 1 console to gNodeB server node, port 3005 (i.e. map localhost:3005 to srv1-in1.sb1.cosmos-lab.org:3005) or follow the instructions to establish a [wiki:UserGuide/RemoteAccess/VPN VPN tunnel] to the [wiki:Architecture/Domains/cosmos_sb1 COSMOS sandbox 1]. This will be used by the browser to connect to a web server running on the server nodes. 1. [wiki:UserGuide/RemoteAccess#RemoteAccess Login] into sandbox 1 console (console.sb1.cosmos-lab.org) with two SSH sessions. The first session will be used for access to gNodeB while the second will be dedicated to handling UE. 1. Make sure all the nodes and devices used in the experiment are turned off: {{{#!shell omf tell -a offh -t srv1-in1,srv1-in2,rfdev2-in1,rfdev2-in2 }}} 1. Load the images on the computing nodes: in each of the two session, start the [Documentation/Short/LoadImage omf image loading] commands: || SSH session 1 for gNodeB || SSH Session 2 for UE || || {{{omf load -t srv1-in1 -i ID-5G-gNB.ndz -r 50 }}} || {{{omf load -t srv1-in2 -i ID-5G-UE.ndz -r 50}}} || 1. Turn the server nodes and NR platform devices on: {{{#!shell-session omf tell -a on -t srv1-in1,srv1-in2,rfdev2-in1,rfdev2-in2 }}} == Experiment Execution == === Configure Routing on the two compute nodes === * Execute NR gNB setup script on srv1-in1 {{{#!shell-session userid@console:~$ ssh root@srv1-in1 root@srv1-in1:~$ ./setup_gNodeB.sh --------------- Make sure to start the gNodeB modem on the MPU by logging in with: sshpass -p root ssh -o StrictHostKeyChecking=no root@192.168.1.2 and running: root@NR-SW-MPU-gNB:~# ./start_all.sh ---------------- net.ipv4.ip_no_pmtu_disc = 1 root@srv1-in1:~# }}} * Execute NR UE setup script on srv1-in2. {{{#!shell-session userid@console:~$ ssh root@srv1-in2 root@srv1-in2:~$ ./setup_UE.sh }}} The two scripts setup routing to push the test traffic from the server that is attached to UE through the 5G-NR link. [[CollapsibleStart(Routing Scripts)]] The two scripts are shown in the table below: {{{#!td align=center setup_gNodeB.sh }}} |--------------- {{{#!td {{{#!shell #!/bin/bash echo "---------------" echo "Make sure to start the gNodeB modem on the MPU by logging in with:" echo "sshpass -p "root" ssh -o StrictHostKeyChecking=no root@192.168.1.2" echo "and running:" echo "root@NR-SW-MPU-gNB:~# ./start_all.sh" echo "----------------" # Turn on routing echo 1 > /proc/sys/net/ipv4/ip_forward sysctl -w net.ipv4.ip_no_pmtu_disc=1 # Add rotuing for the two subnets "behind" the gNodeB ip route add 192.168.2.0/24 via 192.168.1.2 ip route add 192.168.3.0/24 via 192.168.1.2 # as well as for the measurements to come back from the UE over the wire ip route add 192.168.2.1 via 10.37.1.4 dev eth1 # and configure NAT for them iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE iptables -A FORWARD -i eth1 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT }}} }}} |--------------- {{{#!td align=center setup_UE.sh }}} |--------------- {{{#!td {{{#!shell #!/bin/bash echo "---------------" echo "Make sure to start the UE modem on the MPU by logging in with:" echo "sshpass -p "root" ssh -o StrictHostKeyChecking=no root@192.168.2.1" echo "and running:" echo "root@NR-SW-MPU-UE:~# ./start_all.sh" echo "----------------" # Turn on routing echo 1 > /proc/sys/net/ipv4/ip_forward sysctl -w net.ipv4.ip_no_pmtu_disc=1 # Route all internal 5G traffic through the UE MPU ip route add 192.168.3.0/24 via 192.168.2.1 dev eth0 ip route add 192.168.1.0/24 via 192.168.2.1 dev eth0 # Testbed bound traffic stil goes over the control subnet ip r add 10/8 dev eth1 # Everything else goes over the 5G link # (remove default over control network) ip r del default dev eth1 ip r add default via 192.168.2.1 dev eth0 }}} }}} [[CollapsibleEnd]] === Start services on the two MPUs === 1. ssh into gNodeB MPU at 192.168.1.2 with username "root" and password "root" and execute the setup script "start_all.sh": {{{#!shell-session root@srv1-in1:~# sshpass -p "root" ssh -o StrictHostKeyChecking=no root@192.168.1.2 root@NR-SW-MPU-gNB:~# ./start_all.sh root@NR-SW-MPU-gNB:~# metal: info: metal_uio_dev_open: No IRQ for device 3ed80000.shm. root@NR-SW-MPU-gNB:~# exit logout Connection to 192.168.1.2 closed. root@srv1-in1:~# }}} 1. ssh into UE MPU at 192.168.2.1 with username "root" and password "root: and execute the setup script "start_all.sh": {{{#!shell-session root@srv1-in2:~# sshpass -p "root" ssh -o StrictHostKeyChecking=no root@192.168.2.1 root@NR-SW-MPU-UE:~# ./start_all.sh root@NR-SW-MPU-UE:~# metal: info: metal_uio_dev_open: No IRQ for device 3ed80000.shm. root@NR-SW-MPU-UE:~# exit logout Connection to 192.168.2.1 closed. root@srv1-in2:~# }}} The two scripts will start the appropriate firmware on both NR Platform devices and the script "stop_all.sh" can be used to terminate execution. The NR Platform will automatically “connect” a link between the gNB and UE. This can be verified via the NR GUI. === Check the Routing === Use '''mtr''' utility to confirm that traffic is going through the NR link: {{{#!shell-session root@srv1-in2:~# mtr -c 1 -r 8.8.8.8 Start: Fri Sep 22 13:14:06 2023 HOST: srv1-in2.sb1.cosmos-lab.org Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.2.1 0.0% 1 0.2 0.2 0.2 0.2 0.0 2.|-- 192.168.3.1 0.0% 1 1.1 1.1 1.1 1.1 0.0 3.|-- 192.168.1.1 0.0% 1 1.2 1.2 1.2 1.2 0.0 4.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 5.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 6.|-- vlan-102-ar01-winlab-fsci 0.0% 1 1.9 1.9 1.9 1.9 0.0 7.|-- 172.29.50.14 0.0% 1 2.1 2.1 2.1 2.1 0.0 8.|-- 172.29.6.25 0.0% 1 1.8 1.8 1.8 1.8 0.0 9.|-- 172.29.6.58 0.0% 1 3.2 3.2 3.2 3.2 0.0 10.|-- 172.29.6.96 0.0% 1 2.7 2.7 2.7 2.7 0.0 11.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0 12.|-- 142.251.249.125 0.0% 1 3.6 3.6 3.6 3.6 0.0 13.|-- 216.239.40.167 0.0% 1 2.7 2.7 2.7 2.7 0.0 14.|-- dns.google 0.0% 1 2.8 2.8 2.8 2.8 0.0 root@srv1-in2:~# }}} === GUI Setup & Instructions === Note that the GUI web server has already been started during Linux startup. Open the browser and point it at '''http://localhost:3005''' (if using ssh tunneling) or at '''http://srv1-in1.sb1.cosmos-lab.org:3005''' if using VPN; login credentials for the GUI are: '''admin'''/'''admin'''. Once the page comes up, navigate to the DEBUG tab of the GUI: [[Image(Select-Debug.png, 400px)]] and click on the gNB Applink and UE1 Applink circles (LEDs) located under the Status bar: [[Image(Status-Buttons.png, 400px)]] When selected, the LED will turn “green” indicating the GUI is communicating with the respective node. [[CollapsibleStart(Enable Logging to DataBase)]] The configuration file WEBSERVER/server/config.js located on srv1-in1.sb1.cosmos-lab.org can be modified to enable logging to the database as follows: {{{#!td align=center config.js }}} |--------------- {{{#!td {{{#!shell //MongoDB Database parameters dbAvailable: true, dbStore: true, dbCollectionName: "nr_guidata", //This is not a fixed value and can be changed dbHost: "127.0.0.1", //Address of databases server dbPort: 27017, //End Database parameters }}} }}} |--------------- Once the configuration file has been saved the server will automatically restart and all data streamed to the GUI will be saved in the database. The Mongo client can be used to browse or extract data. The data model in use can be found in the following file: WEBSERVER/server/backend/models/tlvDataItem.js [[CollapsibleEnd]] === Sending iperf data between the gNB and UE === The following commands are provided as an example of how to send Downlink iperf data traffic on the NR 28GHz mmW OTA link via the compute machines that are attached to the gNB and UE platforms * In the UE console session start the iperf server {{{#!shell iperf3 -s -i 1 }}} * In the gNodeB session strat the iperf client {{{#!shell iperf3 -c 192.168.2.2 -i 1 -u -b 20M -l 1472 -t 30 }}} The results should be (the actual link performance will also depend on the physical antenna alignment): {{{#!td align=center '''gNodeB''' (srv1-in1) }}} {{{#!td align=center '''UE''' (srv1-in2) }}} |--------------- |--------------- {{{#!td align=left valign=top {{{#!shell-session root@srv1-in1:~# iperf3 -c 192.168.2.2 -i 1 -u -b 20M -l 1472 -t 3 Connecting to host 192.168.2.2, port 5201 [ 4] local 192.168.1.1 port 41249 connected to 192.168.2.2 port 5201 [ ID] Interval Transfer Bandwidth Total Datagrams [ 4] 0.00-1.00 sec 2.15 MBytes 18.0 Mbits/sec 1532 [ 4] 1.00-2.00 sec 2.38 MBytes 20.0 Mbits/sec 1698 [ 4] 2.00-3.00 sec 2.39 MBytes 20.0 Mbits/sec 1699 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-3.00 sec 6.92 MBytes 19.3 Mbits/sec 0.141 ms 69/4929 (1.4%) [ 4] Sent 4929 datagrams iperf Done. root@srv1-in1:~# }}} }}} {{{#!td align=left valign=top {{{#!shell-session root@srv1-in2:~# iperf3 -s -i 1 ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.1.1, port 55250 [ 5] local 192.168.2.2 port 5201 connected to 192.168.1.1 port 41249 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 2.12 MBytes 17.7 Mbits/sec 0.130 ms 25/1532 (1.6%) [ 5] 1.00-2.00 sec 2.35 MBytes 19.7 Mbits/sec 0.142 ms 25/1698 (1.5%) [ 5] 2.00-3.00 sec 2.36 MBytes 19.8 Mbits/sec 0.141 ms 19/1699 (1.1%) [ 5] 3.00-3.04 sec 0.00 Bytes 0.00 bits/sec 0.141 ms 0/0 (-nan%) - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-3.04 sec 6.92 MBytes 19.1 Mbits/sec 0.141 ms 69/4929 (1.4%) ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- }}} }}} === Analyze the NR link performance via the NR GUI === Navigate to the DEBUG tab of the GUI and select the gNB Applink and UE1 Applink buttons located under the Status bar. When selected the LED will turn “green” indicating the GUI is communicating with the respective node. [[Image(NR-gui.png, 800px)]] == Shutting Down Resources == Turn off the server nodes and NR platform devices: {{{#!shell-session omf tell -a offh -t srv1-in1,srv1-in2,rfdev2-in1,rfdev2-in2 }}} == Saving the Node Images == If you modify software on any of the server nodes save the image by the [omf save image commands] but don't forget to first prepare the node by executing (this is the example for srv3-lg1 node): {{{#!shell-session root@srv1-in1:~# ./prepare.sh }}} This script will “clean” and “dump” old log files and any other unnecessary files that do not need to be saved as part of the image, thereby reducing the size of the saved image. At the end of this script, the node will shutdown (it might block your ssh window forcing you to open another one); once back on the console, you can save that node disk image with {{{#!shell-session username@console:~# omf save -n srv1-in1.sb1.cosmos-lab.org }}} (please note the -n option and the FQDN for the node both of which are mandatory for successful saving operation).