[[Include(WikiToC)]] = !SigComm 2022 Cosmos Testbed Tutorial = == Setting an Optical Network on COSMOS Testbed == This wiki page contains the tutorial of setting up an optical network on the COSMOS Test-bed. Authors: Zehao Wang, Duke University \\ Tingjun Chen, Duke University === Description === Cosmos test-bed provides a possibility to create and use optical networks of various topologies. An example of how an optical network could be configured and used is provided. A simple experiment on switching of optical paths is described. === Experiment Topology and Resources Required === || [[Image(sigcomm2022tutorial_topology.png, width=850px)]] || Figure 1. Topology setup of the experiment. This experiment demonstrates optical switching between the short (1-hop) path, between {{{srv1-co1}}} and {{{srv1-lg1}}}, and the long (2-hop) path, between {{{srv1-co1}}} and {{{srv2-lg1}}}. It represents changing of the light path in C-RAN when a "Client" wants to dynamically change its base-band processing location between a nearby "Edge Cloud" and a further away "Central Cloud". This experiment uses the following hardware resources and configurations: * 3 compute servers and 4 ROADM units in the {{{bed.cosmos-lab.org}}} domain: {{{ srv1-co1.bed, srv1-lg1.bed, srv2-lg1.bed rdm1-co1 (10.104.1.1), rdm2-co1 (10.104.1.2), rdm1-lg1 (10.104.2.1), rdm2-lg1 (10.104.2.2) }}} * Interfaces on ToR switch {{{sw-da-co1}}}: {{{ Eth 1/1/1:1 (vlan 1066) srv1-co1 data1 Eth 1/1/66 (vlan 1066) 10g dwdm tunable (connected to Calient 5.7.3) }}} * Interfaces on ToR switch {{{sw-da-lg1}}}: {{{ Eth 1/1/1:1 (vlan 1033) srv1-lg1 data1 Eth 1/1/1:2 (vlan 1034) srv2-lg1 data1 Eth 1/1/33 (vlan 1033) 10g dwdm tunable (connected to Calient 5.7.5) Eth 1/1/34 (vlan 1034) 10g dwdm tunable (connected to Calient 5.7.6) }}} * Comb source connected to Calient 5.8.5 and 5.8.8 via a 1x2 optical splitter. == Setting Up Optical Topology == Setting up an optical topology consists in connecting ROADM units accordingly to the needs of the experiment. ROADMs need to be connected correctly to each other and to ToR switch. All interconnections are realized by Calient S320 switch. In order to support the logical connections and to realize emulate real distance between a nearby Edge Cloud and a further Central Cloud, one could choose to pass the optical signal through 1 hop (via a pair of 10km fiber spools) or 2 hops (via a pair of 10km fiber spools with the 32km Manhattan dark fiber). The following table summarizes all connections required for this experiment: * For the comb source: || ID |||| Ports Connection |||| Devices Connection |||| Comment || || 1 |||| 5.8.5->1.1.1||||comb_source_1 to ROADM1-CO1.port_1 |||| Forming background traffic into ROADM1-CO1 add port 1 || || 2 |||| 5.8.8->1.4.1||||comb_source_2 to ROADM2-CO1.port_1 |||| Forming background traffic into ROADM2-CO1 add port 1 || || 4 |||| 5.7.3<->1.1.1||||srv1-co1 with ROADM1-CO1.port_2 |||| Connecting srv1-co1 with ROADM1-CO1 add/drop port 1 || || 5 |||| 1.1.8->5.5.1||||ROADM1-CO1.line_port to fiber_spool_1 |||| Connecting ROADM1-CO1 line out to the 10 km fiber spool || || 6 |||| 5.5.2->1.1.8||||fiber_spool_2 to ROADM1-CO1.line_port |||| Connecting the 10 km fiber spool to ROADM1-CO1 line in || || 7 |||| 5.5.1->1.4.8||||fiber_spool_1 to ROADM1-LG1.line_port |||| Connecting the 10 km fiber spool to ROADM1-LG1 line in || || 8 |||| 1.4.8->5.5.2||||ROADM1-LG1.line_port to fiber_spool_2 |||| Connecting ROADM1-LG1 line out to the 10 km fiber spool || || 9 |||| 2.1.8->4.6.1||||ROADM2-LG1.line_port to dark_fiber_1 |||| Connecting ROADM2-LG1 line out to the 34 km dark fiber || || 10 |||| 4.6.2->2.1.8||||dark_fiber_2 to ROADM2-LG1.line_port |||| Connecting the 34 km dark fiber to ROADM2-LG1 line in || || 11 |||| 4.6.1->2.4.8||||dark_fiber_1 to ROADM2-CO1.line_port |||| Connecting the 34 km dark fiber to ROADM2-CO1 line in || || 12 |||| 2.4.8->4.6.2||||ROADM2-CO1.line_port to dark_fiber_2 |||| Connecting ROADM2-CO1 line out to the 34 km dark fiber || * For the short route (1-hop, 10km): || ID |||| Ports Connection |||| Devices Connection |||| Comment || || 1 |||| 2.4.1<->5.7.5 |||| ROADM1-LG1.port_2 with srv1-lg1 |||| Connecting ROADM1-LG1 add/drop port 2 to srv1-lg1 to emulate the connection to the edge cloud || * For the long route (2-hop, 10km + 32km): || ID |||| Ports Connection |||| Devices Connection |||| Comment || || 1 |||| 2.4.1<->5.7.6 |||| ROADM2-CO1.port_2 with srv2-lg1 |||| Connecting ROADM2-CO1 add/drop port 2 to srv2-lg1 to emulate the connection to the center cloud || === ROADMs Configuration === All of these configurations can be performed by Python scripts developed to work with the COSMOS test-bed. The Python commands send NETCONF commands to the ROADM. ==== Setting “Snake” Connection ==== Correct ROADM operation requires Line In port of a ROADM to always receive a light. That is why there is a dedicated transceiver (tengigabitethernet 1/33 on ToR) that sends light through all ROADMs by passing through loop-back connection on Calient S320 (port 5.5.1) and redirecting back, so the light is received on the same transceiver. This kind of connection is called “Snake”. In order maintain this “Snake” for “Experiment_1” next connections form Table 1 must be in place: 1,3,5,6,8,9. ==== tengigabitethernet 1/33/1 on ToR configuration ==== Snake Interface (to passe through all ROADMs in loop): 60 (DWDM Channel C60) 1529,55 nm 196,00 Thz with frequency range [195.95,196.05] Thz ==== ROADM MUX/DEMUX configuration ==== * {{{rdm1-co1}}}: DEMUX ports: [5202 (drop2)]; MUX ports: [4101 (add1), 4102 (add2)] * {{{rdm2-co1}}}: DEMUX ports: [5202 (drop2)]; MUX ports: [4101 (add1), 4102 (add2)] * {{{rdm1-lg1}}}: DEMUX ports: [5201 (drop1), 5202 (drop2)]; MUX ports: [4101 (add1), 4102 (add2)] * {{{rdm2-lg1}}}: DEMUX ports: [5201 (drop1)]; MUX ports: [4101 (add1)] ==== ALS Disable Sequence (for 60 seconds) ==== 1. ROADM1-CO1 booster, 2. ROADM2-CO1 booster, === Setting Optical Connections === ==== Configuring the the background traffic using comb source ==== * Channel setting 1. Comb source open channels (channels_comb_open): {{{10-19,40-44,46-49,80-89}}} 2. Comb source through channels (channels_comb_through): {{{10-19,40-49,80-89}}} 3. Transiver channels (channels_trans): 62 * Forward direction 1. Add connection from comb source 1 to ROADM1-CO1 add port 4101 with channels_comb_open pass through 2. Add connection from srv1-co1 transceiver to ROADM1-CO1 add port 4102 with channels_trans pass through 3. Connect ROADM1-CO1 line out to 10 km fiber spool 1 4. Connect 10 km fiber spool 1 to ROADM1-LG1 line in 5. Make demux of ROADM1-LG1 to pass both channels_comb_through and channels_trans to drop port 5211 6. Make mux of ROADM2-LG1 to pass both channels_comb_through and channels_trans to add port 4111 7. Connect ROADM2-LG1 line out to 34 km dark fiber 1 8. Connect 34 km dark fiber 1 to ROADM2-CO1 line in 9. Make demux of ROADM2-CO1 pass only channels_trans to drop port 5202 * Backward direction 1. Add connection from comb source 2 to ROADM2-CO1 add port 4101 with channels_comb_open pass through 2. Add connection from srv2-co1 transceiver to ROADM2-CO1 add port 4102 with channels_trans pass through 2. Connect ROADM2-CO1 line out to 34 km dark fiber 2 3. Connect 34 km dark fiber 2 to ROADM2-LG1 line in 4. Make demux of ROADM2-LG1 to pass both channels_comb_through and channels_trans to drop port 5211 5. Make mux of ROADM1-LG1 to pass both channels_comb_through and channels_trans to add port 4111 6. Connect ROADM1-LG1 line out to 10 km fiber spool 2 7. Connect 10 km fiber spool 2 to ROADM1-LG1 line in 8. Make demux of ROADM1-CO1 pass only channels_trans to drop port 5202 ==== Configuring the short route (1-hop) ==== * ROADMs 1. Make demux of ROADM1-LG1 drop only channels_trans to port 5202 2. Add connection from srv1-lg1 to ROADM1-LG1 add port 4102 * Calient switch 1. Add connections between srv1-lg1 and ROADM1-LG1 add port 4102 and drop port 5202 ==== Configuting the long route (2-hop) ==== * ROADMs 1. Add demux of ROADM1-LG1 to pass channels_trans to drop port 5211 2. Add mux of ROADM1-LG1 to pass channels_trans to add port 4111 3. Add demux of ROADM2-CO1 drop only channels_trans to port 5202 4. Add mux of ROADM2-CO1 to add only channels_trans to port 4102 * Calient switch 1. Add connections between srv2-lg1 and ROADM2-CO1 add port 4102 and drop port 5202 === Configuring the Compute Nodes === 1. Install net-tools: {{{#!shell-session sudo apt install net-tools }}} 2. Configure interfaces the {{{DATA1}}} interface on each server and assign IP addresses: {{{#!shell-session root@srv1-co1:~$ sudo ifconfig DATA1 10.111.1.1 netmask 255.255.0.0 up root@srv1-lg1:~$ sudo ifconfig DATA1 10.111.2.1 netmask 255.255.0.0 up root@srv2-lg1:~$ sudo ifconfig DATA1 10.111.2.2 netmask 255.255.0.0 up }}} == Experiment Execution == 1. Establish the long route (2-hop) 2. Ping from {{{srv1-co1}}} to {{{srv2-lg1}}}. From {{{srv1-co1}}} to {{{srv1-lg1}}} should not be pingable at this moment. {{{#!shell-session root@srv1-co1:~# ping 10.111.2.2 PING 10.111.2.2 (10.111.2.2) 56(84) bytes of data. 64 bytes from 10.111.2.2: icmp_seq=1 ttl=64 time=0.555 ms 64 bytes from 10.111.2.2: icmp_seq=2 ttl=64 time=0.508 ms }}} 3. Switch to the short route (1-hop). 4. Ping from {{{srv1-co1}}} to {{{srv1-lg1}}}(10.111.2.1). From {{{srv1-co1}}} to {{{srv2-lg1}}}(10.111.2.2) should not be pingable at this moment.