[[TOC(hardware*,depth=3)]]


== Network Architecture ==
There exist three primary networks that are accessible by experimenters in COSMOS. Device connections to these networks are determined by the number of interfaces and their maximum bandwidth. Every device has at least a connection to the [#ControlPlane Control Plane]. Then depending on the number of additional interfaces, the device may have a connection to either one or both of the [#DataPlane Data Planes]. Please see the device specific pages for more detailed information.

=== Control Plane ===
The control plane is a physically separate network designed to facilitate infrastructure control and user access to end devices. The control plane network runs on top of Dell S4048-ON switches providing up to 10G connectivity between devices. Please note that performance on the control plane is not guaranteed or deterministic and the network is only meant for experiment coordination and debugging.

=== Data Plane ===
There exist two parallel data plane networks that are available to experimenters. The data plane networks are interconnected with Dell Z9100-ON switches providing up to 100G connectivity between devices. The switches are built on the Broadcom Tomahawk chipset and support Openflow 1.3 as well as ONIE. The data planes are meant for latency and bandwidth sensitive experiment traffic where performance is intended to be deterministic.

=== User facing network architecture ===
[[Image(cosmos_logical_net.png, 900px)]]

All radio devices and servers are connected to the control and data logical networks, described above.
The data network is further connected to the optical underlay. By making requests to the appropriate service, users can configure logical connections to use a particular path across both ethernet switches and and the underlying optical fabric.

Non-user facing networks connect services, and manage hardware.

==== Locality within the network ====
While all devices within the network may be connected, there are bandwidth and latency considerations.
For convenience, devices within the same physical location / topology have matching suffixes.
For example, sdr1-s1-lg1 and sdr2-s1-lg1 share the same next hop switches for their data planes, as do srv1-lg1 and srv2-lg1.

Explanation of suffixes is at [wiki:naming names of things]


=== Isolation and infrastructure

We enforce a split between "Control", "Data", and "Infrastructure" networks. Control and Infrastructure share common physical hardware (Switches, cables, etc.), using VLANs to provide isolation. Together these provide services and user access to the testbed.

The Data networks operate on physically separate hardware, to preserve repeatability of experiments, and to prevent experiments from interfering with testbed services.

L2 Networks do not span sites, with access being done via L3 tunnels, or via public internet.

The Instrumentation network is special, in that it is accessible to experiments, but is managed by infrastructure services. This provides, for example, optical or RF devices that need direct L2 access to multiple domains. Access to this network is granted on a case by case basis to experiments. This is also the only L2 network that spans sites, via L2 Tunnel.

Consoles give users access from outside and communicate with services on the DMZ network, and access to the testbed devices over control. 

=== Network Roles
[[Image(COSMOS Net Arch.png, 900px, align=right)]]
Each Domain has a control network, that is statically configured, as well as two configurable data plane networks. Imaging and other configuration tasks is done via the control network, while experiment and radio traffic is passed on the data networks.

In addition, each "site" has some infrastructure networks.

* CM: used for power control of nodes, not directly accessible to users, instead is managed by services
* Inside: Private network for service traffic, VM Storage traffic, etc.
* DMZ: User accessible network for service traffic
* MGMT: Private network for control of network devices, VM Cluster synchronization, etc.


=== Logical Sites ===

Currently, there are two logically distinct sites.
* The NYC COSMOS Deployment
 * bed.cosmos, including exterior radios and optical underlay
 * sb2.cosmos
* WINLAB, 671 Rt 1 South, North Brunswick, NJ
 * sb1.cosmos
 * grid.orbit
 * outdoor.orbit
 * sb1 - sb10.orbit

=== Vlan Assignment

{{{
#!Markdown

| vlan id | subnet        | name           | location |
|---------|---------------|----------------|----------|
| 101     | 10.101.0.0/16 | mgmt_win       | winlab   |
| 102     | 10.102.0.0/16 | inside_win     | winlab   | 
| 103     | 10.103.0.0/16 | cm_win         | winlab   | 
| 104     | 10.104.0.0/16 | dmz_win        | winlab   | 
| 105     | 10.105.0.0/16 | mgmt_nyc       | nyc      | 
| 106     | 10.106.0.0/16 | inside_nyc     | nyc      | 
| 107     | 10.107.0.0/16 | cm_nyc         | nyc      | 
| 108     | 10.108.0.0/16 | dmz_nyc        | nyc      | 
| 109     | 10.109.0.0/16 | instrument     | both     | 
| 110     | 10.110.0.0/16 | bed_ctrl       | nyc      | 
| 111     | 10.110.0.0/16 | bed_data1      | nyc      | 
| 112     | 10.110.0.0/16 | bed_data2      | nyc      | 
| 113     | 10.110.0.0/16 | sb1_ctrl       | winlab   | 
| 114     | 10.110.0.0/16 | sb1_data1      | winlab   | 
| 115     | 10.110.0.0/16 | sb1_data2      | winlab   | 
| 116     | 10.110.0.0/16 | sb2_ctrl       | nyc      | 
| 117     | 10.110.0.0/16 | sb2_data1      | nyc      | 
| 118     | 10.110.0.0/16 | sb2_data2      | nyc      | 

}}}