wiki:OMFServicesDeployment

Version 2 (modified by editor, 7 days ago) ( diff )

Updating ORBIT/COSMOS service deployment topology

ORBIT/COSMOS Service Deployment Topology

Overview

Two-site testbed architecture:

  • North Brunswick, NJ (Rutgers University): IPs 10.0.0.0 - 10.63.255.255
  • New York City, NY (Columbia University): IPs 10.64.0.0 - 10.127.255.255
  • Connected via IP tunnel for management traffic

19 Ruby/Sinatra microservices + 1 Python/Flask service + 1 CLI tool (omf-expctl) manage all testbed operations.

North Brunswick Hosts

am1 (10.50.0.41) — Aggregate Manager Proxy

Service Port Version Description
omf-agg-mgr-proxy 5054 v1-1 Service discovery, API proxy
omf-rf-control 5001 v0-2 RF attenuator control
omf-rf-switch 5002 v0-2 RF switch matrix control
omf-xy-table 5003 - XY table proxy (→ RPi controllers)
omf-array-mgmt 5004 - Antenna array management (Python/Flask)

am4 — Development Server

  • All service source repositories at /home/seskar/omf-*
  • Build host for all .deb packages
  • Not running production services

am5 (10.50.0.45) — Core Services

Service Port Version Description
omf-cmc 5013 v1-9 Chassis management (IPMI, HTTP CM, PDU)
omf-scheduler 5016 v1-1 Reservation scheduler with LDAP host management
omf-rfmatrix 5020 v1-2 RF attenuator matrix control
omf-status 5021 v1-1 Testbed node power status aggregation

repository2 (10.50.0.22) — Imaging & User Services

Service Port Version Description
omf-account-mgmt 5017 v1-3 User/group registration, LDAP lifecycle
omf-frisbee 5011 v1-4 Frisbee multicast daemon management
omf-pxe 5010 v1-6 PXE boot configuration
omf-saveimage 5012 v1-4 Disk image capture via netcat
omf-user-stats 5015 v1-4 Usage statistics (disk, scheduler, imaging)
iso-server 8080 - ISO image HTTP server

mgmt1 (10.250.0.8) / mgmt2 (10.250.0.9) — Network Infrastructure

  • DHCP: ISC DHCP Server 4, 2,145 static hosts, failover pair
  • DNS: BIND 9, 2,563 A records, 25 forward zones, 53 reverse zones
  • Kea DHCP: Installed but stopped (migration pending)

Other NJ Infrastructure

Host Service Purpose
web1 cosmos-portal React SPA + Apache reverse proxy
db1 (10.0.0.51) LibreNMS Network monitoring (190 devices)
mysql1 MySQL Shared database server
amqp.orbit-lab.org RabbitMQ MQTT broker for EC↔RC communication
gitlab.orbit-lab.org Git Lab Source code repository

ORBIT Console Servers (9)

Each runs omf-cmonitor (port 5000) + omf-expctl:

Console Domain
grid.orbit-lab.org 20x20 indoor grid
sb1.orbit-lab.org Sandbox 1
sb2.orbit-lab.org Sandbox 2
sb3.orbit-lab.org Sandbox 3
sb4.orbit-lab.org Sandbox 4
sb7.orbit-lab.org Sandbox 7
sb9.orbit-lab.org Sandbox 9
outdoor.orbit-lab.org Outdoor network
instrument.orbit-lab.org Instrumentation

New York City Hosts

am-crf.cosmos-lab.org — NYC Aggregate Manager

Service Port Version Description
omf-agg-mgr-proxy 5054 v1-1 Local AM proxy (mirrors am1)
omf-frisbee 5011 v1-4 Local frisbee (127.0.0.1)
omf-pxe 5010 v1-6 Local PXE (127.0.0.1)
omf-saveimage 5012 v1-4 Local saveimage (127.0.0.1)

COSMOS Console Servers (9)

Each runs omf-cmonitor (port 5000) + omf-expctl:

Console Domain
osc.cosmos-lab.org Open Square Campus
indigo.cosmos-lab.org Indigo
accord.cosmos-lab.org ACCORD
sb1.cosmos-lab.org COSMOS Sandbox 1
sb2.cosmos-lab.org COSMOS Sandbox 2
weeks.cosmos-lab.org Weeks
rrail.cosmos-lab.org RRAIL
bed.cosmos-lab.org BED
nebula.cosmos-lab.org Nebula

Specialized NYC Hardware

Host Service Port Description
pi1-auden.sb1.cosmos-lab.org omf-cosmos-cm, omf-auden 5018, 5019 Radio not responding (HW issue)
pi2-auden.sb1.cosmos-lab.org omf-cosmos-cm, omf-auden 5018, 5019 Working, auto-recovery
xytable1.sb1.cosmos-lab.org omf-xytable-ctrl 80 XY table controller (RPi)
xytable2.sb1.cosmos-lab.org omf-xytable-ctrl 80 XY table controller (RPi)

Service Dependency Map

                    ┌─────────────────────┐
                    │   cosmos-portal     │
                    │     (web1)          │
                    └──────────┬──────────┘
                               │ Apache reverse proxy
                    ┌──────────▼──────────┐
                    │  omf-agg-mgr-proxy  │
                    │  am1:5054 / am-crf  │
                    └──────────┬──────────┘
           ┌───────────────────┼───────────────────┐
           │                   │                   │
    ┌──────▼──────┐    ┌──────▼──────┐    ┌──────▼──────┐
    │ omf-cmc     │    │ omf-frisbee │    │ omf-pxe     │
    │ am5:5013    │    │ repos:5011  │    │ repos:5010  │
    └─────────────┘    └─────────────┘    └─────────────┘

    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
    │ omf-sched   │    │ omf-acct    │    │ omf-status  │
    │ am5:5016    │    │ repos:5017  │    │ am5:5021    │
    └──────┬──────┘    └──────┬──────┘    └─────────────┘
           │                  │
     ┌─────▼─────┐     ┌─────▼─────┐
     │  MySQL    │     │   LDAP    │
     │  mysql1   │     │   ldap1   │
     └───────────┘     └───────────┘

Port Registry

Port Service Deployment
5000 omf-cmonitor Console servers (18 instances)
5001 omf-rf-control am1
5002 omf-rf-switch am1
5003 omf-xy-table am1
5004 omf-array-mgmt am1
5010 omf-pxe repository2, am-crf
5011 omf-frisbee repository2, am-crf
5012 omf-saveimage repository2, am-crf
5013 omf-cmc am5
5015 omf-user-stats repository2
5016 omf-scheduler am5
5017 omf-account-mgmt repository2
5018 omf-cosmos-cm Pi nodes
5019 omf-auden Pi nodes
5020 omf-rfmatrix am5
5021 omf-status am5

Network Architecture

IP Ranges

Range Site Usage
10.0.0.0/16 NJ Management infrastructure
10.1.0.0/16 NJ Control network (ORBIT domains)
10.50.0.0/16 NJ Server infrastructure
10.250.0.0/16 NJ DHCP/DNS management
10.64.0.0 - 10.127.0.0 NYC COSMOS networks

MQTT Broker

  • Server: amqp.orbit-lab.org (10.50.0.44), port 1883
  • Used by: omf-expctl (EC) ↔ omf-resctl (RC on PXE-booted nodes)
  • Topic format: omf/<fqdn-with-dashes>/{cmd,reply,status}
  • Web Socket: port 15675/ws (proxied through web1 for portal)

Configuration Management

All services load configuration in this order (later overrides earlier):

  1. default/<service>.yml — built-in defaults (shipped in .deb)
  2. /etc/omf-services/common.yml — global settings (MySQL, LDAP)
  3. /etc/omf-services/<service>.yml — service-specific overrides
  4. ./config.yml — development-only (not installed)

Shared Components

Git Submodules

Submodule Purpose
omf-common Sinatra base class, DSL, XML helpers, config loading
omf-logging-db Active Record database modules (usage, imaging, login audit)
omf-ldap LDAP client wrapper (used by omf-account-mgmt, omf-scheduler)

PXE Images

Image Kernel Purpose
omf-5.8 6.18 LTS (Alpine 3.23) Regular PCs + Dell servers
omf-5.9 5.15 LTS Super Micro/Thinkmate (kernel 6.18 crashes)
omf-5.10 TBD Hardware inventory scanning (with UHD)

Last updated: March 2026

Note: See TracWiki for help on using the wiki.