Network Control Protocol

Syntalos can coordinate recording runs across several computers on the same local network with applications that speak its network control protocol. One machine acts as the controller and broadcasts synchronization commands; all other machines are listeners that start and stop their own local recordings in response.

This page covers the protocol specification for developers who want to integrate external software with a Syntalos fleet.

For users, check out the Network Control Tutorial instead.

Specification

The reference demo scripts in tools/netctl-demo/ implement this protocol in Python.

Transport

The controller binds both ports. Listeners connect to the controller’s address.

Every command message is a two-frame ZMQ multipart message:

Frame 0: b"sy.cmd"           (topic, ASCII)
Frame 1: <UTF-8 JSON object> (payload)

ACK messages are single-frame and sent over the PUSH socket.

Common JSON fields

All messages (commands and ACKs) carry:

Command: prepare

Broadcast by the controller before a run starts. Listeners should apply the provided parameters and prepare the run.

{
  "v": 1,
  "type": "prepare",
  "sender": "rig-ctrl",
  "run_id": "019312ab-...",
  "project": "my-project",
  "subject_id": "M42",
  "subject_group": "control",
  "experiment_id": "novel-object-1"
}

When everything is prepared and the listener is ready to start acquiring data, it must send a prepare ACK.

Command: start

Broadcast after the controller has received all required prepare ACKs and has started its own master timer.

{
  "v": 1,
  "type": "start",
  "sender": "rig-ctrl",
  "run_id": "019312ab-...",
  "ts_start_us": 1737000000123456
}

Each listener uses ts_start_us to align its master clock so that t = 0 is shared across all machines. Alignment requires synchronized system clocks (NTP/PTP).

Command: stop

Broadcast when the controller’s run ends (normally or due to failure).

{
  "v": 1,
  "type": "stop",
  "sender": "rig-ctrl",
  "run_id": "019312ab-...",
  "success": true
}

ACK message

Listeners send ACKs on the PUSH socket.

{
  "v": 1,
  "type": "ack",
  "sender": "rig-cam",
  "run_id": "019312ab-...",
  "ack_for": "prepare",
  "success": true
}

On failure, add an "error" field:

{
  ...
  "success": false,
  "error": "Module 'camera' failed to initialize"
}

ACK timing

Run state machine

Controller                                 Listener(s)
──────────                                 ──────────
broadcast: PREPARE()                ──▶    receive PREPARE
  block: wait for N prepare-ACKs           [engine prepares all modules]
                                    ◀──    prepare-ACK  (success / fail)

broadcast: START(ts_start_us)       ──▶    receive START
  run starts locally                       align master timer to ts_start_us
                                    ◀──    start-ACK  (immediate)

  … run in progress …                      … run in progress …

broadcast: STOP()                   ──▶    receive STOP
  run stops locally                        [engine stops, data flushed]
                                    ◀──    stop-ACK

Controller ACK handling:

• Prepare ACKs - the controller blocks until it has received the expected number of ACKs (configurable) or times out. A timeout or a negative ACK causes the controller to abort and broadcast stop so listeners can clean up.
• Start ACKs - collected passively; the run is not delayed waiting for them. If too few arrive within the timeout, an error is raised.
• Stop ACKs - informational; the controller does not wait for them.

Listener timeout:

If a listener does not receive the start command within 30 seconds of sending its prepare ACK, it aborts its local run, sends a negative prepare ACK, and resets to idle.

Self-filter

Every instance ignores messages whose sender matches its own instance ID. This prevents an instance that has both controller and listener modes active from acting on its own broadcasts.

Versioning

Messages with "v" other than 1 are silently dropped. Future protocol versions will use a different v value and remain backwards-compatible at the transport level.