Running a load-test scenario

The operating guide for the nl6 load-test scenario subsystem — the lifecycle, fidelity mode, run isolation, reconciliation, and troubleshooting. New to the feature? Start with the overview. For copy-pasteable recipes by use case, see the runbooks; for endpoints and shapes, the REST API and report schema.

Scope: one active scenario at a time, over any one of the seven shipped push protocols — syslog, SNMP trap/inform, NetFlow v5/v9, IPFIX, sFlow, and gNMI dial-out. Each protocol opts a device in via its own export config; the scenario gates whichever protocol it targets.

Run a fidelity check

The lifecycle is submit → arm → start → (window elapses) → stop → report.

NL6=http://localhost:8080

# 1. Submit — validate + fingerprint. Returns the scenario id.
ID=$(curl -sf -X POST $NL6/api/v1/scenarios -H 'Content-Type: application/json' -d '{
  "participants": ["10.42.0.1","10.42.0.2","10.42.0.3"],
  "protocol": "syslog", "rate": 10, "window": "30s", "drain": "2s", "seed": 42
}' | jq -r .id)

# 2. Arm — resolve participants; check the excluded list before starting.
curl -sf -X POST $NL6/api/v1/scenarios/$ID/arm | jq

# 3. Start — freezes the fleet, opens the window at T0. Refused at 0/N armed.
curl -sf -X POST $NL6/api/v1/scenarios/$ID/start | jq

# 4. Wait for the window (it self-closes at T1), then finalize + fetch report.
sleep 33
curl -sf -X POST $NL6/api/v1/scenarios/$ID/stop | jq

# 5. Reconcile: report `sent` (in_window + drain) vs your collector's received.

stop is idempotent, so it is safe to call after the window has already auto-closed — you get the same report back.

To reconcile, sum in_window + drain per counters[] row and compare to your monitor's received count for the same (protocol, source_ip, collector) tuple. send_failures vs dropped separates "nl6 could not send" from "nl6 sent but the wire lost it". See the report schema.

Prefer a browser? Add ?format=html for a self-contained page (stat cards, a loss-localization bar chart, and the participant table) you can eyeball or attach to a run report; ?format=csv gives the flat join-ready projection. JSON (no format) stays the machine source of truth.

Fidelity mode

-fidelity — a silent fleet. By default every device starts pushing its background telemetry — flow, SNMP traps, syslog, gNMI dial-out — on its own cadence the moment it comes up. So a scenario's window is mixed in with steady background noise, and the fleet keeps emitting after the run ends. Start nl6 with -fidelity to invert that: the fleet is silent — no autonomous push leaves any device — except during a running scenario's [T0,T1) window, where only that scenario's gated traffic flows. Silence before the run, only the scenario during it, silence again after.

• Devices still answer polls (SNMP / SSH / HTTPS) normally — fidelity mutes only autonomous push telemetry.
• Explicit on-demand fires (POST /devices/{ip}/{trap,syslog}) still go through — a deliberate action, not background chatter.
• Fleet-wide and static: set once at startup, off by default.
• It mutes autonomous push. A gNMI dial-in subscription is client-initiated (the collector Subscribed), so it keeps streaming — cancel the subscription if you need the gNMI path quiet too.

The payoff is the cleanest possible measurement environment — a report you can diff against a collector with zero background contamination on either side. Add -fidelity to any launch command in the runbooks.

NetFlow v5 run isolation (time-window + source-IP only)

NetFlow v5 is a fixed-format protocol — no templates, no option records, no in-band field a scenario could tag. So a v5 fidelity run is isolated purely by the measurement window [T0,T1) and the participant source IPs: the collector attributes scenario traffic by (source_ip, arrival time), not by any per-flow marker. Keep this in mind when reconciling — filter the collector's v5 records to the participant IPs and the window before diffing against the report's sent. (The other protocols carry richer identity — v9/IPFIX templates, sFlow agent/sub-agent, trap varbinds — but the report's join tuple (protocol, source_ip, collector) is the same across all of them.)

Run tagging — isolating experiment traffic

On a shared collector — one that also receives background/production telemetry — you must separate this run's traffic from the noise before reconciling. Each protocol is isolated by its native lever; the report's metadata.run_tags records the mechanism + value so you know how to filter. This is tag-what-exists: the levers below are already carried by the wire encoders (or, for PEN-dependent ones, degrade cleanly).

Protocol	mechanism	Lever	PEN?
NetFlow v9	netflow9_source_id	filter received flows by the device's Source ID	no
IPFIX	ipfix_odid	filter by the Observation Domain ID (enterprise IE is a secondary, PEN-only lever)	no
sFlow v5	sflow_sub_agent_id	filter by sub_agent_id	no
gNMI dial-out	gnmi_synthetic_path	dial-out stamps the device IP in Notification.Prefix.Target; filter by target	no
Syslog 5424	syslog_sd_param	RFC 5424 SD-PARAM [nl6@<PEN> runId="<id>"]	yes
SNMP trap/inform	snmp_enterprise_varbind	enterprise varbind under the nl6 PEN	yes
NetFlow v5	window_source_ip	no taggable field — isolate by participant source IPs + [T0,T1)	n/a

• In every case the measurement window [T0,T1) plus the participant source IPs already narrow the traffic; the per-protocol lever adds a second, in-band discriminator where one exists.
• PEN-dependent levers degrade gracefully. Syslog SD-PARAM and the SNMP enterprise varbind need an IANA Private Enterprise Number. Without one (-scenario-pen unset, the default), run_tags.mechanism is window_source_ip and run_tags.degraded is true — you fall back to window
  • source-IP isolation, which is always available. Set -scenario-pen <n> once your PEN is registered to activate the clean levers; run_tags.value then carries the runId a receiver keys on.
• nl6-reconcile's PHANTOM status flags received traffic that is not in the report — usually background noise that leaked past your filter. Tighten the filter using the run_tags mechanism above.

Reconciliation walkthrough

The instrument's one job is to make loss measurable. The arithmetic:

sent       = in_window + drain          (per counters[] row, or summed)
received   = your monitor's count for the same (protocol, source_ip, collector)
loss_ratio = (sent − received) / sent

nl6-reconcile — one command, not a spreadsheet

nl6-reconcile does the join for you. It is read-only: give it a saved report and your collector's received-counts export, and it outer-joins on (protocol, source_ip, collector) and prints loss_ratio per key with an in-flight tolerance band. See Getting nl6-reconcile.

# Report as JSON (or CSV via ?format=csv); received as a collector CSV export.
curl -sf $NL6/api/v1/scenarios/$ID/report > report.json

nl6-reconcile -report report.json -received collector.csv
# PROTOCOL  SOURCE_IP   COLLECTOR     SENT  RECEIVED  DELTA  LOSS%   STATUS
# syslog    10.42.0.1   10.0.0.9:514  1000  1000      0      0.00%   OK
# syslog    10.42.0.2   10.0.0.9:514  1000  950       50     5.00%   LOSS
#
# Summary: 2 keys | 1 OK | 1 flagged | tolerance 0.50% | sent=2000 received=1950 fleet_loss=2.50%

• Inputs. -report takes the report JSON or the flat CSV projection (auto-detected). -received takes a CSV (protocol,source_ip,collector,received columns) or a Prometheus range-query result (/api/v1/query_range JSON — the last sample of each series is the received count, keyed by its protocol/source_ip/collector labels). Either input may be - for stdin.
• Tolerance. -tolerance 0.005 (default 0.5 %) is the in-flight band: |loss_ratio| within it is OK — records still on the wire at T1 cause a tiny delta that is not real loss. Widen it for lossy paths, tighten to 0 for an exact check.
• Statuses. OK · LOSS (received < sent) · DUP (received > sent — duplication) · MISSING (in the report, no received row — total loss or a join gap) · PHANTOM (received but not in the report — background noise leaked into your export; see the run-tagging notes to isolate it).
• Exit code. 0 when every key is OK, 1 when any key is flagged — so nl6-reconcile drops straight into a CI gate. Use -format csv|json for a machine-readable diff.

Getting nl6-reconcile

nl6-reconcile is a small, stateless, cross-platform Go CLI — it runs wherever you diff (your laptop, a CI runner, the monitoring host), not on the Linux-only simulator host. Three ways to get it:

• Download a release binary (recommended). Grab nl6-reconcile-<os>-<arch> from the releases — linux / darwin / windows × amd64 / arm64 — then chmod +x and put it on your PATH.

• go install (any platform with a Go toolchain):

  go install github.com/labmonkeys-space/nl6/go/cmd/nl6-reconcile@latest

• From source: make reconcile builds go/nl6-reconcile; run it as ./go/nl6-reconcile or copy it onto your PATH.

nl6-reconcile -version prints the build version (stamped from the same release tag as the simulator, so a report's metadata.nl6_version and the diff tool's version together pin a reconciliation).

To reconcile by hand instead:

1. Join the report's counters[] (or the CSV projection) against your monitor's received-counts export on (protocol, source_ip, collector) — the report is keyed by exactly that tuple so the join is 1:1.
2. Per row, compute loss_ratio. 0 = perfect fidelity. A positive ratio localizes to that source_ip; a negative ratio (received > sent) means duplication in the path (retransmits, a misconfigured fan-out).
3. Loss model. sent is the authoritative denominator — it is exact and, with a fixed seed, reproducible. Any shortfall is real wire/collector loss, not measurement noise. The simulator proves this: an injected X% drop is recovered by this arithmetic within ±1 pp (0 % exactly) — see the examples/scenario-syslog-fidelity "injecting known loss" section.

When the numbers don't add up

• received < sent but the network is fine → the collector host is dropping datagrams before your monitor counts them. Check the kernel UDP drop counters: nstat -az | grep -i Udp or cat /proc/net/snmp | grep Udp: — a rising UdpRcvbufErrors / InErrors means the receive buffer overflowed under burst. Raise the collector's SO_RCVBUF (or net.core.rmem_max) and re-run; that loss is on the collector, not the wire.
• received == 0 despite sent > 0 → almost always the collector host's rp_filter dropping 10.42.0.0/16 source IPs (see Collector unreachable below), not real loss.
• Always confirm the join tuple matches: a mismatched collector column means you are diffing against the wrong receiver.
• sent far below informational.requested → the shared global cap throttled the run: the shortfall is in informational.deferred (fires the cap had no token for — not fired, not lost). Deferral is not loss and is excluded from sent (the loss denominator), so loss_ratio stays honest. Raise the cap, lower the profile rate, or accept the throttle — but don't read it as pipeline loss.

Clock sync (chrony/NTP) — required for time localization

Reconciliation totals (sent vs received) need no clock agreement — a counter is a counter. But time localization (sub_windows) does: nl6 buckets each send by its write-return time relative to T0, and to line your received data up against those buckets you must bucket your records by receive-time relative to the same T0. If nl6's host and the collector's host disagree on the wall clock, the two bucketings shear and a loss that is really in [T0+30s, T0+45s] appears smeared across neighbours.

• Run chrony (or ntpd) on both the simulator host and every collector host, disciplined to the same upstream sources. Confirm with chronyc tracking — keep the estimated offset well under one sub_window_duration (a 30 s window → 3 s buckets → sub-second sync is ample; tighten for shorter windows).
• The report's metadata.t0/t1 are the simulator's clock. Bucket your received records against that t0, not your own start time.
• Fleet totals and loss_ratio are unaffected by skew — only the per-bucket sub_windows attribution is. If localization looks smeared but totals reconcile, suspect clock skew before suspecting the pipeline.

Troubleshooting arm failures

arm never fails wholesale for a bad participant — it reports each one in the readiness excluded[] list as {device, reason, remediation_hint}. Check that list before start.

Symptom	Cause	Fix
excluded[].reason = "device not found"	The participant IP is not in the live fleet.	Create the device (POST /api/v1/devices) before arming, or remove it from participants.
excluded[].reason = "device has no syslog exporter"	The device exists but syslog export is not enabled on it.	Enable syslog export on the device — the -syslog-collector seed flag (auto-start batch) or a per-device syslog block in POST /api/v1/devices.
excluded[].reason = "device deleted between arm and start"	The device was deleted in the arm→start gap (before the freeze).	Re-arm after the fleet is stable.
start → 409 … 0/N participants armed	Every declared participant was excluded.	Fix the exclusions above; you cannot start a scenario with no armed devices.
POST /scenarios → 409 a scenario is already active	Another scenario is still non-terminal (only one active at a time).	Stop / abort / DELETE the active scenario first (GET /api/v1/scenarios/{id} to see its phase).
start / stop → 409 cannot … in phase …	Illegal lifecycle transition.	The 409 body names the current phase and the resolving verb; follow the phase/verb matrix.
report → 409 … available only after stop or abort	The scenario has not finalized yet.	Stop it (or wait for the window to auto-close at T1), then re-request the report.
Device create/delete → 409 fleet … frozen by running scenario	Membership is frozen while a scenario runs, so counter deltas can't be corrupted mid-window.	Wait for the scenario to finish, or stop it.

Collector unreachable

If the scenario runs but your monitor receives nothing:

• report.send_failures is high → nl6 itself could not send (socket bind failure, no route). Check the device's configured collector host:port and that the simulator container can reach it.
• report.sent is high but the collector received 0 → the datagrams left nl6 but were dropped en route. The usual culprit is the collector host's reverse-path filter rejecting UDP with 10.42.0.0/16 source IPs when -syslog-source-per-device=true (the default). Relax it (net.ipv4.conf.*.rp_filter=0 or 2) or run with -syslog-source-per-device=false so datagrams carry the simulator's own source IP.
• Confirm the device's collector matches where your monitor listens (counters[].collector in the report).

Heartbeat / silence-alert warning

⚠️ A scenario suppresses the fleet's ordinary background telemetry cadence for its participants for the entire time it is armed and running. Before T0, background fires from armed participants are generation-suppressed (informational.background_suppressed in the report counts them). This is intentional — it keeps the measurement window clean — but it means:

• Silence-based alerts (heartbeat monitors, "no syslog in N minutes" detectors) watching a participant may fire during a scenario because the device's normal chatter is paused. Schedule fidelity checks in a maintenance window, suppress those alerts for the participants, or keep windows short.
• Non-participant devices are unaffected — their background cadence continues.

Graceful abort produces a finalized report

A SIGTERM/SIGINT during a running scenario does not silently discard the run. The shutdown path aborts the scenario through the same drain-and-finalize pipeline a normal stop uses (bounded by the drain grace), so:

• The report is finalized and marked phase: "aborted", with metadata.t0 and a metadata.t1 equal to the abort instant (the window that actually ran, always earlier than the planned T1). duration reflects the truncated window.
• It is immutable and served by GET /api/v1/scenarios/{id}/report exactly like a stopped report — same schema, same ledger identity, same counters[]. Only the phase differs.
• The fleet freeze is released as part of the abort, so device CRUD works again.

The catch is timing: the report lives in memory (see the non-goal below), so a graceful abort finalizes it but the process still exits. If you need the abort report, fetch it before the process is fully gone — or drive the abort by other means and read the report while nl6 is still up.

SIGKILL (or a crash / OOM) makes no promise beyond the in-memory-loss non-goal below: there is no abort pipeline, so no report is produced.

Non-goal: scenarios are in-memory

Scenarios do not persist. The active scenario, its ledger, and its finalized report live entirely in the simulator process's memory. They do not survive a restart:

• Restarting nl6 (or the container) discards any active scenario and any finalized-but-not-yet-fetched report. Fetch the report before restarting.
• There is no scenario history or store — GET /api/v1/scenarios/{id} returns 404 for any ID the current process did not allocate, and IDs (s-000001, …) reset on restart.
• On a graceful shutdown (SIGTERM/SIGINT) a running scenario is aborted first — it finalizes its report (still in-memory) and releases the fleet freeze — so the shutdown is clean, but the report is still lost once the process exits.

Persistence is a deliberate non-goal for this subsystem: a fidelity check is a short, operator-driven experiment whose result is consumed immediately (diffed against a monitor), not an audit log. Capture the report JSON yourself if you need to keep it.