Measuring a collector's ingest ceiling

nl6 can tell you what it put on the wire. It cannot tell you what your collector accepted. This page is the method for finding that second number, and for not fooling yourself while you do.

The worked example throughout is OpenNMS Horizon syslog ingest, because that is the case the numbers came from. The method generalises to any push protocol nl6 exports.

Why a delivered number is not an answer

A load generator reports what it sent. UDP has no failure signal, so "sent 60,000, failures 0" is true even when the collector received nothing at all. Every claim about capacity therefore needs a second measurement taken on the collector side, and the gap between the two has to be explained rather than assumed.

The first attempt at this measurement got the shape of the answer right and the mechanism wrong. It reported a rate and named a Kafka partition count as the likely limiter, on the strength of a config value and a core that was not CPU-bound. The real mechanism was arithmetic nothing had measured: messages per work unit divided by the time to process one. The difference matters because the wrong cause implies the wrong fix.

That said, the first attempt's instinct was better than its reasoning, which is worth admitting because it is the more common shape of being wrong. Partitions turned out to be the only tuning that moved the number at all, so a correction that dismissed them as the wrong suspect over-corrected. The mechanism really was batch size over dispatch cost, and the lever the throughput numbers pointed to really was partitions. Those are different questions, and it is easy to answer one while believing you answered the other.

That lever was then disqualified on correctness, so nothing above should be read as a recommendation to add partitions. See the ordering check for why, and for what is left afterwards.

Vocabulary

One quantity travels under four names, and conflating them is the most common way this measurement goes wrong.

term	meaning here
work unit	whatever the collector processes as one indivisible chunk
record, batch, dispatch	the same thing, named by whichever layer you are looking at
message	one datagram nl6 emitted
event	one row the collector persisted

A work unit usually holds several messages. One message does not necessarily become one event: a collector may coalesce or expand. Where the two differ, say so, or the identity check below will fail for a reason you never modelled.

The OpenNMS worked example uses "Core" for the product component and "vCPU" for processor count, because "core" means both.

Two different numbers, both called "the ceiling"

number	what it means	how you get it
service rate	how fast the pipeline drains a backlog	offer a burst, then measure the drain
sustained ceiling	the highest rate the pipeline keeps up with	ramp the offered rate, watch the queue slope

These are not the same, and the second is almost always the one an operator wants. "Can my collector keep up with 30,000 devices?" is a question about steady state.

A result must state which one it measured. They are not interchangeable, and a number reported without that label is unusable by anyone who did not run it.

Define the sustained ceiling as the highest offered rate at which the collector's input queue depth stays bounded over a sustained window. Bounded, not zero: queue depth oscillates normally. The test is the slope. Sample queue depth at a fixed interval across a 15 minute window and fit a line. A slope that stays positive across the whole window means the offered rate is above the ceiling; a slope oscillating around zero means it is at or below.

That makes the ceiling a search rather than a single run, and a cell is not finished until its final rate has held flat for the full window. A rate that holds for 60 seconds and degrades at 10 minutes is not a ceiling.

Three-point reconciliation

Two measurement points cannot distinguish loss from slowness. Three can.

 (1) nl6 `sent`             what reached the wire
      |  gap = loss between generator and queue
 (2) queue input            what arrived
      |  gap = backlog, or drops inside the collector
 (3) persisted records      what was accepted

Point (1) is the report's sent field, which is in_window + drain. It is deliberately not emitted: that also counts send_failures, dropped and suppressed_pre_window, none of which reached the wire, so using it would charge nl6-side non-sends to the network. See the report schema, which names sent as the loss denominator.

Report all three. A run that reports only (1) and (3) cannot tell a lossy pipeline from a slow one, and those have opposite fixes.

The (1) to (2) gap covers everything between the generator and the queue: datagrams dropped in the network, a receiver whose buffer overran, and any loss inside the collector's own pre-queue path. The (2) to (3) gap is backlog or in-collector drops, and a consumer-side drop counter sits on this segment, not the first.

When zero loss is required, and when it is not

Loss instruments must read zero on runs that claim a rate: the loss-isolation control and every at-or-below-ceiling cell.

They will not read zero while searching above the ceiling, and that is expected. Probing above the ceiling is how the search finds it, and on a UDP path an overrun receiver is the signal you are looking for. An above-ceiling probe with loss is a valid probe; it is simply not a result.

The loss-isolation control

Run at roughly half the measured ceiling, sustained, with the queue flat throughout. There (1) and (3) must agree within the reconciliation tolerance nl6 itself uses (nl6-reconcile defaults to 0.5%), not bit-exactly: (3) is a database count and carries its own noise.

If they do not agree, the pipeline is lossy at any rate. That invalidates the ceiling, rather than annotating it.

Instrumentation

Find the collector's equivalents of these before running anything. Without them you are measuring a black box and guessing at the mechanism.

you need	OpenNMS Horizon equivalent
arrival count, point (2)	producer-side offset delta on the sink topic
queue depth over time	Kafka consumer-group LAG for the sink topic
work-unit service time	..._sink_consumer_syslog_dispatchtime_*
messages per work unit	..._sink_consumer_syslog_messagesize_*, plus offset delta versus messages sent
drop counter	..._sink_consumer_syslog_dropped_count
accepted count, point (3)	windowed, filtered count of the run's events

Two cautions on the OpenNMS names, both of which will cost you a matrix if you skip them.

They are one deployment's rendering, not canonical: the all-lowercase dispatchtime and messagesize appear that way only when the Prometheus JMX exporter runs with lowercaseOutputName: true. Confirm the names on your build before trusting a run, since a rename between versions invalidates a whole session silently.

The accepted count must be windowed and filtered. An unqualified select count(*) from events is a lifetime total across every event source, including the collector's own internal events, so it can never match an offered rate. Take a delta across the run window and restrict it to the events the run produced.

Making the identity reproduce

throughput = (messages per work unit / service time per work unit) × concurrent workers

The worker term matters: on a single-consumer pipeline it is 1 and disappears, but any run that varies parallelism must carry it or the identity under-predicts by exactly the worker count.

Service time is wall clock, not CPU, and the difference decides how the identity scales. Where the per-message cost is dominated by waiting (database round trips, network), adding workers overlaps those waits and throughput rises without CPU rising with it. Measured here, 4 workers took ~130/s to ~320/s, a 2.4x gain from 4x the workers, while the collector used only 1.2 of 4 available vCPU. Treat that figure as an illustration of scaling arithmetic and not as a capacity result: the same configuration is disqualified on ordering further down. Expect sub-linear scaling, and treat a large gap between predicted and measured as a pointer to the next constraint rather than a broken model: in this case the database, the most loaded component at that rate.

State which statistic you used. The mean and the median give different answers (4 / 0.030 = 133, 4 / 0.0294 = 136), so a reader replicating your check needs to know which one you meant.

If the identity does not reproduce the measured rate, something is missing from the picture, and the ceiling number is not yet understood.

Before any run: silence the generator

Background emission inflates (3) without touching (1). It is the one failure mode three-point reconciliation cannot self-detect, because it makes the pipeline look like it is running ahead of the load.

nl6's per-device interval and tick_interval are accepted, echoed back, and not honored (nl6#445): every device fires at the simulator-wide cadence regardless. A long per-device interval therefore does not silence anything, and reading the value back confirms a setting that is not in force.

Start nl6 with -fidelity, then verify the generator is actually silent before offering load: its sent counter must not move while idle. A measurement that skips this check is measuring its own background noise.

Staged gates

Run the cheapest test that can invalidate the most work first. Each gate can end the investigation early, which is a successful outcome.

G1  Is service time FIXED per work unit, or proportional to its contents?
     vary the work-unit size, plot service time
     |- proportional  -> batching is not a lever, drop that axis
     '- mostly fixed  -> batching is the lever

G2  Does the collector parallelise when given more partitions?
     add partitions, re-read the consumer-group assignment
     |- consumer count unchanged -> that axis is inert, drop it
     '- consumer count rises     -> the axis is real

The matrix that follows is not a gate; it is the work these two gates decide the shape of.

G2 has two traps, and the second one loses data.

First, on Kafka num.partitions only affects newly created topics. Raising it without an explicit --alter on the existing topic changes nothing, which looks exactly like the collector failing to parallelise and would retire a live axis on no evidence.

Second, and more serious: the producer starts using new partitions immediately, while a running consumer may never discover them. Measured on Horizon 36.0.3, going from 1 to 4 partitions left the new three carrying thousands of records that did not appear in the consumer group at all, with no consumer, no committed offset and no lag tracked. They were never processed until the collector was restarted.

So a reader who alters a live topic and then checks the assignment gets both a data-stranding incident and a false "inert" reading, because the assignment is still what it was. Restart the collector after altering, re-read the assignment, and only then judge the axis. Then confirm the added consumers are distinct instances doing real work rather than one consumer holding several partitions.

Do not perform this on a production topic without planning for the restart. Partition counts also cannot be reduced, so the change is one-way.

If both gates fail, the honest answer is "this collector's ceiling is X and only a code change moves it". That is worth knowing and costs about an hour.

Load shape is a variable, not a detail

An aggregate rate does not determine the work the collector sees.

Where the collector aggregates per source and flushes on an interval, batch size follows the PER-DEVICE rate:

   batch size  ~  per_device_rate x flush_interval

Measured on Horizon 36.0.3, whose syslog sink keys aggregation per host and flushes at roughly 500 ms, at one fixed aggregate rate of ~3000/s:

devices	per-device rate	messages per work unit
500	6/s	4.0
63	47/s	24.0

More devices at the same aggregate rate makes the collector slower, because it shreds the batches. That is the opposite of the intuition that a fleet is just a rate.

Two consequences for any result:

• A ceiling figure is meaningless without the device count and per-device rate that produced it. Report them beside the number, and carry them in the manifest as controls.
• Comparing two runs at the same aggregate rate but different fleet sizes compares two different workloads, not two configurations.

The matrix

Declare exactly one independent variable per run. The axes are whichever gates survived, so the cell count follows from them.

axis	levels (OpenNMS example)	survives if
A work-unit size	the sink's batch-size setting, at three levels	G1 passed
B consumer parallelism	partitions and consumers: 1, 4	G2 passed

Both axes gives six cells, one axis gives three, neither gives none. The baseline cell is the untouched configuration, and it is run twice: once first, once last, so drift across the session is visible rather than assumed absent.

Discard any cell whose observed messages-per-work-unit drifted materially from its declared level. That cell is not the configuration it claims to be. Note that a queue's batch-size setting is often a soft target rather than a hard cap, so decide and record what "materially" means before running, instead of adjudicating it afterwards.

Everything else is a control, and controls belong in the manifest so two runs can be compared: generator version and device count, protocol and message format, collector version, VM sizes, JVM and GC flags, database durability settings, and whether devices are provisioned as nodes. Two runs are comparable only when every control matches and exactly one axis differs.

The manifest

Gather at run time, never reconstruct afterwards. It must be able to express the result the method demands, which means carrying the three reconciliation counts and the correctness check, not only the tuning:

{
  "measured_quantity": "sustained_ceiling",
  "sut": {
    "queue": { "partitions": 1, "batch_setting": "<property>=<value>" },
    "consumers": { "count": 1, "assignment": ["syslog-0"] }
  },
  "controls": { "generator_version": "...", "devices": 500, "collector_version": "..." },
  "reconciliation": {
    "sent": 0, "arrived": 0, "persisted": 0,
    "drop_counters": { "consumer_dropped": 0 },
    "loss_isolation_run": { "sent": 0, "persisted": 0, "within_tolerance": true }
  },
  "observed": {
    "service_time_ms": { "statistic_used": "mean", "p50": 0, "mean": 0, "min": 0, "max": 0 },
    "work_unit_size_bytes": { "mean": 0, "max": 0 },
    "messages_per_work_unit": 0,
    "concurrent_workers": 1,
    "queue_slope": "flat|positive",
    "ordering_preserved": true
  }
}

messages_per_work_unit and service_time_ms are derived controls. If a cell's observed values drifted from its declared level, that cell must be discarded rather than reported.

Correctness confounds

If a tuning change can alter the collector's output, throughput alone is not a result.

Parallelism can reorder events. Message queues typically guarantee order only within a partition. Unless records are keyed by device, raising the partition count lets two messages from one device be processed out of order.

For a monitoring system that is semantically loaded, not cosmetic. A linkDown and linkUp pair delivered inverted leaves an interface latched down that is actually up.

So the parallelism axis carries a mandatory ordering check: emit a known-ordered per-device sequence and assert the order survives. Where the collector's persisted records cannot reconstruct that sequence, assert instead that all of one source's work units land on a single partition, which is sufficient because the ordering guarantee holds only within a partition. A result showing higher throughput with broken ordering is reported as a regression, not as a gain.

Work-unit size changes alter packing rather than order, so that axis carries no equivalent confound, which is a further reason to test it first.

The check is not a formality: it fired

The assertion was run against Horizon 36.0.3 at 4 partitions, and it failed.

Forty strictly sequential messages from a single device produced seven work units, spread across three of the four partitions. Messages are aggregated per host within a work unit, so ordering holds inside one record. It is the spread of records across partitions that breaks it, because with one consumer per partition those records are processed concurrently.

What was measured is the spread, not its cause. Records from one device are evidently not partitioned by device, but this test does not establish why. An unkeyed record and a record keyed by something other than the device would both produce this result, and modern queue clients do not necessarily round-robin unkeyed records anyway. Naming a mechanism here would be the same unmeasured-cause error this page opens by criticising. The disqualification does not depend on the cause: spread across partitions is sufficient, because the ordering guarantee exists only within a partition.

The consequence is concrete rather than theoretical. A linkDown in one record can be persisted after the linkUp in the next, leaving an interface latched down while it is actually up. Whether anything downstream would catch that was not measured, and on this deployment nothing obviously would.

So on that deployment the only tuning that raised throughput, 130/s to ~320/s, is reported as a regression. The other axis had already failed: service time was proportional to batch contents at roughly 3 ms fixed plus 6.7 ms per message, so n / (3 + 6.7n) asymptotes near 149/s and batching buys about 12%. That leaves no tuning-level lever at all, since keying the records by source would be a code change.

A throughput-only report would have recommended four partitions. That is the entire reason this check is mandatory rather than advisory.

On instruments. A collector's persisted events may not be able to verify ordering end to end. Here, unmatched syslog persists a generic message carrying neither device identity nor original content, so the sequence cannot be reconstructed downstream. Partition spread is then the usable instrument, and it is the stronger disqualifier: it proves the guarantee is absent regardless of whether an inversion happened to occur during the run. It is correspondingly weaker as evidence that messages were misordered, which is a distinction worth keeping when reporting the result.

Worked example: OpenNMS Horizon 36.0.3

Measured on the opennms-benchmark KVM lab: single Minion, 4 vCPU Core, single-partition sink.

This example reports a service rate, not a sustained ceiling. It was obtained by offering a burst far above capacity and measuring the drain, with queue depth growing throughout. It is shown because it is where the mechanism came from, and it is exactly the substitution this page tells you to avoid.

 nl6 --UDP--> Minion --produce--> Kafka --consume--> Core --> Postgres
  40,003/s sent                   LAG growing         |
                                                      '- the constraint

 service time   p50 29.4 ms | mean 30.0 ms | min 23.6 ms | max 54.5 ms
 work unit      ~4 messages per record, ~994 B mean, 1069 B max
 model          (4 messages / 0.030 s mean) x 1 worker  =  133 msg/s
 measured       ~130 events/s drained

The model reproduces the measurement, so the mechanism is understood: one consumer thread, roughly four messages per record, about 30 ms per dispatch. The example assumes one message persists as one event; a collector that coalesces would break that step.

Two honest limits on this example.

The per-message figure of about 248 bytes is 994 / 4, a queue-side number that includes record framing. nl6's syslog datagrams are smaller on the wire, roughly 130 to 210 bytes depending on the catalog entry, so 248 must not be read as a datagram size.

The batching is a count cap plus an interval flush, not a byte cap, and the interval is what actually binds here. SyslogSinkModule exposes an aggregation policy with getBatchSize() (a count) and getBatchIntervalMs(); neither is configured in this lab, so both run at defaults, and the observed batch tracks per-device rate rather than record size. An earlier draft of this page called it a byte cap on the strength of 994 / 4, which was the same mistake it warns about elsewhere: reasoning from a derived average instead of measuring.

This is one topology's number, and only its single-partition state. The same lab was later measured at 4 partitions, in the ordering check, where the throughput was higher and the configuration disqualified. A single Minion and a 4 vCPU Core with a single-partition sink is not a tuned production deployment, and a manifest has to say so as plainly as the number does.

What this would take with real devices

A ceiling only earns its place in a capacity plan if it survives contact with the fleet it is meant to describe, and this one does not survive it intact.

Capacity here rises with offered rate. Both rows below were measured on the same 500-device fleet at the same 4 partitions, so worker count is held fixed and only the per-device rate differs:

offered	per device	mean batch	drain rate
160/s	0.32/s	1.0	(reconciled exactly; below ceiling)
900/s	1.8/s	~1	~144/s
3000/s	6/s	~4	~320/s

The obvious explanation does not survive this page's own model, and that is worth stating rather than smoothing over. Batch size tracks per-device rate, so batching is the natural suspect for the 2.2x jump. But at 3 ms + 6.7 ms/message, going from batch 1 to batch 4 is 103/s to 134/s per worker — about 30%, and the ordering section puts it at 12% against the asymptote. Neither is 2.2x. Held the other way the model overshoots: four workers at batch 1 predicts ~412/s against a measured 144/s.

So the identity does not reproduce either row, and by this method's own rule that means the mechanism is not yet understood — the offered-rate effect is real and measured, but its cause is not established as batch size. The database was the most loaded component at these rates and is the standing candidate. Resolving it needs a run that varies per-device rate with batch size pinned, which this session did not do.

What survives without the mechanism is the shape of the dependency, which is measured directly: capacity depends on per-device rate, not only on the fleet's total. Total offered rate alone does not predict whether a fleet is inside the ceiling; concentration does.

So "~320/s" is not a number to plan a real deployment against. Reaching it requires every device sustaining roughly 6 syslog messages per second, which is a device in a fault storm, not a device in steady state. A production fleet at rest sits at the low-per-device-rate end of the table, and its usable figure is the lower one.

This inverts the usual sizing question. The number to check is not "how many messages per second will the fleet send" but "how many messages per second will the busiest devices send during an incident", since that is both when volume peaks and where the measured capacity is highest.

Two honest limits on the above. The three rows are measured; the 10,000-device extrapolation is inference from them and has not been run. It assumes only that the measured dependence on per-device rate continues down to 0.03/s — deliberately not that batching causes it, since the paragraph above shows that attribution failing. It is consistent with the 160/s row (batch exactly 1.0) but unconfirmed at fleet scale. And the 320/s configuration is disqualified anyway on ordering grounds (the check fired), so it bounds what the tuning could buy, not what it is safe to run.

Related

• Runbooks for the scenario mechanics each run is built from, including how the offered rate is set
• Scenarios for the lifecycle and fidelity mode
• Report schema for sent versus emitted, and what nl6 reports about its own side of the measurement