Simulate 1000 MQTT Clients in C# (mqtt-churn, Part 1)

mqtt-churn series, part 1. Video: 📹Simulate 1000 MQTT Clients in C# · Code: mqtt-churn at tag b1

In MQTT Retained Messages and Last Will, one sensor lost power and an AI agent learned about it from a retained offline Will within a heartbeat. One sensor is a demo. A production line loses power all at once: does the broker fire 500 Wills on time, and how long is the agent blind?

No tool answered that, so we're building one in the open: mqtt-churn. Part 1 builds the fleet, and the first measurement taught three lessons before we killed a single client.

Table of Contents

• Why not an existing tool?
• The fleet in 75 lines
• Paced connects, not a burst
• Leaving the broker clean
• Percentiles you can explain
• The numbers, and what they actually measured
• Key takeaways

Why not an existing tool?

Before writing code, I checked what already exists:

All three measure throughput and connections. None of them reports what the AI agent cares about: the time between a device dropping and a subscriber seeing its retained offline. That is the gap.

The fleet in 75 lines

Every simulated device is a real IMqttClient with a retained Last Will on its own topic, churn/{runId}/{i}/status:

var options = new MqttClientOptionsBuilder()
    .WithTcpServer(o.Broker, o.Port)
    .WithClientId($"churn-{o.RunId}-{i}")
    .WithProtocolVersion(MqttProtocolVersion.V500)
    .WithKeepAlivePeriod(TimeSpan.FromSeconds(o.KeepAlive))
    .WithWillTopic(topic)
    .WithWillPayload("offline")
    .WithWillQualityOfServiceLevel(MqttQualityOfServiceLevel.AtLeastOnce)
    .WithWillRetain()
    .Build();

await client.ConnectAsync(options, ct);
await client.PublishStringAsync(topic, "online", MqttQualityOfServiceLevel.AtLeastOnce, retain: true, ct);

The measured time is CONNECT plus the QoS 1 publish of the retained online, because that is when the device becomes visible to the agent.

Paced connects, not a burst

A thousand clients connecting in the same millisecond tests your laptop's SYN queue, not a broker. Real fleets arrive at a rate. Client i starts at i / rate seconds:

public static TimeSpan StartAt(int i, double rate) => TimeSpan.FromSeconds(i / rate);

var wait = StartAt(i, o.Rate) - clock.Elapsed;
if (wait > TimeSpan.Zero) await Task.Delay(wait, ct);

It is a pure function, so it's the first thing under test.

Leaving the broker clean

A load test that leaves 1000 retained online messages behind poisons the next run. On exit, each client clears its retained topic with an empty retained payload, then sends a normal DISCONNECT, so the broker does not fire its Will:

await client.PublishBinaryAsync(topic, [], MqttQualityOfServiceLevel.AtLeastOnce, retain: true);
await client.DisconnectAsync(new MqttClientDisconnectOptions());

After every run, mosquitto_sub -t 'churn/#' times out with nothing to show.

Percentiles you can explain

Averages hide the slow tail that pages people at 3 am. mqtt-churn reports nearest-rank percentiles, defined as the smallest value with at least p% of samples at or below it:

public static double Percentile(IReadOnlyList<double> sorted, double p)
{
    if (sorted.Count == 0) throw new ArgumentException("No samples.", nameof(sorted));
    var rank = (int)Math.Ceiling(p / 100 * sorted.Count);
    return sorted[Math.Clamp(rank, 1, sorted.Count) - 1];
}

There is no interpolation, so every reported number is a latency one real client actually had. It prints a text histogram too, and --csv dumps one row per client.

The numbers, and what they actually measured

Setup: Mosquitto 2.1.2 in Podman 5.8 on Windows 11, MQTTnet 5.2, .NET 10, Core Ultra 7 265H.

None of the first four rows is a Mosquitto number:

1. localhost costs 2 seconds per client. On Windows, localhost resolves to ::1 first. Nothing answers the container port there, so the OS retries for about 2 s before falling back to IPv4. That's why the tool defaults to 127.0.0.1.
2. The ~45 ms floor is the harness. From Windows, CONNECT alone takes 2–3 ms but the QoS 1 PUBACK round trip takes about 41 ms. The same binary inside the container network gets its PUBACK in about 0.2 ms. The client already sets NoDelay, and Mosquitto's set_tcp_nodelay changed nothing, so the delay sits in Podman's Windows port forwarder.
3. The 1989-connection ceiling is the forwarder too. From Windows, exactly 1989 clients get in. The rest see Connection closed, and the broker logs nothing (its nofile limit is 65536). Inside the network, 5000 of 5000 connect.

So real broker numbers come from running the fleet next to the broker:

dotnet publish src/MqttChurn -c Release -r linux-x64 --self-contained false -o out
podman run --rm --network mqtt-churn_default -v ${PWD}/out:/app -w /app `
  mcr.microsoft.com/dotnet/sdk:10.0 dotnet MqttChurn.dll --broker mosquitto --clients 5000 --rate 500

One more: the very first run after a cold start was about 10× slower (p50 748 ms against 52–56 ms warm). Throw the first run away.

Key takeaways

• Pace your connects. A burst measures your machine; a rate measures the system.
• Clean up retained state with an empty retained payload before a normal DISCONNECT.
• Report percentiles, not averages, and keep every raw sample.
• Measure your harness first. Here, DNS resolution and a port forwarder cost more than the broker did.

Next up, Part 2: Server-Sent Events in C# + React: Watch 5,000 Devices Die. We kill the fleet and watch it die, live in the browser: an ASP.NET Core watcher streams the fleet's state over Server-Sent Events to a React heatmap. On the first run, three deaths out of four never showed up.

Resources

• Code for this part: github.com/Romain-OD/mqtt-churn/tree/b1
• MQTTnet
• MQTT 5.0 specification: Will Message
• Mosquitto configuration (mosquitto.conf)