Computer & IT
Log Ingestion Throughput Calculator
Translate event rate into wire and replicated peak throughput, size ingest shards, and estimate compressed replicated hot-retention storage.
Sustained payload throughput (MB/s)-
Sustained wire throughput (MB/s)-
Peak wire throughput (MB/s)-
Peak replicated write throughput (MB/s)-
Raw payload per day (GB)-
Stored replicated data per day (GB)-
Hot-retention storage (TB)-
Planned throughput per shard (MB/s)-
Required ingest shards-
Events per day-
Decision view
Event-to-shard ingestion river and retention reservoir
Event-to-shard ingestion river and retention reservoirA widening peak-and-replication flow feeds planned shard lanes and a storage reservoir sized by retention.
Exact scenario comparisonPeak burst factor changes while all other entered assumptions remain constant.
| Peak burst factor | Sustained payload throughput (MB/s) | Sustained wire throughput (MB/s) | Peak wire throughput (MB/s) | Peak replicated write throughput (MB/s) | Raw payload per day (GB) | Stored replicated data per day (GB) | Hot-retention storage (TB) | Planned throughput per shard (MB/s) | Required ingest shards | Events per day |
|---|
How to use Log Ingestion Throughput Calculator
- Measure serialized bytes after enrichment.
- Use a burst factor from a high-percentile interval.
- Keep replication, compression, and index overhead aligned with the actual platform.
Calculator guide
Understanding Log Ingestion Throughput Calculator
Log platforms have two different sizing problems: peak replicated write throughput and retained stored volume. Event size, protocol overhead, burst, replication, compression, indexing, and utilization affect different stages.
Detailed calculation process
Detailed log-ingestion and retention calculation
The default stream carries 42,000 events/s at 850 bytes with 12% wire overhead, a 2.2× burst, and three replicas.
General formula: Q_p=E*b/1048576Q_w=Q_p*(1+o)Q_peak=Q_w*f_bQ_r=Q_peak*RV_d=Q_p*86400/1024V_s=V_d*R*(1-c)*(1+i)S=V_s*d/1024q_s=q_r*uN_s=ceil(Q_r/q_s) Peak write and retained storage share the same payload baseline but apply different multipliers in their own order.
What each symbol means
Worked substitution with the default inputs
The default plan needs 24 ingest shards and about 91.9 TB of hot replicated storage.
Worked situations
Practical examples
- Forty-two thousand 850-byte events equal 34.05 MiB/s of payload.
- After overhead, 2.2× burst, and replication, the peak write path is about 251.7 MiB/s and needs 24 planned shards.
Better inputs
Useful tips
- Separate hot retention from archive storage.
- Test shard throughput using the real mapping and refresh interval.
- Budget replay/backfill as a separate scenario.
Before relying on the result
Limitations and common mistakes
- No queue lag, parsing CPU, flush latency, merge amplification, or shard-size ceiling is modeled.
- Compression varies by field cardinality and codec.
- The model assumes one average event size.
Reference
Key terms
- Wire throughput
- Payload plus framing and protocol overhead.
- Replication factor
- Number of stored/write copies in the modeled ingest tier.
- Shard utilization
- Planning share of tested per-shard throughput.
Important note
Validate with production-like load tests, replay behavior, mapping/cardinality, failure recovery, storage watermarks, and vendor-specific shard guidance.
Frequently asked questions
Why multiply peak throughput by replication?
Each logical write produces the entered number of physical copies.
Why does compression not reduce network throughput?
The entered reduction is modeled at storage; use a smaller serialized event size if transport is compressed.
Is shard count also a node count?
No. Node placement depends on per-node CPU, memory, disk, and shard limits.