It's difficult to model every workload, so your experience may vary based on a wide range of factors, however, we have gone out of our way to model realistic parameters for both Kafka and WarpStream. Our objective is not to skew the results in favor of WarpStream, and we have tried to model configurations that make sense for most use cases, but if you would like a custom estimate, please contact us.

The cost estimator assumes:

• ~10 MiB/sec of compressed write throughput per CPU core can be achieved for both WarpStream Agents and Kafka Brokers.
• We assume a 4:1 compression ratio for producers.
• Both Kafka and WarpStream utilize highly available three availability zone deployments, unless otherwise specified.
• Kafka is deployed with 3x replication factor.
• Kafka uses a minimum instance size of r4.xlarge.
• For both Kafka and WarpStream, we assume 3x consumer fanout.
• For Kafka, target EBS utilization is 40%, which is generally recommended to avoid filling disks if write throughput increases suddenly. This also ensures that Kafka brokers are provisioned with sufficient EBS storage to handle their write throughput and IOPS needs. Reasoning below:
  • AWS GP2 volumes have IOPS capacity based on the amount of provisioned storage.
  • AWS GP3 volumes have a fixed amount of baseline IOPS capacity and users must explicitly pay extra for higher write throughput and IOPS. The calculator assumes that users do not pay for these extra capabilities, even though many real-world Kafka deployments would require it.
  • In general, when using cloud provider VMS with directly attached storage (for example i3 series on AWS) the amount of IOPS the disks are capable of is directly proportional to the size of the disks.
• EBS / local storage volume pricing is based AWS MSK's per-broker storage pricing of $0.1/GiB. This pricing is equivalent to the pricing of AWS's GP2 EBS volume type, but slightly higher than the pricing of AWS's GP3 EBS volume type. We believe this is the correct assumption to make because even if users self-host Apache Kafka on GP3 EBS volume types, unlike GP2, IOPS do not increase on GP3 volume types automatically as provisioned storage is increased, and users must manually provision (and pay!) for additional IOPS and throughput.
• No more than 16TiB of EBS storage can be attached to a single broker. EBS has a hard-limit of 64TiB, but even at 16 TiB broker restarts and partition rebalancing will become unmanageable and most managed Kafka services like AWS MSK won't let you attach more than 16TiB of local EBS storage anyways.
• For Kafka, target EC2 network utilization is 33%. This is important to ensure the cluster has some capacity to tolerate an increase in workload throughput and still have enough idle network capacity to rebalance partitions if a node needs to be replaced or the cluster needs to be expanded.
• Kafka leader partitions are zone-aligned with producers for 1/3 of write throughput.
• With Fetch From Follower disabled, 1/3 of Kafka read throughput is zone-aligned.
• With Fetch From Follower enabled, 100% of Kafka read throughput is zone-aligned.
• With MSK, inter-zone networking for replication between broker replicas is free.
• WarpStream Agents are zone-aligned for all produce and consume throughput.
• WarpStream Agents flush to object storage every 250ms (this is the default flush interval).

These assumptions are intended to be generally accurate for most use cases, however, your workload may have different characteristics and therefore you may obtain different results in practice. For a detailed analysis of your specific workload, contact us.

WarpStream is compatible with the Apache Kafka® protocol, but we are not running Kafka. Instead, we run a stateless Agent that has no local disks, and writes directly to object storage, which avoids 100% of cross-AZ replication charges. These stateless Agents are also much easier to operate than Kafka brokers, so you can drastically reduce the amount of time that you spend managing your streaming infrastructure by switching to WarpStream.

WarpStream effectively uses object storage as both the storage layer and the network layer, which avoids much of the cost associated with running Kafka in cloud environments. By writing directly to object storage, WarpStream avoids replicating data between Agent nodes. Instead, data is durably persisted to object storage before WarpStream provides an acknowledgement to the producer. Once data is written to object storage, replication is handled by the object store. If you use Amazon S3 as the object store, for example, this means that the data that you write to WarpStream has an eleven nines (99.999999999%) durability guarantee. And because your data is not replicated between Agents before reaching object storage, you don't pay anything extra for this durability.

In addition, the WarpStream Service Discovery system ensures that your clients are 100% zone-aligned with Agents for both Produce() and Fetch() requests, which completely avoids cross-AZ traffic in both the write path and read path by default. Kafka's Fetch from Follower feature helps reduce cross-AZ traffic in the read path, but because Kafka clients must always write to a leader partition, cross-AZ traffic in the write path is unavoidable. WarpStream Agents do not have the concept of leader partitions, so any producer can produce to any Agent.

Finally, because the compute layer is stateless, you can autoscale the Agents, which means you don't need to overprovision compute for your cluster to be able to handle peak load. This stateless model also enables WarpStream to run on smaller instance types with lower memory requirements for lower-throughput workloads. For example, whereas Kafka is recommended to run on at least d2.xlarge or r4.xlarge EC2 instances in AWS, WarpStream can use much smaller instances in less expensive instance families, and smaller instance sizes as well.

No. WarpStream's stateless compute model, consumption-based billing system, and automatic scale-to-zero functionality make idle WarpStream clusters free. For BYOC clusters you will need to scale the Agent nodes to zero yourself.

Keep in mind that when a WarpStream is scaled to zero, the data is not gone. The data is still persisted in object storage, and will be available as soon as the cluster scales back up.

WarpStream cluster-minutes are accrued in 15 minute intervals, so you are only charged for a 15-minute interval of cluster-minutes if there was traffic during that period. You are not charged for idle clusters, and there are no per-partition charges, either.

WarpStream is SOC 2 Type I and Type II compliant, making it ideal for companies of all sizes across multiple verticals from finance to video games to observability and more. You can learn more about our security controls, subprocessors and request our SOC 2 reports via our Trust Center.

Normally, networking charges associated with running Kafka are accrued based on compressed network throughput. However, WarpStream charges for uncompressed data written because we want to offer predictable pricing, and we believe that your bill should not fluctuate based on which compression algorithm you choose to use in your client. This also aligns incentives so that we are encouraged to reduce your cloud infrastructure costs for the WarpStream BYOC product. This philosophy is also why we don't charge fees per Agent or per core.

BYOC clusters are only charged for uncompressed writes, cluster-minutes, and storage. There are no network ingress or egress charges for self-hosted clusters.

No, unlike other vendors with BYOC or self-hosted software deployment models, WarpStream has no per-Agent, per node, or per-vCPU charges. Feel free to scale your clusters to best fit your workload, without needing to worry about the cost implications of doing so. You can even set up autoscaling, and your cluster will scale out and in as quickly as your autoscaler can respond and provision containers or instances.

Unlike other Kafka and Kafka-compatible systems, WarpStream does not charge extra for replication. That's because there is no local data to replicate. Object storage is the primary and only storage, so replication is handled transparently by the object store itself. Writes are durably persisted to object storage and committed to the metadata store before providing acknowledgement to the client, so you can be confident that your data is being replicated behind the scenes.

Keep in mind that WarpStream's storage fees are all-inclusive, meaning the advertised price is what you will actually pay. Some other vendors list unit prices pre-replication, which means that the actual fees that you pay are 3x higher.

No. Unlike some other Kafka-compatible systems, WarpStream has no per-partition charges. And unlike Kafka, there is no requirement to increase the number of WarpStream Agents when the number of partitions increases. However, similar to Kafka and related systems, performance is improved by not using an excessive number of partitions.

No. WarpStream has no limits on the number of partitions per Agent because WarpStream doesn't have partition "leaders" like Kafka does. We also don't charge any additional fee per partition.

Idle partitions do not contribute to to Agent resource utilization at all, but the number of active partitions (i.e., how many partitions are being written to at any given moment) does contribute towards Agent resource utilization.

By default, WarpStream clusters have a limit on the total number of partitions in a cluster, but these can be raised upon request.

WarpStream supports Schema Validation via external schema registries and services like AWS Glue. It also has its own BYOC-native Schema Registry. 

Like all Schema Registries, the WarpStream BYOC Schema Registry ensures data compatibility and compliance by validating schemas during data production and consumption. This helps minimize downstream data issues and enables schemas to evolve without breaking consumers.

In addition, it has unique features that are only possible with WarpStream’s stateless, zero-disk architecture, such as native integration with the WarpStream Agents, data retrieval via object storage with no intermediate disks, easy scaling, usage of zone-aware routing to avoid interzone networking fees, and no need to wait on “leaders” to be elected, as consensus is handled by WarpStream's metadata store, and Agents can read and write.

Learn more about WarpStream BYOC Schema Registry via our docs and announcement blog.