Introducing WarpStream Managed Data Pipelines for BYOC clusters

Stream processing, made even more operationally mundane

We previously launched embedded support for Bento in the WarpStream Agent, which enables WarpStream users to use Bento without managing any additional infrastructure.

Today, we’re excited to announce that we have taken this feature a step further with WarpStream Managed Data Pipelines. A new feature for WarpStream BYOC clusters, Managed Data Pipelines provide a fully-managed SaaS user experience for Bento, without sacrificing any of the cost benefits, data sovereignty, or deployment flexibility of the BYOC deployment model.

Bento is a lightweight stream processing framework that offers much of the functionality of Kafka Connect, as well as additional stream processing functionality like single message transforms, aggregations, multiplexing, enrichments, and more. It also has native support for WebAssembly (WASM) for more advanced processing. 

Previously, WarpStream was just a replacement for Apache Kafka®, but with Managed Data Pipelines WarpStream can do a lot more out of the box. For example, the WarpStream Agents can now directly connect with external systems, stream data between topics, perform on-the-fly data transformation, stream data into downstream systems, and much more, all with a simple YAML configuration.

In addition to all the native features of Bento, WarpStream Managed Data Pipelines also provide:

• A user interface in the WarpStream Console for creating and editing pipelines.
• The ability to pause and resume pipelines on demand.
• Version control and branching, giving you the ability to easily deploy changes, or roll configurations backwards as need be.
• Automatic handling of SASL authentication, ACLs, AZ-aware routing, and many other WarpStream-native features.
• Control over concurrency, with distribution managed by WarpStream and controlled with a single line in your configuration.

Managed Data Pipelines is the natural evolution of WarpStream’s BYOC product. In the same vein that WarpStream is a novel implementation of an open protocol with no vendor lock-in, Managed Data Pipelines is a cloud native, BYOC-managed version of a popular open-source stream processing framework, enhanced with WarpStream’s signature data plane / control plane split. Pipeline configuration, version control, clustering, and pipeline deployment are all administered remotely using a SaaS UX, but the actual pipelines run in your cloud account, using your compute resources and your object storage buckets. Raw data never leaves your account.

To learn more about how to use Managed Data Pipelines, check out the docs, and if you have any questions, feel free to join our Slack community. You can get started with WarpStream for free, with no credit card required, and start streaming with Managed Data Pipelines in just a few minutes.