Enterprises’ thirst for real-time data is driving profound changes in data architecture. However, building modern, efficient data streaming platforms generally faces two major bottlenecks:
- Apache Kafka, as the central hub for real-time data, exposes severe cost and operational challenges in cloud environments due to its traditional architecture. High holding costs, near-zero elasticity, and complex partition migrations force enterprises to compromise between cost and performance.
- The selection of object storage—the cornerstone of data persistence—is equally challenging. Traditional distributed storage solutions, while powerful, are complex in architecture and have extremely high deployment and operational barriers; while lightweight solutions use the AGPL license, posing potential compliance risks and restrictions for enterprise commercial use.
Enterprises need a new data architecture that balances high performance, low cost, ease of maintenance, and license friendliness.
To address these challenges, AutoMQ and RustFS have announced a strategic partnership. The two parties will deeply integrate AutoMQ’s cloud-native stream processing capabilities, which are 100% compatible with the Apache Kafka protocol, with RustFS, a high-performance distributed object storage built on Rust, licensed under Apache 2.0, and compatible with S3.
This collaboration aims to provide global enterprises with a next-generation Diskless Kafka platform that offers a superior architecture, lower costs, higher performance, and complete avoidance of licensing risks, thereby fundamentally solving the cost and efficiency challenges of real-time data stream processing in the cloud era.
About AutoMQ
AutoMQ is a next-generation cloud-native Kafka-compatible stream processing platform dedicated to solving the core pain points of traditional Kafka in cloud environments: high cost, limited elasticity, and complex operation and maintenance.
AutoMQ employs an advanced compute-storage separation architecture, directly persisting streaming data to S3-compatible object storage, while the compute layer (Broker) is completely stateless. This revolutionary design ensures 100% compatibility with the Apache Kafka protocol, guaranteeing seamless migration to existing ecosystems such as Flink and Spark, while delivering a hundredfold increase in elasticity efficiency and second-level partition migration capabilities, helping enterprises run real-time streaming services with up to 90% savings in Total Cost of Ownership (TCO).
💡 Core Advantages
- Extreme Cost Efficiency: Achieves unlimited storage and pay-as-you-go pricing based on object storage, and completely eliminates expensive cross-Availability Zone (AZ) data replication traffic through a multi-point write architecture, achieving up to 90% TCO savings.
- Hundredfold Elasticity Efficiency: Stateless Broker supports second-level auto-scaling of compute resources. Partition migration time has been reduced from several hours to 1.5 seconds, enabling truly seamless cluster expansion for business operations.
- 100% Kafka Compatibility: Fully compatible with the Apache Kafka protocol and ecosystem, supporting zero-downtime migration from existing clusters without any code modifications.
- Fully Managed and Zero-Maintenance: Built-in automatic data rebalancing and fault self-healing capabilities, providing a BYOC (Bring Your Own Cloud) deployment mode. Data remains 100% within the customer’s VPC, ensuring data privacy and security.
About RustFS
RustFS is a high-performance distributed object storage system developed in Rust and compliant with the S3 protocol. It aims to provide a robust data foundation for AI/ML, big data, and cloud-native applications.
Unlike heavyweight architectures like Ceph, RustFS adopts a lightweight “metadata-free” design, where all nodes are equal, greatly simplifying deployment, operation, and scaling, and avoiding single points of failure in metadata. Leveraging the memory safety, high concurrency, and high performance advantages of the Rust language, RustFS achieves extremely high read/write performance and memory stability while providing exabyte-scale scalability.
💡 Core Advantages
- Apache 2.0 Friendly License: Utilizing the Apache-2.0 open-source license, it is completely friendly to enterprise commercial use, avoiding the intellectual property and compliance risks associated with licenses such as AGPL.
- High Performance and High Stability: Developed in Rust, it naturally possesses advantages in memory safety and high concurrency; under equivalent configurations, its read/write performance far surpasses Ceph, and its memory usage is stable with no high-concurrency jitter.
- Lightweight, metadata-free architecture: Deployment and maintenance are extremely simple, requiring no dedicated metadata server. Scaling is easy, requiring only a single command to start, significantly lowering the operational threshold.
- 100% S3 Compatibility: Fully compatible with the S3 API, supporting existing toolchains and SDKs. It also supports enterprise-level features such as version control, object read-only, and cross-region replication, seamlessly replacing existing S3 storage solutions.
Figure 2: RustFS Architecture (Source: RustFS Official Website)
AutoMQ × RustFS: Building a secure, scalable, cross-cloud Diskless Kafka architecture
The deep integration of AutoMQ’s compute-storage separation architecture and RustFS’s high-performance distributed storage achieves end-to-end technical synergy across three dimensions: cross-cloud support, security and reliability, and unlimited scalability, jointly forming a next-generation Diskless Kafka platform.
Building a Unified, Highly Available Data Flow
Both architectures are designed specifically for multi-cloud environments. AutoMQ provides a flexible BYOC (Bring Your Own Cloud) deployment model, supporting compute instances deployed in multi-cloud environments such as Amazon Web Services, Google Cloud, and Azure, as well as private data centers (IDCs). RustFS provides “true multi-cloud storage” capabilities at the storage layer, supporting bucket-level proactive-proactive cross-region replication. The combination enables enterprises to build unified, vendor-locked real-time data services.
Achieving End-to-End Data Security and Privacy
Both technologies jointly construct end-to-end security and reliability. AutoMQ provides TLS/mTLS encryption at the access layer and deploys the data plane within the user’s VPC using BYOC mode. RustFS provides high-performance object storage security encryption at the storage layer and ensures data integrity through version control and WORM mechanisms. Together, they provide a strict read-after-write consistency model.
Independent flexibility in computing and storage
The architectures of both solutions completely resolve the pain point of traditional Kafka’s difficulty in scaling. RustFS supports EB-level storage capacity and “unlimited scaling.” AutoMQ, with its storage-compute separation, achieves auto-scaling at the compute layer. The perfect combination allows enterprises to independently expand compute resources (to cope with traffic surges) or storage resources (to cope with data growth).
Figure 3: AutoMQ & RustFS Collaborative Architecture
What It Brings
The deep technical synergy between AutoMQ and RustFS brings enterprises a high-performance, low-cost, secure, and compliant real-time data infrastructure.
- True unlimited elasticity and high performance: Both architectures support independent scaling. AutoMQ reduces partition migration time from hours to seconds, maintaining P99 latency <10ms. RustFS provides EB-level storage capacity.
- End-to-end security, reliability, and compliance: AutoMQ ensures data never leaves the VPC. RustFS provides encryption, version control, and importantly, the Apache 2.0 license, allowing enterprises to avoid “license traps” like AGPL.
- A Fully Open-Source Joint Solution: Both parties have built on the Apache License, collaborating to create a 100% full-stack Apache-licensed solution for enterprises.
- Extreme Cost Optimization: AutoMQ can save up to 90% on Kafka TCO. RustFS’s lightweight architecture and license friendliness further reduce storage and software costs.
- Cross-Cloud Support and Simplified Operations: Both companies provide “true multi-cloud storage,” helping enterprises prevent single-cloud vendor lock-in and reducing operational complexity.
Open Source Resources
- AutoMQ GitHub: AutoMQ Open Source Project
- RustFS GitHub: RustFS Open Source Project (Note: Please verify this link matches your repository)
Looking Ahead
AutoMQ and RustFS will continue to deepen their technological integration, jointly driving the development of cloud-native real-time data infrastructure. Together, they will provide global enterprises with lower-cost, higher-performance, easier-to-maintain, and compliance-risk-free data streaming solutions.
