Welcome

Welcome to the Databend documentation!

Databend is an open-source, elastic, and workload-aware cloud data warehouse, a cutting-edge

alternative to Snowflake

Open in the new tab

. Developed with

Rust

Open in the new tab

, it's engineered for complex analysis of the world's largest datasets.

This welcome page guides you through the features, architecture, and other important details about Databend.

Why Databend?

Performance

• Blazing-fast data analytics on object storage.
• Leverages data-level parallelism and instruction-level parallelism technologies for
  optimal performance

  Open in the new tab
  .
• No indexes to build, no manual tuning, and no need to figure out partitions or shard data.

Data Manipulation

• Supports atomic operations such as SELECT, INSERT, DELETE, UPDATE, REPLACE, COPY, and MERGE.
• Provides advanced features such as Time Travel and Multi Catalog (Apache Hive / Apache Iceberg).
• Supports
  ingestion of semi-structured data

  Open in the new tab
  in various formats like CSV, JSON, and Parquet.
• Supports semi-structured data types such as
  ARRAY, MAP, and JSON

  Open in the new tab
  .
• Supports Git-like MVCC storage for easy querying, cloning, and restoration of historical data.

Object Storage

• Supports various object storage platforms. Click here to see a full list of supported platforms.
• Allows instant elasticity, enabling users to scale up or down based on their application needs.

Databend Architecture

Databend's high-level architecture is composed of a meta-service layer, a query layer, and a storage layer.

Meta-Service Layer

Databend efficiently supports multiple tenants through its meta-service layer, which plays a crucial role in the system:

• Metadata Management: Handles metadata for databases, tables, clusters, transactions, and more.
• Security: Manages user authentication and authorization for a secure environment.

Discover more about the meta-service layer in the

meta

Open in the new tab

on GitHub.

Query Layer

The query layer in Databend handles query computations and is composed of multiple clusters, each containing several nodes. Each node, a core unit in the query layer, consists of:

• Planner: Develops execution plans for SQL statements using elements from
  relational algebra

  Open in the new tab
  , incorporating operators like Projection, Filter, and Limit.
• Optimizer: A rule-based optimizer applies predefined rules, such as "predicate pushdown" and "pruning of unused columns", for optimal query execution.
• Processors: Constructs a query execution pipeline based on planner instructions, following a Pull&Push approach. Processors are interconnected, forming a pipeline that can be distributed across nodes for enhanced performance.

Discover more about the query layer in the

query

Open in the new tab

directory on GitHub.

Storage Layer

Databend employs Parquet, an open-source columnar format, and introduces its own table format to boost query performance. Key features include:

• Secondary Indexes: Speeds up data location and access across various analysis dimensions.

• Complex Data Type Indexes: Aimed at accelerating data processing and analysis for intricate types such as semi-structured data.

• Segments: Databend effectively organizes data into segments, enhancing data management and retrieval efficiency.

• Clustering: Employs user-defined clustering keys within segments to streamline data scanning.

Discover more about the storage layer in the

storage

Open in the new tab

on GitHub.

Community

The Databend community is open to data professionals, students, and anyone who has a passion for cloud data warehouses. Feel free to click on the links below to be a part of the excitement:

• Slack:
  https://link.databend.rs/join-slack

  Open in the new tab
• GitHub:
  https://github.com/datafuselabs/databend

  Open in the new tab
• Twitter:
  https://twitter.com/DatabendLabs

  Open in the new tab
• LinkedIn:
  https://www.linkedin.com/company/datafuselabs

  Open in the new tab
• YouTube:
  https://www.youtube.com/@DatabendLabs

  Open in the new tab

Roadmap

• Roadmap v1.3

  Open in the new tab
• Roadmap v1.2

  Open in the new tab
• Roadmap v1.1

  Open in the new tab
• Roadmap v1.0

  Open in the new tab
• Roadmap v0.9

  Open in the new tab
• Roadmap 2023

  Open in the new tab