Can data be stored on disk in MySQL Cluster 5.1?

Yes. In 5.1, data can be stored on disk for non-index fields while keeping indexes in main memory. Checkpointing to disk allows recovery after power loss, and future releases plan disk-based indexes as well. (Timestamp: 1500)

What is online index generation and why is it useful?

Online index generation builds an index without copying all rows or locking the table for long periods. This allows existing queries to run while the index is created in the background, significantly reducing downtime for large tables. (Timestamp: 1620)

How does partitioning work in 5.1? Can I define custom partitions?

Partitioning distributes data across node groups. By default it uses a hash-like partitioning, but 5.1 also explores user-defined partitioning, allowing you to partition by key, range, or list in future releases. This gives you control over data placement. (Timestamp: 1860)

How does failover work in an NDB cluster?

Failover involves detecting a failed node, transferring ownership of fragments to another node, and resuming processing. Epoch-based checkpoints coordinate recovery, and a backup master/slave setup can promote a standby cluster if the primary fails. (Timestamp: 2440)

What is the epoch concept in cluster failover?

An epoch is a synchronization checkpoint across storage nodes representing a consistent point to recover to after a total failure. It is used to determine where to resume applying transactions in a failover scenario. (Timestamp: 2428)

Can data nodes be added online in the future?

Yes. The team indicated that online data node addition is planned for future releases; the initial rollout will add nodes in two phases: bringing new nodes into the cluster and repartitioning data across the existing nodes. (Timestamp: 624)

What is the NDB injector thread?

The NDB injector thread in the MySQL Server subscribes to cluster events and writes row-based updates to the MySQL binary log. This creates a canonical log stream for replication across the cluster, including updates from NDB API clients. (Timestamp: 2195)

Key Moments

A Googly MySQL Cluster Talk

Google Talks

Education3 min read56 min video

Aug 22, 2012|4,372 views|17

googlevideo

Save to Pod

Key Moments

TL;DR

MySQL Cluster 5.1: in-memory core with disk data, online ops, and robust failover.

Key Insights

Architecture is a shared-nothing, synchronous cluster: storage nodes hold data, MySQL servers handle SQL, and a management node distributes config and coordinates failover.

5.1 adds data on disk and variable-sized rows, enabling larger datasets with better memory efficiency and new disk-based indexing plans.

Online operations are emphasized: online add/drop of indices, online table-partitioning control, and online table space and log file management.

Performance enhancements include engine condition pushdown, batch read interfaces, and tight query-cache integration for cluster tables.

Cross-cluster replication and NDB API integrations enable geo-redundancy and offload/reporting workloads while maintaining a unified binary log for replication.

Failover and recovery are epoch-based and highly automatic in design, but some operations (like automatic failover) are still evolving, with manual steps documented.

INTRODUCTION AND CONTEXT

The speaker, Stuart Smith, introduces the topic of MySQL Cluster (NDB) and places it within the lineage of MySQL storage engines. He outlines the progression from 4.1’s early in-memory clustering to 5.0’s performance improvements and 5.1’s new features, including disk-backed data and richer functionality. The talk is framed as part of a trilogy, with emphasis on practical architecture, deployment scenarios, and how a cluster storage engine integrates with standard MySQL servers. The goal is to give attendees a working mental model of what a cluster does, how it is structured, and where it adds value for high-availability, high-throughput workloads.

ARCHITECTURE AND COMPONENTS OF MYSQL CLUSTER

The core architecture is laid out: storage nodes form the data backbone, MySQL server nodes handle SQL traffic, and a separate management server handles configuration distribution and monitoring interfaces. Data is distributed across node groups and replicated to achieve high availability. The model is shared-nothing, with commodity hardware and Ethernet interconnects. A transport layer (TCP by default, with optional SCI). The data path is clear: client MySQL servers issue queries, which are translated into operations on data nodes; all transactions are synchronously replicated to replicas to keep consistency.

NEW FEATURES IN 5.1: DISK-BASED DATA AND VARIABLE-SIZED ROWS

5.1 introduces disk-backed data (dis data) alongside the traditional in-memory model, enabling larger datasets without needing all data in RAM. Variable-sized rows reduce wasted memory, improving RAM efficiency and capacity for big workloads. The talk also explains disk-based index considerations and the groundwork for more flexible index storage. The architecture maintains fast recovery via checkpoints, and the memory footprint becomes more predictable with the ability to size data and index memory more precisely.

ONLINE OPERATIONS AND PARTITIONING

A major theme is online manageability: you can add and drop indexes online without copying entire tables, create or alter table spaces and log file groups, and perform user-defined partitioning. Partitioning by key, range, or list allows explicit control over data placement across node groups, enabling load balancing and targeted performance tuning. Online index creation and drop avoid the traditional lock-and-copy approach, significantly reducing maintenance windows and improving availability during schema changes.

PERFORMANCE OPTIMIZATIONS AND QUERY PROCESSING

The talk covers several performance enhancements: engine condition pushdown sends unindexed-field predicates down to data nodes, enabling parallel evaluation and reducing wire traffic. Batch read interfaces reduce the number of network hops by fetching multiple keys in one request. Query cache integration and improved metadata handling speed up common patterns for cluster tables. The combination of in-node processing and batched lookups yields tangible speedups on typical workloads.

REPLICATION MODEL, NDB API, AND FAILOVER SCENARIOS

Beyond intra-cluster replication, there is replication between clusters for geo-redundancy. The NDB API allows programming direct data operations in C++ and feeding changes into the cluster’s binary log via a new injector thread, resulting in a canonical, row-based binary log for slaves. Failover is epoch-based, with a defined sequence to synchronize binary log positions and epochs across master and backup instances. While some failover automation is still evolving, the architecture supports robust redundancy through multi-channel replication and manual failover workflows.

Mentioned in This Episode

●Products

●Software & Apps

●Books

●Concepts

●People Referenced

MySQL Cluster 5.1 Quick Do/Don't

Practical takeaways from this episode

Do This

Use online index operations to avoid full table copies and locks during schema changes.

Leverage partitioning (including user-defined options) to optimize data distribution.

Consider enabling engine condition pushdown to accelerate unindexed-field queries by executing work on data nodes.

Plan for disk-backed data where appropriate to balance memory usage and persistence.

Utilize cross-cluster replication for DR and offload reporting workloads.

Avoid This

Don’t assume online index changes are instantaneous on very large datasets without planning their impact.

Don’t rely on repeatable-read semantics in this transactional engine; it’s not implemented.

Common Questions

NDB Cluster is the storage engine that provides a high-availability, in-memory distributed database. It partitions data across storage nodes and replicates it, enabling fast primary key/index lookups with configurable redundancy. It also supports a MySQL front-end and a C++ API for direct data access. (Timestamp: 81)