Frazer Clement

No longer posting here, follow link for new content

[Frazer Clement @frazerclement]

I have decided to publish further #MySQL Ndb Cluster content on a different platform.
Hope to see you there.
https://t.co/rZlbmNdqpa
#DistributedSQL

[Frazer Clement @frazerclement]

Open source #MySQL Ndb Cluster software can be deployed on bare metal, VMs, containers, on premise, on any cloud, on any Kubernetes cluster service and on your laptop.
There is no rent to pay, no forced upgrades, no mandatory downtime.
#DistributedSQL

[Frazer Clement @frazerclement]

@jim_dowling Those are nice to have too!
Ndb + Erlang share some common ancestors.
Ndb does not yet offer live code patching though!

[Frazer Clement @frazerclement]

MySQL Ndb Cluster has supported almost twenty years of zero-downtime upgrades through redundancy, protocol + file system format compatibility.
https://t.co/ittKv2PT35
#DistributedSQL #MySQL #HighAvailability

[Frazer Clement @frazerclement]

SQL processing + connectivity scale out with many MySQL Servers connected to a #MySQL Ndb Cluster.
Individual MySQL server failures affect only their clients.
Rolling restart allows software upgrade with SQL service throughout.
#DistributedSQL

[Frazer Clement @frazerclement]

@sunbains @eatonphil OpenLDAP used to use Berkeley DB (BDB) internally which was also the basis for a MySQL storage engine.
Howard implemented LMDB much later, primarily for OpenLDAP but it has been used for many other things.
He has written quite a lot about the background to the project.

[Frazer Clement @frazerclement]

#MySQL Ndb Cluster software can be upgraded via rolling restart.
New versions keep protocol + storage compatibility.
Normal redundancy, failover + recovery mechanisms ensure availability.
Changes affecting compatibility are enabled after all nodes are upgraded.
#DistributedSQL

[Frazer Clement @frazerclement]

Redundancy in #MySQL Ndb Cluster ensures service availability automatically over both unexpected failures + planned maintenance tasks such as :
- Soft state reset
- Storage rebuild
- HW + OS upgrades
- Config changes
- Software upgrade
- Node migration
- Scale up
#DistributedSQL

[Frazer Clement @frazerclement]

#MySQL Ndb Cluster data node Undo logs contain ops to undo (disk stored column) tablespace page writes.
Undo is a WAL, flushed before tablespace pages, guaranteeing recoverability.
Redo logs are a WBL, flushed after transaction commit + before or after checkpoint.
#DistributedSQL

[Frazer Clement @frazerclement]

Durability in #MySQL Ndb Cluster is ensured using the distributed GCP protocols to :
- Define consistent transaction boundaries (epochs)
- Wait for epochs to finalise
- Wait for epoch redo logs to be durable + fsynced
Effectively async distributed group commit.
#DistributedSQL

[Frazer Clement @frazerclement]

Each Redo log part in #MySQL Ndb Cluster is a configurable number + size of Redo log files, initialised at data node start, storing a circular log. Transaction prepare + commit records move the head pos forward. LCP completion moves the tail pos forward.
#DistributedSQL

[Frazer Clement @frazerclement]

The Redo log in each #MySQL Ndb Cluster data node is actually a configurable number (default 4) of independent redo logs called parts.
Each table partition logs operations to one Redo log part, with partitions balanced across parts for increased I/O parallelism
#DistributedSQL

[Frazer Clement @frazerclement]

Checkpoints + Redo logs provide durability in #MySQL Ndb Cluster data nodes for :
- Consistent recovery of data and changes after a total outage via System Restart (SR)
- Faster recovery when a data node is restarted during normal operation via Node Restart (NR)
#DistributedSQL

[Frazer Clement @frazerclement]

Single Writer, Multiple Reader (SWMR/MRSW) partition locks are only used by #MySQL Ndb Cluster if Query Threads are configured.
Lock tenure is confined to one non-blocking microtask execution.
No locks are held while waiting for disk I/O, row locks or buffers.
#DistributedSQL

[Frazer Clement @frazerclement]

@EmericHUNT3R Yes, it is built in to open source MySQL Ndb Cluster 8.0, so everyone can use it. If a cluster is configured with Query Threads then this parallelism is enabled.

[Frazer Clement @frazerclement]

Each #MySQL Ndb Cluster LDM thread executes interleaved microtasks implementing concurrent lookups, scans, writes + txn commits on its partition replicas.
Query threads add parallel execution of lookup + scan microtasks, using SWMR locking per partition replica.
#DistributedSQL

[Frazer Clement @frazerclement]

@rajappaiyer You are right.
I was thinking of the author Mike Burrows' reported quote about Paxos : “There is only one consensus protocol, and that’s Paxos. All other approaches are just broken versions of Paxos”.
May be true but I think it led to overuse of Paxos for everything.

[Frazer Clement @frazerclement]

Good article from Adam Prout about modes of consensus usage in distributed DBs.
#MySQL Ndb Cluster is a CfM system by this categorisation.
Agree that 'Use Paxos/Raft for everything' originates from the Chubby paper + 'Copy Google' trend.
#DistributedSQL https://t.co/u2bYktskX9

[Frazer Clement @frazerclement]

Query threads in #MySQL Ndb Cluster 8.0 allow multithreaded reads + scans of the table/index partitions owned by one LDM instance.
Read scaling within a partition helps balance read hotspots and avoids the costs of using many partitions for scaling.
#DistributedSQL
#MultiParallel

[Frazer Clement @frazerclement]

Quarterly releases of #MySQL Ndb Cluster :
9.1.0 Innov : https://t.co/jMB5HHrWLu
8.4.3 LTS : https://t.co/1MdPtdwzTi
8.0.40 LTS : https://t.co/SwheML9mfj
7.6.32 LTS : https://t.co/zhoD9YYjZY
GPL downloads here : https://t.co/ozf60vnM4l
#DistributedSQL

[Frazer Clement @frazerclement]

@sunbains OK looks like TiKV can handle/ignore node failures at the 'Data' level by normal Raft mechanisms (CfD, requiring 2f+1 replicas), and PD can separately reconfigure to restore redundancy later if necessary?