Patterns of Distributed Systems

• Distributed systems - An implementation perspective

  Type of platform/framework	Example
  Databases	Cassandra, HBase, Riak
  Message Brokers	Kafka, Pulsar
  Infrastructure	Kubernetes, Mesos, Zookeeper, etcd, Consul
  In Memory Data/Compute Grids	Hazelcast, Pivotal Gemfire
  Stateful Microservices	Akka Actors, Axon
  File Systems	HDFS, Ceph

Problems and Their Recurriing Solutions

Serveral things can go wrong when data is stored on multiple servers.

• Process crashes

  • It can be taken down for routine maintenance by system administrators
  • It can be killed doing some file IO because the disk if full and the exception is not properly handled.
  • It cloud environments, it can be even trickier, as some unrelated events can bring the servers down.

    > Flushing data to the disk is one of the most time consuming operations, not every insert or update to the storage can be flushed to disk.
    
    # Write-Ahead Log
    > Servers store each state change as a command in an append-only file on a hard disk. Appending a file is generally a very fast operation, so it can be done without impacting performance. This gives a durability guarantee. The data will not get lost even if the server abruptly crashes and then restarts
    
    > But clients will not be able to get or store any data till the server is back up. lack availability in the case of server failure. One of the obvious solutions is to store the data on multiple servers. So we can replicate the write ahead log on multiple servers.

• Network delays(延迟)

  In the TCP/IP protocol stack, there is no upper bound on delays caused in transmitting messages across a network.

  • A particular server can not wait indefinitely(无期限) to know if another server has crashed.

    > To tackle the first problem, every server sends a HeartBeat message to other servers at a regular interval.

  • There should not be two set of servers, each considering another set to have failed, and therefore continuing to serve different sets of clients. This is called the split brain(脑裂)

    In gneral, If we want to tolerate f failures we need to cluster size of 2f + 1.
    
    Leader and Followers is used in this situation. One of the servers is elected a leader and the other servers acts as followers.

• Process Pauses

  • Generation Clock

    A monotenically increasing number indicating the generation of the server.

• Unsynchronized Clocks and Ordering Events

  The main reason we can not use system clocks is that system clocks across servers are not guaranteed to be synchronized. A time-of-the day clock in a computer is amanaged by a quartz crystal and measures time based on the oscilations of the crystal.
  The clocks across a set of servers are synchronized by a service called NTP. This service periodically checks a set of global time servers, and adjusts the computer clock accordingly.

  • Lamport(烂波特) Clock

    Use logical timestamps as a version for a value to allow ordering of values across servers.

Pattern Sequence

In very simple terms, Consensus(共识) refers to a set of servers which agree on stored data, the order in which the data is stored and when to make that data visible to the clients.

• Pattern Sequence(序列) for implementing consensus

  Consensus implementations use state machine replication to achieve fault tolerance.

Examples

• Raft
• Zab
• Cassandra
• Epoch’s in Kafka