How does blockchain work?

A lot of the ambiguity about blockchain comes from the inability to visualize or form a mental image of what blockchain is.  When we talk about computers, we can image it - a black box with flashing lights, humming disks and processors; the same can be said about storage devices or cell phones.  Even software which is a more abstract concept can be seen via its user interfaces: a human resource software will have lists and details about employees and buttons to add new employees or new organizations.  Perhaps a better match for blockchain would be the humble database - but even the database can somehow be visualized (even if incorrectly) as rows and columns of data.

So how can we concretely understand blockchain?  I have often compared blockchain in many ways to the internet - both are foundational technologies and seem abstract and while both would hard to visualize, the internet has become so ubiquitous that we use it unquestionably; and yet few would be able to explain what exactly the internet is.  

In this article we will try to understand how blockchain works - to be more specific since blockchain is a generic technology, we will try to understand how majority of contemporary blockchain based technologies work - which includes Bitcoin, Ethereum, Hyperledger and Multichain amongst others.  

Overview

At a high level we can think of blockchain as a network of interconnected nodes.  In this way we can think of blockchain’s structure as composed of two parts:

• A software stack which resides in a computer/device (lets call this a “node”)
• A network of nodes which synchronizes data between the nodes  

Lets delve into each in a bit more detail.

Blockchain Node

The blockchain node, as illustrated in figure 1 is a software stack (layers of software) that taken together function as a blockchain system or product.  The node software stack can be thought of as three layers:

1. The blockchain store contains the principal low level workings of blockchain and gives blockchain technology its core capabilities:  storing data in an immutable manner and maintaining collection of data (blocks) in a chain we call the “blockchain”.
2. Protocol/Platform/Middleware.  The next layer acts primarily as a processor to define the rules of accessing data, changing its state as well as managing inter-node communication.  While the implementation of previous layer would be relatively common across platforms, this layer implements custom technology for any platform. Hence Bitcoin implements simple processes that handle token transfers while Ethereum runs an entire virtual machine to process much more generic range of transactions as well as execute code.  
3. Dapps or distributed applications are applications that run on the blockchain platform.  Smart contracts in Ethereum are examples of Dapps. Unlike centralized applications that reside on a single server or via centralized technologies such as cloud, Dapps once created are available and can be executed at all nodes.  Not all blockchain technologies support Dapps fully: Ethereum which has a turing complete virtual machine as part of it’s platform gives users the ability to create arbitrary computer programs (Dapps) and deploy them. Other products may offer a more limited set of Dapps.

Network and Consensus

While the principal characteristics of blockchain are encoded in the software stack of the individual nodes, the distributed P2P nature of blockchain implies that the network is an equally important component.  As will be explained in detail in section X below, once a node is booted up, a network discovery process kicks in to connect the individual node to its peers and automatically synchronize data. Its only once this process is complete that the node can accept transactions and conduct normal operations.

Once the initial synchronization process is complete, the network will continuously share data amongst its nodes via rules governed by a consensus algorithm which ensures eventual data consistency.  Most consensus algorithm implementations impart on blockchain a fairly high degree of fault tolerance - including in the presence of the most stringent form of faults. Those who need a highly technical understanding of fault tolerance in blockchain systems are referred to my earlier article here.

To get a feel of what a blockchain network may look like, lets take a practical example in capital markets which consists of 7 stages of a complete trade lifecycle as illustrated in Figure 2 below.  In the case below, it requires coordination of a minimum of six entities, ranging from the stock exchange to depository institutions, in order to complete a single transaction.

Typically as each entity maintains its own database, with the lack of any automated synchronization mechanism, any change by an organization needs time to propagate to the different entities (this is one of the reason it takes trade time to execute and settle).  Furthermore, depending on the process at work, as partner entities do not share databases, they may need to compare records at specific times to ensure that the information is valid - inter-bank reconciliation is a classic example here which requires diverse stakeholders to share information to ensure that the system as a whole is working without errors.

Figure 3 is an illustration of how the system would look if blockchain network replaced the traditional systems in the capital markets example.  Each involved entity would keep one or more nodes which would automatically synchronize data across the network. One of the greatest beneficiary here would be the regulatory agencies who would get access to unbiased, complete and auditable view of all transactions automatically.  The consensus algorithm ensures here that the data at each node is exactly the same given enough time.

Blockchain Processes

Having looked at the structure of blockchain nodes and network we are now ready to understand its 4 key processes.  Between the structure and the key processes we will be in a good position to understand how blockchain works.

The four key processes are illustrated above in Figure A: (1) Network formation/connectivity,    (2)Transaction, (3) Block formation and (4) Block commitment along with the triggering event which starts the process and the final event corresponding to the completion of the process.  

We will got through each one to understand how they fit together.    

Blockchain network formation process

As was noted earlier, Blockchain at its lowest level is a network of nodes; each node contains in itself a software stack to store and process information; and being a distributed ledger, each blockchain node at steady state must contain identical information.  While we will get into how the blockchain makes information identical later, for now let us focus on the obvious question about any network: how does it become a network?

Network formation in blockchain is driven by a network formation process:  the process of finding and joining the network.

The canonical blockchain network formation process can be broken down into four stages:

Once a network formation and connectivity process is complete, the node is ready to accept transactions and commence normal operations.  To see how it does that let us next review the transaction processes.

Transaction Processes

Transactions in our context are external instructions that perform operations in the blockchain platform: In Bitcoin a transaction could be an instruction for A to pay B a certain certain amount of Bitcoins; in Ethereum, a transaction could be to execute a certain escrow smart contract.  Either way handling transactions are highly custom operations specific to each platform.

In a way this would be obvious; while all blockchain systems are networks; while all store information in blocks and while all synchronize information via consensus algorithms, their operations and hence the transactions (which are the instructions to perform operations) would be somewhat unique to each platform.  

In any case, what is common for transactions in any blockchain platform is that each individual transaction, once ready, is transmitted by a node to its peers.  Each node that receives the transaction, in turn validates it and adds the transaction to memory pool. As we shall see in the next section, nodes rely on this memory pool to provide a list of uncommitted transactions which are then fed into blocks and eventually end up etched into the blockchain.

To get a better feel of the transaction process lets look at what transaction means for Bitcoin and Ethereum.

Bitcoin transactions

Bitcoin has a very specific protocol with only a single capability: to transfer tokens from sender to receiver.  The transaction process here is fairly straightforward:

Once the transaction is broadcast to its peers, peer will check validity and is valid will put the transaction in the memory pool to be committed to a block.  

Ethereum Transactions

Ethereum is a more complex platform compared to Bitcoin.  While Bitcoin has a single purpose, Ethereum was made to be a platform complete with a virtual machine to ensure turing completeness: i.e the ability for any type of programs to be developed and executed on the platform.  This makes Ethereum transactions more complex; unlike Bitcoin there are 4 types of transactions:

• Transactions to create smart contracts (smart contracts are programs that execute on the Ethereum blockchain platform)
• Transactionsto execute smart contract
• Transactions to send Ether (currency in Ethereum)
• Transactions to send messages

While a detailed description of the transactions described above is outside the scope of this article they all at a high level follow a common process which consists of creating the transaction creation, signing the transaction and finally culminating in transmitting the transactions to peers.

Block Formation Process

A block in blockchain is a collection of transactions.  Each uncommitted transaction in the memory pool needs to be added to the block, which in turn is added into a linked list kind of structure.  This chainification of blocks is called the “blockchain”. The block formation process consists of the former (when a transaction is added to the block) while the block commitment process, which we will review in the next section, will cover the latter.

The block formation process is highlighted in figure 7 above.  As we noted earlier, a block in blockchain is an aggregation of transactions.  There are two subtle points to note here which guarantees immutability in blockchain:

1. Within the block, transaction immutability (no changes to the transactions list) is guaranteed by a cryptographic structure called a Merkle tree (or a variant of it).  We will leave the discussion of Merkle trees and immutability to a separate article, but suffice to say that the cryptographic tree structure maintains the validity of the contents of the block.
2. Block immutability in turn (no one can replace an entire block) is guaranteed by a linked list structure whereby the newest block’s metadata contains a reference to the last block.

Once a block is created, the block is broadcast to its connected peers.  The next section details the final process: how nodes accept the block and add it to the blockchain.

Block Commitment Process

Once the transmitted block is received by peers, the block commitment process commences and is the final process; once the block is committed to the blockchain, the information or transaction is considered “etched into stone”.

Each node after it receives the block validates it according to the rules set by the consensus algorithm prior to appending the block to the blockchain.  Consensus rules typically call for validation of the block, its meta-data, individual transactions contained in the block as well as specific information contained in the block such as timestamp.

Once the validation is complete the block is appended to the local copy of blockchain, but with the understanding the blockchain is identical to one held by each non-faulty node in the network.  This is guaranteed by the consensus algorithm within some bounds of how faulty the network can be. Technically inclined readers who are Interested readers on how consensus works in presence of faults are referred to my previous article here.

Putting it all together

So lets review what we have done so far.

• We understood the general structure of blockchain as composed of a node software stack and a network.
• We understood what the node software stack entails (Storage, Protocol/Platform and Dapps)
• We then looked at four common processes that all blockchain systems have
• Network formation which describes how the blockchain network forms and how new nodes connect to an existing network.
• Transactions which describes how instructions to perform any operation on the blockchain are sent to the nodes.
• Block formation which describes how transactions are recorded onto blocks.
• Block commitment which describes how blocks are added into the blockchain.

Through this journey we have seen how an individual transaction is executed by the platform, how its stored and how the “chain” in the blockchain forms.  

We hope blockchain now looks a bit less intimidating than it did before :)

QED.