The Actor Model

The actor model has allowed for systems that are scalable, concurrent and fault-tolerant, and has shaped how modern applications deal with massive workloads. The actor model has perhaps one of the most well known success stories around it in the core at PayPal, where it enabled them to achieve 1 billion transactions a day working on just 8 virtual machine system. The stateless execution model provides technical abstraction that might sound too good to be true, is explained in detail around this stateless execution model, how they use it at PayPal, and how the technology could lead to extreme scalability and efficiency.

Dealing with the Scaling Challenges of PayPal

In the early 2000s, PayPal was growing like a weed, processing over 1 million transactions per day. To handle the load that was growing, the company used a horizontal scaling approach by adding more servers, and they were able to meet demand. At its height, PayPal’s infrastructure had over 1,000 virtual machines running — but that wasn’t without its drawbacks:

Latency and Network Overhead

The growing number of servers needs more network hops, causing latency and more operational cost.

Resource Waste

Countless servers under utilized, burning CPU and RAM

Maintenance became difficult

They had lots of deployments, autoscaling and monitoring for so many servers.

The Need for a New Approach

As transaction volume grew to the point where cut-and-extend hardware was no longer sustainable, PayPal realised it required a different approach. They needed a software solution to utilize resources effectively, scale efficiently and automatically reduce operating costs. They opted to change systems to one based on the actor model, which they adroitly implemented using the Akka framework, affording phenomenal throughput at the price of just 8 virtual machines.

What Is the Actor Model?

The actor model decomposes computation into a large number of small, independent actors, each of which is a lightweight unit of computation. Each actor:

• Serial processing of messages: Each actor processes exactly one message at a time — that is, no threads are being utilized; hence actors are thread-safe without requiring locks.
• State: State per actor and users avoid the pitfalls of race conditions.
• Messages pass in one direction: communication between actors occurs only through the exchange of asynchronous messages, leading to loose coupling.
• Lightweight: Actors are extremely lightweight, much lighter than a thread, and millions of actors can coexist in a single system.
• Fault Tolerance: Actors can supervise other actors and restart or replace them in case of failure.

This makes building systems with highly distributed, scalable and fault-tolerant property very natural and straightforward for the actor model.

The Actor Model-Inspired Domain Model for PayPal

PayPal embraced the actor model to build a high-throughput, fault-tolerant system of systems to safely and effectively process transaction volumes in the millions per second. Taking advantage of the Akka framework that supports both Scala and Java, Akka "Supervisors" used the actor model seamlessly to create stunning performance.

Here’s a summary of what PayPal learned about using the actor model:

High-Level Architecture

Paypal designed their system around the actor model, enabling them to process concurrent processes quite well and scale much more easily. When statistics are stored, they use Akka to maintain the state of these actors.

Main Aspects of PayPal’s Actor-Based Architecture

Actor Hierarchy

• PayPal’s system had a hierarchical structure of actors, with parent actors overseeing child actors.
• This hierarchy allowed for fault tolerance; failing child actors could be spawned by parents at will.

Message-Driven Communication

• Actors were completely message-driven and asynchronous in their communication.
• By this it ensures components are loosely coupled and some easy design and scalable systems.

Thread Pool Management

• The actor model automatically takes care of threads — it also guarantees that an actor gets to process a single message at once.
• No more thread management and no more locks, therefore less complexity and improved performance.

Distributed Deployment

• PayPal deployed their actors to multiple VMs, resulting in a distributed architecture.
• With this, the system can manage bigger workloads at low latency and high throughput.

The format and specifications of the Virtual Machines (VMs)

Though there are no public details on the specs of PayPal’s VMs, they were probably tuned for high-throughput actor workloads. A standard setup would look something like:

• CPU: High core count CPUs (32+ cores preferred) for Parallelism Processing
• Memory: This could be in the order of 64–128 GB of RAM for handling in-memory computations and storing actor states.
• Storage: Fast SSDs for logging and temporary files.

With these specifications, the VMs were capable of handling the demanding requirements of PayPal’s actor-based system, delivering low latency and high throughput.

In my next post, I'll share how to implement the Actor Model in Node.js.

Sources

https://newsletter.systemdesign.one/p/actor-model

https://stately.ai/blog/what-is-the-actor-model