Load balancing Nextcloud

Venturing even deeper into uncharted territory I have decided to try a setup which I have not found any prior examples to:

Use HAProxy for fastCGI reverse proxy and loadbalancing for Nextcloud!

A common setup when reverse proxying in front of Nextcloud you serves the Nextcloud PHP application from a webserver - typically NGINX or Apache2. However for NGINX setup the best solutions is actually serving the local Nextcloud application over php-fpm reverse proxying to a unix socket. So now that HAProxy from version 2.1 supports loadbalancing and reverse proxying fastcgi natively, then why not skip the NGINX al together? This could solve one of the headaches regarding properly load balancing a cluster of Nextcloud Servers as the session as maintained by the load balancer all the way and not conflicting with a local webserver on each Nextcloud node. Further it removes all pains in regards to dabbling with the proxy protocol for forwarding real connecting source IP as the HAProxy understands this protocol natively - if it is added by another proxy before reaching HAProxy - and because HAProxy will be the first component to receive the connection and hence acting as webserver, builds the HTTP headers and passes the stream through TCP to a receiving FastCGI server.

This also means that HAProxy will terminate the TLS traffic and the downstream data flow to the backend PHP servers can go through internal VPN tunnels - if that trafic really has to be encrypted. Here wireguard is a good option where the HAProxy server acts as Wireguard VPN server and each node is a VPN client connecting via certificates. For my lab setup then I use LXC container for the PHP-FPM server running on the same physical box as the HAProxy, so the network is fully virtualized and never exposed "on the wire" so I will not use Wireguard.

As I will experiment with having a cluster of PHP servers I will need to find a way using the same data storage. This can be accomplished (at least in theory) by mounting a zfs storage pool to both servers. This is tricky for two reasons:

1. The data can be accessed by more than one server at the time.
2. If not sharing only Nextcloud files storage for user files it will be impossible to run two different nodes using the same file system.

As Nextcloud data is access/written/modified or deleted from not only a web browser, but also from clients on devices it is already in a classic setup, possible to access the same file at the same time. The mechanisms in Nextcloud solves this already but how does it react if not the same host - which can lock a file to a specific worker thread? Here I think it will only be possible to solve with mirrored data, which synchronizes on events, and can handle versioning and merging of files. SyncThings could be a good solution.

As each node is its own little server then the OS cannot share filesystem with another. They each needs their own separated storage.

I do not have much spare storage laying around, so this will not be a massive test setup. On top of that I also has to take into account what to do in regards to Database. I have two options.

1. MariaDB
2. PostgreSQL

Maria DB is proven and officially recommended over PostgreSQL by Nextcloud however reading through community then PostgreSQL is winning more and more terrain and is fully supported. With PostgreSQL I get open source enterprise grade DB server with a loads of features which could easily be the no brainer for my particular setup. On top of that it is extremely stable and has native support for load balanced setups and multiple different replication modes and backup features. I will go with PostgreSQL.

When done

If successful I will use this setup with just a single node for my existing Nextcloud solution with a failover node. Or I might differentiate the purpose of the nodes so one node is for collaboration (OnlyOffice, Talk, Deck etc.) and the other is for the rest. That way I can spread the load so the user experience of browsing the front end will not be to bothered by any heavy and high consuming process like a Talk session with more than 5 participants.

Have you used the FastCGI feature in HAProxy before?