In Memory Trading - prototype

We all love low latency solutions. When customer sends his request or order, nowadays it really is matter of microseconds, which decides about success. There are simply too many users at the same time, which needs to be served also at the same time. In case of trading systems, slow reactions might cost you a fortune. I don't want to discuss morality of algotrading, as everything, it has pros and cons. I was just thinking about technical concept in background. What is the fastest solution possible? (SPOILER: Really interesting part is the very end!)

So let's create a 'no latency' model. Let's have some liquidity generator, some database representing an orderbook, and let's have all of these components hosted in the very same memory chip - putting these components closest to each other, as physically possible. You simply cannot get faster than this!

Week 1

I started to study Redis database. Well, honestly, it is not your traditional SQL database, where you can structure data and query them later. In fact, Redis is described as key-value store. Practically it means, that Redis is perfect for searching keys and for storing values into these keys. It cannot do much more - but what it does, it does really quickly, because there is no communication with harddrive or anything - all is in the memory.

That means, you must rethink your normal orderbook structure. Instead of having order containing MYCOMPANY BUY WOOD 100 as four independent values in one row, you have to come up with something different. And here it comes - it was my colleague's idea, to have a key name as a string 'buy-wood-100' and then this key would contain list of customers, waiting in the line for fulfilling this order. Brilliant idea, Stan!

From here, we figured out which Redis functions are applicable to our use case. Concept has basically two phases - first is, that customer would like to fulfill an order. Second is, that customer would like to add himself into the queue for specific order. So imagine, that you have list of items (keys) like buy-wood-1, buy-wood-2, buy-wood-3... up to buy-wood-100 (and the same for sell side and for other products) - and each of this items contains list of customers in order of their arrival. There are two commands - RPUSH puts customer at the end of the waiting line and LPOP will remove first customer from the waiting line, making this operation basically a trade! Obviously, when you would like to check, who is the first guy in the queue, you have to use third command LRANGE, which will give you content of the key, all values or just a first one.

I have installed Redis to EC2 instance on Amazon. I do know, that AWS actually offers managed Redis service (Elasticache), but for sake of learning I wanted to do it first manually, so I actually learn details. Installation and configuration was matter of minutes, thanks to excellent documentation. I discovered, that there are two basic ways, how to connect. Via normal network interface (TCP/IP) or via Unix Socket. Because I wanted to create fastest solution possible, I decided to use socket connection.

Next was liquidity generator, simulating algotrader. Because I was so successful with NodeJS, I wrote simple program in Javascript and my colleague Stan did merely the same thing, but in Python, so we could compare results. When running locally, first tests were positive. Both applications were able to generate 100k orders in less then 40 seconds (it was slowest EC2 instance offered by Amazon of course). However, that was not the end. I mentioned that above - I wanted to try to create usable product, meaning I need to isolate customer and backend, while I would keep everything in one memory.

Logical step forward for application isolation was Docker. We put both - NodeJS and Python applications into containers and run them on the same host. Finally, something went wrong! Woohoo! Time to learn new stuff, so exciting! So - you cannot connect Docker container to socket, you have to use network interface. Lesson learned: from speed perspective, it is not a big difference between TCP/IP and socket, so this was actually useful discovery. But things were still not working as expected - good, another opportunity to try new stuff.

We ended up with Redis running in Docker container as well and we only linked our algotraders with it. This solution turns out to be brilliant, as Docker use it's own naming for resources, so regardless of IP address provided by Amazon, our services were always using the same script to run them. Ohh, I love this!

Week 2

Last major step was to make automatic deployment. Our other colleague from SysAdmin team provided us Ansible script and boy / girl, it is awesome! You know, Ansible can do almost the same stuff as Puppet or Chef, but it does stuff differently. First - it uses standard SSH to do all of it's magic, no agent needs to be installed on target machine. Second - Ansible describes 'desired state'. So you only define "I want this service to be up and running" and send it to the cluster of machines (doesn't has to be in cloud). Ansible will login to all of them and check current status of the service and based on this status, it will do some magic (install packages, configures them as a service and run them) or it will do nothing (in case service is already up and running).

Our Ansible script spins up new EC2 instance (in correct VPC, with correct security groups etc.), it installs basic EPEL repository, then Docker, then it pulls all necessary Docker images (Redis, our algotraders - all from public DockerHub) and run them with correct configuration. Whole process takes around minute, while longest is obviously EC2 provisioning. This is what I'm talking about, 'everything as a code' - infrastructure, database and application. Nice! And it can be improved in many directions, I hinted above usage of managed services (but think about latency), core usage separation for Docker and Redis on the host and so on!

Week 3

Almost done. The plan was to also provide simple webpage, something, what would visualize successful trading of algotraders. It is always nice to have some bells and whistles, but we decided to do something different (also I never developed a GUI). Our server was capable of accepting orders from inside the host (Docker or local unix socket), also remotely via internet and also remotely from Lambda, server-less solution from Amazon (and therefore I was able to connect this application to Slack, I already wrote something about it earlier). However, even if technically impressive, I wasn't sure, if this is what will blow away our customers. That calls for customer visit.

And this is the best part, as promised on the very beginning. We have visited one of the most successful algotraders in the world, which happens to sit in the Prague as well. We show them the concept and talked about it more. They were not impressed at all, as they already are fastest on the market :), but that was okay, I didn't expect them to celebrate our simple prototype. However, their feedback was rather inspirational. That one hour of talking together moved us forward more than I would expect. We talked about ways, how to create more fair trading environment for whole world. How to enable customers to do their own computations inside the backend, while backend would provide much simpler functionality (and therefore easier to manage). We considered also non-existent decentralized trading systems. So much fun!

So, this was VERY productive month. We learned so much of new stuff and we got enough food for thoughts to continue!