What It's Like to Build Systems That Trade Billions

The first time I really understood what sub-millisecond meant, I was not staring at a performance benchmark. I was standing in the colocation facility, looking at a row of servers in a cage, and a senior engineer explained to me that the distance between our rack and the exchange's matching engine — physically, in meters — added measurable latency to our order execution. Speed of light through fiber. We had actually optimized the length of the cable.

That is the world of market making at IMC Financial Markets, where I spent more than eight years across two stints. IMC is one of the top three options market makers in the world, operating across more than ninety exchanges globally. The firm's entire business model depends on being able to buy and sell financial instruments at quoted prices, continuously, faster and more accurately than anyone else. The technology is not a support function for that business. The technology is the business.

What a Market Maker Actually Does

Before the engineering, the context. A market maker provides liquidity. When you buy an option on the S&P 500, somebody on the other side of that trade probably does not want to own that position forever — they are a market maker who priced the option, took the trade, and will now hedge it and unwind it through a series of offsetting trades. The market maker makes money on the spread: the difference between the price they will buy at and the price they will sell at. Do this millions of times a day, across thousands of instruments, across dozens of exchanges, and you have a business. Do it one millisecond slower than your competitors and you have a very expensive problem.

IMC operates on the CME in Chicago, Eurex in Frankfurt, ICE in Atlanta, the London Stock Exchange, NYSE Arca in New York, OPRA — the Options Price Reporting Authority — and many more. My first role, which I started in 2007, was on the exchange connectivity team. My job was to make sure that our software could speak the native protocol of each of those exchanges: each one has its own message format, its own session management rules, its own sequencing guarantees and failure modes. Getting those protocols right is the foundation on which everything else is built. If you drop a message, misinterpret a sequence number, or handle a disconnect incorrectly, the consequences range from bad fills to regulatory violations to positions you did not mean to hold.

The Goldman Sachs Reconciliation Problem

In 2014, IMC acquired Goldman Sachs' NYSE Designated Market Maker rights for more than 630 securities. That is a significant business event — overnight, IMC became responsible for making markets in hundreds of additional equities on the NYSE floor. It also created an immediate engineering problem: all of those trades and positions needed to be reconciled. Goldman Sachs and ABN Amro Clearing were involved in the settlement chain. How do you know at the end of each day — actually, continuously throughout the day — that your books match the exchange records, and that both match the clearing house?

Before I built the centralized reconciliation system, this was a fragmented process. Different desks had different views of their positions. Breaks — discrepancies between what you think you own and what the clearing house says you own — had to be identified and resolved manually. In a firm trading thousands of instruments across dozens of venues, that is not sustainable.

The system I built consolidated trade and position data from all of IMC's global operations into a single reconciliation engine, with direct integrations into Goldman Sachs' and ABN Amro's clearing systems. It ran continuously, flagging breaks in real time, and it went from inception to live production in four months. I am still proud of that timeline, not because four months is impressively fast as an abstract fact, but because it was fast in a context where the stakes were real: every day the system did not exist, breaks were being identified manually and resolved at significant cost. Four months mattered.

Three Months to an ETF Desk

I came back to IMC for a second stint in 2016. By then, the ETF market had grown dramatically — Exchange Traded Funds had become a dominant investment vehicle, and market making in ETFs required a specific capability: real-time basket valuation. An ETF is a fund that holds a basket of underlying securities. To price the ETF accurately — to know what it is worth right now, not at end-of-day settlement — you need to continuously compute the net asset value of every security in the basket, in real time, accounting for the fact that those underlying securities might trade on different exchanges in different time zones.

IMC did not have an ETF trading desk when I started that second stint. Launching one was the objective. I built the real-time dynamic ETF basket valuation system using Java with Akka — an actor model framework that handles concurrent, distributed computation particularly well — integrated with Bloomberg for reference data and real-time price feeds. Within three months, the desk was live and trading. Three months from no ETF capability to an operational trading desk that was handling real market flow.

The news aggregation system I built alongside that — pulling data from Bloomberg terminals, Reuters feeds, Federal Reserve announcements, jobs data releases, and a custom Twitter aggregator — was designed to give traders a unified view of market-moving information. The latency between a Fed announcement and our system's awareness of it had to be as short as possible. A jobs report that comes in worse than expected moves markets in milliseconds. You need to know before your quotes become stale.

What Goes Wrong at Nanosecond Scale

Here is what most people outside this industry do not understand about building trading systems: the failure modes are not the same as in ordinary software. In a web application, a slow database query degrades user experience. In a trading system, a slow database query is a price that stays stale while the market moves, which means you are quoting a price you should not be quoting, which means adverse selection — smart traders trade against you and you lose money.

The pathologies are different too. Memory allocation matters because garbage collection pauses — the JVM deciding to reclaim memory — can take several milliseconds. Several milliseconds is an eternity. The pattern we used extensively was ring buffers: pre-allocated, fixed-size data structures that trade messages by moving a pointer rather than allocating new memory. The LMAX Disruptor pattern was foundational to our approach. You write once to a position in the ring, a consumer reads from that position, and no memory allocation happens during the hot path. This is not a micro-optimization. It is the difference between a system that behaves deterministically under load and one that has unpredictable latency spikes.

I spent serious time in the R&D work on low-latency frameworks: kernel bypass networking with DPDK, CPU affinity settings that pinned threads to specific cores to avoid context switching, NUMA-aware memory allocation on multi-socket servers, the careful elimination of every lock in the critical path. Modern CPUs have a deeply complex hierarchy of caches — L1, L2, L3 — and data that does not fit in cache has to be fetched from main memory, which is roughly a hundred times slower. Understanding your data access patterns at the cache line level is not optional in this domain. It is part of the job.

The Weight of Getting It Right

There is a particular pressure to working on systems where the cost of a bug is immediate and financial. A mistake in the exchange connectivity layer can cause erroneous orders. An erroneous order at scale — the wrong side, the wrong quantity, the wrong instrument — is called a fat finger, and they can move markets. Firms have gone from profitable to bankrupt inside an afternoon because of software defects in trading systems. Knight Capital famously lost $440 million in 45 minutes in 2012 due to a deployment error that activated old code.

You feel this weight differently than you feel the weight of building consumer software. A bug on a website is embarrassing. A bug in healthcare software can cost a life. A bug in a trading system at a market maker operating on ninety exchanges can cost millions of dollars in minutes and trigger regulatory scrutiny. The discipline required — the code review culture, the simulation environments, the careful change management, the circuit breakers and hard limits that shut the system down if it behaves outside expected parameters — is not bureaucracy. It is the only thing that stands between a mistake and a disaster.

What I miss most about that world is the clarity. The performance numbers are real. The P&L impact is real. You build something, you measure it, and the market tells you immediately whether you got it right. There is no ambiguity, no quarterly review, no management deciding that the KPIs look good enough. The exchange does not care about your good intentions. It only cares about the speed and correctness of your messages.