The AMM test: a true look at L1 performance

Haseeb Qureshi, Mar 2022

Multichain is now a reality. Ethereum’s lack of scalability has caused a mass migration to a new generation of L1s. Most of these L1s use the EVM (the Ethereum Virtual Machine), which makes them compatible with Ethereum wallets and developer tools. But Solana has completely rebuilt its stack from the ground up. Solana claims to be the fastest blockchain in existence. So it begs the question: Just how much faster is Solana than the EVM chains?

Most benchmarks released by L1s themselves measure TPS of simple value transfers — i.e., transferring coins from one account to another. Simple transfers are extremely cheap and thus produce big numbers, and everyone loves big numbers. But no blockchain is actually bottlenecked on transfers like this, and this kind of activity doesn’t reflect real-world usage patterns. Furthermore, many of these numbers are generated on devnets or testnets rather than on mainnet. We don’t care about what someone’s software can do in the abstract: we care about what is possible on current mainnets.

Okay, fine. So how should we actually measure L1 performance?

That’s a tricky question, because performance has multiple dimensions.

But let’s say we want to ignore decentralization and purely focus on performance. Well, benchmarking blockchain performance is notoriously hard because most new chains have very poor data visibility.

7 years in, Ethereum performance is highly studied and very well-understood. But as you start exploring newer chains, most of them have much less tooling, poor observability, and are constantly evolving. By the time you read this, these benchmarks will probably be out of date.

Furthermore, benchmarking is always arbitrary and riddled with pitfalls. The best you can do is pick a benchmark that measures something valuable, and then qualify your results as carefully as you can. That’s what we’ll be attempting to do here.

But what do we even mean by performance? There are two aspects to performance: throughput and latency.

You can visualize blockchain performance like water flowing through a pipe. The transactions are the water — you want lots of transactions flowing through the pipe at once. But the length of the pipe is what determines its latency — if it takes a long time for a transaction to get confirmed, even if lots of transactions can get confirmed at once, that’s not ideal.

Latency can be subdivided into block time (how long between blocks) and time to finality (how long until a block definitely won’t be reverted). Block time and time to finality are easy to measure.

But to actually measure throughput you need a standard unit of measure. Throughput of what?

We chose this benchmark because 1) it’s simple and easy to measure, 2) every blockchain has a Uniswap V2-style AMM live in production, 3) it’s typical of common smart contract usage patterns.

For most chains that have a gas model, this back-of-the-envelope exercise should be straightforward. First, find the block gas limit and the block time to derive the gas/sec throughput of the chain; next, find an Uniswap v2-style AMM and pick a SwapETHforTokens equivalent transaction; lastly, divide the first number by the second to arrive at how many tx/sec would it achieve if its blocks were stuffed full of identical AMM trades.

Note: this is not a perfect benchmark! It’s idiosyncratic, it doesn’t account for parallelizable transactions (since Uniswap trades on the same pool must be linearized), and it’s not representative of every usage pattern. But smart contract usage is always power-law distributed, and the most used Dapps tend to be AMMs, so within a suite of benchmarks, we believe this is illustrative in getting a holistic view of performance.

So without further ado, let’s go down the list.

We will be using a similar calculation for other EVM chains on the list.

(Note: we are ignoring rollups with this methodology since all smart contract L1s are capable of adding rollups.)

Time to finality: There are two notions of finality on Polygon

Time to finality: ~1.75s after the block is produced

Avalanche is relatively hard to compare due to its block production mechanism being so different from Ethereum and the PoS chains. For Avalanche, there’s a large spread between what it can perform at maximum throughput and what it performs at average throughput. (Chains like Ethereum that have implemented EIP-1559 are bounded by 2x of their average throughput.)

This concludes the benchmarking of the EVM blockchains — the blockchains whose virtual machine is modeled on Ethereum’s. Since all EVM chains use the same gas model, we can look at gas/sec as a benchmark for throughput. The solid bars denote target throughput, and hollow bars represent the limit.

Gas/sec for EVM chains

You can imagine that Binance Smart Chain is what happens when you run the EVM at its absolute limit. If you want to get higher performance out of smart contracts, you’ll have to move away from the EVM entirely.

Orca trades per second: 273.34

Block time: 590 ms

Here’s how we calculated this number. This one’s a doozy.

We first wanted to find a “gas limit” equivalent for Solana. You can’t find any number like that on block explorers. We started by asking some Solana developers we knew, but nobody seemed to know definitively if there even was such a limit. So we rolled up our sleeves and went on a trip to find out for ourselves.

Second, only a limited number of CUs are writable to a single account in a single block. This limit is to prevent too many transactions writing to the same account, therefore reducing a block’s parallelism — though this is exactly what happens during mass congestion, such as during a popular IDO, when all transactions are competing to use a single contract.

This number seems lower than expected! For us to trust this number, we’d want to verify this approach empirically.

We spammed the Orca SOL-ORCA trading pair on the devnet to see how many Orca swaps we could land in a single block, and then extrapolated to the max throughput.

The highest number we managed to hit was 184 swaps in a single block. Assuming a block time of 380 ms, this gives us 484.21 swaps/second on the devnet. (Note that block times are not exact, so there is some jitter in these numbers. If you average across the 3 blocks where we landed the most transactions, it looks more like 381 swaps/second, which seems more reasonable). This seems to confirm that our analytical approach was correct (~10–15% delta), which therefore implies Solana’s mainnet can likely perform about 273 swaps/second on an AMM.

The AMM test: Uniswap v2 style swaps/sec performance

So what’s the upshot of all this?

First, don’t take this as gospel. Do the math yourself.

Second, remember that all these blockchains are moving targets. They’re continually being optimized, and the technology is evolving rapidly, while any benchmark is a moment-in-time snapshot. We’d love to see more independent organizations creating standardized benchmarks, but this is our best attempt.

Third, notice that the spread in performance between these blockchains is not as big as advertised. The performance difference between Ethereum and the very best chain is about 10–25x, not 100x or 1000x. Nobody is getting that great performance out of linearized VM transactions; that will require a lot more work and optimization.

Fifth, users aren’t that sensitive to performance considerations on non-Ethereum L1s right now. They care a lot more about the overall strength of an ecosystem, good UX, and low fees. These blockchains are not currently competing on performance because none of them are actually being used to capacity except during rare spikes, such as during IDOs or market meltdowns.

We expect that all of the major L1s will improve in their performance over time, as the dev teams spend more and more time tuning the performance across typical usage patterns. It should be no surprise that in their early days, each of these blockchains is poorly optimized!

But overall I come away with this impression: Ethereum is the MS-DOS of smart contract operating systems. But the current era of blockchains takes us into the Windows 95 era.

Next-generation blockchains represent a marked improvement, but there’s much further to go from here to get to mainstream adoption.