blockchaindistributed-systemsJune 30, 2025schedule7 min read

The Off-Chain Stack Your DeFi Protocol Can't Ship Without

Smart contracts handle the rules. Everything else — liquidation keepers, price feed relayers, indexers, monitoring — lives off-chain. Here's the production infrastructure that makes DeFi protocols actually work, built with the same distributed systems patterns as any high-reliability backend.

Eliecer Sánchez
Eliecer Sánchez

Chief Architect

Distributed Web3 backend infrastructure diagram

Every DeFi protocol has a dirty secret: the smart contract is the smallest part of the system.

The Solidity or Rust code that lives on-chain handles the invariants — what’s allowed, what isn’t, what state changes are valid. But a lending protocol that doesn’t liquidate undercollateralized positions in time, an AMM that can’t update its price feeds under congestion, or a bridge that silently drops events during a reorg — none of those fail at the smart contract level. They fail in the off-chain infrastructure that most teams build as an afterthought.

This post is about that infrastructure: keeper bots, on-chain indexers, event listener pipelines, and the operational patterns that separate protocols that stay live under stress from those that don’t.

What “Off-Chain” Actually Means

The EVM (and Solana, and every other smart contract platform) is a state machine. It transitions state deterministically in response to transactions. But a state machine can’t initiate anything — it only responds to inputs. Someone or something has to submit those transactions.

Off-chain infrastructure is the set of services that:

  1. Watch chain state for conditions that require action
  2. Compute what action to take (and when)
  3. Submit transactions that trigger on-chain logic
  4. Index on-chain events so your application can query historical state without re-scanning the chain

For a typical lending protocol like Aave or Compound, this means:

  • A liquidation keeper that monitors health factors and submits liquidation transactions when positions become undercollateralized
  • A price feed relayer that pushes oracle updates on-chain before they go stale
  • An event indexer that tracks deposits, borrows, repayments, and liquidations for the frontend and analytics
  • A monitoring layer that alerts on reorgs, missed events, and keeper failures

Miss any of these and the protocol breaks — not slowly, but in ways that cost users real money.

Building a Liquidation Keeper with Elixir

A liquidation keeper is a great example because the requirements look simple but the production constraints are brutal. You need to:

  • Poll or subscribe to chain state continuously
  • Compute health factors for thousands of positions
  • Submit liquidation transactions with competitive gas pricing during spikes
  • Handle chain reorgs without double-submitting or missing positions
  • Recover from RPC failures without dropping positions

This is a textbook OTP problem. Each position under monitoring can be a supervised process. Reorg detection is a GenServer tracking the latest finalized block. The submission pipeline is a Broadway consumer with backpressure.

defmodule Keeper.PositionMonitor do
  use GenServer

  @check_interval 12_000  # one ETH block

  def start_link(position_id) do
    GenServer.start_link(__MODULE__, position_id,
      name: {:via, Registry, {Keeper.Registry, {:position, position_id}}}
    )
  end

  def init(position_id) do
    schedule_check()
    {:ok, %{position_id: position_id, last_health: nil}}
  end

  def handle_info(:check, %{position_id: id} = state) do
    with {:ok, health_factor} <- Keeper.Chain.get_health_factor(id),
         true <- health_factor < Decimal.new("1.0") do
      Keeper.LiquidationQueue.enqueue(id, health_factor)
    end

    schedule_check()
    {:noreply, %{state | last_health: health_factor}}
  end

  defp schedule_check, do: Process.send_after(self(), :check, @check_interval)
end

Each position runs as an independent supervised process. If one crashes — bad RPC response, unexpected nil in the health factor calculation — the supervisor restarts it. The rest of the positions continue unaffected.

The liquidation queue feeds into a Broadway pipeline that handles the actual transaction submission:

defmodule Keeper.LiquidationPipeline do
  use Broadway

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module: {Broadway.DummyProducer, []},
        concurrency: 1
      ],
      processors: [
        default: [concurrency: 5]  # 5 parallel liquidation attempts
      ]
    )
  end

  def handle_message(_, %{data: {position_id, health_factor}} = msg, _) do
    case Keeper.Executor.liquidate(position_id) do
      {:ok, tx_hash} ->
        Logger.info("Liquidated #{position_id}: #{tx_hash}")
        msg

      {:error, :already_liquidated} ->
        # Lost the race — another keeper got there first
        Broadway.Message.failed(msg, :race_condition)

      {:error, reason} ->
        Broadway.Message.failed(msg, reason)
    end
  end
end

The concurrency: 5 in processors means we can attempt five liquidations simultaneously — important when a market drop creates hundreds of liquidatable positions at once. Broadway’s backpressure model means we won’t overwhelm our RPC provider with requests either.

Reorg Handling: The Part Everyone Gets Wrong

An Ethereum reorg happens when the network reorganizes its chain tip — blocks you thought were confirmed get replaced. For a keeper, this means:

  • A liquidation you submitted in block N might be in a block that gets reorged out
  • Positions you thought were liquidated might be “un-liquidated” on the new chain tip
  • Events you indexed from reverted blocks need to be removed

Most keeper implementations I’ve audited handle the happy path and ignore reorgs entirely. This is fine until a reorg happens during high volatility — exactly when your keeper is most critical.

defmodule Keeper.ChainMonitor do
  use GenServer

  @reorg_depth 12  # consider blocks final after 12 confirmations

  def init(_) do
    {:ok, %{finalized_block: 0, pending: %{}}}
  end

  def handle_info({:new_block, block}, state) do
    # Check if any pending blocks got reorged
    {valid, reorged} = state.pending
      |> Map.split_with(fn {_hash, number} ->
        number <= block.number - @reorg_depth or
        Keeper.Chain.block_exists?(hash)
      end)

    # Reprocess positions affected by reorged blocks
    Enum.each(reorged, fn {_hash, _number} ->
      Keeper.PositionReconciler.reconcile_affected_positions()
    end)

    new_pending = valid |> Map.put(block.hash, block.number)
    {:noreply, %{state | pending: new_pending, finalized_block: block.number - @reorg_depth}}
  end
end

The key insight: treat nothing as final until it’s at least 12 blocks deep. Keep a map of pending blocks. When you see a block at depth 12 that doesn’t exist in the chain anymore, you had a reorg — reconcile.

Indexing: Why Not Just Use The Graph?

The Graph is excellent for standard ERC-20 and ERC-721 patterns. For complex protocol logic — multi-hop routing, cross-contract state aggregation, conditional queries — you often need a custom indexer.

The architecture is simpler than it looks:

defmodule Indexer.EventListener do
  use GenServer

  def init(config) do
    {:ok, filter_id} = Indexer.RPC.subscribe_logs(%{
      address: config.contract_address,
      topics: config.event_topics
    })

    {:ok, %{filter_id: filter_id, last_block: config.from_block}}
  end

  def handle_info(:poll, state) do
    {:ok, logs} = Indexer.RPC.get_filter_changes(state.filter_id)

    logs
    |> Enum.map(&Indexer.Decoder.decode_log/1)
    |> Enum.each(&Indexer.Store.upsert_event/1)

    schedule_poll()
    {:noreply, %{state | last_block: max_block(logs, state.last_block)}}
  end
end

Store decoded events in Postgres. Index by block number, transaction hash, and the domain fields you query against. A lending protocol typically needs queries like “all borrow events for account X” and “total deposits by asset in the last 7 days” — neither of which is cheap to compute by re-scanning chain history every time.

The important detail: always store the block number and log index alongside every event. This is what lets you handle reorgs — when block 14,502 gets reorged, you delete all events where block_number = 14502 and re-index from that block.

What Actually Breaks in Production

After running keepers and indexers on mainnet across multiple protocols, here’s what causes real incidents:

RPC provider failures. Single-provider setups fail. Use at least two — Alchemy and Infura, or one managed and one self-hosted node. Switch on failure. The Elixir circuit breaker pattern applies directly here.

Gas price spikes during liquidation cascades. When multiple positions become liquidatable simultaneously (a flash crash), gas prices can spike 10–50x in seconds. Your keeper needs dynamic gas pricing — either EIP-1559 tip adjustment or a gas oracle. A keeper that submits at baseFee + 1 gwei during a spike will never get included.

Memory accumulation in long-running indexers. Indexers that process millions of events accumulate state over time. Profile heap size after 72 hours of continuous operation, not just during development. GenServer processes in Elixir have their own GC, which helps — but the Postgres write queue can still back up if your indexer processes faster than it can write.

Missing events during RPC websocket reconnects. Websocket connections drop. During the reconnect window, you miss events. The solution: always maintain a last_indexed_block cursor and backfill missed events on reconnect using the eth_getLogs RPC method with a block range.

Production Checklist

Before going live:

  • Two independent RPC providers with automatic failover
  • Reorg detection with configurable finality depth (12 for mainnet, 3 for optimistic rollups)
  • Keeper competition strategy — are you MEV-protected, or do you accept racing?
  • Gas oracle integration with floor and ceiling limits
  • Event backfill on indexer restart
  • Alert on missed heartbeats from keeper processes
  • Separate deployment for indexer (read) and keeper (write) — they have different failure modes and scaling characteristics

The on-chain protocol is the easy part. The off-chain infrastructure is where protocols actually live or die.