remora

Remora

Remora is a scale-out execution engine for blockchain validators. It uses an asymmetric architecture where a single primary (coordinator) node handles consensus and scheduling, while a pool of proxy nodes execute smart contracts in parallel with strict determinism guarantees.

Overview

Remora addresses the performance bottleneck in modern blockchain systems where validators must execute transactions sequentially. The system provides:

The implementation consists of ~13k lines of Rust code and supports multiple workload types including synthetic benchmarks (Fake), database benchmarks (TPC-C), and real blockchain transactions (Sui).

Architecture

Remora implements the following key components:

Primary (Coordinator) Node

Code: src/primary/, src/bin/remora.rs

The primary is the single point of coordination that interfaces with consensus. It contains:

The primary assigns unique versions to each object access within a consensus batch, ensuring deterministic execution order across distributed proxies.

Proxy Node

Code: src/proxy/, src/bin/remora.rs

Proxy nodes execute transactions assigned by the primary. Each proxy:

The runtime (src/proxy/core.rs) uses tokio::sync::Notify primitives to enforce version-based dependencies, allowing transactions to execute in parallel while respecting the consensus-established order.

Executor Backends

Code: src/executor/

Remora supports three executor types:

All executors implement the Executor trait (src/executor/api.rs) which defines transaction generation, execution, and verification interfaces.

Load Generator (Client)

Code: src/client/, src/bin/load_generator.rs

The load generator submits transactions to the primary at a configured rate and measures end-to-end latency.

Directory Structure

remora/
├── src/
│   ├── bin/
│   │   ├── remora.rs              # Main binary (primary/proxies)
│   │   └── load_generator.rs      # Client load generator
│   ├── primary/                   # Primary implementation
│   │   ├── shared_obj_txn_forwarder.rs  # Scheduling policies
│   │   ├── load_balancer.rs       # Stateless routing
│   │   ├── mock_consensus.rs      # Consensus module
│   │   └── node.rs                # Primary node logic
│   ├── proxy/                     # Proxy implementation
│   │   ├── core.rs                # Proxy runtime and DAG execution
│   │   └── node.rs                # Proxy node logic
│   ├── executor/                  # Executor backends
│   │   ├── api.rs                 # Executor trait definition
│   │   ├── fake.rs                # Fake executor
│   │   ├── tpcc/                  # TPC-C executor
│   │   └── sui.rs                 # Sui executor
│   ├── client/                    # Load generator
│   ├── networking/                # Network communication
│   ├── config.rs                  # Configuration structures
│   └── metrics.rs                 # Prometheus metrics
├── assets/                        # Configuration examples
└── tests/                         # Integration tests

Prerequisites

Building

Build the release version:

cargo build --release

The binaries will be in target/release/:

Build for development (faster compilation, slower execution):

cargo build

Configuration

Remora requires two configuration files: validator config (network/node setup) and benchmark config (workload parameters).

Validator Configuration

See assets/example-validator.yml:

# Primary network addresses
proxy_server_address: "127.0.0.1:8080"    # primary listens for proxies
client_server_address: "127.0.0.1:8081"   # primary listens for clients
metrics_address: "127.0.0.1:9090"         # Prometheus metrics

# Proxy configurations
proxies:
  - proxy_id: 0
    listen_proxy_address: 127.0.0.1:19000      # Proxy-to-Proxy communication
    listen_primary_address: 127.0.0.1:19002    # Proxy-to-Primary communication
    metrics_address: 127.0.0.1:19090
  - proxy_id: 1
    listen_proxy_address: 127.0.0.1:19001
    listen_primary_address: 127.0.0.1:19003
    metrics_address: 127.0.0.1:19091

# System parameters
validator_parameters:
  load_balancing_policy: !LSDS    # Scheduling policy: LSDS, SDS, RSDS, Zeus
  separation_mode: !NoSeparation  # Stateless/stateful separation mode

Scheduling Policies (see src/primary/shared_obj_txn_forwarder.rs):

Separation Modes:

Benchmark Configuration

See assets/example-benchmark.yml:

load: 10000                # Transactions per second
duration:
  secs: 10                 # Benchmark duration
  nanos: 0
workload: !FakeZipfian     # Workload type
  alpha: 0.5               # Zipfian skew (0.0 = uniform, higher = more skewed)
  number_of_inputs: 5      # Objects per transaction

Workload Types:

Running

Local Single-Machine Setup

Terminal 1 - Start the primary:

RUST_LOG=info cargo run --release --bin remora -- \
  --validator-config assets/example-validator.yml \
  --benchmark-config assets/example-benchmark.yml \
  primary

Terminal 2 - Start proxy 0:

RUST_LOG=info cargo run --release --bin remora -- \
  --validator-config assets/example-validator.yml \
  --benchmark-config assets/example-benchmark.yml \
  proxy --proxy-id 0

Terminal 3 - Start proxy 1:

RUST_LOG=info cargo run --release --bin remora -- \
  --validator-config assets/example-validator.yml \
  --benchmark-config assets/example-benchmark.yml \
  proxy --proxy-id 1

Terminal 4 - Run the load generator:

RUST_LOG=info cargo run --release --bin load_generator -- \
  --validator-config assets/example-validator.yml \
  --benchmark-config assets/example-benchmark.yml

The load generator will submit transactions for the configured duration and report throughput/latency metrics.

Multi-Machine Setup

For distributed deployment:

  1. Update network addresses in validator config to use actual IPs instead of 127.0.0.1
  2. Run the primary on one machine: remora ... primary
  3. Run each proxy on a separate machine: remora ... proxy --proxy-id N
  4. Run the load generator from a client machine

Use --binding-address <IP> flag to override the binding address:

cargo run --release --bin remora -- \
  --validator-config assets/example-validator.yml \
  --binding-address 0.0.0.0 \
  primary

Testing

Run the test suite:

cargo test

Run a specific test:

cargo test remote_proxy_sui

Integration tests (tests/common_case.rs) start a primary, multiple proxies, and a load generator to verify end-to-end functionality.

Metrics and Monitoring

Prometheus Metrics

Each node exposes metrics on its configured metrics_address:

View metrics:

curl http://127.0.0.1:9090/metrics

Key Metrics

See src/metrics.rs for all metrics. Important ones:

Grafana Dashboard

Import assets/grafana-dashboard.json into Grafana to visualize:

The primary also prints periodic metric summaries to stdout when run with RUST_LOG=info.

Research Context

Key concepts and their code locations:

Paper Concept Code Location
Primary src/primary/
Proxy src/proxy/
Version assignment src/primary/shared_obj_txn_forwarder.rs (assign_versions)
Subgraph-first scheduling (SFS) src/primary/shared_obj_txn_forwarder.rs (PreConsensusSchedulingPolicy::LSDS)
DAG-based execution src/proxy/core.rs (dependency tracking)
Object ownership tracking src/primary/shared_obj_txn_forwarder.rs (metadata maps)
Stateless/stateful separation src/executor/api.rs (PrimaryToProxyMessage enum)
Periodic snapshotting src/proxy/core.rs (epoch-based state reporting)
Elasticity (scale-out/scale-in) src/primary/load_balancer.rs

License

Apache License 2.0. See source file headers for details.

This site is open source. Improve this page.