This document describes performance considerations for administrators and developers of
stellar-core. It may also be interesting for a broader audience interested in understanding system performance and scaling limits and avenues for future work.
The performance of a
stellar-core node varies in two main dimensions:
What it is configured to do:
How it is physically and logically configured to do its job:
This document will begin with a brief structural review of subsystems of
stellar-core and their performance considerations. Then for each of the two dimensions above, discuss it from technical-design perspective, then contain empirical advice from operators of existing
stellar-core nodes, and finally note some areas where the current developers anticipate doing future performance work.
For purposes of understanding performance, the following subsystems of
stellar-core are worth keeping in mind:
For more detail on these subsystems, see the architecture.md document.
Nodes that are participating in consensus – basic and full validators – will typically cause higher load on the overlay, herder and SCP subsystems than nodes not participating in consensus, especially if they are themselves relied-on in many other nodes’ quorum sets. This load is usually in the form of bi-directional network traffic and CPU usage.
Furthermore, configuring a node with a larger quorum set – whether the node is participating in consensus or not – will typically cause higher load on the overlay, herder and SCP subsystems, and thus network and CPU usage.
Finally, regardless of consensus participation, the numerous
*_CONNECTIONS parameters that govern the overlay subsystem’s connection management strategy will influence load on overlay directly, and herder and SCP indirectly through increased broadcast traffic (even if mostly redundant). The general rule is: more connections means more load, though connections do need to be kept numerous enough to maintain low latency and connectivity.
Consensus-participation (or lack thereof) will typically not alter the load caused on the transaction, ledger, database, history or bucket list subsystems: every node that is even watching the network will apply all the transactions to the ledger, so needs an equally performant database.
A caveat to the previous paragraph is that if the SCP, herder or overlay subsystems are overloaded, a node may fall out of sync with its peers, which will then cause a load spike on the history subsystem (and subsequently on transaction, ledger, database and bucket list subsystems) as it initiates catchup.
Nodes that are saving historical records to a history archive will incur higher load on the history subsystem. This load is usually in the form of temporary disk space on the archiving node while queueing checkpoints, temporary spikes in outgoing network traffic every 5 minutes (as checkpoints are formed), minor CPU overhead while running upload processes (up to
MAX_CONCURRENT_SUBPROCESSES worth), and long-term continuous growth of (relatively cold) storage use in the archive.
The performance of each such process will vary depending on how it is configured: typically a
put command for a history archive will just be an invocation of
aws or such command-line tool, so you should ensure that your node has adequate memory and CPU to run
MAX_CONCURRENT_SUBPROCESSES worth of that tool, whatever it is.
History archiving (or lack thereof) will typically not alter the load caused on any other subsystems. Since history archiving is essential to having adequate backups of the state of the network, it is strongly recommended that nodes archive history if they can afford to.
Nodes configured to support a Horizon server are typically under significant load at the database level caused by Horizon. This means that the resulting database performance available to
stellar-core's database, ledger and transaction subsystems is greatly reduced. Configuring dedicated non-consensus nodes to support Horizon, or running Horizon on a read replica of a
stellar-core database, may be a reasonable option to alleviate this contention.
Several key steps in the
stellar-core operating loop latency-sensitive: it will fail to keep up / fall out of sync if it cannot complete various actions in a timely enough fashion. It is therefore important to the wellbeing of the server that several physical dimensions be selected for low-latency:
Some key configuration choices concerning storage access will greatly affect performance:
BUCKET_DIR_PATH config option sets the location that
stellar-core places its buckets while (re)writing the bucket list. This should be located on a relatively fast, low-latency local disk. Ideally SSD or NVMe or similar. The faster the better. It does not need to be very large and should not grow in usage very fast, though
stellar-core will fail if it fills up, so keep an eye on its utilization and make sure there’s plenty of room.
DATABASE config value controls not only which kind of database the node is performing transactions against, but also where the database is located. Unlike with many database-backed programs, the content of the database in a
stellar-core installation is somewhat ephemeral: every node has a complete copy of it, as does every history archive, and the database can always be restored / rebuilt from history archives (it is in fact being continuously backed up every 5 minutes). So the main thing to optimize for here is latency, especially on nodes doing consensus. We recommend either:
SQLite on a fast, local disk. This is probably the fastest option and is perfectly adequate for many types of node. Note: if you are running Horizon, it will need to access stellar-core’s database to ingest data. It is not compatible with SQLite.
The newest version of PostgreSQL supported (a minimum version is listed in installation instructions but we usually test with newer versions as well).
postgresql://dbname=core host=/var/run/postgresql. This may require adjustments to PostgreSQL’s
For illustration sake, the following table shows some latency numbers around ledger close-times measured on a test cluster running in AWS, with varying database configurations:
|Database connection type||median||p75||p99||max|
|SQLite on NVMe instance storage||1ms||2ms||2ms||12ms|
|Local PostgreSQL on NVMe instance storage, Unix socket||3ms||3ms||3ms||31ms|
|Local PostgreSQL on NVMe instance storage, TCP socket||3ms||4ms||4ms||50ms|
|Local PostgreSQL on SSD EBS, Unix socket||5ms||20ms||27ms||169ms|
|Local PostgreSQL on SSD EBS, TCP socket||4ms||19ms||54ms||173ms|
|Remote PostgreSQL on RDS, TCP socket||27ms||87ms||120ms||170ms|