Moved to jepsen.tests.adya.

Validates that a history is correct with respect to some model.

Helps analyze clock skew over time.

Supporting functions for performance analysis.

Renders an HTML timeline of a history.

Command line interface. Provides a default main method for common Jepsen functions (like the web interface), and utility functions for Jepsen tests to create their own test runners.

Applies operations to a database.

Serializes and deserializes objects to/from bytes.

Provides control over a remote node. There’s a lot of dynamically bound state in this namespace because we want to make it as simple as possible for scripts to open connections to various nodes.

A CLJ-SSH powered implementation of the Remote protocol.

Provides the base protocol for running commands on remote nodes, as well as common functions for constructing and evaluating shell commands.

The recommended way is to use SSH to setup and teardown databases. It’s however sometimes conveniet to be able to setup and teardown the databases using docker exec and docker cp instead, which is what this namespace helps you do.

The recommended way is to use SSH to setup and teardown databases. It’s however sometimes conveniet to be able to setup and teardown the databases using kubectl instead, which is what this namespace helps you do. Use at your own risk, this is an unsupported way of running Jepsen.

Network control functions.

SSH client libraries appear to be near universally-flaky. Maybe race conditions, maybe underlying network instability, maybe we’re just doing it wrong. For whatever reason, they tend to throw errors constantly. The good news is we can almost always retry their commands safely! This namespace provides a Remote which wraps an underlying Remote in a jepsen.reconnect wrapper, catching certain exception classes and ensuring they’re automatically retried.

Built-in JDK SSH libraries can be orders of magnitude slower than plain old SCP for copying even medium-sized files of a few GB. This provides a faster implementation of a Remote which shells out to SCP.

An sshj-backed control Remote. Experimental; I’m considering replacing jepsen.control’s use of clj-ssh with this instead.

Utility functions for scripting installations.

Entry point for all Jepsen tests. Coordinates the setup of servers, running tests, creating and resolving failures, and interpreting results.

Allows Jepsen to set up and tear down databases.

Libfaketime is useful for making clocks run at differing rates! This namespace provides utilities for stubbing out programs with faketime.

Some systems Jepsen tests are expensive or time-consuming to set up. They might involve lengthy compilation processes, large packages which take a long time to download, or allocate large files on initial startup.

In a Nutshell

Generators work with an immutable context that tells them what time it is, what processes are available, what process is executing which thread and vice versa, and so on. We need an efficient, high-performance data structure to track this information. This namespace provides that data structure, and functions to alter it.

This namespace interprets operations from a pure generator, handling worker threads, spawning processes for interacting with clients and nemeses, and recording a history.

This namespace contains functions for testing generators. See the jepsen.generator-test namespace in the test/ directory for a concrete example of how these functions can be used.

We burn a lot of time in hashcode and map manipulation for thread names, which are mostly integers 0…n, but sometimes non-integer names like :nemesis. It’s nice to be able to represent thread state internally as purely integers. To do this, we compute a one-time translation table which lets us map those names to integers and vice-versa.

Some tests are expensive to check–for instance, linearizability–which requires we verify only short histories. But if histories are short, we may not be able to sample often or long enough to reveal concurrency errors. This namespace supports splitting a test into independent components–for example taking a test of a single register and lifting it to a map of keys to registers.

Lazyfs allows the injection of filesystem-level faults: specifically, losing data which was written to disk but not fsynced. This namespace lets you mount a specific directory as a lazyfs filesystem, and offers a DB which mounts/unmounts it, and downloads the lazyfs log file–this can be composed into your own database. You can then call lose-unfsynced-writes! as a part of your database’s db/kill! implementation, likely after killing your DB process itself.

A nemesis which combines common operations on nodes and processes: clock skew, crashes, pauses, and partitions. So far, writing these sorts of nemeses has involved lots of special cases. I expect that the API for specifying these nemeses is going to fluctuate as we figure out how to integrate those special cases appropriately. Consider this API unstable.

EXPERIMENTAL: provides standardized support for nemeses which add and remove nodes from a cluster.

This namespace defines the protocol for nemesis membership state machines—how to find the current view from a node, how to merge node views together, how to generate, apply, and complete operations, etc.

Functions for messing with time and clocks.

Controls network manipulation.

Protocols for network manipulation. High-level functions live in jepsen.net.

Controls operating system setup and teardown.

Common tasks for CentOS boxes.

Common tasks for Debian boxes.

Common tasks for SmartOS boxes.

Common tasks for Ubuntu boxes. Tested against Ubuntu 18.04.

Stateful wrappers for automatically reconnecting network clients.

Helper functions for mucking around with tests!

Prints out stuff.

Supports tests where each node has a single, distinct role. For instance, one node might run ZooKeeper, and the remaining nodes might run Kafka.

Persistent storage for test runs and later analysis.

Jepsen tests are logically a map. To save this map to disk, we originally wrote it as a single Fressian file. This approach works reasonably well, but has a few problems:

Supports serialization of various Jepsen datatypes via Fressian.

Provide utilities for writing tests using jepsen.

Generators and checkers for tests of Adya’s proscribed behaviors for weakly-consistent systems. See http://pmg.csail.mit.edu/papers/adya-phd.pdf

Helper functions for doing bank tests, where you simulate transfers between accounts, and verify that reads always show the same balance. The test map should have these additional options:

Checks for a strict serializability anomaly in which T1 < T2, but T2 is visible without T1.

Tests based on transactional cycle detection via Elle. If you’re looking for code that used to be here, see elle.core.

Detects cycles in histories where operations are transactions over named lists lists, and operations are either appends or reads. See elle.list-append for docs.

A test which looks for cycles in write/read transactions. Writes are assumed to be unique, but this is the only constraint. See elle.rw-register for docs.

This workload is intended for systems which behave like the popular Kafka queue. This includes Kafka itself, as well as compatible systems like Redpanda.

Common generators and checkers for linearizability over a set of independent registers. Clients should understand three functions, for writing a value, reading a value, and compare-and-setting a value from v to v’. Reads receive nil, and replace it with the value actually read.

Tests for an anomaly in parallel snapshot isolation (but which is prohibited in normal snapshot isolation). In long-fork, concurrent write transactions are observed in conflicting order. For example:

Kitchen sink

Web server frontend for browsing test results.