jepsen.nemesis.combined

A nemesis which combines common operations on nodes and processes: clock skew, crashes, pauses, and partitions. So far, writing these sorts of nemeses has involved lots of special cases. I expect that the API for specifying these nemeses is going to fluctuate as we figure out how to integrate those special cases appropriately. Consider this API unstable.

This namespace introduces a new abstraction. A nemesis+generator is a map with a nemesis and a generator for that nemesis. This enables us to write an algebra for composing both simultaneously. We call checkers+generators+clients a “workload”, but I don’t have a good word for this except “nemesis”. If you can think of a good word, please let me know.

We also take advantage of the Process and Pause protocols in jepsen.db, which allow us to start, kill, pause, and resume processes.

clock-package

(clock-package opts)

A nemesis and generator package for modifying clocks. Options as for nemesis-package.

compose-packages

(compose-packages packages)

Takes a collection of nemesis+generators packages and combines them into one. Generators are combined with gen/any. Final generators proceed sequentially.

db-generators

(db-generators opts)

A map with a :generator and a :final-generator for DB-related operations. Options are from nemesis-package.

db-nemesis

(db-nemesis db)

A nemesis which can perform various DB-specific operations on nodes. Takes a database to operate on. This nemesis responds to the following f’s:

:start :kill :pause :resume

In all cases, the :value is a node spec, as interpreted by db-nodes.

db-nodes

(db-nodes test db node-spec)

Takes a test, a DB, and a node specification. Returns a collection of nodes taken from that test. node-spec may be one of:

nil - Chooses a random, non-empty subset of nodes :one - Chooses a single random node :minority - Chooses a random minority of nodes :majority - Chooses a random majority of nodes :minority-third - Up to, but not including, 1/3rd of nodes :primaries - A random nonempty subset of nodes which we think are primaries :all - All nodes “a”, … - The specified nodes

db-package

(db-package opts)

A nemesis and generator package for acting on a single DB. Options are from nemesis-package.

default-interval

The default interval, in seconds, between nemesis operations.

f-map

(f-map lift pkg)

Takes a function lift which (presumably injectively) transforms the :f values used in operations, and a nemesis package. Yields a new nemesis package which uses the lifted fs. See generator/f-map and nemesis/f-map.

f-map-perf

(f-map-perf lift perf)

Takes a perf map, and transforms the fs in it using lift.

file-corruption-nemesis

(file-corruption-nemesis db)(file-corruption-nemesis db bitflip truncate)

Wraps jepsen.nemesis/bitflip and jepsen.nemesis/truncate-file to corrupt files.

Responds to:

{:f :bitflip  :value [:node-spec ... ; target nodes as interpreted by db-nodes
                      {:file "/path/to/file/or/dir" :probability 1e-5}]} 
{:f :truncate :value [:node-spec ... ; target nodes as interpreted by db-nodes
                      {:file "/path/to/file/or/dir" :drop {:distribution :geometric :p 1e-3}}]} 

See jepsen.nemesis.combined/file-corruption-package.

file-corruption-package

(file-corruption-package {:keys [faults db file-corruption interval], :as _opts})

A nemesis and generator package that corrupts files.

Opts:

{:file-corruption
 {:targets     [...] ; A collection of node specs, e.g. [:one, ["n1", "n2"], :all]
  :corruptions [     ; A collection of file corruptions, e.g.:
   {:type :bitflip
    :file "/path/to/file"
    :probability 1e-3},
   {:type :bitflip
    :file "path/to/dir"
    :probability {:distribution :one-of :values [1e-3 1e-4 1e-5]}},
   {:type :truncate
    :file "path/to/file/or/dir"
    :drop {:distribution :geometric :p 1e-3}}]}}

:type can be :bitflip or :truncate.

If :file is a directory, a new random file is selected from that directory on each target node for each operation.

:probability or :drop can be specified as a single value or a distribution-map. Use a distribution-map to generate a new random value for each operation using jepsen.util/rand-distribution.

See jepsen.nemesis/bitflip and jepsen.nemesis/truncate-file.

Additional options as for nemesis-package.

grudge

(grudge test db part-spec)

Computes a grudge from a partition spec. Spec may be one of:

:one Isolates a single node :majority A clean majority/minority split :majorities-ring Overlapping majorities in a ring :minority-third Cleanly splits away up to, but not including, 1/3rd of nodes :primaries Isolates a nonempty subset of primaries into single-node components

nemesis-package

(nemesis-package opts)

Takes an option map, and returns a map with a :nemesis, a :generator for its operations, a :final-generator to clean up any failure modes at the end of a test, and a :perf map that can be passed to checker/perf to render nice graphs.

This nemesis is intended for throwing a broad array of simple failures at the wall, and seeing “what sticks”. Once you’ve found a fault, you can restrict the failure modes to specific types of faults, and specific targets for those faults, to try and reproduce it faster.

This nemesis is not intended for complex sequences of faults, like partitionining away a leader, flipping some switch, adjusting the clock on an unrelated node, then crashing someone else. I don’t think I can devise a good declarative langauge for that in a way which is simpler than “generators” themselves. For those types of faults, you’ll write your own generator instead, but you may be able to use this nemesis to execute some or all of those operations.

Mandatory options:

:db The database you’d like to act on

Optional options:

:interval The interval between operations, in seconds. :faults A collection of enabled faults, e.g. :partition, :kill, … :partition Controls network partitions :packet Controls network packet behavior :kill Controls process kills :pause Controls process pauses and restarts :file-corruption Controls file corruption

Possible faults:

:partition :packet :kill :pause :clock :file-corruption

Partition options:

:targets A collection of partition specs, e.g. :majorities-ring, …

Packet options:

:targets A collection of node specs, e.g. :one, :all :behaviors A collection of network packet behaviors, e.g. {:delay {}}

Kill and Pause options:

:targets A collection of node specs, e.g. :one, :all

File corruption options:

:targets A collection of node specs, e.g. :one, :all :corruptions A collection of file corruptions, e.g. {:type :bitflip, :file “/path/to/file” :probability 1e-3}

nemesis-packages

(nemesis-packages opts)

Just like nemesis-package, but returns a collection of packages, rather than the combined package, so you can manipulate it further before composition.

node-specs

(node-specs db)

Returns all possible node specification for the given DB. Helpful when you don’t know WHAT you want to test.

noop

A package which does nothing.

packet-nemesis

(packet-nemesis db)

A nemesis to disrupt packets, e.g. delay, loss, corruption, etc. Takes a db to work with db-nodes.

The network behavior is applied to all traffic to and from the target nodes.

This nemesis responds to:

{:f :start-packet :value [:node-spec   ; target nodes as interpreted by db-nodes
                          {:delay {},  ; behaviors that disrupt packets
                           :loss  {:percent :33%},...}]} 
{:f :stop-packet  :value nil}

See jepsen.net/all-packet-behaviors.

packet-package

(packet-package opts)

A nemesis and generator package that disrupts packets, e.g. delay, loss, corruption, etc.

Opts:

{:packet
 {:targets      ; A collection of node specs, e.g. [:one, :all]
  :behaviors [  ; A collection of network behaviors that disrupt packets, e.g.:
   {}                         ; no disruptions
   {:delay {}}                ; delay packets by default amount
   {:corrupt {:percent :33%}} ; corrupt 33% of packets
   ; delay packets by default values, plus duplicate 25% of packets
   {:delay {},
    :duplicate {:percent :25% :correlation :80%}}]}}

See jepsen.net/all-packet-behaviors.

Additional options as for nemesis-package.

partition-nemesis

(partition-nemesis db)(partition-nemesis db p)

Wraps a partitioner nemesis with support for partition specs. Uses db to determine primaries.

partition-package

(partition-package opts)

A nemesis and generator package for network partitions. Options as for nemesis-package.

partition-specs

(partition-specs db)

All possible partition specs for a DB.