Appends an element to a BigVector block writer. This function is asynchronous and returns as soon as the writer’s queue has accepted the element. Close the writer to complete the process. Returns writer.

Takes a FressianStreamBlockWriter and a Clojure value. Appends that value as Fressian to the stream. Returns writer.

Takes a handle, a block ID, and its corresponding offset. Updates the handle’s block index (in-memory) to add this mapping. Returns handle.

Takes a handle, a optional block ID, and the maximum number of elements per chunk. Returns a BigVectorBlockWriter which can have elements appended to it via append-to-big-vector-block!. Those elements, in turn, are appended to a series of newly created FressianStream blocks, each of which is stitched together into a BigVector block with the given ID. As each chunk of writes is finished, the writer automatically writes a new block index, ensuring we can recover at least part of the history from crashes.

The writer is asynchronous: it internally spawns a thread for serialization and IO. Appends to the writer are transferred to the IO thread via a queue; the IO thread then writes them to disk. Closing the writer blocks until the transfer queue is exhausted.

Loop which writes values from a BigVectorBlockWriter’s queue to disk.

How many elements should we write to a chunk of a BigVector before starting a new one?

How many bytes do we use to store a bigvector’s count?

How many bytes do we use to store a bigvector element’s index?

Compute the checksum of a block, given two bytebuffers: one for the header, and one for the data.

Computes the checksum of a block, given a ByteBuffer header, and an already-computed CRC32 checksum of the data. Useful for streaming writers which compute their own checksums while writing. Mutates data-crc in place; I can’t figure out how to safely copy it.

Where do we write a checksum in the block header?

How long is the checksum for a block?

Returns a blank ByteBuffer for a block header. All fields zero.

Fetches the checksum of a block header.

Takes a block type and a ByteBuffer of data, and constructs a block header whose type is the given type, and which has the appropriate length and checksum for the given data.

An optimized way to construct a block header, a block type, the length of a data region (not including headers) and the CRC checksum of that data. Mutates the checksum in place.

Fetches the length of a block header.

How long is a block header?

Returns the type of a block header, as a keyword.

How many bytes per block ID?

Takes a block index and returns the number of bytes required for that block to be written, NOT including headers.

Where in the file do we write the offset of the index block?

Where do we write a block length in a block header?

How long is the length prefix for a block?

How many bytes per block offset address?

Constructs a new BlockRef object pointing to the given block ID.

Takes a handle and a block ID, and returns the set of all block IDs which that block references. Right now we do this by parsing the block data; later we might want to move references into block headers. With no block ID, returns references from the root.

A map of block types to integer codes.

Where do we store the block type in a block header?

How long is the type for a block?

Verifies the checksum of a block, given two ByteBuffers: one for the header, and one for the data.

Takes a Handle and reads the magic bytes, ensuring they match.

Takes a Handle and reads the version. Ensures it’s a version we can decode, and updates the Handle’s version if it hasn’t already been set.

Closes a Handle

Takes two handles: a reader and a writer. Copies the root and any other referenced blocks from reader to writer.

The current file version.

Version 0 was the first version of the file format.

Version 1 added support for FressianStream and BigVector blocks.

A little helper function for finding BlockRefs in a nested data structure. Returns the IDs of all BlockRefs.

Where in the file the first block begins.

Flushes writes to a Handle to disk.

How many bytes should we buffer before writing Fressian data to disk?

How do we read Fressian data?

Takes a handle. Creates a new block ID, and prepares to write a new FressianStream block at the end of the file. Returns a FressianStreamBlockWriter which can be used to write elements to the FressianStream. When closed, the writer writes the block header and updates the handle’s block index to refer to the new block.

How do we write Fressian data?

Garbage-collects a file (anything that works with io/file) in-place.

How big does a file region have to be before we just mmap it instead of doing file reads?

Takes a handle, reloads its block index from disk, and returns handle.

The magic string at the start of Jepsen files.

Where the magic is written

Bytes it takes to store the magic string.

Takes a handle and returns a fresh block ID for that handle, mutating the handle so that this ID will not be allocated again.

Takes a handle and returns the offset of the next block. Right now this is just the end of the file.

Constructs a new handle for a Jepsen file of the given path (anything which works with io/file).

Called when we read anything from a handle. Ensures that we’ve checked the magic, version, and loaded the block index.

Called when we write anything to a handle. Ensures that we’ve written the header before doing anything else. Returns handle.

Takes a handle and a ByteBuffer for a big-vector block. Returns a lazy vector (specifically, a soft chunked vector) representing its data.

Takes a handle and a logical block id. Looks up the offset for the given block and reads it using read-block-by-offset (which includes verifying the checksum).

Takes a Handle and the offset of a block. Reads the block header, validates the checksum, and interprets the block data depending on the block type. Returns a map of:

{:type The block type, as a keyword :offset The offset of this block :length How many bytes are in this block, total :data The interpreted data stored in this block—depends on block type}

Takes a Handle and the offset of a block. Reads the block header and data, validates the checksum, and returns a map of:

{:header header, as bytebuffer :data data, as bytebuffer}

Fetches the ByteBuffer for a block’s data, given a block header stored at the given offset.

Fetches the ByteBuffer for a block header at the given offset.

Takes a ByteBuffer and reads a block index from it: a map of

{:root root-id :blocks {id offset, id2 offset2, …}}

Takes a handle and returns the current root block index offset from its file. Throws :type ::no-block-index if the block index is 0 or the file is too short.

Returns a ByteBuffer corresponding to a given file region. Uses mmap for large regions, or regular read calls for small ones.

Takes a handle and a ByteBuffer of data from a Fressian block. Returns its parsed contents.

Takes a handle and a ByteBuffer of data from a FressianStream block. Returns its contents as a vector.

Takes a handle and a ByteBuffer for a partial-map block. Returns a lazy map representing its data.

Takes a handle. Looks up the root block from the current block index and reads it. Returns nil if there is no root.

Reads a test from a handle’s root. Constructs a lazy test map where history and results are loaded as-needed from the file. Leave the handle open so this map can use it; it’ll be automatically closed when this map is GCed. Includes metadata so that this test can be rewritten using write-results!

Sets the checksum in a block header. Returns the block header.

Sets the length in a block header. Returns the block header.

Sets the type (a keyword) in a block header. Returns the header.

Takes a handle and a block ID. Updates the handle’s block index (in-memory) to point to this block ID as the root. Returns handle.

A map of integers to block types.

Takes a handle and a test created with write-initial-test!, and returns a BigVectorBlockWriter for writing operations to the history. Append elements using append-to-big-vector-block!, and .close the writer when done.

Returns the version of a Handle.

Where in the file the version begins

Bytes it takes to store a version.

Takes a handle, a block ID, a count, and a vector of initial-index block-id chunks. Writes a BigVector block with the given count and chunks to the end of the file. Records the freshly written block in the handle’s block index, and returns ID.

Writes a block to a handle at the given offset, given a block type as a keyword and a ByteBuffer for the block’s data. Returns handle.

Writes block data to the given block offset (e.g. the address of the header, not the data itself) in the file, backed by the given handle. Returns handle.

Writes a block header to the given offset in the file backed by the given handle. Returns handle.

Writes a block index for a Handle, based on whatever that Handle’s current block index is. Automatically generates a new block ID for this index and adds it to the handle as well. Then writes a new block index offset pointing to this block index. Returns handle.

Takes a handle and the offset of a block index block to use as the new root. Updates the file’s block pointer. Returns handle.

Takes a FileChannel, an offset, and a ByteBuffer. Writes the ByteBuffer to the FileChannel at the given offset completely. Returns number of bytes written.

Takes a handle, an optional block ID, and some Clojure data. Writes that data to a Fressian block at the end of the file, records the new block in the handle’s block index, and returns the ID of the newly written block.

Takes a handle, a byte offset, and some Clojure data. Writes that data to a Fressian block at the given offset. Returns handle.

Takes a FileChannel, an offset, a checksum, and a data structure as Fressian. Writes the data structure as Fressian to the file at the given offset. Returns the size of the data that was just written, in bytes. Mutates checksum with written bytes.

Takes a Handle and writes the initial magic bytes and version number. Initializes the handle’s version to current-version if it hasn’t already been set. Returns handle.

Writes an initial test to a handle, making the test the root. Creates an (initially nil) block for the history. Called when we first begin a test. Returns test with additional metadata, so we can write the history and results later.

Takes a handle, a Clojure map, and the ID of the block which stores the rest of the map (use nil if there is no more data to the PartialMap). Writes the map to a new PartialMap block, records it in the handle’s block index, and returns the ID of this block itself. Optionally takes an explicit ID for this block.

Takes a handle, a byte offset, a Clojure map, and the ID of the block which stores the rest of the map (use nil if there is no more to the PartialMap). Writes the map and rest pointer to a PartialMap block at the given offset. Returns handle.

Writes an entire test map to a handle, making the test the root. Useful for re-writing a completed test that’s already in memory, and migrating existing Fressian tests to the new format. Returns handle.

Takes a handle and a test created with write-initial-test!, and writes it again as the root. Used for rewriting a test after running it, but before analysis, in case there’s state that changed. Returns test.

Takes a handle and a test created with write-initial-test!, and appends its :results as a partial map block: :valid? in the top tier, and other results below. Writes test using those results and history blocks. Returns test, with ::results-id metadata pointing to the block ID of these results.