What's New in 4.0

CogDB 4.0 replaces the storage layer with a new binary format called Spindle, rewrites the read path to use memory-mapped I/O, and adds an in-memory cache for graph traversals. The same graph takes roughly half the disk space, and repeated traversals are about 7× faster.

Install

pip install --upgrade cogdb
# or pin it:
pip install cogdb==4.0.0

Something broken? Open an issue or find us on Discord.

What changed

Spindle replaces marshal

CogDB 3.x serialized values using Python's marshal. Marshal is fast, but it has two properties that make it a poor fit for a database format. First, CPython makes no stability guarantee across versions, so a file written under one Python may fail to open under the next. Second, loading marshal data from a source you don't trust is unsafe. For an embedded database where files may outlive the Python version that created them, both of these are problems.

Spindle is a binary format written for CogDB. Every store file opens with a 23-byte header: the magic bytes COGDB\0, a format version byte, the file's creation time as an int64 nanosecond timestamp, and 8 reserved bytes. This means corruption, version mismatches, and format drift can be detected at open time rather than mid-read.

Typed values

Spindle tags each value with a type code and writes a fixed layout. There are six supported types:

Two of these carry the API today: graph triples are strings, and put_embedding writes a numeric list. The remaining types are already specified and implemented. We have plans for richer numeric support on the roadmap, including math functions.

Where the space savings come from

Spindle stores the same graph in roughly half the space. Three changes account for most of it:

• Index slots went from 32 bytes to 8. They used to hold ASCII text; now they hold a binary int64. The hash index file is 4× smaller for the same capacity.
• Edge keys carry a 1-byte direction prefix (\x00 for out, \x01 for in) in place of the 3.x string suffixes __:out:__ and __:in:__. That saves 8–9 bytes on every edge record.
• Framing costs 2 bytes per field. A 32-byte key with a 32-byte value packs into a 68-byte payload.

Faster reads and traversals

The read path has three changes that work together:

• Memory-mapped reads. A random record lookup is a copy out of the page cache rather than a seek-and-read syscall. Only writes go through the regular file handle. This removes a layer of system call overhead on every read.
• Decoded record cache. An LRU cache holds deserialized records, so a cache hit skips both I/O and parsing.
• In-memory adjacency cache. Enabled by default via Graph(..., use_memory_view=True). It loads edges by demand paging, falls back to a single-key disk read on a miss, and applies writes inline so it never drifts out of sync with what is on disk.

Performance

We ran the benchmark suite (test/benchmark.py) against 3.8.2 and 4.0.0 on the same machine. Writes and single-shot queries are within run-to-run noise between the two versions. The gains show up in the two places the new format and cache are built to help: on-disk size and repeated traversals.

On-disk size

The same 37,825-edge social graph, written by each version:

About 1.9× more compact, before any compression.

Repeat traversals

The same graph (~3,000 nodes), repeated single-hop traversals:

About 7× faster. Unlike 3.x, throughput holds steady as the graph grows rather than falling off once reads start spilling to disk. Two-hop traversals with a warm cache see a similar improvement.

Migrating from 3.x

CogDB ships a migrator that converts a database in place and rewrites the graph edge keys so traversals keep working:

import cog.migrate

cog.migrate.migrate("/path/to/your/database")

Your original files are kept next to the database as .v3_backup, so you can roll back if anything looks wrong.

The release candidates went through a round of crash-robustness and migration fixes before 4.0.0 went final. Migrate with the final release rather than an rc.

Full release notes

The format and storage work landed in PR #89. The complete changelog is on the v4.0.0 release page.