🛠 the Autonomi builders dev room

What is actually happening

An mmap'd database does random point reads through the page cache. The Linux block layer applies read-ahead to sequential-ish patterns: when it thinks you're scanning, it reads ahead of you. Read-ahead is sized per block device by /sys/block/<device>/queue/read_ahead_kb (KiB). When that number is huge — some storage drivers and desktop-tuned distributions set it to tens of megabytes — every random 4 KiB point read drags a multi-megabyte read behind it. Your random-read workload becomes a sequential-scan workload at the device level. On spinning rust or a bus-limited USB device, that read amplification swamps you. The counter-intuitive part: read-ahead feeds the page cache, which is why a bigger cache can make it worse — the kernel fills RAM with bytes you never asked for, evicting pages you actually reuse.

The diagnostic, step by step

• Find the block device behind your data directory. df /path/to/data gives the mount; lsblk maps it to the device. The queue tunable lives on the whole-disk node.
• Read the current read-ahead: cat /sys/block/<device>/queue/read_ahead_kb
• Form the ratio. Divide read_ahead_kb by your workload's access granularity — the machine's page size (getconf PAGE_SIZE: 4 KiB on most ARM64/x86_64, 16 KiB on Apple Silicon and some ARM64 kernels). Working threshold measured and pinned in our node preflight tooling on 2026-08-23: above 1024 KiB is a Fail for a random-read database workload; the boundary sits exactly at 1024 KiB (1024 passes, 1025 fails). A read_ahead_kb of 65532 KiB against 4 KiB pages is a ~16,000:1 read-amplification ratio on every miss.
• Corroborate with iostat -x 1 while the workload runs: average request size (rareq-sz) will sit in the read-ahead range, not your access range.

The fix — and why a restart is required

Cap read-ahead persistently with a udev rule, so it survives replug and reboot:

# /etc/udev/rules.d/90-read-ahead-cap.rules
ACTION=="add|change", KERNEL=="<device>", ATTR{queue/read_ahead_kb}="1024"

Then reload and trigger:

sudo udevadm control --reload && sudo udevadm trigger

You must restart the database process afterwards. The read-ahead window in effect for a file is captured when the file is opened — a process holding the data files open keeps its oversized read-ahead behaviour until it closes and reopens them. Verifying the sysfs value shows the new cap while your still-running process quietly keeps the old behaviour; that gap has fooled people into "the fix didn't work." Note: the udev ATTR path takes KiB and is the stable form — writing the sysfs file directly takes KiB on older kernels, bytes on some newer ones.

Credits

The read-ahead finding was made by storage_guy in the Beehive Nature Relay community diagnostics thread (2026-08); this article is the write-up of that finding, with thresholds from the bmesh-hwfit preflight probe (landed 2026-08-23).

This article is a diagnostic procedure, not a cause announcement. Follow it in order and the failing wall will identify itself. Whether an ARM64 kernel's virtual-address width (VA_BITS) is the wall on any given machine is established only by step 5, never assumed.

Step 0 — confirm it's the mmap failing

strace -f -e trace=mmap,mmap2 <your program> 2>&1 | tail

A failing call looks like:

mmap(NULL, 137438953472, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cannot allocate memory)

Write down three numbers: the requested length (here 137,438,953,472 bytes = 128 GiB); the protection and flags (PROT_NONE + MAP_NORESERVE = a pure reservation, no RAM backing needed yet); whether it is a fixed-address request (MAP_FIXED with non-NULL first argument — fixed mappings can fail where a free-range request of the same size would succeed). A PROT_NONE|MAP_NORESERVE reservation of 128 GiB consumes essentially no RAM — it only needs address space to exist. That is why free RAM proves nothing.

Step 1 — check resource limits (the most common and cheapest wall)

ulimit -v   # RLIMIT_AS: total address space per process, KiB
ulimit -a

If unlimited, skip ahead. If it is a number smaller than the requested mapping, this is your wall — and the fix belongs in the service definition (LimitAS= in the systemd unit, or the launching shell's ulimit), not in the kernel.

Step 2 — page size (ARM64 is not uniform here)

getconf PAGE_SIZE

ARM64 kernels boot with either 4 KiB (CONFIG_ARM64_PAGE_SHIFT=12) or 16 KiB (PAGE_SHIFT=14) pages. Page size changes what a given reservation costs in page-table entries and, on 16 KiB-page systems, effectively halves the reach of the same VA width. Record it; you need it in step 5.

Step 3 — read the error correctly

ENOMEM from mmap means one of: the process is out of address space, RLIMIT_AS was hit, overcommit refused the reservation, or (with MAP_FIXED) the requested range was unavailable. The kernel does not tell you which. That is what the remaining steps are for.

Step 4 — overcommit

cat /proc/sys/vm/overcommit_memory

0 heuristic (default), 1 always allow, 2 refuse reservations beyond (swap + RAM × overcommit_ratio). Mode 2 on a machine with little swap refuses large MAP_NORESERVE reservations even though free RAM exists — a frequent mimic of this symptom.

Step 5 — measure the actual address-space ceiling (VA_BITS)

zcat /proc/config.gz | grep CONFIG_ARM64_VA_BITS   # always matches the RUNNING kernel
cat /boot/config-$(uname -r) | grep CONFIG_ARM64_VA_BITS   # fallback; must match the running release

CONFIG_ARM64_VA_BITS=39 → user address space is 512 GiB · =48 → 256 TiB · =52 → 4 PiB. If your failing reservation (step 0) exceeds the ceiling implied by the measured value, you have your wall — no hypothesis needed, it is arithmetic. If it fits comfortably inside the ceiling, VA_BITS is not your problem; go back to steps 1 and 4. Fencing notes on measurement: /boot/config-* files can lag the running kernel; /proc/config.gz is authoritative when present; when absent many distributions ship kernel config in a separate package. If neither source exists, the honest verdict is unknown, not pass — refuse to guess. What we are NOT claiming: that a 39-bit VA kernel is the cause of any specific "out of memory" report. It is one measurable ceiling among several. Every cause claim must come from your own step-5 arithmetic against your own step-0 mmap line.

If step 5 arithmetic does name VA_BITS as the wall

The width is fixed at kernel build and boot time; it cannot be raised at runtime. The remedy is to run a kernel built with 48-bit (or 52-bit) VA — for example a mainstream distribution arm64 kernel, most of which build 48-bit — or to reduce the mapping's size in software.

Credits

The ARM64 probe procedure and its output shape were contributed by TT3, and the tooling that automates the VA_BITS/page-size/read-ahead checks with typed Pass/Fail/Unknown verdicts is the bmesh-hwfit preflight probe (landed 2026-08-23). Community diagnostics thread, Beehive Nature Relay, 2026-08.

Why one byte cascades

Content addressing names data by the cryptographic hash of its bytes. A large file is split into chunks; each chunk's address is the hash of that chunk's exact bytes; the file is reassembled from a list of chunk addresses. Dedup happens when — and only when — a chunk's bytes are identical to a chunk the store already holds.

• Fixed-size chunking: every chunk boundary after the change point shifts by the edit offset. A one-byte insertion at the front means no chunk after chunk 1 matches the old chunks. Nearly the whole file re-stores.
• Content-defined chunking (CDC, rolling-hash boundaries at content-dependent offsets): boundaries after the edit re-anchor at the next stable rolling-hash window, so the cascade stops some bounded distance after the change — typically a few chunk sizes, not the whole file. This is mitigation, not elimination: the affected window still re-stores, because its bytes genuinely differ.

The store is behaving correctly. Two byte strings hash equal only if they are equal. Any tool that "duplicates" your data without matching bytes is matching on something weaker than content — and none of those are dedup guarantees. The same law cuts the other way, and it's a feature: identical bytes stored by a million different users, under a million different names, are stored once.

The design law: untagged stream + metadata sidecar

The failure mode to design against is tagging the stream. If you embed identity metadata — owner, version, sequence numbers, timestamps, permissions — inside the bytes that get chunked and hashed, then: every metadata change re-hashes the content it labels, even when the content didn't change; two users' copies of the same content hash differently because their metadata differs, so dedup silently dies; and encryption that mixes metadata into the content layer makes the divergence permanent — the ciphertexts differ, so nothing downstream can ever match. The law: the stream you content-address must be the untagged bytes, and everything else lives in a metadata sidecar that is stored/referenced alongside, never interleaved. Concretely:

• Chunk and hash the raw content stream — nothing prepended, nothing interleaved.
• Carry names, ownership, versions, permissions, and content-type in a separate small record that references the content's chunk list. Updating the sidecar touches only the sidecar.
• Encrypt content before addressing if the scheme requires it, but keep metadata out of the content ciphertext's inputs except where the scheme's security genuinely demands it, and know exactly which of those inputs break byte-identity.
• If your format must be self-describing, put the description after the addressed payload, or in a header excluded from chunk hashing by construction — never mid-stream.

Operational checklist

• Confirm your store chunks with a content-defined algorithm if edits-then-reupload is your normal workload; fixed-size chunking is fine for write-once data.
• Audit any pre/append/interleave your application does to bytes before they reach the chunker. Each one is a dedup break.
• When measuring dedup ratios, change content and metadata separately. A metadata-only change that re-stores content means the layering is wrong.
• Expect the cascade, size it (a few chunk lengths for CDC), and schedule uploads accordingly.

Credits

The untagged-stream + metadata-sidecar design law and the exact-bytes cascade observations come from the Beehive Nature Relay community diagnostics of 2026-08 (AntTP findings by traktion; W@tch findings ↗ by aautonomicc); this article is the write-up of their findings.