containerd's EROFS snapshotter carries the highest activity risk — 3 functions first

Three of containerd's five highest activity-weighted risk functions sit in plugins/snapshots/erofs/erofs.go, alongside hotspots in the v2 shim and the CRI sandbox runner.

Stephen Collins ·
Generated by hotspots · free & open source
pip
$ pip install hotspots-cli
Activity Risk2.38Low
Hottest FunctionCreate

Run this on your own codebase

See if your own repo has a Create-style hotspot — run this in any local git repo:

pip
$ pip install hotspots-cli
npm
$ npm install -g @stephencollinstech/hotspots
Run in any repo
$ hotspots analyze .
★ Star on GitHub

Key Points

What makes the fan-out of 87 in RunPodSandbox a problem in containerd?

Fan-out is the count of distinct functions a function calls directly, and a value of 15 or more is already a strong coupling signal. RunPodSandbox in internal/cri/server/sandbox_run.go calls 87 distinct functions, so it is coupled to a very wide slice of the codebase. That makes it hard to test in isolation, because a unit test has to account for many collaborators, and it means a contract change in almost any of those callees can ripple into this function. Combined with a cyclomatic complexity of 85, a reader also has to track many execution paths at once.

How do I reduce fan-out and cyclomatic complexity in go?

The main technique is extract-method: pull each coherent phase of a long function into a named function that takes explicit inputs and returns explicit results. A cyclomatic complexity above 15 warrants splitting, and above 30 warrants immediate attention. A concrete first step for NewSnapshotter in erofs.go, which has a cyclomatic complexity of 45, is to move all configuration validation into a single validation function and test it with a table of invalid option combinations. That alone should remove well over half the branches from the constructor.

Is containerd actively maintained?

The data says yes for the code I examined. All five top hotspots are in the active tier, with 22 touches in the last year for each of the three erofs.go functions (last changed 20 days before the analyzed commit), 16 touches for Create in shim.go (last changed 6 days before), and 9 touches for RunPodSandbox (last changed 5 days before). Of 4,745 functions, 1,911 are in active code. Active development and high structural debt are not mutually exclusive, and this repo shows both: 944 functions fall in the critical band.

How do I reproduce this analysis?

Use the hotspots CLI from GitHub and check out the exact commit I analyzed, 79e2374. Then run `hotspots analyze . --mode snapshot --force` from the repository root. The same command works on any local git repository without configuration.

What does activity-weighted risk mean?

Activity-weighted risk combines structural complexity (cyclomatic complexity, nesting depth, and fan-out) with how much and how recently the code has changed. Functions that are both hard to understand and in changing code score highest. A function with a cyclomatic complexity of 80 that has not been touched in two years scores much lower than one with a complexity of 20 touched every week, because the dormant one has lower near-term regression risk. This helps teams spend refactoring effort where it reduces the chance of introducing bugs now, not just where code looks complicated in the abstract.

I would start with plugins/snapshots/erofs/erofs.go, because it holds 3 of the top 5 hotspots (createSnapshot, mounts, and NewSnapshotter), each with an activity-weighted risk score between 2.33 and 2.37, 22 touches in the last year, and only 20 days since last change. That makes this a live regression risk concentrated in one module, not a backlog item spread across the codebase. containerd is the container runtime underneath much of the Kubernetes ecosystem, and my analysis of commit 79e2374 covered 4,745 functions, of which 944 landed in the critical band. The other two top hotspots are Create in core/runtime/v2/shim.go and RunPodSandbox in internal/cri/server/sandbox_run.go, which has the most extreme shape in the set: a cyclomatic complexity of 85 and a fan-out of 87.

The table below ranks functions by activity-weighted risk — a score that multiplies structural complexity by recent commit frequency. A function that is both hard to understand (high cyclomatic complexity) and actively changing is a higher priority than one that is complex but untouched. CC = cyclomatic complexity (independent execution paths); ND = max nesting depth; FO = fan-out (distinct callees).

Top 5 Hotspots

FunctionFileRiskCCNDFO
Createcore/runtime/v2/shim.go2.420321
createSnapshotplugins/snapshots/erofs/erofs.go2.415530
mountsplugins/snapshots/erofs/erofs.go2.330414
NewSnapshotterplugins/snapshots/erofs/erofs.go2.345315
RunPodSandboxinternal/cri/server/sandbox_run.go2.385487
Activity Tier Distribution
Active1911Quiet2834

4,745 functions analyzed

The tier split above shapes how I read the list. Of 4,745 functions, 1,911 sit in active code and 2,834 are quiet. All five of my top hotspots are in the active tier, so none of what follows is dormant structural debt. These functions are complex and are being edited right now. Every one of the five scores between 2.32 and 2.38, so the ranking among them is tight. I would treat them as a set rather than fight over order.

A note on interpretation: a high score means complex code in a changing area. It does not mean the code is buggy or that the maintainers are doing something wrong. The historical signals on these files are mostly mild, and I call that out per function below.

The erofs.go cluster: three functions, one file

The three erofs.go functions share a file, so they share its history: 26 total commits, 22 touches in the last year per function, a bug-fix rate of 7.7%, no reverts, no bug-linked commits, and 5 distinct authors in the last 90 days. That is a low defect signal, so I read this cluster as maintainability debt under active development rather than a trouble spot. The 5 distinct authors in 90 days is the number I would watch. Five people editing three complex, tightly related functions in one file is where merge friction and conflicting assumptions tend to show up.

Create — shim.go

Create
core/runtime/v2/shim.go
2.38
critical
CC 20
ND 3
FO 21
touches/yr 16

This is the highest-scoring function on the list at 2.38, and it is the one with the most revert history: the file has 1 revert among 35 total commits, a bug-fix rate of 25.7%, and 3 distinct authors in the last 90 days. It was last changed 6 days ago. I would not call that alarming, but among my five it is the file with the strongest hint of past correction, so it gets the strongest review priority from the history side.

The function creates a task through the shim: it builds a create request from the options, copies rootfs mounts, calls the task service, and then, when restoring from a checkpoint, unpacks a rootfs-diff archive and starts the task. The restore branch is the part I would look at. The excerpt shows file opening, decompression, archive application, and a start call all inside the create path, and an author comment in the source admits some uncertainty about why restore goes through create at all. I will not speculate beyond that, but a comment like that on an actively changing function is a flag that the restore flow is partly a workaround.

There is also something in the error path worth a second look in review: one error message formats the archive name from a file handle in the branch where opening that file just failed. That is a small thing I noticed in the excerpt, and I would verify it rather than assert it.

A fan-out of 21 with a cyclomatic complexity of 20 is moderate on both counts. My recommendation: move the checkpoint restore logic (rootfs-diff unpacking) into its own method, so that the normal create path stays short and the restore path can be tested on its own.

createSnapshot — erofs.go

createSnapshot
plugins/snapshots/erofs/erofs.go
2.37
critical
CC 15
ND 5
FO 30
touches/yr 22

This is the EROFS snapshotter’s entry point for creating a snapshot, and the excerpt shows it doing a lot in one body: a layer content cache lookup, a dm-verity metadata check, a deferred cleanup block that removes temp and snapshot directories on error, a metadata transaction, and ownership handling for user-namespace-remapped containers in non-block mode.

The cyclomatic complexity of 15 is moderate. The shape that stands out is nesting depth 5 combined with a fan-out of 30. Nesting depth 4 and above is where I start to find code hard to reason about, and a fan-out of 30 means the function is coupled to roughly thirty distinct callees. The deferred cleanup closure and the transaction closure both capture and mutate named variables (the snapshot, the temp directory, the path, the error), which is the kind of structure where an edit in one branch can quietly change what cleanup does in another.

My recommendation: extract the cache-hit handling (lookup, dm-verity check, label attachment) into its own helper, and the user-namespace ownership block into another. That removes two of the deeper nesting levels and should cut fan-out meaningfully without touching the transaction logic.

mounts — erofs.go

mounts
plugins/snapshots/erofs/erofs.go
2.34
critical
CC 30
ND 4
FO 14
touches/yr 22

This function builds the mount list a snapshot hands back. From the excerpt it branches on whether the snapshot has parents, whether a recorded layer blob resolves, whether fsverity applies, whether the snapshot is a view, and whether block mode is on. Block mode alone returns a two-entry mount list with ext4 formatting options, while the non-block path returns a plain bind mount.

Cyclomatic Complexity 30
threshold: 10

A cyclomatic complexity of 30 is the high end of what I consider workable: 30 independent paths means 30 test cases to cover each branch once. The error handling is explicit in the Go style, with several early returns, and each early return is a path someone has to test. The nesting depth of 4 and fan-out of 14 are modest by comparison, so complexity is the dominant problem here, not coupling.

My recommendation: split by mount strategy. The no-parent case (blob-backed, block-mode, or bind) and the parent-chain case are different jobs, and a function per case would drop each one well under 15 paths. The block-mode mount construction is also a self-contained literal that can move into a helper on its own.

NewSnapshotter — erofs.go

NewSnapshotter
plugins/snapshots/erofs/erofs.go
2.33
critical
CC 45
ND 3
FO 15
touches/yr 22

The constructor has the highest cyclomatic complexity of the three at 45, with a shallow nesting depth of 3. That combination fits what the excerpt shows: a long, mostly flat sequence of configuration validation. It validates the dm-verity mode, checks dm-verity support, creates the root directory, checks root compatibility, rejects incompatible combinations of layer content caches with fsverity and immutability, requires absolute cache paths, checks fsverity support, and restricts the immutable flag to Linux. Each guard adds a branch.

Flat and long is a different risk from deeply nested. Each check is individually easy to read, so I would not call this hard to understand line by line. The cost is that 45 paths is a lot to cover with tests, and every new snapshotter option adds another guard to the same function. Given the file’s recent activity, new options are likely.

My recommendation: pull the configuration validation into a dedicated validation function that takes the config and returns an error, then test it with a table of invalid combinations. That separates ‘is this configuration legal’ from ‘build the snapshotter’, and the constructor body shrinks to resource setup.

RunPodSandbox — sandbox_run.go

RunPodSandbox
internal/cri/server/sandbox_run.go
2.32
critical
CC 85
ND 4
FO 87
touches/yr 9

This is the extreme shape in the set. A cyclomatic complexity of 85 puts it well past the 30+ threshold I treat as high, and a fan-out of 87 means it is directly coupled to 87 distinct functions.

Fan-out 87
threshold: 15

The excerpt shows why. In one function it starts a tracing span, generates an ID, reserves the sandbox name, creates a lease, resolves the OCI runtime, generates runtime options, and registers multiple deferred cleanups that each depend on named return values and on a separately tracked cleanup error. The comment in the source explains the cleanup design: if cleanup fails, the sandbox is left in a not-ready state for the kubelet’s next sync to handle. That is a thoughtful design, but it means correctness depends on several deferred closures interacting in the right order.

The activity profile is the lowest of my five at 9 touches in the last year, but it was last changed 5 days ago, and the file has a 22.7% bug-fix rate across 22 commits with 2 distinct authors in the last 90 days. No reverts and no bug-linked commits. Fewer touches and fewer authors lower the day-to-day collision risk, but the blast radius of any single edit is the largest on the list, because a change in one of 87 callees’ contracts can surface here.

My recommendation: this is the function I would break up first even though it is not first by score. Extract the setup phases into named steps (name and lease reservation, runtime resolution, runtime options, and so on), and keep the deferred cleanup logic in one place that the steps report into. Even splitting it into four or five phase functions would bring each below the point where I would worry about it.

How I would order the work

  1. RunPodSandbox for size: it has the largest structure and will pay back the most from decomposition.
  2. The erofs.go cluster as one coordinated effort, because three functions in one file edited by 5 authors in 90 days should be refactored together, not in three conflicting pull requests.
  3. Create in shim.go for the restore path extraction, given the revert and the 6-day-old change.

For framing, the quiet tier looks different: structurally complex functions with few recent touches carry lower activity-weighted risk, and I would treat them as structural debt rather than a current regression risk.

Reproduce This Analysis

git clone https://github.com/containerd/containerd
cd containerd
git checkout 79e23749d4840f5f12d7b4fc54147479f2254f96
hotspots analyze . --mode snapshot --force

To run the same analysis on your own codebase, run hotspots analyze . --mode snapshot in any local git repo — no configuration required.

I use Hotspots to highlight structural and activity risk — not “bad code.” I treat these findings as a prioritization aid, not a bug predictor. Editorial policy →

Was this useful? Tell me more →

Related Analyses