At commit 429e2ad, DeepSpeed — the large-scale distributed training library — has 6,952 analyzed functions, 579 of which score in the critical band. Every one of the top five hotspots lands in the fire quadrant: high structural complexity combined with recent commit activity, which makes these live regression risks rather than backlog cleanup items. I would start with get_accelerator in accelerator/real_accelerator.py, which carries an activity-weighted risk score of 20.67, a cyclomatic complexity of 112, and was last modified 15 days ago — the structural load alone is extreme, and the function is not dormant.
The table below ranks functions by activity-weighted risk — a score that multiplies structural complexity by recent commit frequency. A function that is both hard to understand (high cyclomatic complexity) and actively changing is a higher priority than one that is complex but untouched. CC = cyclomatic complexity (independent execution paths); ND = max nesting depth; FO = fan-out (distinct callees).
Top 5 Hotspots
| Function | File | Risk | CC | ND | FO |
|---|---|---|---|---|---|
get_accelerator | accelerator/real_accelerator.py | 20.7 | 112 | 4 | 20 |
_save_moe_checkpoint | deepspeed/runtime/engine.py | 19.7 | 89 | 6 | 63 |
step | deepspeed/runtime/fp16/onebit/zoadam.py | 18.9 | 115 | 9 | 35 |
offload_opt_states_inc | deepspeed/compile/passes/offload_adam_states.py | 18.7 | 83 | 6 | 33 |
load_moe_state_dict | deepspeed/runtime/engine.py | 18.6 | 48 | 6 | 29 |
Large Repo Analysis
DeepSpeed is a large repository. To stay within memory constraints, this analysis used hybrid touch mode: structural complexity — CC, ND, FO — is measured precisely for every function. Git activity is tracked at the function level (via git log -L) only for files with 5 or more commits in the last 30 days; other files use a file-level approximation. Rankings therefore surface functions that are both structurally complex and in the most actively-changing parts of the codebase. Dormant code with high structural complexity will rank lower than it would under a full per-function analysis — to surface it, run hotspots analyze . --per-function-touches on a machine with sufficient memory.
Repository snapshot
6,952 functions analyzed
Every function in this analysis falls into either the fire or watch quadrant — there is no debt quadrant at all, meaning DeepSpeed has no complex-but-dormant code at rest. The 1,870 fire-quadrant functions are all both structurally complex and recently active. The five I’m prioritizing below are the highest-scoring among the 579 critical-band functions.
High cyclomatic complexity — many independent execution paths, each a potential bug surface and required test case.God Function×5God Function
Calls an unusually large number of distinct functions (high fan-out), making it the structural centre of gravity for a subsystem.Long Function×5Long Function
Function body is too long to review in a single pass; likely contains multiple distinct responsibilities.Exit Heavy×4Exit Heavy
Multiple return or throw paths dispersed through the body — each exit needs separate test coverage.Deeply Nested×4Deeply Nested
Control structures nested 4+ levels deep, making it hard to reason about the full execution state at inner branches.Hub Function×1Hub Function
Many other functions call this one — a change here ripples widely through callers.
Every function in the top five matches the god_function and long_function patterns simultaneously. That combination — a single function doing too much, expressed over too many lines — is what drives the extreme cyclomatic complexity values you’ll see below. Four of the five are also deeply_nested and exit_heavy, which multiplies the test-coverage burden: each exit path is a branch that needs its own test case, and each nesting level makes it harder to reason about which path you’re actually on.
get_accelerator — real_accelerator.py
This function is the hardware-abstraction gateway for the entire DeepSpeed runtime. From the source excerpt I can see it works in two phases: first, it checks the DS_ACCELERATOR environment variable and validates the requested backend (XPU, CPU, NPU, SDAA, MPS, HPU, MLU, SUPA, and more) with per-backend import probes and assertion checks; second, if no override is set, it runs automatic hardware detection. Each supported accelerator gets its own conditional branch with its own import guard and error path.
That structure is what produces a cyclomatic complexity of 112 — one of the highest individual values in the repository. Every accelerator variant adds at least two paths (import succeeds / import fails), and the auto-detection phase adds more. The fan-out of 20 means the function reaches directly into 20 distinct callees; in Python, where types are resolved at runtime, that coupling is wider in practice than the number suggests, because each imported module brings its own dynamic dispatch surface.
The hub_function pattern is the most consequential label here. get_accelerator is called from across the codebase — offload_opt_states_inc calls it directly, as does the step function in the 0/1 Adam optimizer. A change to the accelerator selection logic or the validation conditions ripples into every consumer. It was touched once in the last 30 days, 15 days ago, which means it is not dormant.
Recommendation: Decompose the per-accelerator validation into a registry or dispatch table — one callable per accelerator name — rather than a single chained if/elif. Each validator becomes independently testable and the main function’s cyclomatic complexity drops to roughly the number of supported accelerators rather than their combined branch count. The exit_heavy pattern (multiple raise-on-failure paths) maps cleanly onto this: each validator owns its own failure mode.
_save_moe_checkpoint — engine.py
This is the most actively changed function in the top five: 4 touches in the last 30 days, with the most recent just 2 days ago. Its job is serializing Mixture-of-Experts checkpoint state to disk, but the source excerpt reveals it is doing far more than that. It handles at least two distinct MoE implementation paths — native DeepSpeed MoE layers and AutoEP (automatic expert parallelism) layers — with mutual-exclusion checks between them and ZeRO Stage 3. It iterates over all named modules, identifies expert layers, remaps local-to-global expert IDs, filters parameters by name patterns, clones tensors to avoid shared storage, and dispatches saves through a pluggable checkpoint engine.
A cyclomatic complexity of 89 and a maximum nesting depth of 6 are both strong signals that this function has absorbed too many responsibilities. The fan-out of 63 is the highest in the top five: 63 distinct functions called from one place, in Python, means that the blast radius of a logic error here is enormous — any of those 63 callees could behave differently than expected under a new MoE configuration, and the function itself provides no obvious seam to isolate the failure.
The external signals add context without proving defect: the file has seen 4 commits from 4 distinct authors in the last 90 days, and half of that recent activity has been tagged as bug fixes. The author spread suggests this is a shared ownership area, worth noting when prioritizing review.
Recommendation: Extract the native MoE serialization path and the AutoEP serialization path into separate private methods, each callable from a thin dispatch wrapper. This cuts the nesting depth by at least two levels immediately, reduces the main function’s cyclomatic complexity to roughly the number of top-level decisions (ZeRO compatibility checks, path selection), and gives each path its own test surface. Given 4 touches in 30 days, this refactoring has a near-term payoff.
step — zoadam.py
The step function in onebit/zoadam.py implements a single optimization step for the 0/1 Adam optimizer — a communication-compressed variant designed for large distributed training runs. The source excerpt shows it doing a significant amount of work in a single call frame: it normalizes the grads argument across three different shapes (None, a generator, a flat list, or a list of lists), then iterates over parameter groups and individual parameters, handles sparse gradient rejection, initializes a substantial amount of per-parameter state on first call (exponential moving averages, error buffers, momentum accumulators, chunk sizes for server/worker partitioning), and then proceeds into the actual update logic.
The cyclomatic complexity of 115 is the highest in the top five. The maximum nesting depth of 9 is also the highest across all five functions — ND 9 means there are code paths buried nine control-structure levels deep, which makes it effectively impossible to reason about without executing the code. The fan-out of 35 reflects calls into PyTorch tensor operations, distributed communication primitives, and accelerator-specific utilities. In Python, the dynamic dispatch on tensor types and the isinstance checks on gradient shapes add implicit branching that the CC value alone does not fully capture.
The file has no bug-linked commits, no reverts, and its single commit in the last 30 days was not tagged as a bug fix. The structural risk here is a maintainability concern, not a documented defect history.
Recommendation: The state initialization block — the section that allocates error buffers, sets chunk sizes, and calls get_accelerator().empty_cache() — is a natural extraction target. Moving it into a _initialize_state helper immediately reduces the main function’s nesting depth and cyclomatic complexity by isolating the first-call path. The gradient normalization preamble is a second extraction candidate. Together these two extractions would bring the main step function to a complexity level that is actually testable in isolation.
offload_opt_states_inc — offload_adam_states.py
This function lives in deepspeed/compile/passes/, which places it in the compilation pipeline — specifically a graph transformation pass that schedules offloading of Adam optimizer states (exponential moving averages) to reduce peak memory during training. The source excerpt shows it working at the level of an FX Graph object: it first strips previously scheduled offload/reload nodes from the graph, then profiles peak memory across nodes in forward or backward order, and then decides when to insert new offload and reload operations based on a memory budget.
A cyclomatic complexity of 83 reflects the number of decisions this pass makes: forward vs. backward traversal order, first-graph vs. subsequent-graph special-casing, per-parameter checks for exp_avg and exp_avg_sq state keys, memory threshold comparisons, and task scheduling. The nesting depth of 6 is consistent with loops-within-conditionals-within-conditionals, which is typical for graph passes that need to reason about both the graph structure and runtime memory simultaneously. The fan-out of 33 means the function reaches into graph manipulation utilities, profiling result structures, accelerator memory queries, and the optimizer state management layer.
This function was touched once in the last 30 days, 15 days ago, and the file has no historical bug-linked commits or reverts. The risk is structural: the next engineer who needs to add a new offload strategy or support a new memory model will be modifying a function with 83 independent execution paths.
Recommendation: The forward-pass and backward-pass scheduling logic are already separated by a bwd flag — that is a natural split point for two dedicated methods. The first-graph initialization block (syncing offload/reload states, building the task list) is a third extraction candidate. Separating these reduces the function to a coordinator that delegates to well-named helpers, making the overall scheduling strategy legible without tracing through 83 paths.
load_moe_state_dict — engine.py
This is the counterpart to _save_moe_checkpoint — it loads MoE expert state from disk back into the model. The source excerpt shows it handling three distinct loading paths: a legacy format (old_moe_load), the AutoEP format with metadata validation, and a module-traversal path for native MoE layers. The legacy path includes a global-to-local expert ID remapping. The AutoEP path begins with a detailed metadata validation loop that checks for required fields, duplicate layer IDs, and type correctness before attempting any I/O.
At a cyclomatic complexity of 48 it is the lowest CC in the top five, but it shares the same file as _save_moe_checkpoint, the same 4 touches in 30 days, the same 2 days since last change, the same 4-author spread, and the same rate of bug-fix-tagged commits. The deeply_nested and exit_heavy patterns appear here too: nesting depth 6 and a structure that raises RuntimeError on multiple validation failure conditions. With fan-out of 29, the function calls into checkpoint engine, group communication utilities, module introspection helpers, and state dict manipulation — a broad coupling surface for a function that is actively being modified.
The co-location with _save_moe_checkpoint in engine.py means changes to either function frequently travel together, which is consistent with the 4-author, 4-commit pattern on the file. Any engineer touching the save path should treat the load path as in-scope for the same review.
Recommendation: The AutoEP metadata validation block is self-contained and currently embedded inline — extracting it into a _validate_autoep_metadata function would immediately make the main loading logic readable and give the validation its own test surface. The three loading paths (legacy, AutoEP, native MoE) should then each become their own private method, reducing the top-level function to a dispatch switch. This mirrors the recommendation for _save_moe_checkpoint and the two refactorings could be done in the same pull request.
What the context functions tell me
A handful of watch-quadrant functions — update and get_seq_length in deepspeed/utils/static_cache.py, log_dist in deepspeed/utils/logging.py, __post_init__ in deepspeed/runtime/rollout/base.py, and is_zero_param in deepspeed/runtime/zero/utils.py — are all active in the last 30 days with low structural complexity. None of them require refactoring attention right now, but is_zero_param with 2 touches in the last 4 days is worth keeping on a watch list given how broadly the ZeRO utilities are referenced across the codebase.
Patterns Found
Antipatterns detected across the top functions in this snapshot:
| Pattern | Occurrences |
|---|---|
complex_branching | 5 |
god_function | 5 |
long_function | 5 |
exit_heavy | 4 |
deeply_nested | 4 |
hub_function | 1 |
These labels belong to two tiers — Tier 1 (structural): complex_branching, deeply_nested, exit_heavy, long_function, god_function. Tier 2 (relational/temporal): hub_function, cyclic_hub, middle_man, neighbor_risk, stale_complex, churn_magnet, shotgun_target, volatile_god.
Reproduce This Analysis
git clone https://github.com/deepspeedai/DeepSpeed
cd DeepSpeed
git checkout 429e2ad212c1b8fcd16739b126c74beabc2238cf
hotspots analyze . --mode snapshot --explain-patterns --force --hybrid-touches 5To run the same analysis on your own codebase, run hotspots analyze . --mode snapshot in any local git repo — no configuration required.
I use Hotspots to highlight structural and activity risk — not “bad code.” I treat these findings as a prioritization aid, not a bug predictor. Editorial policy →