At commit 9547025, polars has 21,718 analyzed functions, 937 of which are in the critical band — and all five of the top hotspots are in the ‘fire’ quadrant, meaning they are both structurally complex and actively changing right now. The top-ranked function, lower_ir, carries an activity-weighted risk score of 21.18 against a cyclomatic complexity of 80 and was last modified two days ago. I would start there, but the projection-pushdown optimizer’s pushdown function — with a cyclomatic complexity of 215 — is the single most structurally demanding function in this set and deserves immediate parallel attention.
The table below ranks functions by activity-weighted risk — a score that multiplies structural complexity by recent commit frequency. A function that is both hard to understand (high cyclomatic complexity) and actively changing is a higher priority than one that is complex but untouched. CC = cyclomatic complexity (independent execution paths); ND = max nesting depth; FO = fan-out (distinct callees).
Top 5 Hotspots
| Function | File | Risk | CC | ND | FO |
|---|---|---|---|---|---|
lower_ir | crates/polars-stream/src/physical_plan/lower_ir.rs | 21.2 | 80 | 9 | 65 |
f | py-polars/src/polars/io/cloud/credential_provider/_builder.py | 21.0 | 65 | 4 | 18 |
to_alp_impl | crates/polars-plan/src/plans/conversion/dsl_to_ir/mod.rs | 19.7 | 74 | 6 | 62 |
pushdown | crates/polars-plan/src/plans/optimizer/projection_pushdown/mod.rs | 19.6 | 215 | 6 | 69 |
_to_dataset_scan_impl | py-polars/src/polars/io/iceberg/_dataset.py | 19.1 | 73 | 6 | 37 |
Large Repo Analysis
polars is a large repository. To stay within memory constraints, this analysis used hybrid touch mode: structural complexity — CC, ND, FO — is measured precisely for every function. Git activity is tracked at the function level (via git log -L) only for files with 5 or more commits in the last 30 days; other files use a file-level approximation. Rankings therefore surface functions that are both structurally complex and in the most actively-changing parts of the codebase. Dormant code with high structural complexity will rank lower than it would under a full per-function analysis — to surface it, run hotspots analyze . --per-function-touches on a machine with sufficient memory.
Quadrant and Pattern Overview
21,718 functions analyzed
Every function Hotspots flagged lands in either ‘fire’ or ‘watch’ — there is no dormant structural debt in this snapshot. That means the complexity described below is not a backlog problem; it is a live shipping risk. The five critical functions share a consistent set of antipatterns:
High cyclomatic complexity — many independent execution paths, each a potential bug surface and required test case.Exit Heavy×5Exit Heavy
Multiple return or throw paths dispersed through the body — each exit needs separate test coverage.God Function×5God Function
Calls an unusually large number of distinct functions (high fan-out), making it the structural centre of gravity for a subsystem.Long Function×5Long Function
Function body is too long to review in a single pass; likely contains multiple distinct responsibilities.Deeply Nested×4Deeply Nested
Control structures nested 4+ levels deep, making it hard to reason about the full execution state at inner branches.Hub Function×1Hub Function
Many other functions call this one — a change here ripples widely through callers.
All five are flagged as god functions and long functions, all five exhibit complex branching with multiple exit paths, and four of the five reach nesting depths that make local reasoning hard. The branching and exit-heavy patterns together mean that any given change exercises only a fraction of the possible execution paths — a direct burden on test coverage.
lower_ir — lower_ir.rs
lower_ir converts logical intermediate representation (IR) nodes into physical streaming execution nodes. The excerpt shows it annotated with #[recursive::recursive], using a local lower_ir! macro to simplify recursive self-calls, and dispatching across a large match on IR node variants — SimpleProjection, Select, HStack, Slice, and more. Each branch translates one logical plan variant into a physical streaming counterpart, threading mutable arenas, caches, and context across every call.
The numbers are serious. A cyclomatic complexity of 80 means 80 independent execution paths, each a required test case and a potential regression surface. Max nesting depth of 9 pushes well past the threshold where a reader must mentally track multiple levels of control flow simultaneously — the excerpt alone shows a nested macro call inside a conditional inside a match arm, which is typical of how that depth accumulates here. Fan-out of 65 means this single function calls 65 distinct functions, so a change to any one of those callees can propagate effects back through lower_ir in non-obvious ways.
The #[allow(clippy::too_many_arguments)] suppression on a 9-argument function signals that the function’s surface area has grown beyond what the language’s own linting considers healthy. In Rust, mutable references to multiple arenas passed alongside a context struct (StreamingLowerIRContext) mean ownership and aliasing rules are doing real work here — structural complexity that CC alone understates.
This file also carries the highest reviewer-comment density of the top five, on a single total commit. That one commit attracted significant reviewer attention, consistent with a function this complex.
Recommendation: The match arms over IR variants are the natural decomposition boundary. Each variant arm (SimpleProjection, Select, HStack, etc.) could become its own lower_ir_<variant> function, reducing the central function to a dispatch table and bringing CC to a manageable level. That also isolates test cases to individual IR node types rather than requiring end-to-end plan execution to exercise a single branch.
f — _builder.py
f is a locally-defined closure inside the credential provider builder — the logic that decides which cloud credential provider to return, or whether to return one at all, based on the caller’s credential_provider argument, the detected cloud scheme (Azure, AWS, GCP), and any storage_options key-value pairs. It is effectively the entire decision engine for auto-selecting credentials, compressed into a single function.
A cyclomatic complexity of 65 in Python credential-selection logic means 65 paths through cloud provider detection, storage option parsing, and provider instantiation. The excerpt shows the function checking for None, for an already-initialized CredentialProviderBuilder, for the literal string "auto", for a DEFAULT_CREDENTIAL_PROVIDER, for the path scheme, and then branching across Azure, AWS, and GCP cloud families — with nested iteration over storage_options keys inside the Azure branch. Every early return is a potential path a test suite never reaches.
The hub_function pattern is the distinctive flag here. With fan-out of 18, this function imports and calls helpers from across the cloud I/O subsystem (_first_scan_path, _get_path_scheme, _is_aws_cloud, _is_azure_cloud, _is_gcp_cloud, plus provider constructors) — it is the hub through which all credential resolution flows. Adding a new cloud provider will require surgery here.
Every commit recorded against this file has been classified as a bug fix. With only one total commit that is a small sample, but the only recorded change to this file was corrective.
Recommendation: The cloud-family branches (_is_azure_cloud, _is_aws_cloud, _is_gcp_cloud) are the clearest extraction targets — each deserves its own _build_azure_credential, _build_aws_credential, _build_gcp_credential function. That moves the per-provider storage_options parsing logic out of the hub and into testable, isolated units. The top-level f then becomes a scheme router with no embedded business logic of its own.
to_alp_impl — mod.rs
The doc comment in the excerpt is explicit: to_alp_impl converts a DslPlan (the user-facing DSL representation) into IR nodes, adding expressions and sub-plans to their respective arenas as it recurses, and returning the root node. It is the boundary function between what users write and what the optimizer sees — every scan, union, join, projection, and Python-backed plan must pass through it.
to_alp_impl was touched twice in the last 30 days and last modified 0 days ago — it is changing today. A cyclomatic complexity of 74, nesting depth of 6, and fan-out of 62 on a function being actively edited is the definition of live regression risk. The excerpt shows a match over DslPlan variants that includes feature-gated arms (#[cfg(feature = "python")]), concurrent metadata fetching via ASYNC.block_in_place_on, and recursive self-calls through to_alp_impl(lp, ctxt). The #[recursive] attribute indicates the recursion depth can be significant enough to require stack management.
Two authors have touched this file in the last 90 days and reviewer comments have been active on it, suggesting collaboration on a function where shared mental models are already difficult to maintain at CC 74.
Recommendation: The match arms over DslPlan variants are the natural split. Each plan type (Scan, Union, PythonScan, Join, etc.) should have its own to_alp_<variant> conversion function accepting the relevant fields and the DslConversionContext. The central to_alp_impl becomes a thin dispatcher. Feature-gated variants (#[cfg(feature = "python")]) are also easier to isolate in testing when they live in their own functions rather than inside the full conversion pipeline.
pushdown — mod.rs
pushdown is the workhorse of the projection pushdown optimizer — the pass that eliminates unnecessary column reads by pushing column-selection requirements down toward the data sources. Based on the excerpt, it operates as a visitor method on IR nodes, using edge-based graph traversal to determine which projections can be safely pushed through each node type.
Cyclomatic complexity of 215 is the figure that dominates this analysis. That is not a typo. 215 independent execution paths through a single function makes exhaustive path testing impractical — the combinatorics are prohibitive. The excerpt shows why: inline function definitions (unlink_current_node), multiple macros defined inside the function body (unlink_current_node_and_return!, projected_names_subset_or_return!, pushdown_with_added_names), and a large match or conditional chain over IR node types below the excerpt. Each macro invocation hides additional branching that contributes to the true complexity.
Fan-out of 69 is the highest in the top five. In Rust, that breadth of coupling means lifetime and borrow constraints from many different types must all be satisfied simultaneously — any refactoring must respect ownership rules that the current monolithic structure may be implicitly relying on.
Two authors, two touches in 30 days, last modified two days ago. This is not structural debt waiting for a future sprint — it is a 215-CC function being actively changed right now.
Recommendation: The inline macro definitions and nested function declarations are the first thing to extract — moving unlink_current_node and pushdown_with_added_names out of the function body and into module-level private functions immediately reduces local cognitive load and makes them independently testable. After that, the per-IR-node projection logic should be extracted into a ProjectionPushdown trait impl per node type, following the visitor pattern the code already hints at.
_to_dataset_scan_impl — _dataset.py
_to_dataset_scan_impl handles the translation of an Iceberg dataset scan request into either a native Iceberg scan or a PyIceberg-backed scan — or None if the scan cannot proceed. Based on the excerpt, it manages snapshot resolution, schema ID lookup, PyArrow predicate conversion, PyIceberg filter construction, and verbose diagnostic logging across all of those steps.
A cyclomatic complexity of 73 in a function that touches snapshot metadata, schema versioning, and predicate pushdown against an external catalog reflects how many things can go differently: the snapshot may not exist, the schema ID may be missing, the PyArrow predicate may not convert, the use_metadata_statistics and use_pyiceberg_filter flags may be in any combination. The nesting depth of 6 is visible in the excerpt — conditionals nest inside snapshot existence checks, which nest inside predicate conversion logic, which nest inside verbosity guards.
Fan-out of 37 means this function calls into PyIceberg internals, PyArrow schema conversion, Polars internal utilities, and logging helpers. PyIceberg in particular is an external dependency — any API change there requires changes here.
This file matches the pattern seen in the credential provider file: its single recorded commit was itself corrective. Two separate I/O-layer files showing the same pattern is worth noting as historical context, even with a small sample.
Recommendation: Snapshot resolution (ID → schema ID → iceberg_schema) and predicate conversion (pyarrow_predicate → iceberg_table_filter) are the clearest extraction candidates — each is a self-contained concern with its own failure modes. Pulling them into _resolve_snapshot_schema and _build_iceberg_filter functions reduces the main function’s branching and makes the PyIceberg-specific logic easier to stub in tests.
Context: Watch-Quadrant Signals
Several context-only functions are worth a brief mention. parse_into_list_of_expressions in py-polars/src/polars/_utils/parse/expr.py, rechunk in series.py, and len in group_by.py are all ‘watch’-quadrant — active in recent commits but low structural complexity, so none are refactoring priorities. Their activity means they share the same recent commit windows as the critical functions above. If a change to a critical function adjusts expression semantics, the expression-parsing and aggregation helpers are worth a cross-check.
Patterns Found
Antipatterns detected across the top functions in this snapshot:
| Pattern | Occurrences |
|---|---|
complex_branching | 5 |
exit_heavy | 5 |
god_function | 5 |
long_function | 5 |
deeply_nested | 4 |
hub_function | 1 |
These labels belong to two tiers — Tier 1 (structural): complex_branching, deeply_nested, exit_heavy, long_function, god_function. Tier 2 (relational/temporal): hub_function, cyclic_hub, middle_man, neighbor_risk, stale_complex, churn_magnet, shotgun_target, volatile_god.
See more analyses with these patterns: complex_branching, deeply_nested, exit_heavy, god_function, long_function.
Reproduce This Analysis
git clone https://github.com/pola-rs/polars
cd polars
git checkout 954702563262a7974014e59e79174ba018061914
hotspots analyze . --mode snapshot --explain-patterns --force --hybrid-touches 5To run the same analysis on your own codebase, run hotspots analyze . --mode snapshot in any local git repo — no configuration required.
I use Hotspots to highlight structural and activity risk — not “bad code.” I treat these findings as a prioritization aid, not a bug predictor. Editorial policy →