Cryptography's ASN.1 and serialization code — 5 hotspots to fix first

An analysis of pyca/cryptography finds five functions in the Rust backend, declarative ASN.1 module, and test utilities that combine high structural complexity with recent commit activity, led by pkey_private_bytes at an activity-weighted risk score of 18.11.

Stephen Collins ·
Generated by hotspots · free & open source
pip
$ pip install hotspots-cli
Activity Risk18.11Low
Hottest Functionpkey_private_bytes

Antipatterns Detected

complex_branching5exit_heavy5long_function4deeply_nested3god_function3hub_function1

Run this on your own codebase

See if your own repo has a pkey_private_bytes-style hotspot — run this in any local git repo:

pip
$ pip install hotspots-cli
npm
$ npm install -g @stephencollinstech/hotspots
Run in any repo
$ hotspots analyze .
★ Star on GitHub

Key Points

What is a god function and why does it matter in cryptography?

A god function is a function that has grown to handle too many responsibilities at once, usually visible as a combination of high cyclomatic complexity and high fan-out — calling many other functions directly. In this codebase, `write` in the declarative ASN.1 encoder calls 13 distinct functions with a cyclomatic complexity of 76, and `_normalize_field_type` calls 23 distinct functions with a complexity of 40. Three functions in my top five carry this pattern, which matters because a single change to add or fix one ASN.1 type's handling risks touching logic that every other type shares in the same function body.

How do I reduce cyclomatic complexity in python and rust ASN.1 code?

The direct technique is extract-method: pull each branch of a large if-chain or match statement into its own named function, turning one function with complexity 40 or 76 into several functions each under 10. As a rule of thumb, complexity above 15 warrants splitting during the next touch, and complexity above 30 — like `write` at 76 or `load_pkcs1_vectors` at 73 — warrants attention now rather than at the next scheduled refactor. A concrete first step: in `_normalize_field_type`, replace the sequential `if hasattr(...)`/`elif _is_union(...)` chain with a small dispatch table keyed on field-type category, which would cut its branch count by roughly half without changing any return values.

Is pyca/cryptography actively maintained?

Yes — every function in my top five sits in the fire quadrant, meaning high structural complexity paired with recent commit activity, and all five last changed 19 days ago with one touch each in the last 30 days. None of the 2,087 functions I scanned fell into the debt quadrant of complex-but-dormant code. That combination — recent development concentrated in exactly the areas carrying the most structural complexity, like the declarative ASN.1 encoder and private-key serialization path — means engineering attention is present, but it also means regression risk is live rather than parked in a backlog.

How do I reproduce this analysis?

I ran the open-source hotspots CLI against commit `67946b9` of pyca/cryptography. After `git checkout 67946b9`, the exact command is `hotspots analyze . --mode snapshot --explain-patterns --force`, and the same command runs unmodified on any local git repository without extra configuration.

What does activity-weighted risk mean?

Activity-weighted risk combines structural complexity — cyclomatic complexity, nesting depth, and fan-out together — with recent commit activity, so functions that are both hard to understand and recently changed score highest. A function with cyclomatic complexity 80 that hasn't been touched in two years would score much lower than `pkey_private_bytes`, which combines a complexity of 47 with a recent touch and produces a score of 18.11. This prioritization is meant to focus review effort on code where a bug is likely to be introduced in the next few commits, not just code that looks complicated in isolation.

Out of 2,087 functions I scanned in pyca/cryptography, 430 land in the ‘fire’ quadrant — structurally complex code that is also seeing recent commits — and none land in pure structural debt. The top of that list is pkey_private_bytes in src/rust/src/backend/utils.rs, with a cyclomatic complexity of 47, a nesting depth of 7, and an activity-weighted risk score of 18.11 — meaning it is not just complicated, it was last changed 19 days ago with one touch in the last 30 days. I’d start my review there this week, then work down through the ASN.1 encoding path, since four of the five functions in my top five sit in serialization or encoding logic that all last changed 19 days ago.

The table below ranks functions by activity-weighted risk — a score that multiplies structural complexity by recent commit frequency. A function that is both hard to understand (high cyclomatic complexity) and actively changing is a higher priority than one that is complex but untouched. CC = cyclomatic complexity (independent execution paths); ND = max nesting depth; FO = fan-out (distinct callees).

Top 5 Hotspots

FunctionFileRiskCCNDFO
pkey_private_bytessrc/rust/src/backend/utils.rs18.14773
writesrc/rust/src/declarative_asn1/encode.rs17.076513
_normalize_field_typesrc/cryptography/hazmat/asn1/asn1.py16.840423
load_pkcs1_vectorstests/utils.py16.273414
identify_key_typesrc/rust/src/x509/sign.rs15.61182

Large Repo Analysis

cryptography is a large repository. To stay within memory constraints, this analysis used hybrid touch mode: structural complexity — CC, ND, FO — is measured precisely for every function. Git activity is tracked at the function level (via git log -L) only for files with 5 or more commits in the last 30 days; other files use a file-level approximation. Rankings therefore surface functions that are both structurally complex and in the most actively-changing parts of the codebase. Dormant code with high structural complexity will rank lower than it would under a full per-function analysis — to surface it, run hotspots analyze . --per-function-touches on a machine with sufficient memory.

<BandChart fire=‘430’ debt=‘0’ watch=‘1657’ ok=‘0’ caption=‘Risk quadrant distribution across 2,087 functions’ />

Zero functions fell into the debt quadrant, and 430 fell into fire. That’s a meaningful signal on its own: nothing in this top set is complex-and-dormant. Every high-complexity function in my top five was last touched within the same recent window, which means the risk concentrated there is in things people are actively shipping, not backlog cleanup. The other 1,657 functions sit in the watch quadrant — active but not structurally complex enough to warrant refactoring priority.

<PatternCloud patterns=‘complex_branching:5,exit_heavy:5,long_function:4,deeply_nested:3,god_function:3,hub_function:1’ />

Across the top five, the same shapes repeat: five functions with complex branching, five with heavy exit points, four long enough to warrant extraction, three deeply nested, and three that read as god functions doing too much in one place. That repetition across a 138-function critical set tells me this isn’t one bad file — it’s a pattern in how the Rust/Python boundary and the ASN.1 layer have grown.

pkey_private_bytes — src/rust/src/backend/utils.rs

<FunctionCard fn=‘pkey_private_bytes’ file=‘src/rust/src/backend/utils.rs’ risk=‘18.11’ band=‘critical’ cc=‘47’ nd=‘7’ fo=‘3’ touches=‘1’ />

This is the top-ranked hotspot in the repository, and the source confirms why. It’s a private-key serialization function branching across encoding types (Raw, PKCS8, TraditionalOpenSSL), format checks, password validation, and encryption-algorithm instance checks — a cyclomatic complexity of 47 with nesting reaching 7 levels deep and a fan-out of 3. The #[allow(clippy::too_many_arguments)] annotation on the signature itself is a tell: eight parameters feeding a single function that has to validate combinations of encoding, format, and encryption together. It carries one touch in the last 30 days and last changed 19 days ago, and with one author in that period and zero bug-linked commits on the file, this reads as fresh, deliberate work rather than a function accumulating fixes over time. My recommendation: split the Raw-key path, the PKCS8 path, and the TraditionalOpenSSL path into separate helper functions — each currently nested inside one long if-chain, and each independently testable once extracted.

write — src/rust/src/declarative_asn1/encode.rs

<FunctionCard fn=‘write’ file=‘src/rust/src/declarative_asn1/encode.rs’ risk=‘16.97’ band=‘critical’ cc=‘76’ nd=‘5’ fo=‘13’ touches=‘1’ />

This is the highest cyclomatic complexity in my top five at 76, and the source shows why: it’s a match statement over ASN.1 type variants — Sequence, SequenceOf, Set, SetOf, and more — each branch building its own writer closure and iterating over fields. A fan-out of 13 means this one function directly calls into thirteen distinct functions, and the ‘god_function’ and ‘long_function’ patterns both flag it. That combination of high fan-out and a large match block means a bug in one ASN.1 type’s encoding path is unlikely to be isolated — the function’s size makes it hard to verify each branch gets equivalent test coverage. It last changed 19 days ago with one touch in that window and one author, so this reads as active feature work on the declarative ASN.1 encoder, not decay. I’d extract each match arm (Type::Sequence, Type::SequenceOf, Type::Set, Type::SetOf, and whatever remains below the excerpt) into named helper functions — that alone would cut the top-level complexity by more than half.

_normalize_field_type — src/cryptography/hazmat/asn1/asn1.py

<FunctionCard fn=’_normalize_field_type’ file=‘src/cryptography/hazmat/asn1/asn1.py’ risk=‘16.84’ band=‘critical’ cc=‘40’ nd=‘4’ fo=‘23’ touches=‘1’ />

This is the fan-out outlier of the group: 23 distinct function calls from a single function, the highest of any in my top five. In a duck-typed language, that number understates the real coupling — the function is resolving Python type-hint machinery (typing.get_origin, typing.Annotated, typing.get_args) at runtime against a growing set of ASN.1 field types (TLV, unions, SIZE-constrained types), and each new supported type shape adds another branch here. Cyclomatic complexity of 40 with nesting depth 4 reflects that this function is the central point where every declarative ASN.1 field definition gets validated and normalized before use — which makes it the natural hub for the whole declarative ASN.1 layer’s correctness. One touch in the last 30 days and one author in that period puts this squarely in the same recent development wave as write above; it also last changed 19 days ago, the same as the others. Given the fan-out, I’d introduce a small dispatch table keyed on field-type category (Annotated, Union/Optional, TLV, root-type) instead of the current sequential if/elif chain — it turns 40 branches into N smaller, independently testable functions.

load_pkcs1_vectors — tests/utils.py

<FunctionCard fn=‘load_pkcs1_vectors’ file=‘tests/utils.py’ risk=‘16.23’ band=‘critical’ cc=‘73’ nd=‘4’ fo=‘14’ touches=‘1’ />

This is a test-fixture parser, not production crypto code, but its complexity is nearly as high as the production ASN.1 encoder above — cyclomatic complexity 73 driven by a long chain of line.startswith(...) checks for PKCS#1 vector-file markers (# Message, # Salt, # Seed, # Signature, # Encryption, # Example), each toggling parser state. The ‘exit_heavy’ and ‘god_function’ patterns both apply: it accumulates state across many elif branches and continues/returns scattered through the loop, which makes it hard to know all the ways a malformed vector file could be silently mis-parsed. Because this feeds cryptographic test vectors into the RSA test suite, a subtle parsing bug here could mask a real regression in the RSA implementation rather than catch one. It shares the same one-touch-in-30-days, 19-day-old change pattern as the others, suggesting it was touched in the same recent wave of edits. I’d break the state-machine parsing into a small explicit parser class with named states, replacing the attr string-flag pattern with an enum — that removes most of the branching without changing behavior.

identify_key_type — src/rust/src/x509/sign.rs

<FunctionCard fn=‘identify_key_type’ file=‘src/rust/src/x509/sign.rs’ risk=‘15.62’ band=‘critical’ cc=‘11’ nd=‘8’ fo=‘2’ touches=‘1’ />

This one is the odd shape in the group: cyclomatic complexity is only 11 — moderate by itself — but nesting depth hits 8, the deepest of any function in my top five, despite a fan-out of just 2. The source shows why: it’s a straight if/else if chain checking private_key.is_instance(...) against seven key types (RSA, DSA, EC, Ed25519, Ed448, ML-DSA-44/65/87), and the nesting comes from that structural mismatch between low branching and how the checks are laid out. This function decides which key type a signing operation is dealing with, so it sits upstream of every certificate-signing code path in the file — low fan-out means it isn’t calling much, but its result gates behavior elsewhere. One touch in the last 30 days, 19 days since last change, one author in that period — consistent with the ML-DSA post-quantum key types being recently added to this dispatch chain. I’d flatten this into a match on a type-check helper or a lookup table over (PyType, KeyType) pairs, since if/else if chains that only ever return a variant are a textbook match-statement candidate in Rust.

Patterns Found

Antipatterns detected across the top functions in this snapshot:

PatternOccurrences
complex_branching5
exit_heavy5
long_function4
deeply_nested3
god_function3
hub_function1

These labels belong to two tiers — Tier 1 (structural): complex_branching, deeply_nested, exit_heavy, long_function, god_function. Tier 2 (relational/temporal): hub_function, cyclic_hub, middle_man, neighbor_risk, stale_complex, churn_magnet, shotgun_target, volatile_god.

See more analyses with these patterns: complex_branching, deeply_nested, exit_heavy, god_function, hub_function, long_function.

Reproduce This Analysis

git clone https://github.com/pyca/cryptography
cd cryptography
git checkout 67946b90c9b8f093caf54125bf484e5e0ac4422b
hotspots analyze . --mode snapshot --explain-patterns --force --hybrid-touches 5

To run the same analysis on your own codebase, run hotspots analyze . --mode snapshot in any local git repo — no configuration required.

I use Hotspots to highlight structural and activity risk — not “bad code.” I treat these findings as a prioritization aid, not a bug predictor. Editorial policy →

Was this useful? Let me know →

Related Analyses