research
24 posts tagged research.
hotspots v1.36.0: risk isn't the only axis
The headline feature here started as a research question, not a product idea: do coupling and ownership actually predict something different from complexity-times-activity, or are they just repeating the same ranking in different units? The answer shaped the entire design — four ranked lists, never blended into one.
When You Don't Have Enough History to Train
Every new repo starts with the same problem: no defect labels, no training signal, no basis for a risk ranking beyond 'how often does this file change?'
The Signal That Evaporated (And the One That Didn't)
Two signals looked nearly identical on paper. One collapsed under temporal holdout. The other survived — and the difference tells you something important about how to design defect prediction features…
Spearman ρ Looked Fine. P@10 Was Zero.
Spearman ρ can be positive and meaningful while the top-ranked files contain zero future bugs. Here's how I found that out, why it happens, and how to catch it before it reaches a user.
Tabular Beats LLM, and That's the Right Answer
On most codebases, a small XGBoost model trained on git history features outperforms a fine-tuned 7B LLM at defect risk ranking — not because the LLM is bad, but because the data is inherently tabular.
The Coupling Signal That Kept Inverting Itself
I tried three formulations of co-change coupling for defect risk ranking. Raw co-occurrence inverted on dense codebases. Mutual information made it worse.
Your Transformer Featurizer Only Helps When Your History Is Broken
Transformer embeddings on source code improved defect ranking on exactly the repos where git history had already failed — and added near-zero or negative signal everywhere else. Here's what that conditional looks like in practice.
Java Code Health: Patterns Across 10 Open-Source Repositories
I analyzed 10 popular Java repositories and found the same three antipatterns in every single one. Here's what that tells us about Java development practices at scale.
The Defect Prediction Trap
I fine-tuned a model that scored Spearman ρ=+0.678 on defect risk. Then I zeroed out the git history fields and it dropped to −0.142. The model wasn't reading code. It was relaying commit metadata back as a prediction.
The Metric That Hides Failure
Spearman ρ measures average rank agreement across all files. Precision@K measures whether the files engineers actually review are in the right place. On mature, stable codebases, a ranker can show positive ρ while returning zero bugs in its top 10.
Which LLM Vendor Wins at Defect Risk Prediction?
When evaluating open-weight models for code analysis, the vendor comparison you run in 2024 might give you the wrong answer in 2025.
When Your Labels Lie: File-Level Bug Attribution Inflates Defect Rates
Most defect prediction pipelines share a quiet assumption: if a file was touched in a bug-fix commit, every function in that file was 'buggy' — and that assumption silently inflates defect labels in ways that undermine fine-tuning before it starts.
How to Know If Fine-Tuning Will Help Before You Run It
Before you spend compute on per-repo fine-tuning, one scalar from the codebase's bug-fix history can tell you whether it's worth running. Here's how it works and what it can't predict.
Go Code Health: Patterns Across 12 Open-Source Repositories
I analyzed 12 Go repositories and found the same structural antipatterns in every single one. Here's what the data shows.
JavaScript Code Health: Patterns Across 19 Open-Source Repositories
I analyzed 19 open-source JavaScript repositories and found exit-heavy functions in 89% of them. Here's what the data reveals about structural debt in the JavaScript ecosystem.
Python Code Health: Patterns Across 12 Open-Source Repositories
I analyzed 12 open-source Python repositories and found complex branching in 100% of them. Here's what the data reveals about Python code health patterns.
Rust Code Health: Patterns Across 11 Open-Source Repositories
I analyzed 11 popular Rust repositories to find the most common code health antipatterns. Every single one had exit-heavy functions and long functions — Rust's ownership model doesn't prevent structural complexity.
TypeScript Code Health: Patterns Across 44 Open-Source Repositories
I analyzed 44 TypeScript repositories and found structural antipatterns in nearly all of them. Here's what the data shows about where complexity accumulates.
Can a Fine-Tuned LLM Learn Which Code Is Risky?
I spent a few weeks running a controlled experiment: can a small language model, fine-tuned on a specific codebase's history, learn to rank code by defect risk better than a hand-tuned heuristic?
TypeScript produces less exit-heavy code than Go or Python — and more branching
exit_heavy averages 5.3 in non-TypeScript repos and 4.5 in TypeScript. complex_branching has the highest max-out rate of any pattern in TS. The flip is structural, not stylistic — and it changes which refactoring techniques help.
Hub functions are a frontend framework problem — here's the data
hub_function barely appears in Go, Rust, or Python repos. In TypeScript UI framework code it's concentrated in coordinator functions with fan-out values of 37–112. Here's why.
Five structural patterns appear in nearly every TypeScript OSS repo I've analysed
Five structural patterns — god_function, long_function, exit_heavy, complex_branching, deeply_nested — appear in 86–100% of the 28 TypeScript OSS repos I've analysed. Here's what the data shows.
Running Hotspots on expo/expo Without Running Out of Memory
expo/expo crashed Hotspots with an OOM. Here's the full account: what broke, the systematic fixes across memory and CPU, and the new escape-hatch flags that make very large repos practical.
What Happens When You Run Hotspots on 102,000 Functions
I stress-tested Hotspots against VS Code (102k functions) and found an O(N³) bug, a sampling tail exclusion bug, and a philosophical question about approximation accuracy — all in one sitting.