machine learning
12 posts tagged machine-learning.
When You Don't Have Enough History to Train
Every new repo starts with the same problem: no defect labels, no training signal, no basis for a risk ranking beyond 'how often does this file change?'
The Signal That Evaporated (And the One That Didn't)
Two signals looked nearly identical on paper. One collapsed under temporal holdout. The other survived — and the difference tells you something important about how to design defect prediction features…
Spearman ρ Looked Fine. P@10 Was Zero.
Spearman ρ can be positive and meaningful while the top-ranked files contain zero future bugs. Here's how I found that out, why it happens, and how to catch it before it reaches a user.
Tabular Beats LLM, and That's the Right Answer
On most codebases, a small XGBoost model trained on git history features outperforms a fine-tuned 7B LLM at defect risk ranking — not because the LLM is bad, but because the data is inherently tabular.
The Coupling Signal That Kept Inverting Itself
I tried three formulations of co-change coupling for defect risk ranking. Raw co-occurrence inverted on dense codebases. Mutual information made it worse.
Your Transformer Featurizer Only Helps When Your History Is Broken
Transformer embeddings on source code improved defect ranking on exactly the repos where git history had already failed — and added near-zero or negative signal everywhere else. Here's what that conditional looks like in practice.
The Defect Prediction Trap
I fine-tuned a model that scored Spearman ρ=+0.678 on defect risk. Then I zeroed out the git history fields and it dropped to −0.142. The model wasn't reading code. It was relaying commit metadata back as a prediction.
The Metric That Hides Failure
Spearman ρ measures average rank agreement across all files. Precision@K measures whether the files engineers actually review are in the right place. On mature, stable codebases, a ranker can show positive ρ while returning zero bugs in its top 10.
Which LLM Vendor Wins at Defect Risk Prediction?
When evaluating open-weight models for code analysis, the vendor comparison you run in 2024 might give you the wrong answer in 2025.
When Your Labels Lie: File-Level Bug Attribution Inflates Defect Rates
Most defect prediction pipelines share a quiet assumption: if a file was touched in a bug-fix commit, every function in that file was 'buggy' — and that assumption silently inflates defect labels in ways that undermine fine-tuning before it starts.
How to Know If Fine-Tuning Will Help Before You Run It
Before you spend compute on per-repo fine-tuning, one scalar from the codebase's bug-fix history can tell you whether it's worth running. Here's how it works and what it can't predict.
Can a Fine-Tuned LLM Learn Which Code Is Risky?
I spent a few weeks running a controlled experiment: can a small language model, fine-tuned on a specific codebase's history, learn to rank code by defect risk better than a hand-tuned heuristic?