# Healthcare *Fraud Detection*

18,764 claims, three model families, and the question of why the best public notebooks on this dataset are too good to be true.

> Project page: https://clemenshoeller.at/en/projects/fraud-detection
> Category: analytics/python

## Key facts

- **Course** · Data Management & Analytics
- **Institution** · WU Vienna, master's
- **Date** · 2026
- **My part** · Classification
- **Result** · ROC-AUC 0.88 without leakage

- Full notebook ↗: https://clemenshoeller.at/projects/healthcare-fraud-detection.html

## /01 The question

Healthcare claim fraud costs payers hundreds of billions per year worldwide. The problem is not that the cases are invisible, but that they sit among millions of entirely ordinary claims while audit teams can only review a fraction of them manually.

The course provided a dataset of 18,764 individual insurance claims. We worked through it along three questions, each mapping to one family of methods:

1. Can claims be segmented into meaningful groups with different risk profiles? (clustering)
2. Can the approved reimbursement amount be predicted from claim characteristics? (regression)
3. Can a claim be flagged as fraudulent **at submission time**, before it has been processed? (classification)

The third question was my part, and it contains the actual point of the project. Public notebooks on this dataset report near-perfect results. The question was whether that reproduces, and whether it means anything at all.

- Claims: 18,764
- Features after prep: 60
- Fraud share: 8.7%
- Clusters: 4

## /02 Method

The work was done in JupyterLab with Python, pandas and scikit-learn. Data preparation was shared; after that each person took one of the three method families.

### Preparation

- **Missing values** in three columns (600, 663 and 828 entries). A correlation analysis confirmed they are missing completely at random and unrelated to the target. Categories were filled with "Unknown", prior visits with 0.
- **Outliers** in the amount columns capped at the 99th percentile rather than dropped, so no observation is lost.
- **Feature engineering:** ten additional variables, including financial ratios, date features and binary patient and provider indicators.
- **Scaling and encoding** produced a final dataset of 18,764 observations and 60 features.

### What the other two parts found

The **clustering** (k-means over eleven features, with the fraud label deliberately excluded and used only for validation) found four groups with very different risk: three large clusters between 3.7% and 10.9% fraud, and one small, clearly separated cluster of 618 claims with a fraud rate of 98.9%. Its markers: high claim ratios, large gaps between claimed and approved amounts, and very short submission delays.

The **regression** on the approved amount reached an R² of about 0.93, but the honest reading is different: the claimed amount alone carries roughly 99% of the coefficient weight. Without it, all four models produced a negative R², performing worse than simply predicting the mean. Patient age, diagnosis, specialty and insurance type contribute virtually nothing.

### My part: the classification

Logistic regression, a decision tree and a random forest were compared. Because only about 8.7% of claims are fraudulent, accuracy would be useless here: a model that always says "not fraud" scores 91%. Evaluation therefore used the confusion matrix, precision, recall, F1 and ROC-AUC for the fraud class, with a stratified split and class_weight="balanced".

The decisive step was running every model under **two feature scenarios**:

- **Full set:** every feature, the way the public notebooks do it.
- **Pre-adjudication:** without the six features that only exist after the insurer has processed the claim, meaning approved amount, claim status and the ratios derived from them. This is the realistic setting.

## /03 Results

The near-perfect scores do reproduce. But they rest on information that does not yet exist at submission time. That is data leakage, and the gap it creates is measurable.

- Full feature set *contains post-adjudication signals*: 0.997 · F1 0.86
- Pre-adjudication only *realistic setting*: 0.88 · F1 0.39

*ROC-AUC of the random forest under both feature scenarios. The gap of roughly 0.11 appears identically across all three model families, so it is a property of the features rather than of any single model.*

The main driver of the leakage is the ratio between claimed and approved amount. It essentially already encodes the insurer's verdict, and thereby answers the question the model is supposed to ask. A five-fold stratified cross-validation confirmed that the 0.88 in the honest scenario is stable and does not depend on a lucky split (standard deviation about 0.007).

### The threshold is the actual lever

In the honest scenario the random forest initially looked useless: at the default threshold of 0.5 it caught almost no fraud. That is not a model failure but a consequence of class imbalance, since with rare positives the averaged probabilities of a forest seldom exceed 0.5. The ranking of cases was good all along; only the cut was in the wrong place.

*Recall (fraud caught)*

- Threshold 0.50 *default setting*: practically unusable
- Threshold 0.21 *tuned for best F1*: 0.51 · Precision 44% F1 0.51

*The same random forest, the same features, the same ROC-AUC. Only the decision threshold was moved. At the tuned point the model flags about 460 of roughly 3,750 test claims, among them about 200 of the 325 actual fraud cases.*

### What the honest model keys on

Three independent views (the random forest's impurity importance, the more robust permutation importance, and the logistic regression coefficients as odds ratios) all point the same way. By far the most important is the **time between service and claim submission**: it leads both importance rankings, and permuting it costs several times more ROC-AUC than any other feature. The second factor is the **claim amount**. The decision tree splits on exactly that at its root.

That makes business sense: fraudulent claims here are submitted strikingly soon after the service (median about 4 days versus 15 for legitimate ones) and tend to be inflated. All remaining demographic and coding features add almost nothing once those two are accounted for.

## /04 Interpretation

Yes, a claim can be flagged at submission time, but far less impressively than the public notebooks suggest. And the honest model is still useful, just for a different job.

The threshold is not a technical detail but a business decision: lower it to catch more fraud at the cost of auditing more clean cases; raise it to keep audits focused. The sensible use is therefore not automatic rejection but triage: the model orders the queue for human review.

> **Limitations** — The dataset is synthetically generated. The near-perfect linear relationship between claimed and approved amount is very likely an artifact of how the data was created rather than a reflection of real reimbursement processes. The results should therefore be read as a methodological demonstration, not as a finding that transfers directly into practice.

### What I took away

The technical work was the smaller part. The larger part was not believing a ROC-AUC of 0.997. A model that already has its own target question sitting in the input looks excellent and is worthless, and that happens constantly in practice because feature tables rarely document when a value actually comes into existence. Since then my first question about any model is no longer how good it is, but at what point in time each feature becomes known.

**Skills gained:** Python · scikit-learn · Classification models · Detecting data leakage · Threshold tuning · Cross-validation · Data analysis & statistics

## /05 Related projects

These projects connect more tightly than the tiles on the home page suggest. Here are all the others, sorted by how closely they relate to this one:

- [Google vs. ChatGPT](https://clemenshoeller.at/projekte/google-vs-chatgpt)
- [Prompt Injection](https://clemenshoeller.at/projekte/prompt-injection)
- [Consulting Cases](https://clemenshoeller.at/projekte/consulting-cases)
- [AURORA](https://clemenshoeller.at/projekte/aurora)
- [Zotter Marketing Plan](https://clemenshoeller.at/projekte/zotter-marketingplan)
- [AI Automations](https://clemenshoeller.at/projekte/ki-automatisierungen)
- [AI Talent Gap](https://clemenshoeller.at/projekte/ai-talent-gap)
- [LLMs & Creativity](https://clemenshoeller.at/projekte/llms-kreativitaet)
- [This Website](https://clemenshoeller.at/projekte/diese-website)
- [VWA: Origin of the Internet](https://clemenshoeller.at/projekte/vwa)

---
*German version: https://clemenshoeller.at/projekte/fraud-detection.md*
*Master's project in Data Management & Analytics (WU Vienna): fraud detection on 18,764 claims with Python and scikit-learn, including a demonstration of data leakage and threshold tuning as a business lever.*