Class Imbalance: When 99% Accuracy Means Nothing

· 2 min read · Syed Omar Faruk Towaha
Class Imbalance: When 99% Accuracy Means Nothing

Here's my favourite machine learning magic trick. I'll build a fraud detection model with 99% accuracy in one line:

def predict(transaction):
    return "legitimate"

Applause, please. If 99% of transactions are legitimate, this model is right 99% of the time. It also catches exactly zero fraud, which is the only thing it was built to do.

Imbalanced data
When one class is rare, accuracy mostly measures the common class.

Why accuracy lies

Accuracy counts all correct predictions equally. When one class dominates, a model can score highly just by always predicting the majority. This happens everywhere: fraud, disease screening, equipment failure, spam on a quiet day, rare defects in manufacturing.

Better questions, better metrics

Instead of "how often are we right?", ask:

from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred, digits=3))
Lazy vs useful model
Lower accuracy, but the useful model actually catches fraud.

There's always a trade-off. Flag more transactions and recall goes up, but precision falls and your customers get annoying calls about their perfectly normal coffee purchase. Which trade-off is right depends on the cost of each mistake: a missed cancer diagnosis and a false spam alert are not equally bad.

Other good tools:

Ways to handle imbalance

Adjust the decision threshold. Many models output a probability. You don't have to flag at 0.5; maybe 0.2 is right for your costs. This is often the cheapest and most effective fix.

proba = model.predict_proba(X_test)[:, 1]
y_pred = (proba >= 0.2).astype(int)

Use class weights, so mistakes on the rare class cost more during training:

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(class_weight="balanced", max_iter=1000)

Resample carefully. Undersample the majority or oversample the minority (techniques like SMOTE create synthetic examples). Only ever resample the training data; your test set must look like the real world.

Get more examples of the rare class, if you can. Sometimes the best model improvement is a conversation with the team that labels the data.

The takeaway

When someone shows you a model with 99% accuracy, your first question should be "what percentage is the majority class?" If the answer is 99%, you're looking at the one-line magic trick, just with more GPUs.

// related

// prefer the terminal?

Open the terminal blog and type read class-imbalance-99-percent-accuracy-means-nothing.