AI Fraud Detection: When Rules Are Enough and When You Need Machine Learning

Fraud teams are often pitched machine learning as the starting point. For most businesses a rules engine is the better first step. Rules are transparent, can be changed in minutes when a new attack appears, and need no historical labels. A velocity limit on new accounts, a block on mismatched billing and shipping countries for high-value orders, or a hold on first-time payouts to a freshly added destination catch a large share of unsophisticated fraud and give you a baseline to beat.

The cost of rules shows up later. As the rule set grows, rules overlap and contradict, nobody remembers why one was added, and every new rule raises the false positive rate a little. That is the signal to consider models, not the first week.

A model combines many weak signals that no individual rule would act on: device and network characteristics, behaviour compared with the customer's own history, the relationship between this account and others, timing patterns. It produces a score rather than a yes or no, so the business can set different actions at different thresholds: allow, step up verification, send to review, decline.

Gradient-boosted trees are a common and sensible default for tabular transaction data, because they handle mixed feature types, need modest tuning and can be explained with feature importance. Graph-based features, such as how many accounts share a device or payout destination, often add more lift than a fancier model on the same inputs.

Models learn from labels, and fraud labels are slow and biased. A chargeback can arrive weeks after the transaction. Transactions you declined never get a label at all, so a model trained only on approved traffic never learns about the fraud you already stop. Define the label window explicitly, keep a small randomly allowed sample where policy permits so you can measure what the current system misses, and track the label delay so evaluation does not treat recent, still-unlabelled traffic as legitimate.

Confirmed outcomes from a manual review team are the cleanest labels you will have. Design the review tool so analysts record a reason code, not just a verdict. Those codes later become features, rules and training data.

Card and wallet decisions happen in a checkout flow, so scoring has to fit a tight latency budget, commonly well under a few hundred milliseconds for the whole pipeline. That pushes the design toward precomputed features in a low-latency store, a lightweight model served close to the transaction service, and a clear fallback when scoring times out. Decide in advance whether a timeout fails open or fails closed, and make that choice per product and per amount.

Keep the decision path and the learning path separate. Real-time scoring uses a frozen model version; offline jobs retrain, evaluate against a held-out period, and promote a new version only when it beats the current one. Log the features and the score for every decision so any outcome can be explained and replayed later.

Accuracy is a poor metric when fraud is a tiny share of traffic. Track the fraud caught at a fixed review capacity, the false positive rate among good customers, the approval rate, and the cost of each outcome: the loss on missed fraud against the revenue lost to wrongly declined orders. The best threshold is the one that minimizes that combined cost, and it will differ by market and product.

Watch for drift. Attackers adapt, seasonal traffic changes the baseline, and a new payment method shifts the feature distributions. Monitor input distributions and score distributions, and alert when they move, rather than waiting for chargebacks to reveal that the model went stale.

A practical rollout: instrument and log everything, ship a small rules set with clear owners, add a review queue and a reason-code taxonomy, then train a first model in shadow mode that scores traffic without acting on it. Compare it with the rules, and only then let it influence decisions, starting with the review queue before automatic declines. Each stage produces data the next one needs.