We are announcing that the first generation of Praeth — our continuous risk engine for application-layer fraud, abuse, and automated attacks — is serving production traffic across the Neuraphic ecosystem. Every authenticated request into our identity, communication, and inference surfaces is now scored in real time before any state-changing action is permitted. This post describes what we built, what we observed during the evaluation period, and what we learned from running the system against live adversarial pressure.
Consistent with the constraints we set out when the program began, this post discusses the system at the level of its design intent and its measured behavior, not at the level of its implementation. We will not describe the specific signals, the specific models, or the specific infrastructure that Praeth uses, because doing so would meaningfully reduce the cost of evading it. The point of publishing what we have learned is not to write a recipe an attacker can follow; it is to advance the state of the public conversation about what is achievable in this space.
What Praeth does
Praeth is an inline scoring service. For every relevant request entering one of our products, an upstream component constructs a feature description of the request — drawing on contextual evidence, behavioral evidence, and the request's relationship to recent and historical activity — and presents that description to Praeth. Praeth returns a calibrated score in the unit interval and a structured set of reasons. The calling product is responsible for interpreting the score against its own policy: above one threshold the request proceeds; between two thresholds the user is invited to complete an additional verification; above the upper threshold the request is refused. The thresholds are vertical-specific, because the cost of a false positive at signup is not the same as the cost of a false positive at password reset.
The component that constructs the feature description and the component that returns the score are deliberately decoupled. The first is owned by the product team and reflects the product's understanding of which observations are meaningful in its context. The second is owned by our applied research group and reflects the current best model for converting those observations into a single number. This separation lets us iterate on the model without coordinating with every product, and it lets product teams add or refine signals without having to retrain anything.
Evaluation methodology
Before turning enforcement on, we ran Praeth in shadow mode for an extended period across all participating surfaces. Shadow mode means the system was scoring every request and storing the result, but its decisions were not enforced — the existing static defenses continued to govern allow or deny. This let us measure two things at once: how often Praeth's decisions disagreed with the existing defenses, and what the consequences would have been had its decisions been the ones in force.
The disagreements broke down into four categories. The largest was the cases in which Praeth would have allowed a request that the existing rules refused; the great majority of these were false positives in the existing rules — legitimate users who tripped a brittle threshold. The second-largest was the cases in which Praeth would have refused a request that the existing rules allowed; on inspection, these were almost entirely automated abuse that the existing rules had no signal to detect. The remaining two categories — Praeth raising a verification prompt where the existing rules took a binary action — were the most informative, because they showed us where the model was uncertain and where additional signal would help.
We evaluated calibration using the standard reliability-diagram methodology and assessed it against held-out traffic that was never used in training. For decisions classified as high confidence, the empirical agreement with ground truth (subsequently labeled by the existing fraud-investigation workflow) was within two percentage points of the score for every bucket above the seventieth percentile. We consider this sufficient to make policy decisions that bind on behalf of users; we will continue to monitor it.
What we observed in production
We share four findings from the deployment that we believe are useful to the broader community without compromising the system.
Aggregate metrics are misleading; per-vertical metrics are not. When measured globally, Praeth's false-positive rate looked excellent and its true-positive rate looked excellent, but the global numbers averaged out a population in which some product surfaces saw essentially zero adversarial traffic and others were under sustained attack. Reporting per-vertical metrics — separately for signup, sign-in, password recovery, outbound communication, and each automation surface — produced numbers that were each significantly worse than the global average and each significantly more actionable. We now treat any global aggregate as a smell.
Coordination signals dominate individual-actor signals. The single most useful class of evidence for distinguishing benign from malicious behavior turned out to be relational: who else is doing the same thing at the same time, who else is acting on the same target, what fraction of the recent population sharing this attribute has subsequently been confirmed adversarial. Individual-actor signals (the characteristics of a single request taken in isolation) carried a meaningful but secondary share of the discriminative power. This was not a surprise to anyone who has worked on anti-abuse before, but the magnitude of the gap was larger than we expected.
Calibrated scores are necessary; binary decisions are insufficient. We had originally assumed that the right output of a risk-scoring system was a verdict — block or allow — and that the calibration was an internal detail. The shadow-mode evaluation convinced us otherwise. Many of the most consequential decisions are in the middle, where neither blocking nor allowing is correct, and the right action is to escalate the request to a verification step that the legitimate user can complete and the adversary cannot. The middle band is where calibrated probabilities pay for themselves, and it is also where the user experience is most fragile. We treat the interaction in this band as a first-class design surface, not as an afterthought.
Adversarial pressure is uneven, asymmetric, and learns. We observed clear evidence that the populations attacking each surface adapted to the deployed defense over the course of the evaluation. The shape of the adaptation differed by surface: some attackers immediately abandoned techniques that no longer worked, others continued at undiminished volume against patterns we had already neutralized, and still others probed slowly to identify the boundaries of the new defense. Any system in this category that does not have a continuous adversarial-evaluation loop running against it will be obsolete on a timescale measured in months, not years.
What we will not publish
We are not publishing the signals Praeth ingests, the model architecture it uses, the exact thresholds in our decision policy, or the per-vertical metrics in absolute terms. Each of these would be useful to a serious adversary and we do not see a path on which the marginal academic value of disclosure exceeds that cost. We will share the methodology of the work, the lessons that generalize, and the operational principles that we believe other practitioners can apply to their own systems without needing to know the details of ours.
What is next
The first generation of Praeth is operational, but the work is not finished. The next phase focuses on the relational layer that turned out to dominate the simpler signals, on closing the loop between the system and the human investigators who curate its training data, and on extending the same approach to surfaces that are not yet covered. External access is being extended on a measured basis; platforms handling sensitive transactions, high-friction flows, or adversarial traffic can request access through the console.
We will continue to report on the work in this newsroom.