Modeled Uncertainty and Decision-Making Under Risk

Why we teach our AI to say "I don't know"

1. Why acknowledging error is one of AI's most important properties

Machine learning models have traditionally been trained with one objective: minimize prediction error. That creates a dangerous bias, especially in financial services, where uncertainty is not noise. It is part of the structure of the problem.

Interest rate shifts, regulatory changes, economic shocks and changes in customer behavior all escape full predictability. Models that ignore these variables, or treat them as exceptions, perform well in backtests and fail when the context changes. And the context always changes.

An obsession with accuracy can be counterproductive. Models overfitted to the past create an illusion of control. In practice, that illusion leads to hidden risk exposure and to decisions that look technical but are dangerously dogmatic.

At Zayon, we start from a more realistic premise:

"Models that admit what they don't know make better decisions than models that pretend to know everything."— Deborah Ribeiro, Founder & CEO, Zayon

So we don't look for definitive predictions. We produce signals with explicit confidence ranges. Instead of hiding uncertainty, we expose it, measure it and make it part of the decision.

This doesn't mean giving up performance. It means performing with awareness of risk. It's the difference between an autopilot that flies straight into a storm and a co-pilot that says: "Poor visibility ahead. Do you want to review the route?"

In financial decisions, a single unexpected event can turn a sound portfolio into a loss.

Recognizing uncertainty is not weakness. It is prudence.

"Well-modeled uncertainty doesn't paralyze. It guides."— Deborah Ribeiro, Founder & CEO, Zayon

In practice, this makes decisions like these possible:

• "Approve a partial limit increase now and request updated financial statements before releasing the rest."

• "Escalate this case to an analyst: the company's profile is outside the patterns the model was trained on."

• "Review exposure to this sector if the default signal moves outside its expected range over the next 30 days."

Errors will happen. The question is whether you want to be surprised or prepared. Models that are uncertain and say so don't just avoid losses. They make decisions defensible.

2. What is modeled uncertainty, and why does it matter?

Modeled uncertainty is a model's ability to quantify and represent its own degree of confidence, or lack of it, in each output, taking into account the limitations of both the data and the model itself. In practical terms, it is the uncertainty we can estimate, compute or simulate using probabilistic and statistical methods.

There are two main types:

• Aleatoric uncertainty: the noise intrinsic to the data. This is variability that cannot be reduced even with more data, such as unpredictable economic shocks or measurement errors in reported figures.

• Epistemic uncertainty: the limits of the model's knowledge, usually caused by insufficient data or by patterns the model never saw during training. This type can be reduced with more data or better model architecture.

Modeling uncertainty draws on techniques such as model ensembles, Monte Carlo dropout, Bayesian methods and probability calibration. At Zayon, our engines use complementary ensembles, each capturing historical and seasonal patterns specific to the segment or product being analyzed.

We don't hide this uncertainty. We make it part of the decision process. In our architecture, no model output enters a decision without its uncertainty attached.

Why does this matter? Because in financial decisions, a single misjudged exposure can erase the margin of an entire portfolio. Deciding well is not about always being right. It is about understanding risk clearly:

• Reliability and safety: knowing when the model is uncertain prevents blind bets and allows for more cautious action or human review when needed.

• Out-of-distribution detection: the system flags when current data falls outside known patterns, signaling higher risk and the need for attention.

• Risk management: incorporating uncertainty into decisions helps prioritize resources and actions, protecting margins and capital.

• Continuous improvement: knowing where the model is unsure directs data collection and technical improvements.

• Calibration: a well-calibrated model expresses its confidence correctly, which makes it easier to integrate into decision systems and earns the trust of the people who use it.

To support this, our models use an adaptive reliability-weighting system, calibrated to the historical behavior of each segment. This dynamic calibration adjusts the confidence of each signal and helps contain the risks that come with market and economic variation.

3. How we teach a model to be honest about what it doesn't know

At Zayon, we combine machine learning and deep learning so that our engines don't just produce signals, but also recognize their own limits. That means two things:

• Updating signals as relevant new data arrives: a rate shock, a regulatory change or any event outside the expected pattern.

• Quantifying the uncertainty attached to each output, so the institution knows exactly where the model is more or less confident.

This doesn't happen by accident. For a model to be honest about what it doesn't know, it has to be forced to show its weaknesses. Here's how we do it.

A. Humility by design: the model is not an oracle.Traditional models assume clean, complete and stable data. In financial services, that is the exception, not the rule. We treat uncertainty as a structural part of modeling. The model is trained to recognize when its inputs don't support a reliable signal, and it responds with an output calibrated to the confidence it has shown in similar situations and to the uncertainty present in the current context.

B. Controlled noise and incomplete data. Financial data is rarely clean, complete and current. Financial statements arrive late. Bureau data has gaps. Exogenous variables such as policy decisions are not captured. Regimes shift abruptly. Ignoring this during training produces fragile models.

Our models are deliberately trained on missing data, noise and simulated extreme events, including:

• economic and interest-rate shocks;

• outdated or incomplete financial statements;

• regulatory and policy changes;

• behavior patterns outside the historical range.

We also use controlled data corruption and missingness-aware training to simulate real conditions. This forces the model to operate under uncertainty from the ground up, without collapsing when data fails.

C. Probabilistic inference and divergent ensembles.Our models don't answer with a single voice. They consult an ensemble of predictors trained on different assumptions, histories and architectures. This internal council works as follows:

• Agreement across models: high confidence in the signal.

• Significant divergence between models: a warning sign of structural uncertainty.

Internal divergence lets the system recognize when it is extrapolating. In those cases, the decision does not proceed automatically. Depending on the institution's policy, it is conditioned or escalated to a human.

D. Adaptive reliability weights.Not every segment behaves the same way. A retail SME is not an agribusiness cooperative member, which is not a payroll-loan customer. The model dynamically adjusts the weight it gives each information source based on:

• the historical seasonality of that segment;

• volatility observed in recent cycles and its statistical dispersion;

• the consistency of past signals in similar contexts.

This fine calibration lets the model adapt its own confidence to the profile being analyzed, reducing the risk of uninformed decisions when predictability is low.

The result: instead of a blind oracle, a system that quantifies risk, treats instability as part of the decision, and slows down when it should. Teaching a model to recognize its limits, without freezing and without faking precision, is what makes decisions defensible. It isn't magic. It is structure, method and respect for real-world complexity.

4. From uncertainty to decision

At Zayon, risk is not a side effect. It is an input. Every decision is anchored in explicit uncertainty, and our decision framework handles it in four steps.

Step 1: Quantify the signal and its uncertainty.We measure the risk behind each decision, going beyond a point estimate. Market variables, external events, historical volatility and the model's own uncertainty all go into the signal.

Step 2: Apply the institution's policy.The model doesn't decide. The institution's policy does. Policies can use uncertainty directly: approve when the signal is within risk appetite even at its upper bound; condition when the central estimate is acceptable but the range is not; escalate when evidence is missing or the case falls outside known patterns.

Step 3: Simulate scenarios.We run simulations that show how a decision behaves under different conditions, from sudden shocks to prolonged trends. This gives visibility into the "what if" and supports adjustments before exposure materializes.

Step 4: Record everything.The decision is recorded together with the signal, its uncertainty, the policy version applied and the outcome. When the decision is reviewed, by an auditor, a regulator or the institution itself, it can be reconstructed exactly as it was made.

A practical example

A small business asks to raise its credit line from R$200,000 to R$350,000.

• The model estimates a low probability of default, 2.1%, but with a wide range: from 1.4% to 6.8%.

• The range is wide for two reasons. The company's sector shows patterns that are underrepresented in the training data (epistemic uncertainty). And its latest financial statements are nine months old.

What happens? The central estimate is within the institution's risk appetite, but the upper bound is not. Under the institution's policy, the decision is CONDITION: approve an increase to R$250,000 now, and release the rest once updated financial statements confirm the signal.

The institution captures the opportunity without taking on exposure it can't justify. And the decision, including why it was conditioned, is on record.

This is the core of decision-making under risk: not avoiding risk, but handling it with evidence, simulation and policy, so the institution can act with confidence even when the future is uncertain.

5. Lessons you can apply today

Working on decisions under uncertainty has taught us some lessons that apply to any institution using AI in consequential decisions.

• Error is not incompetence. Modeled uncertainty is a competitive advantage.Errors are part of the process. What matters is modeling and quantifying uncertainty so that error becomes useful information. Understanding when and why a signal may fail is not weakness. It prevents blind decisions, reduces losses and makes fast adjustments possible.

• Uncertainty is dynamic and must be recalibrated continuously. Markets, regulation and customer behavior change quickly. Uncertainty is not fixed: it expands and contracts with context. Static techniques can't capture that. Adaptive systems that recalibrate reliability weights and reassess risk as new data arrives keep signals aligned with current reality.

• Transparency builds trust. No one wants a fake crystal ball. Showing the model's limitations, communicating risk and explaining the confidence behind each output builds real trust, with analysts, with customers and with regulators.

• Models are not decision-makers. AI delivers signals, scenarios and risks. The decision follows the institution's policy, and accountability stays with people who have context, experience and authority. A model is powerful when it works inside a decision system that knows what to do with its uncertainty.

6. The future belongs to models that can say "I don't know"

A perfect model is a myth. The real difference lies in algorithmic humility.

Models that recognize their own limits avoid blind decisions and become adaptive tools that learn from error and improve in the real world. That honesty is what makes decision systems resilient in complex environments, where incomplete data, disruptive events and volatility are the rule, not the exception.

"By embracing uncertainty and exposing it transparently, we turn hidden risks into clear signals for contingency planning, and turn decisions that look wrong into fuel for continuous improvement of the system and the business."— Deborah Ribeiro, Founder & CEO, Zayon

The future of decision intelligence lies in a model's ability to say "I don't know" with confidence, and in a decision infrastructure that knows what to do next.

Want to see how modeled uncertainty can make your institution's decisions more defensible? Talk to our team →