When an Indian NBFC tells you it has "solved explainability," it almost always means one specific thing: it has wired up SHAP, can produce a feature-attribution waterfall for any rejected applicant, and can render that into a reason code. That is a real capability. It is also the easy half of the problem.
The hard half is this: the explanation you generated for a borrower in March must still be true in November — not true of the model you are running in November, but true of the model that actually produced the March decision. Explainability has quietly migrated from a model-development concern (can I understand my classifier?) to a governance-infrastructure requirement (can I prove, two years later, why this specific person was declined, under the exact model and threshold configuration in force that day?). Most lenders are not architected for the second question at all.
This post is about the gap between generating explanations and governing them.
Section 1: Two Types of Explainability — Technical Definitions
Interpretability literature draws a hard line between two objects that get casually conflated in vendor decks.
Global explainability characterises the model's learned decision function $f$ across the entire input distribution. It answers "how does this model reason in aggregate?" The standard toolkit is permutation feature importance (measuring the increase in loss when feature $j$ is shuffled), SHAP global summary plots (the distribution of per-feature Shapley contributions across a population), and partial dependence plots (the marginal effect of $x_j$ on $f$ holding other features at their expectation). The consumer of global explanations is the model validator: it feeds model documentation, the model risk management file, and regulatory model validation.
Local explainability attributes a single prediction $f(x)$ to specific feature contributions for a specific instance $x$. It answers "why this person, this decision, today?" The dominant methods are SHAP (Shapley Additive exPlanations) and LIME (which fits a sparse linear surrogate in the neighbourhood of $x$). The consumer here is the borrower, the adverse-action notice, the recourse engine, and — critically — the auditor reconstructing a past decision.
These are not interchangeable. A global feature-importance ranking tells you nothing defensible about why Priya was rejected, and a local attribution for Priya tells you nothing reliable about the model's population-level behaviour. Conflating them is the first governance error.
Shapley values, formally
SHAP grounds local attribution in cooperative game theory. Treat each feature as a "player" in a coalition game where the payoff is the model output. The Shapley value $\phi_i$ for feature $i$ is its average marginal contribution across every possible coalition:
$$ \phi_i = \sum_{S \subseteq F \setminus {i}} \frac{|S|!,\bigl(|F| - |S| - 1\bigr)!}{|F|!},\bigl[,f(S \cup {i}) - f(S),\bigr] $$
where $F$ is the full feature set, $S$ ranges over all subsets not containing $i$, $f(S)$ is the model's expected output conditioned on knowing only the features in $S$, and the combinatorial weight is the probability of forming coalition $S$ under a uniformly random feature-ordering. The marginal term $f(S \cup {i}) - f(S)$ is the lift feature $i$ adds when joining coalition $S$.
The reason SHAP became the regulatory default is that the Shapley solution is the unique attribution satisfying efficiency ($\sum_i \phi_i = f(x) - \mathbb{E}[f]$, so contributions reconcile exactly to the prediction), symmetry, dummy, and additivity. For an adverse-action regime, efficiency is the prize: the reason codes provably sum to the decision. No other popular method gives you that guarantee.
Computational considerations: exact vs TreeSHAP vs KernelSHAP
The formula above is a sum over $2^{|F|}$ coalitions — intractable beyond ~20 features. Production credit systems pick one of three approximations, and the choice has governance consequences.
- Exact SHAP is only viable for tiny feature sets or linear models where $\phi_i$ has a closed form. Rare in practice.
- TreeSHAP computes exact Shapley values for tree ensembles (XGBoost, LightGBM, random forests) in polynomial time — $O(TLD^2)$ for $T$ trees, $L$ leaves, depth $D$ — by exploiting tree structure instead of enumerating coalitions. For a credit scorecard built on gradient-boosted trees (the typical Indian NBFC stack), TreeSHAP is the correct default: exact, fast, and deterministic, which matters enormously for reproducibility.
- KernelSHAP is model-agnostic: it estimates $\phi_i$ by weighted linear regression over sampled coalitions. Use it only when the model is a black box you cannot introspect (a neural net, a vendor API). It is slower, stochastic (sample-dependent), and its variance is itself a governance liability — two runs can yield different reason codes for the same applicant unless you fix the seed and sample budget.
For a governance practitioner the takeaway is blunt: prefer TreeSHAP and a tree model not only for speed but because determinism is a compliance property. A reason code that changes when you re-run the explainer is not auditable.
Section 2: The Explanation Drift Problem
Here is the part the industry is not talking about.
Define explanation drift as the divergence between SHAP attributions for the same input vector $x$ across two model versions $v_1$ and $v_2$:
$$ D(x) = \bigl\lVert, \phi(x, v_1) - \phi(x, v_2) ,\bigr\rVert $$
where $\phi(x, v_k)$ is the attribution vector under version $k$ and $\lVert \cdot \rVert$ is your chosen norm ($L_1$ for total attribution shift, $L_\infty$ for worst-case single-feature shift). Crucially, $x$ is held fixed. The borrower has not changed; the explanation of the borrower has.
Consider a concrete case. A thin-file applicant is rejected at $v_1$, and the top local drivers are:
$$ \phi(x, v_1) = [,\text{low_income_stability}: -0.42,;; \text{high_utilisation}: -0.31,; \dots,] $$
The lender issues an adverse-action notice citing income instability and credit utilisation. Three months later the model is retrained — now ingesting Account Aggregator flows and GST data. The same borrower's vector, scored under $v_2$, attributes the decline to entirely different features:
$$ \phi(x, v_2) = [,\text{irregular_GST_filing}: -0.38,;; \text{low_savings_rate}: -0.29,; \dots,] $$
The decision is the same (rejection). The reason the borrower was told is now materially false relative to any model the lender currently runs. The original explanation has not aged; it has been orphaned. There is no longer a live model that would produce it.
This is not an edge case — it is the expected outcome of any retraining that adds features or reweights existing ones. And it creates a two-sided risk. Regulatory risk: under the RBI FREE-AI framework, explanations must remain reconstructable at the point of decision, not at the point of audit. If your only artefact is "here is what the current model says about this person," you cannot reconstruct the March rationale, because the March model is gone. Borrower-facing risk: the recourse you gave ("improve income stability") is now disconnected from any model logic, so a borrower who follows it precisely may still be declined — and you have no defensible account of why.
The deeper point: an explanation is only meaningful when paired with the model that authored it. A SHAP vector with no version binding is a number with no referent.
Section 3: Explanation Stability vs Model Accuracy Trade-Off
There is a genuine tension here, and pretending otherwise is dishonest. Every meaningful model improvement — new features, reweighted splits, a fresh training window — changes the decision function $f$, and therefore changes $\phi(x)$ for some population of borrowers. Model improvement and explanation stability are in direct opposition: the first requires reasoning to change; the second requires reasoning to stay consistent. You cannot maximise both.
The mistake is treating this as a reason to freeze models (you lose accuracy and fairness gains) or to ignore drift (you lose auditability). The resolution is to stop demanding that explanations be stable and instead demand that they be faithful — faithful to whichever model actually produced each historical decision.
This makes explanation fidelity a first-class governance KPI:
$$ \text{Fidelity} = \frac{#{\text{historical decisions whose stored explanation is faithful to its authoring model version}}}{#{\text{total historical decisions}}} $$
A lender that regenerates explanations against the current model is, by construction, driving this metric toward zero every time it retrains. A lender that locks explanations to their authoring version keeps fidelity at 1.0 by design.
That is what version-locked explanation storage buys you. At decision time, persist the immutable tuple:
$$ \bigl(, x,;; v,;; \phi(x, v),;; \theta_v ,\bigr) $$
— the input vector, the model version identifier, the SHAP attribution vector computed under $v$, and the threshold configuration $\theta_v$ in force at that moment (cut-off score, segment-level thresholds, policy overlays). This record is written once and never recomputed. When an auditor, an ombudsman, or the borrower asks "why was I declined on 14 March," you do not re-run a model — you retrieve the row. Fidelity is preserved not by stabilising the model but by freezing the evidence.
Section 4: What Governing Explanations Actually Requires
Version-locked storage is the foundation. A governance layer needs three things built on top of it.
1. Audit-trail architecture — the explanation ledger. To survive regulatory scrutiny, each entry must be self-describing and tamper-evident. Minimally it contains: version metadata (model version, training-data snapshot hash, explainer library version); a feature schema hash so you can prove the input vector's structure and prevent silent schema drift from corrupting replay; the per-feature SHAP values with the base value $\mathbb{E}[f]$ (so efficiency, $\sum_i \phi_i = f(x) - \mathbb{E}[f]$, can be re-verified independently); and confidence intervals on attributions where the explainer is stochastic (mandatory for KernelSHAP). Append-only, hash-chained storage turns the ledger from "our database says so" into "here is cryptographic evidence the record predates the dispute."
2. Monitoring — SHAP distribution surveillance. Explanation drift should be detected, not discovered in litigation. Monitor the distribution of $\phi$ across model versions and trigger when a retrained model shifts feature attribution for a key segment beyond a threshold — e.g. when the mean $|\phi_j|$ for a protected or thin-file segment changes by more than $X%$ between $v_n$ and $v_{n+1}$. This is distinct from data drift and concept drift: a model can have stable accuracy and stable inputs while completely rewiring which features drive declines for a subpopulation. That rewiring is invisible to standard ML monitoring and visible only to attribution monitoring. It is also exactly the signal a regulator cares about, because it is where reason-code consistency and fairness quietly break.
3. Recourse linkage. Local SHAP attributions are the technical bridge between "why rejected" and "what to change." The negative contributors in $\phi(x, v)$ identify the features dragging the score below threshold; counterfactual recourse (e.g. DiCE-style search) then finds the minimal, actionable perturbation of those features that flips the decision under the same model $v$. The non-obvious governance requirement: recourse must be generated against the same version the explanation is bound to. Recourse computed on $v_2$ and attached to a $v_1$ decision is not advice — it is misdirection. The ledger is what makes that binding enforceable.
Generating explanations is not governing them
This is the distinction the Indian market is missing. Open-source tooling — shap, lime, dice-ml — solves explanation generation: given a model and an input, produce $\phi(x)$. That is solved, commoditised, and roughly a weekend of integration work.
It does not touch explanation management: preserving $\phi(x, v)$ as immutable evidence, auditing it against its authoring model, validating that stored reason codes still reconcile to their decisions, and monitoring attribution distributions for drift across the full model lifecycle. Generation is a function call. Governance is an infrastructure — a versioned ledger, a schema-integrity regime, a drift-monitoring pipeline, and a recourse engine all bound by model version.
A lender with SHAP has a screwdriver. A lender with explanation governance has a system of record that can answer, three years and four retrainings later, exactly why one borrower was declined — under the model, threshold, and schema that actually decided it. Under FREE-AI, the second is the obligation. Most of the industry has shipped the first and called it done.
Frequently Asked Questions
Is SHAP sufficient for FREE-AI explainability compliance? SHAP solves explanation generation, not governance. FREE-AI requires that explanations remain reconstructable at the point of decision. That demands version-locked storage of the SHAP vector, model version, and threshold configuration as immutable evidence — a capability SHAP itself does not provide.
What is explanation drift? Explanation drift is the change in SHAP attributions for the same fixed input vector across two model versions, $\lVert \phi(x, v_1) - \phi(x, v_2) \rVert$. The borrower is unchanged, but the explanation of their decision changes — typically as an expected side effect of retraining or adding features.
Why not just regenerate explanations against the current model? Because the current model did not make the historical decision. Regenerating produces an explanation that is faithful to today's model but false relative to the one that actually declined the applicant, driving explanation fidelity toward zero and breaking auditability.
TreeSHAP or KernelSHAP for credit scoring? For tree-based scorecards (XGBoost, LightGBM), use TreeSHAP: it is exact, fast, and deterministic — and determinism is a compliance property. Reserve KernelSHAP for genuine black boxes, and record confidence intervals because its outputs are sample-dependent.
How do SHAP values connect to recourse? The negative SHAP contributors identify the features suppressing a borrower's score. Counterfactual search over those features finds the minimal actionable change that flips the decision — but it must be computed against the same model version the explanation is bound to, or the recourse is invalid.