When I started building the mapping engine at the core of Pensvyne, the first version used keyword matching. It was fast to build, easy to explain, and produced results that looked reasonable in a demo. The problem appeared when we ran it against real policy libraries and real regulatory bulletins: the false negative rate — obligations that should have matched but didn't — was unacceptably high. And in regulatory compliance, false negatives are exam findings.
This post is about why keyword matching systematically fails in the regulatory domain, what concept-level mapping does differently, and the practical engineering decisions behind building a mapping system that compliance teams can actually rely on.
The Structural Problem with Keyword Matching in Financial Regulation
Keyword matching assumes that the same obligation will use the same vocabulary in both the regulatory text and the internal policy. In most domains, this is a reasonable approximation. In financial regulation, it fails for three structural reasons.
Regulatory vocabulary is not policy vocabulary. Regulators write in their own terminology. Your internal policies were written by legal teams, outside counsel, and compliance officers who translated regulatory requirements into institutional language. "Concentration risk limits" in a Fed guidance document may appear in your policy as "large exposure thresholds" or "sector allocation caps." A keyword search for "concentration risk" will miss both.
Obligations evolve faster than policy language. A capital adequacy policy written after the Basel II framework will use Basel II terminology. When Basel III updates arrive — referencing Tier 1 capital ratios, leverage ratio buffers, LCR requirements — the keyword overlap with the existing policy may be minimal, even though the policy is directly implicated. The obligation is the same. The vocabulary shifted.
Policy documents have internal terminology inconsistency. A single policy document might use "AML program," "anti-money laundering procedures," "BSA compliance framework," and "suspicious activity monitoring" to refer to parts of the same regulatory obligation. A keyword search for any one of these terms will surface only part of the relevant policy sections.
What a Keyword Search Actually Catches — and What It Misses
To make this concrete: we ran a test during Pensvyne's development using a set of OCC bulletins from 2022-2023 and a synthetic policy library modeled on a community bank with 85 policy documents. The keyword matching approach (exact match plus stemming) correctly identified 58% of the regulatory-to-policy mappings that a human compliance expert confirmed as relevant. The concept-level approach caught 91% of the same mappings.
The 42% that keyword matching missed were not obscure edge cases. They included the entire class of synonym-divergence matches (same obligation, different vocabulary), cross-reference misses (a policy that references another policy which contains the relevant language), and version-drift misses (policies updated after a prior regulatory cycle that no longer use the current regulatory terminology).
We're not claiming that concept mapping is perfect — the 9% we missed in that test represents real work, and we've spent subsequent development time on it. But the difference between 58% and 91% coverage is the difference between a compliance program that passes examination and one that generates findings.
How Concept Mapping Works in Practice
Concept mapping operates on semantic similarity rather than string matching. The approach involves representing regulatory text and policy text as structured concept graphs — clusters of related terms, obligations, and processes — rather than as flat bags of words. When a new regulatory bulletin arrives, the system extracts its concept structure: which regulatory domains it addresses, which obligations it creates or modifies, and which institutional processes it affects.
The policy library is similarly indexed as a concept graph. The mapping asks: which policy sections address the same underlying obligations as the bulletin's concept inventory, regardless of the specific vocabulary used?
In practical terms, this means the system understands that "minimum capital requirement," "Tier 1 capital ratio buffer," "capital adequacy," and "capital floor" are all expressions of the same underlying obligation. It understands that "customer identification program," "CIP procedures," and "know-your-customer documentation" are different institutional expressions of the same BSA/AML obligation. When a regulatory update touches any node in that concept cluster, the relevant policy sections are surfaced — not just the ones that happen to share vocabulary with the update.
The False Positive Problem — and Why It Matters Too
It's worth acknowledging the other failure mode: concept mapping systems can over-match, surfacing policies that are loosely related to a regulatory concept but not actually affected by the specific update. High false positive rates have a different but equally serious compliance impact — they generate noise that desensitizes compliance teams to alerts, and they create unnecessary work.
We're not saying false positives don't matter; they absolutely do. The calibration work in concept mapping is getting precision and recall both high enough to be operationally useful. Too many false positives and compliance officers stop trusting the system. Too many false negatives and they miss real obligations. The target — which is harder to reach than it sounds — is a system where the output of the mapping process is consistently close to what an experienced compliance analyst would have produced manually.
In our development work, the practical threshold we found useful was: if the system's suggested mapping requires more than thirty seconds for a compliance analyst to confirm or reject, the signal-to-noise ratio is probably off. Matches should be immediately legible — "yes, this OCC bulletin on climate risk management touches our credit risk appetite policy, section 4.2" — not "maybe this bulletin on liquidity risk is vaguely related to our IT operations continuity plan."
Implementation Considerations for Different Policy Library Architectures
Concept mapping works better on some policy library architectures than others. Flat, lengthy policy documents — a single 120-page compliance manual covering everything — are harder to map at a granular section level than modular policy libraries where each policy addresses a specific regulatory domain in 10-25 pages. This isn't a dealbreaker, but it's a meaningful difference in mapping quality.
During Pensvyne's onboarding process, we work with institutions to understand their policy architecture before we index the library. For institutions with highly consolidated policy structures, we often recommend a segmentation step — not rewriting the policies, but adding structured metadata (section tags, regulatory domain labels, responsibility assignments) that improves the concept graph fidelity.
The concept mapping approach also handles policy cross-references better than keyword matching. If your Vendor Management Policy contains the phrase "as set forth in the BSA/AML Compliance Program," a keyword search for BSA/AML terms in the Vendor Management Policy will fail. Concept mapping that follows cross-references within the policy library will correctly identify that the Vendor Management Policy section is implicated by FinCEN guidance on third-party relationships.
Where We Are Now and What's Still Hard
The problem we've made the most progress on is the synonym-divergence class. Regulatory terminology shifts every three to five years as frameworks evolve, and maintaining a concept ontology that tracks these shifts is an ongoing editorial problem, not a one-time engineering problem. Rachel on our team maintains the regulatory concept taxonomy — adding new terms, deprecating obsolete ones, and mapping new regulatory vocabulary to existing concept clusters as each new bulletin cycle introduces its own preferred language.
The hard problem that remains is institution-specific interpretation. Some institutions have developed idiosyncratic terminology for common regulatory obligations — terms that appear nowhere in regulatory text and nowhere in industry standards documents, but are well-understood internally. The concept mapping approach handles standard synonymy well. It handles institution-specific jargon only as well as the institutional metadata we're given during onboarding. This is one reason why the onboarding process for Pensvyne is more involved than just uploading documents: we need to understand how your institution has translated regulatory language into your own vocabulary before the mapping layer can reliably find it.