🤖 AI Summary
This work addresses the inefficiency of repeatedly verifying redundant or invalid information in clinical NLP generation–verification pipelines by proposing a lightweight dynamic memory mechanism that learns gating strategies during inference to filter spurious outputs. The study finds that directly learning filtering rules from sparse rejection signals provided by verifiers yields limited performance; instead, integrating clinical ontologies with evidence detection aligned to verification logic enables effective gating. The system employs a dual-model architecture comprising a Llama-3.3 70B generator and an MMed-Llama-3.1 70B verifier, along with a flag-but-not-delete policy for handling suspicious outputs to preserve clinical auditability. Experiments on 5,000 patient records demonstrate that the ontology-driven filter identifies 49,734 violation relations, while the question-answering filter increases the verifier’s rejection probability by 1.84×.
📝 Abstract
We study inference-time pattern-memory gating in a production-scale clinical natural language processing (NLP) pipeline. The pipeline pairs a generator (Llama-3.3 70B) proposing extractions with a verifier (MMed-Llama-3.1 70B) accepting or rejecting them, over 167,034 PMC-Patients narratives, and adds a lightweight memory that learns at deployment which extractions to filter, so the verifier need not re-examine candidates already seen to fail. We report four findings. First, learning filtering rules directly from the verifier's rejections failed at full scale: the relation-extraction filter stayed empty despite 785,797 logged rejections, because they were spread too thinly across too many distinct forms to accumulate. Second, a simpler rule using a fixed clinical ontology produced the same filtering without the verifier, capturing 49,734 ontology-violating relations on a held-out 5,000-patient set. Third, of five versions of the question-answering filter, four failed for distinct, instructive reasons; the fifth succeeded by checking whether a patient's extracted entities support the question asked, and where it applies was 1.84 times likelier to flag an answer the verifier would reject than one it would accept. Fourth, one pattern held across all five: a filter is selective only when it tests the same evidence the verifier weighs, not when it imitates the verifier's output. Together these give a transferable result for any generator-verifier pipeline: the most natural memory design can fail silently at scale, and whether a pre-generation gate is selective is decided before any engineering effort, by whether its signal probes the question the verifier itself answers. Throughout, the system flags suspect extractions rather than deleting them, so every decision stays visible for clinical review. All code and test artefacts are released openly.