The part of my RAG project that refuses to answer

Everyone building with LLMs right now is optimizing for the confident, helpful answer. I spent the last few weeks optimizing for the opposite: getting my system to shut up when it doesn’t actually know.

Key Takeaways

• Refusal is a first-class result, not an error path — build your system to say “no good match” instead of forcing a weak one
• Your threshold is probably a guess — a 9-pair eval set moved my refusal accuracy from 62% to 89%
• A perfect score you can’t reproduce is luck, not proof — non-deterministic generation means single-run metrics can mislead
• Ship the negative result — a failed experiment with a known mechanism is still knowledge

A retrieval system always has an answer for you

ResearchBridge is a side project that matches pre-med students to research and volunteer opportunities by semantic similarity, then writes a grounded explanation of why each one fits. It’s not a medical system. The “health” part is just that the users are pre-med. Underneath, the engineering is the same shape as any RAG problem: given a query and a corpus, when do you trust the match?

Most RAG demos do the easy version: return the top few nearest opportunities and write something plausible about each. It always has an answer, and that’s exactly the problem. Semantic search will always return something, and an LLM will always explain why it’s a great fit, even when it isn’t.

So the feature I actually cared about was refusal: the system returning “no good match” instead of forcing one. Refusal isn’t an error path. It’s a first-class result, a row in the database like any other match. Making it work turned into the most honest engineering I’ve done. Three things I learned:

My refusal threshold was wrong, and only an eval caught it

I’d picked 0.40 as the cosine-similarity cutoff by feel. That cutoff was quietly rejecting genuine matches, being too cautious, which feels safe but is just a different way of being wrong. I only found out because I built a small eval set — 9 hand-labeled student-opportunity pairs (6 match cases, 3 deliberate refusal cases including an astrophysics profile against an all-biomedical corpus) — and measured how often the system’s refusals agreed with mine. At 0.40, they agreed 62% of the time. Recalibrating to 0.32 took that to 89%. The number I’d trusted was a guess wearing a lab coat.

The eval caught the model making things up, and then I caught myself

Next came a faithfulness check: a second model reads the generated explanation and the source it’s meant to be grounded in, and flags anything invented. Sure enough, it caught the generator writing that a student was “passionate about broadening representation,” a nice sentence that appeared in neither the student’s profile nor the opportunity. Grounded-sounding, ungrounded.

But the faithfulness check had its own blind spot. It passed claims as faithful when they were grounded in the wrong source — a perfect sentence about the right student, traced correctly to a retrieved chunk, from an opportunity that wasn’t the one being evaluated. Faithful and true were two different things, and neither the generator nor the judge could tell the difference. I had to add a separate cross-check that the source chunk belonged to the expected opportunity before trusting the score.

One run scored a perfect 1.0 and I almost wrote that down as a win, until I remembered generation is non-deterministic, ran it again, and got 0.91 with real fabrications caught. Across multiple evaluation runs, the faithfulness scores fluctuated — sometimes by 0.1 or more — confirming that a single pass is never enough data. A clean score you can’t reproduce is luck, not proof.

My best idea made things worse, and I shipped that finding anyway

My hypothesis was that distilling each opportunity down to its essence before embedding would sharpen the separation between matches and non-matches. I built it, pre-committed to the prediction, and measured. It made separation worse — the overlap between match and non-match similarity scores actually increased. So I reverted it, and then wrote the negative result into the project’s README, because a failed experiment with a known mechanism is still knowledge, and hiding it would be the exact dishonesty the whole project is trying to engineer against.

The actual job is the honesty around the model

Here’s what ties them together. The interesting part of building with LLMs isn’t getting them to answer. They’ll always answer. It’s building the system, and the discipline, to know when the answer isn’t trustworthy and to say so. That applies to the model’s output (refuse over fabricate), to your own metrics (distrust a score you can’t reproduce), and to your claims about your own work (report the negative result). The refusal row in my database and the negative result in my README are the same move: not prompt-whispering models into sounding smart, but building the evaluation and the honesty around them so you can tell when they’re wrong.

My project matches students to research opportunities. But the part I’m actually proud of is that when it doesn’t have a good answer, it says so. And when I don’t, I try to too.