Field Notes: Six 2025 Ideas That Changed How I Build RAG & Agents

Two bugs broke my RAG system last month. The first was self‑inflicted—I nudged top‑k from 8 to 16 “for coverage” and watched answers get longer and less faithful. The second came from a knowledge base that contradicted itself; my agent cheerfully averaged the disagreement into nonsense. These weren’t model problems. They were system problems. And they pushed me into a week‑long rabbit hole of 2025 papers that, together, changed how I build.

Below is a narrative of what I learned. It’s not a survey; it’s a lab notebook. I’ll explain the idea, link the paper, and show the exact tweak I made. No hype, just the pieces that actually moved the needle.

1) How many passages should we stuff into context?Copy link to section

On Tuesday, I read Guo et al. (2025), who frame RAG as noisy in‑context learning and—finally—give finite‑sample risk bounds for the context you feed the LLM. The moral is refreshingly simple: each passage is an example; more examples add both signal and noise; there’s a bias–variance trade‑off; there’s a sweet spot.

Paper: Retrieval‑Augmented Generation as Noisy In‑Context Learning: A Unified Theory and Risk Bounds, Guo et al., 2025 — arXiv · HTML

I stopped hard‑coding top‑k. Instead, I scored each candidate passage for “noisiness” (age, domain mismatch, lexical mismatch) and let a tiny controller pick k per query.

# a 20‑line controller that paid for itself in a day
from math import exp
 
def choose_k(passages, min_k=4, max_k=12):
    # passages: list of dicts with noise features in [0,1]
    # heuristic risk ~ noise_mean + noise_var; lower risk → larger k
    ns = [0.5*p['age'] + 0.3*p['domain_mismatch'] + 0.2*p['lex_mismatch'] for p in passages]
    mu = sum(ns)/len(ns); var = sum((x-mu)**2 for x in ns)/len(ns)
    risk = 0.7*mu + 0.3*var
    # map risk→k with a smooth squashing; tune constants on logs, not vibes
    frac = 1.0/(1.0 + exp(8*(risk-0.35)))  # center at ~0.35
    return int(min_k + frac*(max_k-min_k))

In my logs, this shaved ~12–18% tokens per answer and reduced “contradiction with sources” flags. More importantly: answers felt calmer. The model wasn’t drowning in barely‑relevant context anymore.

2) What if the evidence fights itself?Copy link to section

Mid‑week, a teammate sent me RAMDocs—a dataset where queries meet ambiguity, noise, and misinformation all at once. The accompanying method MADAM‑RAG uses a light, debate‑style agent setup that asks small critics to surface conflicts before we synthesize.

Paper & data: Retrieval‑Augmented Generation with Conflicting Evidence — Wang et al., 2025 — arXiv · RAMDocs code: GitHub

I borrowed the spirit, not the letter. After retrieval, I spawn two quick “voices”: one tries to resolve ambiguity (which entity/date/formula do we mean?), the other tries to flag misinformation (does any passage contradict the rest?). Only then do I ask the main model to answer, explicitly citing the sub‑conclusions.

The effect is subtle but real: fewer confident wrong answers when the corpus disagrees with itself, and clearer “here are the two plausible interpretations” when things are genuinely ambiguous.

3) Train the process, not just the outcomeCopy link to section

Most of my failures start before generation: poor query rewriting, bad retriever choice, premature stopping. Leng et al. (2025) propose DecEx‑RAG, which treats agentic RAG as a tiny MDP—Decision (what/when to retrieve) then Execution (how to use it)—and adds process supervision so we reward good steps, not just good final answers.

Paper: DecEx‑RAG: Boosting Agentic Retrieval‑Augmented Generation with Decision and Execution Optimization via Process Supervision — Leng et al., 2025 — arXiv · HTML

I instrumented my scaffold to log state → action → observation for each retrieve/rewrite/answer step, then trained a tiny critic that scores those steps post‑hoc. Even a simple linear reward model nudged the agent away from wasteful branches (e.g., redundant query expansions) and toward sequences that produced faithful answers with less context.

4) Small models as the default brainCopy link to section

This one is more of an argument than a result, but it hit home: Belčák (2025) makes the case that Small Language Models (<10B) should drive most agent workloads, with a bigger LLM reserved for the rare, ambiguous synthesis step. If your agent spends 80% of its life searching, filtering, formatting, filling forms, a small model plus sharp tools beats a giant model plus vibes.

Position paper: Small Language Models are the Future of Agentic AI — Belčák, 2025 — arXiv · PDF · Overview: NVIDIA Labs

I rewired the runtime: a 7–13B model handles tool calls and browsing, and I escalate to a larger model only when my critics disagree or confidence is low. Costs dropped; latency tails shrank; nobody missed the extra parameter count.

5) If your agent browses, give it a real testCopy link to section

I used to evaluate browsing by watching a few demos. Then BrowseComp arrived: 1,266 questions that force multi‑page reading, reformulation, and patience. It’s nasty in a good way. Accuracy scales with test‑time compute, which is exactly what we need to tune planning policies.

Benchmark: BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents — Wei et al., OpenAI, 2025 — blog · paper PDF · arXiv

I set a compute schedule (3, 6, 12 page loads/tool calls) and plotted accuracy vs. budget. The curve told me where my agent was too cautious (premature stopping) and where it was lost (looping on the wrong site). A single planning tweak—“when in doubt, reformulate once, then broaden”—bought me nine points.

Related: OpenAI later reported 68.9% with the ChatGPT agent on this benchmark — announcement: Introducing ChatGPT Agent.

6) Security: no more wishful thinkingCopy link to section

Finally, WASP gave me the cold shower I needed. In a sandboxed GitLab/Reddit‑style world, simple human‑written prompt injections frequently pushed agents onto the wrong path (partial success rates up to 86%), even if full attacker goals were rarely completed.

Benchmark: WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks — Evtimov et al., 2025 — arXiv · PDF · code

I added a strict instruction hierarchy (system > developer > page) and tool‑permission gating (writes require explicit elevation). Rerunning WASP made the partial‑success curve drop to something I could live with. Not perfect, but honest.

Bonus: When GraphRAG actually pays offCopy link to section

Two compact overviews helped me decide when to leave classic RAG:

Survey: A Survey of Graph Retrieval‑Augmented Generation — Zhang et al., 2025 — arXiv · PDF

Evaluation: RAG vs. GraphRAG: A Systematic Evaluation and Key Insights — Han et al., 2025 — arXiv · OpenReview PDF

Rule of thumb I now use with teams: if the answer depends on entities and relations (incidents→causes→policies; functions→calls→PRs), GraphRAG or a hybrid is worth the added plumbing. If most passages stand alone, classic RAG is simpler and faster.

A short, practical checklistCopy link to section

I’m allergic to long bullet lists, so here’s the only one you’ll see:

• Make top‑k adaptive with a simple noise‑aware controller, and log the choice.
• Insert a conflict‑resolver micro‑stage (ambiguity + misinformation) before synthesis.
• Instrument the process (state → action → observation) and supervise steps, not just outcomes.
• Default to an SLM‑first scaffold; escalate to a big model only on demand.
• Evaluate browsing on BrowseComp; red‑team the whole stack on WASP.

If you implement even two of these, you’ll feel the system get quieter and more honest. That’s been the theme of my month: fewer knobs, better rails, and models that seem smarter mostly because the system got sharper.