Publications

About

Data curation (pruning and distilling training sets) is standard practice but typically optimized for accuracy. In medical imaging, this is often the wrong target, as curated datasets can maintain accuracy while losing critical properties clinicians rely on: calibration, rare-class sensitivity, and stability under domain shift.

We introduce MedCurate-Bench, a benchmark designed to audit whether curated medical datasets preserve diagnostic validity. Evaluating across five modalities, two paradigms, and cross-hospital shifts, we show that:

• Accuracy is a lagging indicator: Worst-class sensitivity degrades up to 10-20× faster than overall accuracy.
• Coverage awareness is essential: Only coverage-aware selection consistently prevents catastrophic class collapse.
• Curation damage is asymmetric: Calibration error is largely recoverable, but worst-class sensitivity loss is not.
• Coverage rules preserve generalization: Per-class coverage rules maintain class integrity and minimize cross-hospital gaps.

MedCurate-Bench reframes curation evaluation from simple accuracy metrics to comprehensive diagnostic-validity profiles.

Architecture

About

Reasoning-trace curation for supervised fine-tuning typically relies on agreement between independently sampled traces as a proxy for correctness. This assumption works well when there is a single verifiable correct answer, but fails in open-ended multimodal reasoning tasks (e.g., explaining persuasive techniques in video advertisements) where multiple correct answers exist.

This paper shows that standard curation toolkits do not transfer to open-ended settings. The key bottleneck is selection rather than generation: the primary challenge is not the model’s ability to generate valid reasoning traces, but the ability to reliably select correct traces from its generations. Crucially, this selection bottleneck persists and does not close even when the sampling budget is scaled up.

Architecture

About

Large Language Model (LLM) agents acting over multiple episodes often repeat the same mistakes because individual failures are rarely captured in a checkable, reusable memory.

We introduce FAILGROUND, a deterministic pipeline that automatically mines agent errors into typed Failure-Provenance Triples (FPT) stored in a Failure-Aware Knowledge Graph (FAKG). FAILGROUND detects hallucinations deterministically without relying on costly LLM-as-judge or external knowledge graph queries. Investigating when replaying this failure memory is beneficial, we establish the safety boundary of context-invariance:

• Safe Replay: When failures are properties of the environment (e.g., schema violations), replaying them matches strong baselines on InterCode-SQL and improves performance on BIRD-SQL, porting safely across different agents.
• Harmful Replay: When failures are episode-specific, literal replay introduces misinformation, performing worse than having no memory at all.

Architecture

About

At inference time, scaling agent performance typically involves generating multiple candidate trajectories and selecting the best one with a verifier. However, standard temperature-based sampling often produces trajectories that share the same underlying strategic failure modes despite surface-level variations.

We introduce PRAXIS (Principled Reasoning via Agentic Exploration at Inference-time Scale), a training-free framework that generates candidate trajectories under distinct cognitive lenses. These lenses are grounded in linguistic pragmatics and agent failure taxonomies. Evaluating across multiple benchmarks, PRAXIS outperforms temperature-based sampling by 12-17 percentage points, demonstrating consistent log-linear scaling. We also identify critical failure modes in standard verifiers, such as false-consensus and restraint-bias.

Architecture