AMANDA: Agentic Medical Knowledge Augmentation for Data-Efficient Medical Visual Question Answering

This repository contains the implementation of "AMANDA: Agentic Medical Knowledge Augmentation for Data-Efficient Medical Visual Question Answering" (EMNLP 2025). AMANDA augments medical vision-language models with collaborative agents that reason, ask targeted visual follow-up questions, and retrieve grounded biomedical knowledge.

Paper: arXiv:2510.02328

Highlights

• 🧠 Agentic medical reasoning: Perceiver, Reasoner, Evaluator, Explorer, and Retriever agents work together in a fixed medical VQA pipeline.
• 🔎 Targeted visual exploration: low-confidence answers trigger clinically focused follow-up questions for additional image evidence.
• 🧬 Biomedical knowledge augmentation: SPOKE retrieval grounds reasoning with structured medical context.
• 🖼️ Few-shot example selection: visual similarity and text similarity are fused to select relevant in-context examples.
• 🛠️ Training support: Med-BLIVA and Med-InstructBLIP training code is included for users who want to train their own backbones.

Abstract

Medical visual question answering remains challenging when only limited labeled medical data is available. AMANDA addresses this setting with a fixed multi-agent pipeline: a Perceiver first extracts image-level evidence, a Reasoner produces an answer, an Evaluator estimates confidence, an Explorer asks targeted follow-up visual questions when confidence is insufficient, and a Retriever brings in grounded biomedical knowledge from SPOKE. The framework is designed for zero-shot and few-shot medical VQA without committing task-specific model weights to this repository. The few-shot setting uses visual and textual similarity to select relevant in-context examples, while the RAG component supplies structured medical context from a biomedical knowledge graph.

Repository Structure

• src/amanda/: AMANDA pipeline, agents, prompts, SPOKE retrieval, few-shot selection, and evaluation code.
• configs/: example configuration files. They contain environment variable names, public endpoints, and thresholds, but no private paths or secrets.
• scripts/: entry points for inference, few-shot index building, evaluation, and training wrappers.
• training/bliva/: Med-BLIVA and Med-InstructBLIP training code adapted from the BLIVA framework. We use the BLIVA training framework and LLaVA-Med data utilities; trained weights are not included.
• assets/: figures used by the public README.

Large files and private artifacts, including datasets, model weights, generated outputs, paper PDFs, and API credentials, are intentionally excluded.

Data Preparation

AMANDA expects datasets and images to be prepared outside the repository. Keep annotations, images, generated captions, few-shot indices, predictions, and checkpoints outside version control and pass their locations through command-line arguments or environment variables.

Training Data

The Med-BLIVA / Med-InstructBLIP training path has two stages.

Stage 1 uses the LLaVA-Med biomedical concept alignment data:

https://hanoverprod.z21.web.core.windows.net/med_llava/alignment/llava_med_alignment_500k.json

Save the validated annotation file as:

$AMANDA_MEDLLAVA_ANNOTATION_ROOT/llava_med_alignment_500k_valid.json

Stage 2 uses BioMed-VITAL instruction data:

Hugging Face dataset: mao1207/BioMed-VITAL-instructions

Export the instruction records as:

$AMANDA_MEDLLAVA_ANNOTATION_ROOT/BioMed-VITAL-instructions-150K-valid.json

Both stages use images referenced by the annotation records. LLaVA-Med releases the image URL list here:

https://hanoverprod.z21.web.core.windows.net/med_llava/llava_med_image_urls.jsonl

Use the Microsoft LLaVA-Med downloader, or an equivalent copy of it, to fetch PMC articles and extract the referenced images:

mkdir -p data/llava-med

wget https://hanoverprod.z21.web.core.windows.net/med_llava/llava_med_image_urls.jsonl \
  -O data/llava-med/llava_med_image_urls.jsonl

python <LLaVA-Med checkout>/llava/data/download_images.py \
  --input_path data/llava-med/llava_med_image_urls.jsonl \
  --pmc_output_path data/llava-med/pmc_articles \
  --images_output_path data/llava-med/images

Set the image root to the extracted image directory:

export AMANDA_MEDLLAVA_IMAGE_ROOT=data/llava-med/images

The training dataset classes join AMANDA_MEDLLAVA_IMAGE_ROOT with each annotation record's image field, so the relative paths in the JSON must match the downloaded image layout.

Inference and Evaluation Data

For VQA-RAD, SLAKE, and other Med-VQA datasets, AMANDA reads a JSON list. The recommended format follows the LLaVA-style conversation layout:

[
  {
    "id": "sample_id",
    "image": "relative/path/to/image.jpg",
    "answer_type": "open",
    "conversations": [
      {"from": "human", "value": "<image>\nWhat abnormality is shown?"},
      {"from": "gpt", "value": "cardiomegaly"}
    ]
  }
]

answer_type should be open for open-ended questions and close or closed for yes/no or option-style questions. AMANDA also accepts simplified records with question or prompt, answer, question_id, and image, but evaluation is most reliable when the ground truth keeps the LLaVA-style conversations field.

Expected files for the paper-style VQA-RAD and SLAKE workflow are:

train_rad.json
test_rad.json
train_slake.json
test_slake.json

Images should be stored under a dataset image root and referenced by each record's relative image path. Dataset files are not redistributed in this repository; obtain VQA-RAD and SLAKE from their original releases and convert them to the schema above.

Model Training

Training is optional for AMANDA inference with an existing medical VLM backend, but this repository includes the training code needed to reproduce Med-BLIVA / Med-InstructBLIP style backbones. The release keeps the two-stage training path used for the main model preparation.

Install the training package from the repository root:

pip install -e training/bliva

Set data locations:

export AMANDA_MEDLLAVA_ANNOTATION_ROOT=data/llava-med/annotations
export AMANDA_MEDLLAVA_IMAGE_ROOT=data/llava-med/images

Stage 1: Medical Alignment Pretraining

Med-BLIVA with Vicuna:

AMANDA_BLIVA_CFG=train_configs/pretrain_medbliva_vicuna.yaml \
AMANDA_BLIVA_RUN_NAME=pretrain_medbliva_vicuna \
bash scripts/train_bliva_pretrain.sh

Med-InstructBLIP with Vicuna:

AMANDA_BLIVA_CFG=train_configs/pretrain_medblip_vicuna.yaml \
AMANDA_BLIVA_RUN_NAME=pretrain_medblip_vicuna \
bash scripts/train_bliva_pretrain.sh

Stage 2: Medical Instruction Finetuning

Use the stage-1 checkpoint as the initialization for stage 2:

export AMANDA_BLIVA_PRETRAINED=path/to/stage1_checkpoint.pth
export AMANDA_BLIVA_LOAD_PRETRAINED=true

Med-BLIVA with Vicuna:

AMANDA_BLIVA_CFG=train_configs/finetune_medbliva_vicuna.yaml \
AMANDA_BLIVA_RUN_NAME=finetune_medbliva_vicuna \
bash scripts/train_bliva_finetune.sh

Med-InstructBLIP with Vicuna:

AMANDA_BLIVA_CFG=train_configs/finetune_medblip_vicuna.yaml \
AMANDA_BLIVA_RUN_NAME=finetune_medblip_vicuna \
bash scripts/train_bliva_finetune.sh

The public training config set is intentionally small:

training/bliva/train_configs/pretrain_medbliva_vicuna.yaml
training/bliva/train_configs/finetune_medbliva_vicuna.yaml
training/bliva/train_configs/pretrain_medblip_vicuna.yaml
training/bliva/train_configs/finetune_medblip_vicuna.yaml

Generated output/ directories and .pth checkpoints should remain local.

Inference

The default inference backend is Microsoft LLaVA-Med v1.5 through a LLaVA-Med source checkout. BLIVA is kept as an optional backend for users who train or provide compatible BLIVA checkpoints.

Configure the LLM and VLM through environment variables:

export OPENAI_API_KEY=<your key>
export AMANDA_LLM_MODEL=<chat model>
export AMANDA_LLAVA_MED_MODEL_PATH=microsoft/llava-med-v1.5-mistral-7b
export AMANDA_LLAVA_MED_REPO_PATH=<LLaVA-Med checkout>
export AMANDA_DEVICE=cuda

Run AMANDA:

PYTHONPATH=src python scripts/run_amanda.py \
  --config configs/amanda.example.yaml \
  --question-file data/vqa-rad/test_rad.json \
  --image-root data/vqa-rad/images \
  --output-file outputs/vqa_rad_predictions.jsonl

Each prediction row includes the final answer in text, the generated caption, the initial visual answer, the reasoning history, evaluator scores, optional examples, and optional SPOKE RAG context.

Optional BLIVA Backend

To use a trained BLIVA checkpoint for inference, set model.backend: bliva in the config and provide:

export AMANDA_BLIVA_MODEL_NAME=<registered BLIVA model name>
export AMANDA_BLIVA_MODEL_TYPE=<registered BLIVA model type>
export AMANDA_BLIVA_CHECKPOINT_PATH=path/to/checkpoint.pth

Do not commit checkpoint paths or checkpoint files.

Few-Shot Example Selection

Few-shot mode uses dual similarity over question text and images. Build similar_indices with BiomedCLIP embeddings:

PYTHONPATH=src python scripts/build_fewshot_indices.py \
  --train-file data/vqa-rad/train_rad.json \
  --test-file data/vqa-rad/test_rad.json \
  --image-folder data/vqa-rad/images \
  --output-file data/vqa-rad/test_rad_with_similar_indices.json

The selector ranks candidates by text similarity and visual similarity, then fuses the two rankings with average-rank fusion. Runtime few-shot examples are formatted as (caption, question, answer). If --captions-file is not supplied during inference, AMANDA generates missing train captions with the configured VLM.

SPOKE Retrieval

SPOKE retrieval is part of the fixed AMANDA pipeline. The public endpoint and filtering thresholds live in configs/spoke.example.yaml. The retriever performs medical entity extraction, queries SPOKE neighborhoods, converts triples into text, filters KG context, and silently omits empty RAG context from the Reasoner prompt.

Evaluation

Predictions are aligned to ground truth by (question_id, prompt). The evaluator prints only the two paper metrics:

• Open Recall
• Closed Accuracy

PYTHONPATH=src python scripts/evaluate_predictions.py \
  --gt data/vqa-rad/test_rad.json \
  --pred outputs/vqa_rad_predictions.jsonl

Acknowledgements

AMANDA builds on Microsoft LLaVA-Med for medical VLM inference and image-data utilities, BLIVA for the training framework used by the Med-BLIVA / Med-InstructBLIP training code, and SPOKE for biomedical knowledge graph retrieval.

Citation

@article{wang2025amanda,
  title={AMANDA: Agentic Medical Knowledge Augmentation for Data-Efficient Medical Visual Question Answering},
  author={Wang, Ziqing and Mao, Chengsheng and Wen, Xiaole and Luo, Yuan and Ding, Kaize},
  journal={arXiv preprint arXiv:2510.02328},
  year={2025}
}