rigorous tests of reflection strategies to consolidate the framework - detailed experiments

[jd-coderepos]

to make the planning of experiments manageable they are outlined below:

seven LLMs selected: gpt-4o, mistral-large, qwen3-30b-a3b-instruct-2507, qwen3-235b-a22b, Qwen3-0.6B, Qwen3-4B, and Llama-3.3-70B-Instruct

• task 1: self refinement experiments (the minimal scikgextract workflow -- same llm for extraction, reflection, and feedback)
  - total experiments: 7 llms x ___ #datasets = ___ total runs

[jd-coderepos]

question: perhaps we should consider using anthropic models instead of openai models for the proprietary runs...

[SameerSamji]

Update: Model Selection for Extraction Task

I experimented with all the models listed above for the extraction pipeline. During experimentation, I got some issues with smaller models, specifically Qwen3-0.6B and Qwen3-4B. These models frequently produced incomplete or invalid JSON outputs, which violated the expected schema for the extraction task.

To address this, I replaced these models with other smaller models in the similar family:

1. Qwen3.5-9B
2. Ministral 3 3B 2512

These models showed significantly improved performance and consistently generated complete and schema-valid JSON outputs.

[SameerSamji]

Update: Gold-Standard Datasets Selected for SciKGExtract Experimentation

For the evaluation of the SciKGExtract framework, four gold-standard datasets have be chosen from the literature.
The selected datasets are:

1. PolyIE
2. PcMSP
3. BioRED
4. BC5CDR

Dataset Organization
All dataset resources are available here: https://github.com/sciknoworg/scikgextract/tree/main/data/gold-standard-datasets
The directories contains both raw and processed versions of each dataset. Additionally, each dataset folder includes a dedicated README.md file providing detailed descriptions, statistics, and dataset-specific information.

Schema Definition
The corresponding extraction schemas used for SciKGExtract for these datasets are defined in: https://github.com/sciknoworg/scikgextract/tree/main/data/schemas

[SameerSamji]

Update: Anthropic Models for SciKGExtract Extraction

I experimented using Claude Sonnet 4.6 on the ALD/E dataset. However, due to complex and deeply nested schema of ALD/E processes, the model was not able to successfully execute the extraction task.

The primary issue was related to limitations in handling complex and recursive schema, which led to compiled grammar error.

This behavior is consistent with known limitations in Claude’s structured output system for complex schemas, as documented here:
https://platform.claude.com/docs/en/build-with-claude/structured-outputs#json-schema-limitations

Given this, I am currently experimenting Claude Sonnet 4.6 with simpler schemas of the gold-standard datasets (PolyIE, PcMSP, BioRED, and BC5CDR) in order to better assess its performance.

[SameerSamji]

Update: Task 1: Self-Refinement Experiments

I have completed the experiments for knowledge extraction without refinement across all selected datasets and extraction models. The experiments was conducted on the ALD/E dataset as well as the four gold-standard datasets: PolyIE, PcMSP, BioRED, and BC5CDR. In addition, the self-refinement experiments have now also been fully executed across all five datasets. I will push the resultant extractions soon with the quantitative evaluations.

[SameerSamji]

Update: Task 1: Self-Refinement Experiments

The experiments for Knowledge extraction with self-refinement across all selected datasets and extraction models are finished. The resultant extractions together with it quantitative evaluations can be found here: https://github.com/sciknoworg/scikgextract/tree/main/results