diff --git a/docs/hub/eval-results.md b/docs/hub/eval-results.md index 78c6aa7248..08b8834b51 100644 --- a/docs/hub/eval-results.md +++ b/docs/hub/eval-results.md @@ -70,7 +70,7 @@ Anyone can submit evaluation results to any model via Pull Request: 3. Add a `.eval_results/*.yaml` file with your results. 4. The PR will show as "community-provided" on the model page while open. -For help evaluating a model, see the [Evaluating models with Inspect](https://huggingface.co/docs/inference-providers/guides/evaluation-inspect-ai) guide. +For help evaluating a model, see the [Evaluating models with Inspect](https://huggingface.co/docs/inference-providers/guides/evaluation-inspect-ai) guide. Evaluation logs from Inspect can be written directly to [HF Storage Buckets](./storage-buckets-integrations#inspect-ai). > [!TIP] > Community scores are visible while the PR is open. If a score is disputed, the model author can close the PR to remove it. The goal is to surface existing evaluation data transparently while building toward a fully reproducible standard via verified scores. diff --git a/docs/hub/storage-buckets-integrations.md b/docs/hub/storage-buckets-integrations.md index b6c322ae2e..649f2c05b0 100644 --- a/docs/hub/storage-buckets-integrations.md +++ b/docs/hub/storage-buckets-integrations.md @@ -51,6 +51,21 @@ from datasets import load_dataset ds = load_dataset("buckets/username/my-bucket", data_files=["data.parquet"]) ``` +## Inspect AI + +[Inspect AI](https://inspect.aisi.org.uk/) can write evaluation logs directly to a bucket — point its log directory at an `hf://buckets/` path (requires `huggingface_hub>=1.6.0`). Create the bucket and authenticate first (Inspect won't create it for you): + +```bash +hf auth login +hf buckets create username/my-bucket --private + +export INSPECT_LOG_DIR=hf://buckets/username/my-bucket/eval-logs +inspect eval popularity.py --model openai/gpt-4 +inspect view +``` + +See [Inspect's eval logs guide](https://inspect.aisi.org.uk/eval-logs.html#sec-hugging-face-storage-buckets) for details. + ## Filesystem operations For direct file operations, `huggingface_hub` exposes a pre-instantiated [filesystem object](/docs/huggingface_hub/guides/hf_file_system), `hffs`: