Add batch inference benchmarking notebooks by sfc-gh-jiehuang · Pull Request #279 · Snowflake-Labs/sf-samples

sfc-gh-jiehuang · 2026-06-23T21:03:53Z

Summary

Adds two batch inference benchmark notebooks under samples/ml/model_serving/batch_inference_benchmarking/:

run_batch_xgboost_10b.ipynb — XGBoost 10B-row batch inference benchmark
run_batch_st_10m.ipynb — Sentence Transformer 10M-row batch inference benchmark

Add XGBoost 10B-row and Sentence Transformer 10M-row batch inference benchmark notebooks under samples/ml/model_serving/batch_inference_benchmarking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

sfc-gh-gmurlidhar · 2026-06-29T21:11:46Z

+    "\n",
+    "# --- Generate input data ---\n",
+    "SENTENCE_TEMPLATES = [\n",
+    "    \"Machine learning models require diverse training data for optimal performance across domains.\",\n",


Can we use a standard dataset

We would need to rerun the benchmark for all platforms if we update the datasets. I do not feel like it is worth the effort at this point.

sfc-gh-gmurlidhar · 2026-06-29T21:15:21Z

+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "# ╔════════════════════════════════════════════════════════════╗\n# ║  USER CONFIGURATION — fill these in before running         ║\n# ╚════════════════════════════════════════════════════════════╝\nCONNECTION_NAME = \"<connection>\"         # Snowflake connection name (from ~/.snowflake/connections.toml)\nDB_NAME = \"ST_BENCHMARK\"                # Database to create/use for this benchmark\nWAREHOUSE_SIZE = \"4X-LARGE\"             # Warehouse size for data generation\nIMAGE_REPO = \"<db>.<schema>.<repo>\"     # Image repository for SPCS containers\nEVENT_TABLE = \"<db>.<schema>.<table>\"   # Event table for platform metrics (set to None to skip metrics)\n\n# ╔════════════════════════════════════════════════════════════╗\n# ║  BENCHMARK DEFAULTS — change only to explore alternatives  ║\n# ╚════════════════════════════════════════════════════════════╝\nNUM_NODES = 2\nINSTANCE_FAMILY = \"GPU_NV_S\"\nNUM_WORKERS = 2\nMAX_BATCH_ROWS = 256\nREPLICAS = 2\nFUNCTION_NAME = \"encode\"\nINPUT_ROWS = 10_000_000\nGPU_REQUESTS = \"1\"                      # GPUs per worker\nREPEAT = 3\nWARMUP_ROW_COUNT = 1_000\n\nMODEL_ID = \"all-MiniLM-L6-v2\"          # HuggingFace model to download\nMODEL_NAME = \"all_minilm_l6_v2\"\nMODEL_VERSION = \"V1\""


Feel like some of these don't need to be changed by customers. Of course this is the code and they can do whatever but we should draw less attention to it

The users do not need to touch this to run this notebook.

It is the concern that we do not want to draw attention to it? We can move the non mandatory parameters to other sections. What do you think?

sfc-gh-sdas · 2026-06-30T20:19:44Z

+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "# ╔════════════════════════════════════════════════════════════╗\n# ║  USER CONFIGURATION — fill these in before running         ║\n# ╚════════════════════════════════════════════════════════════╝\nCONNECTION_NAME = \"<connection>\"         # Snowflake connection name (from ~/.snowflake/connections.toml)\nDB_NAME = \"ST_BENCHMARK\"                # Database to create/use for this benchmark\nWAREHOUSE_SIZE = \"4X-LARGE\"             # Warehouse size for data generation\nIMAGE_REPO = \"<db>.<schema>.<repo>\"     # Image repository for SPCS containers\nEVENT_TABLE = \"<db>.<schema>.<table>\"   # Event table for platform metrics (set to None to skip metrics)\n\n# ╔════════════════════════════════════════════════════════════╗\n# ║  BENCHMARK DEFAULTS — change only to explore alternatives  ║\n# ╚════════════════════════════════════════════════════════════╝\nNUM_NODES = 2\nINSTANCE_FAMILY = \"GPU_NV_S\"\nNUM_WORKERS = 2\nMAX_BATCH_ROWS = 256\nREPLICAS = 2\nFUNCTION_NAME = \"encode\"\nINPUT_ROWS = 10_000_000\nGPU_REQUESTS = \"1\"                      # GPUs per worker\nREPEAT = 3\nWARMUP_ROW_COUNT = 1_000\n\nMODEL_ID = \"all-MiniLM-L6-v2\"          # HuggingFace model to download\nMODEL_NAME = \"all_minilm_l6_v2\"\nMODEL_VERSION = \"V1\""


why image_repo is needed?

sfc-gh-sdas · 2026-06-30T20:20:19Z

+    "    EVENT_TABLE = None\n",
+    "    print(\"EVENT_TABLE not configured -- platform metrics will be skipped.\")\n",
+    "\n",
+    "WAREHOUSE_NAME = f\"{DB_NAME}_WH\"\n",


take this as input?

sfc-gh-sdas · 2026-06-30T20:21:09Z

+    "    print(f\"Registered: {MODEL_NAME}/{MODEL_VERSION}\")\n",
+    "\n",
+    "# --- Generate input data ---\n",
+    "SENTENCE_TEMPLATES = [\n",


we should probably take more standard dataset from HF

sfc-gh-sdas · 2026-06-30T20:22:17Z

+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": "# ╔════════════════════════════════════════════════════════════╗\n# ║  USER CONFIGURATION — fill these in before running         ║\n# ╚════════════════════════════════════════════════════════════╝\nCONNECTION_NAME = \"<connection>\"         # Snowflake connection name (from ~/.snowflake/connections.toml)\nDB_NAME = \"ST_BENCHMARK\"                # Database to create/use for this benchmark\nWAREHOUSE_SIZE = \"4X-LARGE\"             # Warehouse size for data generation\nIMAGE_REPO = \"<db>.<schema>.<repo>\"     # Image repository for SPCS containers\nEVENT_TABLE = \"<db>.<schema>.<table>\"   # Event table for platform metrics (set to None to skip metrics)\n\n# ╔════════════════════════════════════════════════════════════╗\n# ║  BENCHMARK DEFAULTS — change only to explore alternatives  ║\n# ╚════════════════════════════════════════════════════════════╝\nNUM_NODES = 2\nINSTANCE_FAMILY = \"GPU_NV_S\"\nNUM_WORKERS = 2\nMAX_BATCH_ROWS = 256\nREPLICAS = 2\nFUNCTION_NAME = \"encode\"\nINPUT_ROWS = 10_000_000\nGPU_REQUESTS = \"1\"                      # GPUs per worker\nREPEAT = 3\nWARMUP_ROW_COUNT = 1_000\n\nMODEL_ID = \"all-MiniLM-L6-v2\"          # HuggingFace model to download\nMODEL_NAME = \"all_minilm_l6_v2\"\nMODEL_VERSION = \"V1\""


why do you need IMAGE_REPO?

sfc-gh-sdas · 2026-06-30T20:25:00Z

+    "        CREATE OR REPLACE TABLE {table_name} AS\n",
+    "        SELECT\n",
+    "            ARRAY_CONSTRUCT({array_literal})[MOD(SEQ4(), {num_templates})]::VARCHAR AS SENTENCE\n",
+    "        FROM TABLE(GENERATOR(ROWCOUNT => {row_count}))\n",


can we use wine dataset directly? https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_wine.html

Replace run_batch_st_10m.ipynb and run_batch_xgboost_10b.ipynb with the latest versions from the source benchmarking notebooks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Add batch inference benchmarking notebooks

968cf04

Add XGBoost 10B-row and Sentence Transformer 10M-row batch inference benchmark notebooks under samples/ml/model_serving/batch_inference_benchmarking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

sfc-gh-jiehuang requested review from sfc-gh-gmurlidhar and sfc-gh-sdas June 23, 2026 21:07

sfc-gh-gmurlidhar reviewed Jun 29, 2026

View reviewed changes

sfc-gh-jiehuang requested a review from sfc-gh-gmurlidhar June 30, 2026 13:33

sfc-gh-sdas reviewed Jun 30, 2026

View reviewed changes

Update batch inference benchmarking notebooks

6ad991a

Replace run_batch_st_10m.ipynb and run_batch_xgboost_10b.ipynb with the latest versions from the source benchmarking notebooks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Uh oh!

Conversation

sfc-gh-jiehuang commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sfc-gh-jiehuang commented Jun 23, 2026 •

edited

Loading