Skip to content

Commit 9ed8f50

Browse files
authored
fix: bundle NLTK punkt_tab in Docker image for airgapped environments (open-webui#21165)
Pre-download NLTK punkt_tab during Docker build instead of at runtime. This fixes document extraction failures in offline/airgapped environments where the container cannot download the tokenizer data after restarts. Fixes open-webui#21150
1 parent d3f2cf7 commit 9ed8f50

1 file changed

Lines changed: 2 additions & 0 deletions

File tree

Dockerfile

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,7 @@ RUN pip3 install --no-cache-dir uv && \
143143
python -c "import os; from sentence_transformers import SentenceTransformer; SentenceTransformer(os.environ.get('AUXILIARY_EMBEDDING_MODEL', 'TaylorAI/bge-micro-v2'), device='cpu')" && \
144144
python -c "import os; from faster_whisper import WhisperModel; WhisperModel(os.environ['WHISPER_MODEL'], device='cpu', compute_type='int8', download_root=os.environ['WHISPER_MODEL_DIR'])"; \
145145
python -c "import os; import tiktoken; tiktoken.get_encoding(os.environ['TIKTOKEN_ENCODING_NAME'])"; \
146+
python -c "import nltk; nltk.download('punkt_tab')"; \
146147
else \
147148
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu --no-cache-dir && \
148149
uv pip install --system -r requirements.txt --no-cache-dir && \
@@ -151,6 +152,7 @@ RUN pip3 install --no-cache-dir uv && \
151152
python -c "import os; from sentence_transformers import SentenceTransformer; SentenceTransformer(os.environ.get('AUXILIARY_EMBEDDING_MODEL', 'TaylorAI/bge-micro-v2'), device='cpu')" && \
152153
python -c "import os; from faster_whisper import WhisperModel; WhisperModel(os.environ['WHISPER_MODEL'], device='cpu', compute_type='int8', download_root=os.environ['WHISPER_MODEL_DIR'])"; \
153154
python -c "import os; import tiktoken; tiktoken.get_encoding(os.environ['TIKTOKEN_ENCODING_NAME'])"; \
155+
python -c "import nltk; nltk.download('punkt_tab')"; \
154156
fi; \
155157
fi; \
156158
mkdir -p /app/backend/data && chown -R $UID:$GID /app/backend/data/ && \

0 commit comments

Comments
 (0)