You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
|**meta-llama/Llama-3.1-8B-Instruct**| ❌ | ❌ | ✅ Validated on Dell XE7740 | ✅ Validated on Dell XE7740 |
113
+
|**Qwen/Qwen3-4B-Instruct-2507**| ✅ Validated on Dell XE7740 | ✅ Validated on Dell XE7740 | ❌ | ❌ |
105
114
106
115
### Required API Configuration
107
116
108
117
**For Inference Service:**
109
118
110
119
This application supports multiple inference deployment patterns:
111
120
112
-
-**GenAI Gateway**: Provide your GenAI Gateway URL and API key
121
+
**GenAI Gateway**: Provide your GenAI Gateway URL and API key
122
+
- URL format: https://api.example.com
113
123
- To generate the GenAI Gateway API key, use the [generate-vault-secrets.sh](https://github.com/opea-project/Enterprise-Inference/blob/main/core/scripts/generate-vault-secrets.sh) script
114
-
- The API key is the `litellm_master_key` value from the generated `vault.yml` file
115
-
116
-
-**APISIX Gateway**: Provide your APISIX Gateway URL and authentication token
124
+
- The API key is the litellm_master_key value from the generated vault.yml file
125
+
126
+
**APISIX Gateway**: Provide your APISIX Gateway URL and authentication token
- Note: APISIX requires the model name in the URL path
117
129
- To generate the APISIX authentication token, use the [generate-token.sh](https://github.com/opea-project/Enterprise-Inference/blob/main/core/scripts/generate-token.sh) script
118
130
- The token is generated using Keycloak client credentials
119
131
120
-
### Local Development Configuration
121
-
122
-
**For Local Testing Only (Optional)**
123
-
124
-
If you're testing with a local inference endpoint using a custom domain (e.g., `api.example.com` mapped to localhost in your hosts file):
125
-
126
-
1. Edit `backend/.env` and set:
127
-
```bash
128
-
LOCAL_URL_ENDPOINT=api.example.com
129
-
```
130
-
(Use the domain name from your INFERENCE_API_ENDPOINT without `https://`)
131
-
132
-
2. This allows Docker containers to resolve your local domain correctly.
133
-
134
-
**Note:** For public domains or cloud-hosted endpoints, leave the default value `not-needed`.
135
-
136
132
### Verify Docker Installation
137
133
138
134
```bash
@@ -159,95 +155,28 @@ cd Enterprise-Inference/sample_solutions/DocSummarization
159
155
160
156
### Set up the Environment
161
157
162
-
This application requires **two `.env` files** for proper configuration:
This application requires an `.env` file in the root directory for proper configuration. Create it using [.env.example](./.env.example) with the commands below:
170
159
171
160
```bash
172
161
cp .env.example .env
173
162
```
174
-
175
-
Then edit `.env` if needed, **OR** create it directly:
176
-
177
-
```bash
178
-
# From the DocSummarization directory
179
-
cat > .env <<EOF
180
-
# Docker Compose Configuration
181
-
LOCAL_URL_ENDPOINT=not-needed
182
-
EOF
183
-
```
184
-
185
-
**Note:** If using a local domain (e.g., `api.example.com` mapped to localhost), replace `not-needed` with your domain name (without `https://`).
186
-
187
-
#### Step 2: Create `backend/.env` File
188
-
189
-
You can either copy from the example file:
190
-
191
-
```bash
192
-
cp backend/.env.example backend/.env
193
-
```
194
-
195
-
Then edit `backend/.env` with your actual credentials, **OR** create it directly:
196
-
197
-
```bash
198
-
cat > backend/.env <<EOF
199
-
# Inference API Configuration
200
-
# INFERENCE_API_ENDPOINT: URL to your inference service (without /v1 suffix)
201
-
# - For GenAI Gateway: https://genai-gateway.example.com
202
-
# - For APISIX Gateway: https://apisix-gateway.example.com/Llama-3.1-8B-Instruct
203
-
# Note: APISIX Gateway requires the model name in the URL path
204
-
#
205
-
# INFERENCE_API_TOKEN: Authentication token/API key for the inference service
206
-
# - For GenAI Gateway: Your GenAI Gateway API key
207
-
# - For APISIX Gateway: Your APISIX authentication token
# Set to false only for dev with self-signed certs
234
-
VERIFY_SSL=true
235
-
EOF
236
-
```
163
+
Then modify it as needed, with special consideration to certain environment variables mentioned below. Read through the .env file for full instructions.
237
164
238
165
**Important Configuration Notes:**
239
166
240
-
-**INFERENCE_API_ENDPOINT**: Your actual inference service URL (replace `https://your-actual-api-endpoint.com`)
241
-
- For APISIX/Keycloak deployments, the model name must be included in the endpoint URL (e.g., `https://apisix-gateway.example.com/Llama-3.1-8B-Instruct`)
167
+
-**INFERENCE_API_ENDPOINT**: Your actual inference service URL (replace `https://api.example.com`)
168
+
- For APISIX/Keycloak deployments, the model name must be included in the endpoint URL (e.g., `https://api.example.com/Llama-3.1-8B-Instruct`)
242
169
-**INFERENCE_API_TOKEN**: Your actual pre-generated authentication token
243
170
-**INFERENCE_MODEL_NAME**: Use the exact model name from your inference service
244
-
- To check available models: `curl https://your-api-endpoint.com/v1/models -H "Authorization: Bearer your-token"`
245
-
-**LOCAL_URL_ENDPOINT**: Only needed if using local domain mapping
171
+
- To check available models: `curl https://api.example.com/v1/models -H "Authorization: Bearer your-token"`
172
+
-**LOCAL_URL_ENDPOINT**: Only needed if using local domain mapping (i.e. `api.example.com` mapped to localhost) for Docker containers to resolve correctly.
173
+
- Use the domain name from INFERENCE_API_ENDPOINT without `https://`
174
+
- For public domains or cloud-hosted endpoints, leave the default value `not-needed`
0 commit comments