Skip to content

feat: Add local model worker parameters #2974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shaohuzhang1 merged 2 commits into from
Apr 24, 2025
Merged

feat: Add local model worker parameters #2974

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bdb0775
fix: Ollama maximum output token field
shaohuzhang1 Apr 24, 2025
0986914
feat: Add local model worker parameters
shaohuzhang1 Apr 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion apps/common/management/commands/services/services/local_model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,13 @@ def cmd(self):
os.environ.setdefault('SERVER_NAME', 'local_model')
log_format = '%(h)s %(t)s %(L)ss "%(r)s" %(s)s %(b)s '
bind = f'{CONFIG.get("LOCAL_MODEL_HOST")}:{CONFIG.get("LOCAL_MODEL_PORT")}'
worker = CONFIG.get("LOCAL_MODEL_HOST_WORKER", 1)
cmd = [
'gunicorn', 'smartdoc.wsgi:application',
'-b', bind,
'-k', 'gthread',
'--threads', '200',
'-w', "1",
'-w', worker,
'--max-requests', '10240',
'--max-requests-jitter', '2048',
'--access-logformat', log_format,
Copy link
Copy Markdown
Contributor Author

@shaohuzhang1 shaohuzhang1 Apr 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided Gunicorn command line is mostly correct, but there are a few minor improvements and clarifications that could be made:

  1. Worker Configuration: The worker setting should be more robust. Instead of hardcoding it to 1, you can use a variable that checks if the configuration has been set; otherwise, default it to 1. This ensures that the application works even if the configuration file does not specify this option.

  2. Security Considerations: Ensure that paths in the environment variables like LOCAL_MODEL_HOST, LOCAL_MODEL_PORT, etc., are properly sanitized to avoid security vulnerabilities such as injection attacks if they come from user input.

  3. Logging Format: Double-check the format string for logging (--access-logformat) to ensure it correctly reflects what information needs to be logged by your application. You might want to adjust it based on your specific requirements.

Here’s an improved version of the command with these considerations:

@@ -24,12 +24,14 @@ def cmd(self):
     os.environ.setdefault('SERVER_NAME', 'local_model')
     log_format = '%(h)s %(t)s %(L)ss "%(r)s" %(s)s %(b)s '
     bind = f'{CONFIG.get("LOCAL_MODEL_HOST")}:{CONFIG.get("LOCAL_MODEL_PORT")}'
+    # Set worker using dynamic value or default
+    worker_setting = CONFIG.get("LOCAL_MODEL_HOST_WORKER")
+    if worker_setting is None:
+        worker_setting = 1
    
     cmd = [
         'gunicorn', 'smartdoc.wsgi:application',
         '-b', bind,
         '-k', 'gthread',
         '--threads', '200',
-        '-w', "1",
+        '-w', str(worker_setting),
         '--max-requests', '10240',
         '--max-requests-jitter', '2048',
         '--access-logformat', log_format,

Make sure to replace 'smartdoc.wsgi' with the actual path to your WSGI module if different, depending on how your Flask app is structured.

Expand Down
2 changes: 1 addition & 1 deletion apps/setting/models_provider/impl/ollama_model_provider/credential/llm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ class OllamaLLMModelParams(BaseForm):
_step=0.01,
precision=2)

max_tokens = forms.SliderField(
num_predict = forms.SliderField(
TooltipLabel(_('Output the maximum Tokens'),
_('Specify the maximum number of tokens that the model can generate')),
required=True, default_value=1024,
Copy link
Copy Markdown
Contributor Author

@shaohuzhang1 shaohuzhang1 Apr 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code snippet you provided has a small typo in max_tokens that should be corrected to num_predict. Here's the updated line:

Updated Code Snippet:

num_predict = forms.SliderField(

No other significant issues or optimizations are present in this particular part of the code.

Expand Down
3 changes: 2 additions & 1 deletion apps/smartdoc/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,8 @@ class Config(dict):
'SANDBOX': False,
'LOCAL_MODEL_HOST': '127.0.0.1',
'LOCAL_MODEL_PORT': '11636',
'LOCAL_MODEL_PROTOCOL': "http"
'LOCAL_MODEL_PROTOCOL': "http",
'LOCAL_MODEL_HOST_WORKER': 1

}

Copy link
Copy Markdown
Contributor Author

@shaohuzhang1 shaohuzhang1 Apr 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided code snippet appears to be a section of a configuration dictionary class defined in Python. Here is a brief review for any potential issues:

  1. Typo Correction: There is a typo in 'LOCAL_MODEL_HOST_WORKER'. The value should likely be an integer representing the number of worker processes, but it is currently missing its closing quotes.

  2. Code Formatting: While not strictly necessary, indentation can improve readability for those unfamiliar with Python's syntax.

Suggested corrections:

@@ -93,7 +93,8 @@
     'SANDBOX': False,
     'LOCAL_MODEL_HOST': '127.0.0.1',
     'LOCAL_MODEL_PORT': '11636',
-    'LOCAL_MODEL_PROTOCOL': "http"
+    'LOCAL_MODEL_PROTOCOL': "http",
+    'LOCAL_MODEL_HOST_WORKER': 4  # Assuming you want four workers

These changes address the identified issue and make the code more readable and correct. If there are further optimizations needed, additional context about what these configurations represent (e.g., database settings, API endpoints) would be helpful.

Expand Down