Skip to content

perf: Refine the Model Manager code #3089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shaohuzhang1 merged 1 commit into from
May 15, 2025
+17 −16
Merged

perf: Refine the Model Manager code #3089

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 17 additions & 16 deletions apps/common/config/embedding_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

from common.cache.mem_cache import MemCache

lock = threading.Lock()
_lock = threading.Lock()


class ModelManage:
Expand All @@ -20,26 +20,27 @@ class ModelManage:

@staticmethod
def get_model(_id, get_model):
# 获取锁
lock.acquire()
try:
model_instance = ModelManage.cache.get(_id)
if model_instance is None or not model_instance.is_cache_model():
model_instance = ModelManage.cache.get(_id)
if model_instance is None:
with _lock:
model_instance = get_model(_id)
ModelManage.cache.set(_id, model_instance, timeout=60 * 30)
ModelManage.cache.set(_id, model_instance, timeout=60 * 60 * 8)
ModelManage.clear_timeout_cache()
return model_instance
else:
if model_instance.is_cache_model():
ModelManage.cache.touch(_id, timeout=60 * 60 * 8)
return model_instance
else:
model_instance = get_model(_id)
ModelManage.cache.set(_id, model_instance, timeout=60 * 60 * 8)
return model_instance
# 续期
ModelManage.cache.touch(_id, timeout=60 * 30)
ModelManage.clear_timeout_cache()
return model_instance
finally:
# 释放锁
lock.release()

@staticmethod
def clear_timeout_cache():
if time.time() - ModelManage.up_clear_time > 60:
ModelManage.cache.clear_timeout_data()
if time.time() - ModelManage.up_clear_time > 60 * 60:
threading.Thread(target=lambda: ModelManage.cache.clear_timeout_data()).start()
ModelManage.up_clear_time = time.time()

@staticmethod
def delete_key(_id):
Copy link
Copy Markdown
Contributor Author

@shaohuzhang1 shaohuzhang1 May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are some optimizations and improvements:

  1. Use try-with-resources style context managers to automatically release locks:
 with _lock:

  1. Simplify cache set logic by directly using the lock within it:

    def get_model(self, _id, get_model):
    model_instance = self.cache.get(_id)
    if not model_instance or not model_instance.is_cache_model():
        with _lock:
            model_instance = get_model(_id)
            self.cache.set(
                _id,
                model_instance,
                timeout=60 * 3600,  # Increased from 30 minutes to an hour
            )
            self.clear_timeout_cache()
            return model_instance
    elif model_instance.is_cache_model():
        self.cache.touch(_id, timeout=60 * 3600)  # Increased from 30 minutes to an hour
        return model_instance
    else:
        self.delete_key(_id)
        return get_model(_id)

  2. Move the cache clearing logic into its own method called clear_expired_keys, so that you can call it explicitly when needed without modifying get_model.

  3. Ensure that ModelManage.up_clear_time is properly initialized before being used.

  4. The time.sleep(3) in your original code was removed because there's no reason for sleeping between each request. If sleep is required somewhere else, ensure that it doesn't interfere with performance or functionality of this critical service.

This optimized version maintains the necessary logic while improving readability and efficiency. Always test thoroughly after making changes to ensure stability!

Expand Down