Skip to content

Commit 36389fb

Browse files
committed
fix: protect C++ and C# keywords from NLTK tokenization splitting (steam-bell-92#907)
1 parent 3c7a02d commit 36389fb

1 file changed

Lines changed: 8 additions & 0 deletions

File tree

utilities/AI-Resume-Analyzer/AI-Resume-Analyzer.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,17 @@ def analyze_resume(resume_text: str) -> dict:
2424

2525
resume = resume_text.lower()
2626

27+
# Pre-process to protect C++, C#, and .NET from NLTK tokenization
28+
PROTECTED = {"c++": "cpplang", "c#": "csharplang", ".net": "dotnetlang"}
29+
for raw, placeholder in PROTECTED.items():
30+
resume = resume.replace(raw, placeholder)
31+
2732
# NLP processing
2833
words = word_tokenize(resume)
2934

35+
# Restore protected tokens
36+
words = [PROTECTED.get(w, w) for w in words]
37+
3038
stop_words = set(stopwords.words('english'))
3139
KEEP_AS_IS = {"c++", "c#", ".net"}
3240

0 commit comments

Comments
 (0)