Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Pull request overview
This PR improves job deduplication and keyword handling across backend scraping and frontend filtering, adding support for merged keywords/sources and more robust matching (case/diacritic-insensitive).
Changes:
- Backend: deduplicate equivalent jobs using normalized title/company/location or normalized URL; merge keywords/sources when duplicates are found.
- Frontend: deduplicate jobs client-side and allow filtering/search over combined keywords with normalized text matching.
- Update unit tests to cover cross-platform deduplication and merged keyword behavior; adjust default
environment.jsonkeywords.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| frontend/tests/unit/utils/useJobsFiltering.test.ts | Adds test ensuring equivalent jobs are deduped and keywords are combined. |
| frontend/src/types/jobs.ts | Extends Job type with keywords and sources arrays. |
| frontend/src/hooks/useJobsFiltering.ts | Adds normalization, deduplication, keyword splitting, and improved filtering/search behavior. |
| backend/tests/unit/services/pipeline/scrapeAllSources.test.js | Adds test for deduplication across platforms by title/company/location. |
| backend/src/pipeline/scrapeAllSources.js | Implements normalized dedup keys and improved merge logic for keywords/sources. |
| backend/src/db/environment.json | Changes the persisted default KEYWORDS list. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const mergedKeywords = [...new Set([...splitJobKeywords(existing), ...splitJobKeywords(job)])]; | ||
| const mergedSources = [ | ||
| ...new Set( | ||
| [...(existing.sources ?? []), ...(job.sources ?? []), existing.source, job.source] | ||
| .map((source) => String(source || "").trim()) | ||
| .filter(Boolean), | ||
| ), | ||
| ]; |
There was a problem hiding this comment.
Ao fazer merge de fontes, mergedSources inclui também existing.source e job.source. Como source é definido como mergedSources.join(", "), em merges subsequentes esse valor vira uma string com vírgulas (ex.: "LinkedIn, Greenhouse") e pode acabar entrando como um item adicional em sources, gerando duplicatas e um source cada vez mais poluído (principalmente quando a mesma vaga aparece 3+ vezes). Sugestão: ao montar mergedSources, evite re-inserir existing.source/job.source quando já existe sources, ou faça split de source por vírgula e normalize cada item antes de inserir no Set; e derive source apenas de sources.
| jobs.flatMap((job) => | ||
| [ | ||
| ...(Array.isArray(job.sources) ? job.sources : []), | ||
| job.source, | ||
| ] | ||
| .filter(Boolean) | ||
| .map((source) => String(source).trim()), | ||
| ), |
There was a problem hiding this comment.
getMergedSources agrega job.source diretamente. Depois que um job é deduplicado, mergeKeywords passa a setar source como mergedSources.join(", "); em um terceiro merge, esse source já contém vírgulas (ex.: "LinkedIn, Greenhouse") e vira um novo item no Set, criando sources com entradas compostas e duplicadas, e um source final com repetições. Sugestão: ao coletar fontes, normalize job.source dividindo por vírgula (e trim) antes de inserir, ou então ignore job.source quando job.sources existir e calcule source apenas a partir de sources.
| jobs.flatMap((job) => | |
| [ | |
| ...(Array.isArray(job.sources) ? job.sources : []), | |
| job.source, | |
| ] | |
| .filter(Boolean) | |
| .map((source) => String(source).trim()), | |
| ), | |
| jobs.flatMap((job) => { | |
| const normalizedSources = Array.isArray(job.sources) | |
| ? job.sources.flatMap((source) => String(source || "").split(",")) | |
| : []; | |
| const normalizedSource = String(job.source || "").split(","); | |
| return [...normalizedSources, ...normalizedSource] | |
| .map((source) => String(source).trim()) | |
| .filter(Boolean); | |
| }), |
| { | ||
| "KEYWORDS": [ | ||
| "Java", | ||
| "Spring", | ||
| "RabbitMQ", | ||
| "Docker" | ||
| "Java" | ||
| ] |
There was a problem hiding this comment.
O arquivo versionado environment.json passa a inicializar KEYWORDS com apenas "Java". Como parseKeywords() prefere o conteúdo desse arquivo quando existe e não está vazio, isso muda o comportamento padrão do scraper/app para buscar somente uma keyword (em vez do fallback DEFAULT_KEYWORDS em backend/src/config.js). Se a intenção não é limitar o padrão, considere manter a lista anterior ou deixar KEYWORDS vazio/ausente para permitir o fallback.
No description provided.