Skip to content

fix(grouper): identify events similarity only when titles are same #410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
slaveeks wants to merge 2 commits into from
Closed

fix(grouper): identify events similarity only when titles are same #410

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e9b6f5f
fix(grouper): identify events similarity only when titles are same
slaveeks May 21, 2025
97d842a
fix lint
slaveeks May 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions workers/grouper/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
import { MS_IN_SEC } from '../../../lib/utils/consts';
import DataFilter from './data-filter';
import RedisHelper from './redisHelper';
import levenshtein from 'js-levenshtein';

Check warning on line 17 in workers/grouper/src/index.ts

View workflow job for this annotation

GitHub Actions / ESlint 

'levenshtein' is defined but never used

Check failure on line 17 in workers/grouper/src/index.ts

View workflow job for this annotation

GitHub Actions / ESlint 

'levenshtein' is defined but never used

Check warning on line 17 in workers/grouper/src/index.ts

View workflow job for this annotation

GitHub Actions / ESlint 

All imports in import declaration are unused
import { computeDelta } from './utils/repetitionDiff';
import TimeMs from '../../../lib/utils/time';

Expand Down Expand Up @@ -244,22 +244,24 @@
*/
private async findSimilarEvent(projectId: string, event: EventDataAccepted<EventAddons>): Promise<GroupedEventDBScheme | undefined> {
const eventsCountToCompare = 60;
const diffTreshold = 0.35;

Check warning on line 247 in workers/grouper/src/index.ts

View workflow job for this annotation

GitHub Actions / ESlint 

'diffTreshold' is assigned a value but never used

Check failure on line 247 in workers/grouper/src/index.ts

View workflow job for this annotation

GitHub Actions / ESlint 

'diffTreshold' is assigned a value but never used

Check warning on line 247 in workers/grouper/src/index.ts

View workflow job for this annotation

GitHub Actions / ESlint 

Variable 'diffTreshold' is declared but its value is never read. Allowed unused names must match /^_/

const lastUniqueEvents = await this.findLastEvents(projectId, eventsCountToCompare);
Copy link
Copy Markdown
Member

@neSpecc neSpecc May 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably, we can find event by title using mongo. We dont need to get 60 events for that.


/**
* First try to find by Levenshtein distance
* First try to find similar event by title
*/
const similarByLevenshtein = lastUniqueEvents.filter(prevEvent => {
const distance = levenshtein(event.title, prevEvent.payload.title);
const threshold = event.title.length * diffTreshold;
const similarByTitle = lastUniqueEvents.filter(prevEvent => {
// const distance = levenshtein(event.title, prevEvent.payload.title);

return distance < threshold;
// const threshold = event.title.length * diffTreshold;

// return distance < threshold;
return event.title.toLowerCase() === prevEvent.payload.title.toLowerCase();
}).pop();
Comment on lines +253 to 260
Copy link
Copy Markdown
Member

@neSpecc neSpecc May 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets just comment out grouping by Levenshtein and implement grouping by title below.


if (similarByLevenshtein) {
return similarByLevenshtein;
if (similarByTitle) {
return similarByTitle;
}

/**
Expand Down
18 changes: 9 additions & 9 deletions workers/grouper/tests/index.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -472,17 +472,17 @@
});

describe('Grouping', () => {
test('should group events with partially different titles', async () => {
await worker.handle(generateTask({ title: 'Some error (but not filly identical) example' }));
await worker.handle(generateTask({ title: 'Some error (yes, it is not the identical) example' }));
await worker.handle(generateTask({ title: 'Some error (and it is not identical) example' }));
// test('should group events with partially different titles', async () => {
// await worker.handle(generateTask({ title: 'Some error (but not filly identical) example' }));
// await worker.handle(generateTask({ title: 'Some error (yes, it is not the identical) example' }));
// await worker.handle(generateTask({ title: 'Some error (and it is not identical) example' }));

const originalEvent = await eventsCollection.findOne({});
// const originalEvent = await eventsCollection.findOne({});

expect((await repetitionsCollection.find({
groupHash: originalEvent.groupHash,
}).toArray()).length).toBe(2);
});
// expect((await repetitionsCollection.find({
// groupHash: originalEvent.groupHash,
// }).toArray()).length).toBe(2);
// });

Copy link
Copy Markdown
Member

@neSpecc neSpecc May 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add test for grouping by title

describe('Pattern matching', () => {
beforeEach(() => {
Expand All @@ -490,8 +490,8 @@
});

test('should group events with titles matching one pattern', async () => {
jest.spyOn(GrouperWorker.prototype as any, 'getProjectPatterns').mockResolvedValue([ 'New error .*' ]);

Check warning on line 493 in workers/grouper/tests/index.test.ts

View workflow job for this annotation

GitHub Actions / ESlint 

Unexpected any. Specify a different type
const findMatchingPatternSpy = jest.spyOn(GrouperWorker.prototype as any, 'findMatchingPattern');

Check warning on line 494 in workers/grouper/tests/index.test.ts

View workflow job for this annotation

GitHub Actions / ESlint 

Unexpected any. Specify a different type

await worker.handle(generateTask({ title: 'New error 0000000000000000' }));
await worker.handle(generateTask({ title: 'New error 1111111111111111' }));
Expand Down
Loading