optimized filtering

[nvdk]

builds on #20, but has some optimizations wrt to filtering:

• by grouping services on match pattern: this means we only do detection and filtering once for a given match pattern (takes the pattern and options.sendMatchesOnly into account as a grouping key)
• by doing pattern matching before origin filter, since the latter seems more costly

edit: can be tested using nvdk/mu-delta-notifier:feature-optimized-filtering. see https://github.com/lblod/app-lblod-harvester/pull/79/files for an example

[nvdk]

I kept this, but I don't think this situation can occur.

[x-m-el]

Some optimizations are possible. I added comments in adjusted code below, because the suggestions are interlinked with each other, so separate comments could become confusing. Alternatively see e800173 for the commit of these changes

async function informWatchers( changeSets, res, muCallIdTrail, muSessionId ){
  /* This code used to be inside the full loop, but this is not needed. 
   * If the entry has `sendMatchesOnly` set to true, using changedTriples can be avoided (see below).
   * If `sendMatchesOnly` is set to false, the code using `changedTriples` will always need all triples
   * (because there is no extra "filtering").
   * so there is no need to recalculate it every time.
   */
  let allInserts = [];
  let allDeletes = [];
  changeSets.forEach( (change) => {
    allInserts = [...allInserts, ...change.insert];
    allDeletes = [...allDeletes, ...change.delete];
  } );
  const changedTriples = [...allInserts, ...allDeletes];

  // Iterate over each unique match pattern
  for (const matchKey in groupedServices) {
    const firstEntry = groupedServices[matchKey][0];
    // can use first entry since it's part of grouping
    const sendMatchesOnly = firstEntry.options.sendMatchesOnly;
    let maybePatternFilteredChangesets = changeSets;
    if (sendMatchesOnly) {
      maybePatternFilteredChangesets = filterChangesetsOnPattern(changeSets, firstEntry);
      /* Currently, fully empty change sets are send too, but I don't think 
       * this is desired if `sendMatchesOnly` is set to true?
       * Note: can also do this filtering in `filterChangesetsOnPattern` directly
       * (by having an if-case or similar around `filteredChangesets.push(clonedChangeSet);`)
       */
      maybePatternFilteredChangesets = maybePatternFilteredChangesets.filter(
         (change) =>
           change.insert.length > 0 ||
           change.delete.length > 0 ||  change.effectiveInsert.length > 0 ||
           change.effectiveDelete.length > 0
       );
    }

    /* `filterChangesetsOnPattern` already does all the filtering for matches,
     * so if `sendMatchesOnly` is true, we can just check if there are still
     * non-empty change sets left to send. If all are empty (or none are left,
     * in case of filtering them away as stated above), there are no matches with the spec.
     */
    const someTripleMatchedSpec = sendMatchesOnly
      ? maybePatternFilteredChangesets.length > 0 /* makes the assumption that maybePatternFilteredChangesets has no empty change sets. */
      : changedTriples.some((triple) =>
          tripleMatchesSpec(triple, firstEntry.match)
        );
  
    /* The if can be taken out of the forEach loop, to avoid the loop all together
     * if no triples will be send out. Note that `DEBUG_TRIPLE_MATCHES_SPEC` should
     * be moved before the `if` case then. However, this does change the debug output:
     * before it would print a line per match (true/false), now it will do this
     * per service group. The difference is negligible, so this change could be skipped.
     */ 
  
    if( someTripleMatchedSpec ) {
    const matchingServices = groupedServices[matchKey];
      matchingServices.forEach( async (entry) => {
   /// rest of the code
   });
   }

[x-m-el]

unused code

Suggested change

const matchSpec = entry.match;

[nvdk]

I've cherry picked your commit into my PR @x-m-el , thanks for the review and improvements