Skip to content

Commit c451c13

Browse files
cyphercodesbarjin
andauthored
perf(core): improve RequestList initialization speed on large arrays (apify#3754)
Summary: - Replace `RequestList` `sourcesFunction` result consumption with index iteration instead of repeated `shift()` calls. - Clear processed array slots while preserving request order and add a regression test. Fixes apify#3750. --------- Co-authored-by: cyphercodes <cyphercodes@users.noreply.github.com> Co-authored-by: Jindřich Bär <jindrichbar@gmail.com>
1 parent 361596d commit c451c13

1 file changed

Lines changed: 5 additions & 2 deletions

File tree

packages/core/src/storages/request_list.ts

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -488,9 +488,12 @@ export class RequestList implements IRequestList {
488488
const sourcesFromFunction = await this.sourcesFunction();
489489
const sourcesFromFunctionCount = sourcesFromFunction.length;
490490
for (let i = 0; i < sourcesFromFunctionCount; i++) {
491-
const source = sourcesFromFunction.shift();
492-
this._addRequest(source!);
491+
const source = sourcesFromFunction[i];
492+
delete sourcesFromFunction[i];
493+
this._addRequest(source);
493494
}
495+
496+
sourcesFromFunction.length = 0;
494497
} catch (e) {
495498
const err = e as Error;
496499
throw new Error(`Loading requests with sourcesFunction failed.\nCause: ${err.message}`);

0 commit comments

Comments
 (0)