memory error

[m4riachi]

Hi and thank you for this amazing tool

I want to crawl a big website like 1,000,000 link, when i execute the cmd and after like 2 days i get an error about memory limit.

If you can add the option to create a file after every 50,000 crawled not until the end.

Or if is there any other solution that can help?

Thank you

[lgraubner]

#53 might be something that could solve this.