The Current Situation
The MediaWiki parsing service currently tries to retrieve at the very least 5.000 pages from the various CK Wikis—most of which patently do not exist. This places unnecessarily load on their servers and also lengthens our build times, especially when the cache needs to be purged.
Recommended Solution
We could massively lessen the duration of our load to their servers when a cache purge is inevitably needed once more by keeping a list of pages that exist, separate of whether we've downloaded them or not. A remedy to this problem could also reduce build times in the same cases.
So. Download and parse the entire list of pages. Check against this list before firing off a request would be a great idea. Be sure to update this list whenever a New Page edit is received.
The following API Sandbox link may useful: https://ck.uesp.net/wiki/Special:ApiSandbox#action=query&format=json&list=allpages&aplimit=500
The Current Situation
The MediaWiki parsing service currently tries to retrieve at the very least 5.000 pages from the various CK Wikis—most of which patently do not exist. This places unnecessarily load on their servers and also lengthens our build times, especially when the cache needs to be purged.
Recommended Solution
We could massively lessen the duration of our load to their servers when a cache purge is inevitably needed once more by keeping a list of pages that exist, separate of whether we've downloaded them or not. A remedy to this problem could also reduce build times in the same cases.
So. Download and parse the entire list of pages. Check against this list before firing off a request would be a great idea. Be sure to update this list whenever a New Page edit is received.
The following API Sandbox link may useful: https://ck.uesp.net/wiki/Special:ApiSandbox#action=query&format=json&list=allpages&aplimit=500