Properly import the wikipedia spam blocklist

[GexEnterTheGecko]

last regex was awful, now it actually works, when the evaluator encounters \/, it doesn't spectacularly cause bad domain names

[NotaInutilis]

Seeing the comments, I'll wait until tomorrow before merging that :D

[GexEnterTheGecko]

"it shouldn't cause issue" and then i look at the CI and there's x.comrttereschenko in there! 1d637f4#diff-63a71dc442383a5564cfaeab39744758e49dd6a12c569a85091256b21a9017bb

[NotaInutilis]

wat

[GexEnterTheGecko]

leads to http://форум-казино.рф/

[NotaInutilis]

wat²

[GexEnterTheGecko]

that was in the file, i just re-pasted the thing without running the CI, but basically this is something that leads to a cyrillic casino website thing

[GexEnterTheGecko]

aka: this URL is https://казино-форум.рф/

[NotaInutilis]

Hope you don't mind me adding a couple more very useful comments. There are at least 4 weird lines like that, not sure they are from the sedding or another part of the copying. It's Sunday evening, I refuse to think.

[GexEnterTheGecko]

no worries! most of these lines are a thing called https://en.wikipedia.org/wiki/Percent-encoding !

[NotaInutilis]

Ah yeah, I don't encounter these ones that often. I think there are a couple in this list, but they also work in their original character encoding so I'm not quite sure if they should be normalized in one way or the other.

[NotaInutilis]

ALSO GUESS WHAT THERE'S ANOTHER ISSUE #53

[GexEnterTheGecko]

note to self: don't drink tap water at jerry garcia's

[GexEnterTheGecko]

@NotaInutilis i've added it to the allowlist, please check my commit for correctness when you have the time

[NotaInutilis]

The allowlist is only used in the importation process, not in the list generation. I think I coded that quite roughly so it should clear all files in the _imported folder, I'll have to check that again.