Uzanto:SchlurcherBot

Emergency bot shutoff button Administrators: Use this button if the bot is malfunctioning (direct link)

Function overview: Convert links from http:// to https://

Rationale:

• WPR: Why we should convert external links to HTTPS wherever possible
• WPR: Should we convert existing Google and Internet Archive links to HTTPS?

Programming language: C#

Source code available: Main C# script: commons:User:SchlurcherBot/LinkChecker

Namespaces: This bot only edits on namespace 0 (Main) and 6 (File)

Function details: The link checking algorithm is as follows:

1. The bot extracts all http-links from the parsed html code of a page.
   • It searches for all href elements and extracts the links.
   • It does not search the wikitext, and thus does not rely on any Regex.
   • This is also to avoid any problems with templates that modify links (like archiving templates).
   • Links that are subsets of other links are filtered out to minimize search and replace errors.
2. The bot checks if the identified http-links also occur in the wikitext, otherwise they are skipped.
3. The bot checks if both the http-link and the corresponding https-link is accessible.
   • This step also uses a blacklist of domains that were previously identified as not accessible.
4. If both links redirect to the same page, the http-link will be replaced by the https-link (the link will not be changed to the redirect page, the original link path will be kept).
5. If both Links are accessible and return a success code (2xx), it will be checked if the content is identical.
   1. If the content is identical, and the link is directly to the host, then the http-link will be replaced by the https-link.
   2. If the content is identical but not the host, it will be checked if the content is identical to the host link, only if the content is different, then the http-link will be replaced by the https-link.
      • This step is added as some hosts return the same content for all their pages (like most domain sellers, some news sites or pages in ongoing maintenance).
   3. If the content is not identical, it will be checked if the content is at least 99.9% identical (calculated via the en:Levenshtein distance).
      • This step is added as most homepages use dynamic IDs for certain elements, like for ad containers to circumvent Ad Blockers.
   4. If the content is at least 99.9% identical, the same host check as before will be performed.
   5. If any of the checked links fails (like Code 404), then nothing will happen.

Source for pages: The bot works on the list of pages identified through the external links SQL dump. The list was scrambled to ensure that subsequent edits are not clustered from a specific area.

Further comments: The bot respects the API etiquette and uses both a user-agent header as well as respects the maxlag parameter.

Status: (CentralAuth)

Approved as global bot (per this request) and thus flagged as bot on all projects that did not opt-out (per this list).

Project	Request	Pages	Edit Description Used	Status
commonswiki	Approved	31'145'089	Fix http to https	Running…
dewiki	Approved	1'888'381	Bot: http → https	Running…
enwiki	Approved	8'570'327	Bot: http → https	Running…
eswiki	Approved	2'191'542	Bot: http → https	Running…
frwiki	Approved	2'970'187	Bot: http → https	Running…
itwiki	Approved	2'359'233	Bot: http → https	Running…
jawiki	Allows global bots	994'375	Bot: http → https	Waiting
plwiki	Approved	1'527'763	Bot: http → https	Waiting
ptwiki	Approved	1'214'889	Bot: http → https	Waiting
ruwiki	Allows global bots	1'797'992	Bot: http → https	Waiting
zhwiki	Allows global bots	1'105'051	Bot: http → https	Waiting
dewikinews	Approved	17'280	Bot: http → https	Done
dewikiquote	Approved	5'673	Bot: http → https	Waiting
dewikisource	Approved	97'284	Bot: http → https	Done
dewikiversity	Approved	9'301	Bot: http → https	Done
dewikivoyage	Approved	19'094	Bot: http → https	Done
dewiktionary	Approved	145'334	Bot: http → https	Done
altwiki	Approved (Village Pump)	864	Bot: http → https	Waiting
arywiki	[1]	8'953	Bot: http → https
bnwiki	[2]	172'842	বট: http থেকে https-তে পরিবর্তন করছে
bnwikibooks	Approved (Village Pump)	745	বট: http থেকে https-তে পরিবর্তন করছে	Waiting
bswiki	[3]	79'281	Bot: http → https
cswikibooks	[4]	601	Bot: http → https
cswikisource	[5]	46'231	Bot: http → https
cswikiversity	[6]	1'946	Bot: http → https
dvwiki	[7]	1'088	Bot: http → https
enwikisource	[8]	117'079	Bot: http → https
enwiktionary	[9]	443'242	Bot: http → https
eswikibooks	[10]	3'417	Bot: http → https
eswikinews	[11]	14'339	Bot: http → https
eswikisource	Approved (Village Pump)	5'826	Bot: http → https	Done
frwikibooks	[12]	7'632	Bot: http → https
frwikinews	[13]	19721	Bot: http → https
frwikisource	Approved	42'309	Bot: http → https	Waiting
frwikiversity	[14]	4'126	Bot: http → https
frwikivoyage	[15]	8'536	Bot: http → https
frwiktionary	[16]	532'493	Bot: http → https
fywiki	Approved (Village Pump)	30'516	Bot: http → https	Waiting
glwiki	[17]	213'696	Bot: http → https
hewikibooks	Approved (Village Pump)	1'660	Bot: http → https	Waiting
hewikisource	Approved	98'820	Bot: http → https	Waiting
hewikivoyage	Allows global bots	2'038	Bot: http → https
hewiktionary	Approved (Temporary) (Village Pump)	6'559	Bot: http → https	Done
hiwiktionary	Approved (Village Pump)	4'970	Bot: http → https	Waiting
hrwikibooks	Approved (Village Pump)	428	Bot: http → https	Waiting
hrwikiquote	Approved (Village Pump)	1'254	Bot: http → https	Waiting
huwikibooks	Approved (Village Pump)	18'488	Bot: http → https	Waiting
huwikisource	Approved (Village Pump)	7'222	Bot: http → https	Waiting
idwiki	[18]	673'383	Bot: http → https
iswiki	Approved	30'026	Bot: http → https	Waiting
iswikisource	Approved (Village Pump)	38	Bot: http → https	Waiting
iswiktionary	Approved (Village Pump)	17'145	Bot: http → https	Waiting
itwikinews	[19]	12'880	Bot: http → https
itwikivoyage	[20]	8'563	Bot: http → https
itwiktionary	[21]	80'610	Bot: http → https
jawikibooks	Approved	1'873	Bot: http → https	Done
jawiktionary	[22]	8'834	Bot: http → https
kshwiki	Approved (Village Pump)	1'364	Bot: http → https	Waiting
lawikisource	Approved (Village Pump)	9'453	Bot: http → https	Waiting
liwikisource	Approved (Village Pump)	1'080	Bot: http → https	Waiting
liwiktionary	[23]	86	Bot: http → https
mnwwiki	[24]	1'010	Bot: http → https
mrwiki	[25]	59'852	Bot: http → https
mrwikisource	Approved (Village Pump)	1'372	Bot: http → https	Waiting
mtwiki	[26]	4'626	Bot: http → https
ndswiki	[27]	24'342	Bot: http → https
nlwikivoyage	[28]	2'385	Bot: http → https
nnwiki	Approved (Village Pump)	144'877	Bot: http → https	Waiting
outreachwiki	[29]	6'136	Bot: http → https
plwikiquote	Approved (Temporarily)	10'443	Bot: http → https	Done
plwiktionary	Approved (Temporarily)	92'252	Bot: http → https	Done
ptwikibooks	Approved (Village Pump)	4'917	Bot: http → https	Waiting
rowiki	[30]	608'015	Bot: http → https
rowiktionary	Approved (Temporarily) (Village Pump)	82'646	Bot: http → https	Done
ruwikinews	Approved	833'044	Bot: http → https	Waiting
ruwikisource	[31]	213'855	Bot: http → https
ruwiktionary	[32]	19'274	Bot: http → https
slwiki	[33]	135'056	Bot: http → https
slwikisource	[34]	16'902	Bot: http → https
sourceswiki	Approved (Village Pump)	26'478	Bot: http → https	Waiting
specieswiki	Approved (Village Pump)	640'405	Bot: http → https	Waiting
srwiki	[35]]	659'443	Bot: http → https
srwikibooks	Approved (Village Pump)	935	Bot: http → https	Waiting
svwikisource	[36]	1'791	Bot: http → https
svwikiversity	Approved (Village Pump)	374	Bot: http → https	Waiting
svwikivoyage	[37]	1'327	Bot: http → https
ukwiki	[38]	1'280'019	Bot: http → https
urwiki	[39]	148'606	Bot: http → https
vecwikisource	Approved (Village Pump)	4'875	Bot: http → https	Waiting
viwiki	[40]	1'350'516	Bot: http → https
wuuwiki	[41]	6'016	Bot: http → https
yuewiktionary	Approval (Village Pump)	401	Bot: http → https	Waiting