itkach / mw2slob Goto Github PK
View Code? Open in Web Editor NEWA tool to convert MediaWiki content to dictionaries in slob format
License: GNU General Public License v3.0
A tool to convert MediaWiki content to dictionaries in slob format
License: GNU General Public License v3.0
Relative links inside image maps like this one are not converted:
<div class="noresize" style="height: 329px; width: 300px; "><map name="ImageMap_1_30407246" id="ImageMap_1_30407246">
<area href="/wiki/Frankreich" shape="poly" coords="0,328,16,310,6,296,18,257,11,250,13,236,30,209,37,165,75,118,0,89" alt="Frankreich" title="Frankreich">
<area href="/wiki/Schweiz" shape="poly" coords="2,328,16,312,22,318,43,312,46,319,77,308,82,316,100,313,113,302,111,298,95,300,93,295,98,290,105,286,112,288,117,287,119,293,115,294,115,297,123,297,126,300,130,294,135,302,160,301,192,325,200,324,206,329" alt="Schweiz" title="Schweiz">
<area href="/wiki/%C3%96sterreich" shape="poly" coords="207,328,201,325,217,322,214,317,221,311,235,317,245,328,282,328,278,322,280,311,293,318,297,313,296,318,296,328,207,329" alt="Österreich" title="Österreich">
<area href="/wiki/Bodensee" shape="poly" coords="146,299,160,301,192,324,216,322,213,317,202,312,198,311,194,303,185,300,179,302,169,295,167,290,153,280,151,281,162,289,167,300,164,299,146,290,143,293,148,296" alt="Bodensee" title="Bodensee">
<area href="/wiki/Rheinland-Pfalz" shape="poly" coords="0,90,1,1,88,0,95,12,86,18,93,35,91,37,96,49,94,52,96,54,99,54,98,64,94,69,98,74,87,82,86,102,81,109,79,117,74,118,46,106,25,107,12,102,8,92" alt="Rheinland-Pfalz" title="Rheinland-Pfalz">
<area href="/wiki/Hessen" shape="poly" coords="93,33,87,18,98,9,93,8,90,1,154,1,155,5,159,3,159,7,158,9,160,10,158,18,156,19,154,31,156,34,153,42,151,42,151,45,144,45,141,46,140,49,141,51,136,54,131,58,129,56,131,53,132,49,134,50,135,49,139,46,136,44,130,45,131,43,130,41,126,42,122,41,122,40,120,37,118,30,118,27,108,31,111,38,110,41,107,42,107,41,104,38,101,35,98,32,96,32" alt="Hessen" title="Hessen">
<area href="/wiki/Bayern" shape="poly" coords="157,35,153,31,156,25,154,18,160,17,161,11,158,9,160,7,159,3,154,3,155,1,298,0,296,313,293,314,292,318,287,318,283,316,284,311,279,312,279,319,280,324,281,328,245,328,245,325,240,324,237,316,233,319,226,317,223,316,224,312,221,310,217,312,216,315,215,317,204,312,207,308,218,308,227,300,237,302,244,298,247,299,247,304,251,301,254,296,251,288,247,283,253,280,249,270,253,239,247,214,237,201,269,181,270,169,264,161,267,156,273,164,286,157,279,148,284,123,270,109,264,109,265,96,253,86,255,75,251,64,257,59,252,52,251,44,247,38,244,40,243,45,240,46,233,47,232,33,229,34,227,32,228,29,222,14,221,13,215,17,212,12,208,15,208,17,208,13,209,6,208,3,197,7,193,3,188,6,184,4,179,6,174,6,175,11,176,15,179,15,180,18,185,16,184,21,184,23,181,24,178,22,173,24,173,28,170,32,166,34,160,35,160,34" alt="Freistaat Bayern" title="Freistaat Bayern">
<area href="/wiki/Alb-Donau-Kreis" shape="poly" coords="243,191,263,184,234,166,223,177,201,181,205,198,193,224,214,228,229,215,243,221,243,234,250,232,239,206,232,211,224,202,236,190,242,191" alt="Alb-Donau-Kreis" title="Alb-Donau-Kreis">
<area href="/wiki/Baden-Baden" shape="poly" coords="64,152,77,162,84,153,70,138,68,145,72,151" alt="Baden-Baden" title="Baden-Baden">
<area href="/wiki/Landkreis_Biberach" shape="poly" coords="172,234,178,221,214,231,229,215,243,221,243,233,249,233,251,258,204,257,194,244" alt="Landkreis Biberach" title="Landkreis Biberach">
<area href="/wiki/Landkreis_B%C3%B6blingen" shape="poly" coords="124,181,132,184,162,169,149,139,139,134,129,146,134,154,124,175" alt="Landkreis Böblingen" title="Landkreis Böblingen">
<area href="/wiki/Bodenseekreis" shape="poly" coords="154,284,154,275,172,273,216,302,200,313,170,295" alt="Bodenseekreis" title="Bodenseekreis">
<area href="/wiki/Landkreis_Breisgau-Hochschwarzwald" shape="poly" coords="12,238,13,250,9,288,44,272,72,288,78,275,95,273,60,245,39,247,45,253,45,266,34,259,22,259,27,250,37,245,25,240" alt="Landkreis Breisgau-Hochschwarzwald" title="Landkreis Breisgau-Hochschwarzwald">
<area href="/wiki/Landkreis_Calw" shape="poly" coords="92,165,98,149,93,149,92,138,120,139,129,146,134,153,123,184,116,186,92,165" alt="Landkreis Calw" title="Landkreis Calw">
<area href="/wiki/Landkreis_Emmendingen" shape="poly" coords="13,238,64,249,70,225,22,218" alt="Landkreis Emmendingen" title="Landkreis Emmendingen">
<area href="/wiki/Enzkreis" shape="poly" coords="133,107,104,122,100,140,104,144,118,140,119,137,113,136,113,125,124,126,128,135,121,141,128,146,140,133,137,123,141,122,139,114" alt="Enzkreis" title="Enzkreis">
<area href="/wiki/Landkreis_Esslingen" shape="poly" coords="157,156,163,170,190,181,202,180,194,149,175,147,167,158" alt="Landkreis Esslingen" title="Landkreis Esslingen">
<area href="/wiki/Freiburg_im_Breisgau" shape="poly" coords="28,249,23,259,40,261,41,266,46,266,46,253,36,245,31,250" alt="Freiburg im Breisgau" title="Freiburg im Breisgau">
<area href="/wiki/Landkreis_Freudenstadt" shape="poly" coords="77,203,74,171,91,166,113,184,124,185,130,190,123,202,102,199,98,212,78,202" alt="Landkreis Freudenstadt" title="Landkreis Freudenstadt">
<area href="/wiki/Landkreis_G%C3%B6ppingen" shape="poly" coords="202,180,212,181,235,168,239,153,220,154,216,148,193,151,196,165" alt="Landkreis Göppingen" title="Landkreis Göppingen">
<area href="/wiki/Heidelberg" shape="poly" coords="109,56,113,65,124,64,126,54,117,53" alt="Heidelberg" title="Heidelberg">
<area href="/wiki/Landkreis_Heidenheim" shape="poly" coords="261,184,233,166,239,153,261,145,279,150,288,159,272,164,264,156,263,163,268,176" alt="Landkreis Heidenheim" title="Landkreis Heidenheim">
<area href="/wiki/Landkreis_Heilbronn" shape="poly" coords="130,88,134,109,162,108,163,101,152,87,161,86,174,98,164,101,164,110,192,105,198,98,185,101,179,89,194,71,187,64,130,91" alt="Landkreis Heilbronn" title="Landkreis Heilbronn">
<area href="/wiki/Heilbronn" shape="poly" coords="153,88,163,101,174,99,167,90,160,86" alt="Heilbronn" title="Heilbronn">
<area href="/wiki/Hohenlohekreis" shape="poly" coords="188,100,179,89,196,72,187,64,200,61,202,55,226,62,226,77,201,101,193,97" alt="Hohenlohekreis" title="Hohenlohekreis">
<area href="/wiki/Landkreis_Karlsruhe" shape="poly" coords="98,74,130,89,134,107,103,121,98,140,93,140,80,133,85,125,80,121,78,117,103,122,101,110,86,104,90,82" alt="Landkreis Karlsruhe" title="Landkreis Karlsruhe">
<area href="/wiki/Karlsruhe" shape="poly" coords="79,117,85,104,100,109,103,122,90,120" alt="Karlsruhe" title="Karlsruhe">
<area href="/wiki/Landkreis_Konstanz" shape="poly" coords="111,286,114,276,127,269,152,263,160,274,154,277,154,281,166,299,163,302,155,297,142,290,146,296,135,302,127,291,128,298,121,297,115,297,116,295,120,291,120,286,111,286" alt="Landkreis Konstanz" title="Landkreis Konstanz">
<area href="/wiki/Landkreis_L%C3%B6rrach" shape="poly" coords="11,290,46,272,54,274,45,312,23,319" alt="Landkreis Lörrach" title="Landkreis Lörrach">
<area href="/wiki/Landkreis_Ludwigsburg" shape="poly" coords="138,133,149,140,151,149,158,139,175,134,182,128,182,109,135,110,142,122,137,125" alt="Landkreis Ludwigsburg" title="Landkreis Ludwigsburg">
<area href="/wiki/Main-Tauber-Kreis" shape="poly" coords="176,10,186,23,195,28,194,53,225,63,254,61,247,39,233,47,231,33,225,34,220,14,207,16,207,3,175,9" alt="Main-Tauber-Kreis" title="Main-Tauber-Kreis">
<area href="/wiki/Mannheim" shape="poly" coords="100,58,109,57,107,43,93,33,94,52" alt="Mannheim" title="Mannheim">
<area href="/wiki/Neckar-Odenwald-Kreis" shape="poly" coords="153,42,142,62,152,75,202,63,202,54,193,52,195,27,186,24" alt="Neckar-Odenwald-Kreis" title="Neckar-Odenwald-Kreis">
<area href="/wiki/Ortenaukreis" shape="poly" coords="24,219,82,231,77,166,50,154,31,189" alt="Ortenaukreis" title="Ortenaukreis">
<area href="/wiki/Ostalbkreis" shape="poly" coords="279,151,221,153,206,144,218,139,209,123,212,120,225,127,239,110,263,109,282,123" alt="Ostalbkreis" title="Ostalbkreis">
<area href="/wiki/Pforzheim" shape="poly" coords="120,138,111,135,113,126,122,125,128,134,123,138" alt="Pforzheim" title="Pforzheim">
<area href="/wiki/Landkreis_Rastatt" shape="poly" coords="50,155,75,118,87,125,79,132,98,150,94,165,77,172,51,155,63,153,72,156,75,161,82,159,84,153,80,145,74,138,70,138,67,145,71,149,68,151,63,153" alt="Landkreis Rastatt" title="Landkreis Rastatt">
<area href="/wiki/Landkreis_Ravensburg" shape="poly" coords="211,308,215,298,184,280,186,260,247,257,252,300,225,300" alt="Landkreis Ravensburg" title="Landkreis Ravensburg">
<area href="/wiki/Rems-Murr-Kreis" shape="poly" coords="173,147,167,134,182,126,179,115,192,104,201,107,211,124,217,140,203,147,177,146" alt="Rems-Murr-Kreis" title="Rems-Murr-Kreis">
<area href="/wiki/Landkreis_Reutlingen" shape="poly" coords="158,171,158,202,183,227,192,223,206,193,201,178,170,176,165,171" alt="Landkreis Reutlingen" title="Landkreis Reutlingen">
<area href="/wiki/Rhein-Neckar-Kreis" shape="poly" coords="107,43,109,55,124,52,127,57,123,63,115,66,110,65,111,62,114,58,98,58,98,75,135,92,153,75,142,61,154,44,143,45,134,60,129,58,138,45,117,39,117,28,109,30,112,40" alt="Rhein-Neckar-Kreis" title="Rhein-Neckar-Kreis">
<area href="/wiki/Landkreis_Rottweil" shape="poly" coords="82,201,83,230,101,231,109,245,124,232,114,214,122,201,101,200,98,211" alt="Landkreis Rottweil" title="Landkreis Rottweil">
<area href="/wiki/Landkreis_Schw%C3%A4bisch_Hall" shape="poly" coords="193,103,230,129,241,108,262,109,251,62,225,65,204,98" alt="Landkreis Schwäbisch Hall" title="Landkreis Schwäbisch Hall">
<area href="/wiki/Schwarzwald-Baar-Kreis" shape="poly" coords="105,284,65,250,67,230,99,230,116,257,108,264,113,275" alt="Schwarzwald-Baar-Kreis" title="Schwarzwald-Baar-Kreis">
<area href="/wiki/Landkreis_Sigmaringen" shape="poly" coords="148,262,160,275,185,278,186,260,203,253,173,233,179,220,168,216,156,220,161,225,162,235,147,232,142,241,149,263,157,266" alt="Landkreis Sigmaringen" title="Landkreis Sigmaringen">
<area href="/wiki/Stuttgart" shape="poly" coords="156,139,152,151,169,158,175,146,167,134" alt="Stuttgart" title="Stuttgart">
<area href="/wiki/Landkreis_T%C3%BCbingen" shape="poly" coords="124,196,124,182,157,169,159,203" alt="Landkreis Tübingen" title="Landkreis Tübingen">
<area href="/wiki/Landkreis_Tuttlingen" shape="poly" coords="111,274,150,263,143,239,128,228,108,243,115,259,108,263" alt="Landkreis Tuttlingen" title="Landkreis Tuttlingen">
<area href="/wiki/Ulm" shape="poly" coords="243,191,230,194,227,204,235,212,240,203,236,200" alt="Ulm" title="Ulm">
<area href="/wiki/Landkreis_Waldshut" shape="poly" coords="42,311,55,276,59,275,71,288,77,285,79,276,96,277,103,285,93,301,110,301,107,310,96,313,83,313,73,306,48,317" alt="Landkreis Waldshut" title="Landkreis Waldshut">
<area href="/wiki/Zollernalbkreis" shape="poly" coords="126,197,113,215,124,231,130,227,139,240,143,239,148,233,163,234,163,224,156,218,164,216,167,208,161,202" alt="Zollernalbkreis" title="Zollernalbkreis"></map><img alt="Karte" src="http://upload.wikimedia.org/wikipedia/commons/thumb/9/9e/Baden-W%C3%BCrttemberg_UL_%28town%29.svg/300px-Baden-W%C3%BCrttemberg_UL_%28town%29.svg.png" width="300" height="329" srcset="http://upload.wikimedia.org/wikipedia/commons/thumb/9/9e/Baden-W%C3%BCrttemberg_UL_%28town%29.svg/450px-Baden-W%C3%BCrttemberg_UL_%28town%29.svg.png 1.5x, http://upload.wikimedia.org/wikipedia/commons/thumb/9/9e/Baden-W%C3%BCrttemberg_UL_%28town%29.svg/600px-Baden-W%C3%BCrttemberg_UL_%28town%29.svg.png 2x" data-file-width="1058" data-file-height="1161" usemap="#ImageMap_1_30407246">
<div style="margin-left: 280px; margin-top: -20px; text-align: left;"><a href="http://de.wikipedia.org/wiki/datei:Baden-W%C3%BCrttemberg_UL_(town).svg" title="Über dieses Bild"><img alt="Über dieses Bild" src="http://bits.wikimedia.org/static-1.25wmf12/extensions/ImageMap/desc-20.png" style="border: none;"></a></div>
In some dictionaries images from the network are not loaded.
Dictionary examples are available here:
neolurk-org-20220821.lzma2.slob 364.8MiB
ru.gta.fandom.com-20220822.lzma2.slob 17.7MiB
Commands used to create dictionaries:
mwscrape -c http://admin:admin@localhost:5984 gta.fandom.com/ru --site-path=/ --speed 3
mw2slob scrape http://admin:admin@localhost:5984/gta-fandom-com/ru --no-math --ensure-ext-image-urls -a sklart
mwscrape -c http://admin:admin@localhost:5984 http://neolurk.org/ --speed 5
mw2slob scrape http://admin:admin@localhost:5984/neolurk-org -a sklart -f common wiki
Please help me with this problem. Thanks
Hallo
on the main page under Installation the link for slob is wrong. It is github.org instead of github.com so it goes 404.
Thank.
frank
Pastebin of the error (hit ctrl+c early so you wouldn't have to see miles of the same thing), which was run in a virtualenv dedicated to mwscrape2slob and the new format: https://ptpb.pw/QQle.txt
Possibly related issues: aarddict/tools#39 (comment)
How to reproduce:
mwscrape yugioh.wikia.com --site-path=/ [email protected]
.cardtablespanrow:contains("sets")
.cardtablespanrow:contains("Card appearances")
tr:contains("Anime") + tr
tr:contains("Anime")
tr:contains("Video game statuses") + tr
tr:contains("Video game statuses")
table.collapsible.hslit:contains("pages")
mwscrape2slob http://127.0.0.1:5984/yugioh-wikia-com -f ${yourfilterfilenamehere}
Expected results: No errors and a *.slob file filled with the cards.
Actual results: See the pastebin.
KaTeX is a lot faster than MathJax. The switch may be fairly easy.
Hi,
while comparing two revisions of mwscrape2slob/mw2slob I found that slob archives created with mw2slob contain broken tables, only the right half of the tables are visible in aard 0.53:
working revision: https://github.com/itkach/mw2slob/tree/e01d3e92f0a372ebd0f57390e437a28f9d3c0438 - screenshot:
broken revision: https://github.com/itkach/mw2slob/tree/0358818432fc1404ce2ef7d35da7a59d970b77ef - screenshot:
The slob archives were created from a local CouchDB server using these options:
-f common wiki -c lzma2 -ll en,nl,pl
When dumping a database of a random MediaWiki site created with mwscrape
(apart from wikipedia), the external image URLs generate will be broken, even with the --ensure-ext-image-urls
flag enabled - thus, "Load remote content" on Aard2 clients won't work with the images included in those resulting slob files.
An example:
source: https://wiki.archlinux.org/title/File:Merge-arrows-2.png
target: https://wiki.archlinux.org/images/c/c9/Merge-arrows-2.png
broken result in generated slob: https://wiki.archlinux.org/title//c/c9/Merge-arrows-2.png
(In some other cases, the generated links are working fine, but "Load remote content" still won't work properly)
I suspect that it's related to
Lines 212 to 213 in 078ac83
I've created a .slob with this tool which reproduces this problem: archwiki-20221126.slob not this one but this: archwiki_image_link_test_not_working.slob
Would it be possible to generate TOCs in the articles from the enterprise dumps? They are not included in these dumps. Or should the dumps themselves be fixed?
Cool project, thanks! : )
It just displays small one line placeholder box containing icon and text '{\displaystyle '.
correct should be 'a+b i + c j + d k'
(i,j,k are in bold). Formula is from the top of article https://en.wikipedia.org/wiki/Quaternion:
"Quaternions are generally represented in the form 'a+b i + c j + d k' "
strangely it renders many other more complex formulas flawlessly.
DETAILS:
running under win 10 using command line
Java\bin\java.exe -Dslobber.host="localhost" -Dslobber.port=80 -jar aard2-web-0.8.jar ..*.slob
using portable java
java -version
java version "1.8.0_361"
Java(TM) SE Runtime Environment (build 1.8.0_361-b09)
Java HotSpot(TM) Client VM (build 25.361-b09, mixed mode)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.