itkach / mw2slob Goto Github PK
View Code? Open in Web Editor NEWA tool to convert MediaWiki content to dictionaries in slob format
License: GNU General Public License v3.0
mw2slob's People
Contributors
Stargazers
Watchers
mw2slob's Issues
Links inside image maps are not converted
Relative links inside image maps like this one are not converted:
<div class="noresize" style="height: 329px; width: 300px; "><map name="ImageMap_1_30407246" id="ImageMap_1_30407246">
<area href="/wiki/Frankreich" shape="poly" coords="0,328,16,310,6,296,18,257,11,250,13,236,30,209,37,165,75,118,0,89" alt="Frankreich" title="Frankreich">
<area href="/wiki/Schweiz" shape="poly" coords="2,328,16,312,22,318,43,312,46,319,77,308,82,316,100,313,113,302,111,298,95,300,93,295,98,290,105,286,112,288,117,287,119,293,115,294,115,297,123,297,126,300,130,294,135,302,160,301,192,325,200,324,206,329" alt="Schweiz" title="Schweiz">
<area href="/wiki/%C3%96sterreich" shape="poly" coords="207,328,201,325,217,322,214,317,221,311,235,317,245,328,282,328,278,322,280,311,293,318,297,313,296,318,296,328,207,329" alt="Österreich" title="Österreich">
<area href="/wiki/Bodensee" shape="poly" coords="146,299,160,301,192,324,216,322,213,317,202,312,198,311,194,303,185,300,179,302,169,295,167,290,153,280,151,281,162,289,167,300,164,299,146,290,143,293,148,296" alt="Bodensee" title="Bodensee">
<area href="/wiki/Rheinland-Pfalz" shape="poly" coords="0,90,1,1,88,0,95,12,86,18,93,35,91,37,96,49,94,52,96,54,99,54,98,64,94,69,98,74,87,82,86,102,81,109,79,117,74,118,46,106,25,107,12,102,8,92" alt="Rheinland-Pfalz" title="Rheinland-Pfalz">
<area href="/wiki/Hessen" shape="poly" coords="93,33,87,18,98,9,93,8,90,1,154,1,155,5,159,3,159,7,158,9,160,10,158,18,156,19,154,31,156,34,153,42,151,42,151,45,144,45,141,46,140,49,141,51,136,54,131,58,129,56,131,53,132,49,134,50,135,49,139,46,136,44,130,45,131,43,130,41,126,42,122,41,122,40,120,37,118,30,118,27,108,31,111,38,110,41,107,42,107,41,104,38,101,35,98,32,96,32" alt="Hessen" title="Hessen">
<area href="/wiki/Bayern" shape="poly" coords="157,35,153,31,156,25,154,18,160,17,161,11,158,9,160,7,159,3,154,3,155,1,298,0,296,313,293,314,292,318,287,318,283,316,284,311,279,312,279,319,280,324,281,328,245,328,245,325,240,324,237,316,233,319,226,317,223,316,224,312,221,310,217,312,216,315,215,317,204,312,207,308,218,308,227,300,237,302,244,298,247,299,247,304,251,301,254,296,251,288,247,283,253,280,249,270,253,239,247,214,237,201,269,181,270,169,264,161,267,156,273,164,286,157,279,148,284,123,270,109,264,109,265,96,253,86,255,75,251,64,257,59,252,52,251,44,247,38,244,40,243,45,240,46,233,47,232,33,229,34,227,32,228,29,222,14,221,13,215,17,212,12,208,15,208,17,208,13,209,6,208,3,197,7,193,3,188,6,184,4,179,6,174,6,175,11,176,15,179,15,180,18,185,16,184,21,184,23,181,24,178,22,173,24,173,28,170,32,166,34,160,35,160,34" alt="Freistaat Bayern" title="Freistaat Bayern">
<area href="/wiki/Alb-Donau-Kreis" shape="poly" coords="243,191,263,184,234,166,223,177,201,181,205,198,193,224,214,228,229,215,243,221,243,234,250,232,239,206,232,211,224,202,236,190,242,191" alt="Alb-Donau-Kreis" title="Alb-Donau-Kreis">
<area href="/wiki/Baden-Baden" shape="poly" coords="64,152,77,162,84,153,70,138,68,145,72,151" alt="Baden-Baden" title="Baden-Baden">
<area href="/wiki/Landkreis_Biberach" shape="poly" coords="172,234,178,221,214,231,229,215,243,221,243,233,249,233,251,258,204,257,194,244" alt="Landkreis Biberach" title="Landkreis Biberach">
<area href="/wiki/Landkreis_B%C3%B6blingen" shape="poly" coords="124,181,132,184,162,169,149,139,139,134,129,146,134,154,124,175" alt="Landkreis Böblingen" title="Landkreis Böblingen">
<area href="/wiki/Bodenseekreis" shape="poly" coords="154,284,154,275,172,273,216,302,200,313,170,295" alt="Bodenseekreis" title="Bodenseekreis">
<area href="/wiki/Landkreis_Breisgau-Hochschwarzwald" shape="poly" coords="12,238,13,250,9,288,44,272,72,288,78,275,95,273,60,245,39,247,45,253,45,266,34,259,22,259,27,250,37,245,25,240" alt="Landkreis Breisgau-Hochschwarzwald" title="Landkreis Breisgau-Hochschwarzwald">
<area href="/wiki/Landkreis_Calw" shape="poly" coords="92,165,98,149,93,149,92,138,120,139,129,146,134,153,123,184,116,186,92,165" alt="Landkreis Calw" title="Landkreis Calw">
<area href="/wiki/Landkreis_Emmendingen" shape="poly" coords="13,238,64,249,70,225,22,218" alt="Landkreis Emmendingen" title="Landkreis Emmendingen">
<area href="/wiki/Enzkreis" shape="poly" coords="133,107,104,122,100,140,104,144,118,140,119,137,113,136,113,125,124,126,128,135,121,141,128,146,140,133,137,123,141,122,139,114" alt="Enzkreis" title="Enzkreis">
<area href="/wiki/Landkreis_Esslingen" shape="poly" coords="157,156,163,170,190,181,202,180,194,149,175,147,167,158" alt="Landkreis Esslingen" title="Landkreis Esslingen">
<area href="/wiki/Freiburg_im_Breisgau" shape="poly" coords="28,249,23,259,40,261,41,266,46,266,46,253,36,245,31,250" alt="Freiburg im Breisgau" title="Freiburg im Breisgau">
<area href="/wiki/Landkreis_Freudenstadt" shape="poly" coords="77,203,74,171,91,166,113,184,124,185,130,190,123,202,102,199,98,212,78,202" alt="Landkreis Freudenstadt" title="Landkreis Freudenstadt">
<area href="/wiki/Landkreis_G%C3%B6ppingen" shape="poly" coords="202,180,212,181,235,168,239,153,220,154,216,148,193,151,196,165" alt="Landkreis Göppingen" title="Landkreis Göppingen">
<area href="/wiki/Heidelberg" shape="poly" coords="109,56,113,65,124,64,126,54,117,53" alt="Heidelberg" title="Heidelberg">
<area href="/wiki/Landkreis_Heidenheim" shape="poly" coords="261,184,233,166,239,153,261,145,279,150,288,159,272,164,264,156,263,163,268,176" alt="Landkreis Heidenheim" title="Landkreis Heidenheim">
<area href="/wiki/Landkreis_Heilbronn" shape="poly" coords="130,88,134,109,162,108,163,101,152,87,161,86,174,98,164,101,164,110,192,105,198,98,185,101,179,89,194,71,187,64,130,91" alt="Landkreis Heilbronn" title="Landkreis Heilbronn">
<area href="/wiki/Heilbronn" shape="poly" coords="153,88,163,101,174,99,167,90,160,86" alt="Heilbronn" title="Heilbronn">
<area href="/wiki/Hohenlohekreis" shape="poly" coords="188,100,179,89,196,72,187,64,200,61,202,55,226,62,226,77,201,101,193,97" alt="Hohenlohekreis" title="Hohenlohekreis">
<area href="/wiki/Landkreis_Karlsruhe" shape="poly" coords="98,74,130,89,134,107,103,121,98,140,93,140,80,133,85,125,80,121,78,117,103,122,101,110,86,104,90,82" alt="Landkreis Karlsruhe" title="Landkreis Karlsruhe">
<area href="/wiki/Karlsruhe" shape="poly" coords="79,117,85,104,100,109,103,122,90,120" alt="Karlsruhe" title="Karlsruhe">
<area href="/wiki/Landkreis_Konstanz" shape="poly" coords="111,286,114,276,127,269,152,263,160,274,154,277,154,281,166,299,163,302,155,297,142,290,146,296,135,302,127,291,128,298,121,297,115,297,116,295,120,291,120,286,111,286" alt="Landkreis Konstanz" title="Landkreis Konstanz">
<area href="/wiki/Landkreis_L%C3%B6rrach" shape="poly" coords="11,290,46,272,54,274,45,312,23,319" alt="Landkreis Lörrach" title="Landkreis Lörrach">
<area href="/wiki/Landkreis_Ludwigsburg" shape="poly" coords="138,133,149,140,151,149,158,139,175,134,182,128,182,109,135,110,142,122,137,125" alt="Landkreis Ludwigsburg" title="Landkreis Ludwigsburg">
<area href="/wiki/Main-Tauber-Kreis" shape="poly" coords="176,10,186,23,195,28,194,53,225,63,254,61,247,39,233,47,231,33,225,34,220,14,207,16,207,3,175,9" alt="Main-Tauber-Kreis" title="Main-Tauber-Kreis">
<area href="/wiki/Mannheim" shape="poly" coords="100,58,109,57,107,43,93,33,94,52" alt="Mannheim" title="Mannheim">
<area href="/wiki/Neckar-Odenwald-Kreis" shape="poly" coords="153,42,142,62,152,75,202,63,202,54,193,52,195,27,186,24" alt="Neckar-Odenwald-Kreis" title="Neckar-Odenwald-Kreis">
<area href="/wiki/Ortenaukreis" shape="poly" coords="24,219,82,231,77,166,50,154,31,189" alt="Ortenaukreis" title="Ortenaukreis">
<area href="/wiki/Ostalbkreis" shape="poly" coords="279,151,221,153,206,144,218,139,209,123,212,120,225,127,239,110,263,109,282,123" alt="Ostalbkreis" title="Ostalbkreis">
<area href="/wiki/Pforzheim" shape="poly" coords="120,138,111,135,113,126,122,125,128,134,123,138" alt="Pforzheim" title="Pforzheim">
<area href="/wiki/Landkreis_Rastatt" shape="poly" coords="50,155,75,118,87,125,79,132,98,150,94,165,77,172,51,155,63,153,72,156,75,161,82,159,84,153,80,145,74,138,70,138,67,145,71,149,68,151,63,153" alt="Landkreis Rastatt" title="Landkreis Rastatt">
<area href="/wiki/Landkreis_Ravensburg" shape="poly" coords="211,308,215,298,184,280,186,260,247,257,252,300,225,300" alt="Landkreis Ravensburg" title="Landkreis Ravensburg">
<area href="/wiki/Rems-Murr-Kreis" shape="poly" coords="173,147,167,134,182,126,179,115,192,104,201,107,211,124,217,140,203,147,177,146" alt="Rems-Murr-Kreis" title="Rems-Murr-Kreis">
<area href="/wiki/Landkreis_Reutlingen" shape="poly" coords="158,171,158,202,183,227,192,223,206,193,201,178,170,176,165,171" alt="Landkreis Reutlingen" title="Landkreis Reutlingen">
<area href="/wiki/Rhein-Neckar-Kreis" shape="poly" coords="107,43,109,55,124,52,127,57,123,63,115,66,110,65,111,62,114,58,98,58,98,75,135,92,153,75,142,61,154,44,143,45,134,60,129,58,138,45,117,39,117,28,109,30,112,40" alt="Rhein-Neckar-Kreis" title="Rhein-Neckar-Kreis">
<area href="/wiki/Landkreis_Rottweil" shape="poly" coords="82,201,83,230,101,231,109,245,124,232,114,214,122,201,101,200,98,211" alt="Landkreis Rottweil" title="Landkreis Rottweil">
<area href="/wiki/Landkreis_Schw%C3%A4bisch_Hall" shape="poly" coords="193,103,230,129,241,108,262,109,251,62,225,65,204,98" alt="Landkreis Schwäbisch Hall" title="Landkreis Schwäbisch Hall">
<area href="/wiki/Schwarzwald-Baar-Kreis" shape="poly" coords="105,284,65,250,67,230,99,230,116,257,108,264,113,275" alt="Schwarzwald-Baar-Kreis" title="Schwarzwald-Baar-Kreis">
<area href="/wiki/Landkreis_Sigmaringen" shape="poly" coords="148,262,160,275,185,278,186,260,203,253,173,233,179,220,168,216,156,220,161,225,162,235,147,232,142,241,149,263,157,266" alt="Landkreis Sigmaringen" title="Landkreis Sigmaringen">
<area href="/wiki/Stuttgart" shape="poly" coords="156,139,152,151,169,158,175,146,167,134" alt="Stuttgart" title="Stuttgart">
<area href="/wiki/Landkreis_T%C3%BCbingen" shape="poly" coords="124,196,124,182,157,169,159,203" alt="Landkreis Tübingen" title="Landkreis Tübingen">
<area href="/wiki/Landkreis_Tuttlingen" shape="poly" coords="111,274,150,263,143,239,128,228,108,243,115,259,108,263" alt="Landkreis Tuttlingen" title="Landkreis Tuttlingen">
<area href="/wiki/Ulm" shape="poly" coords="243,191,230,194,227,204,235,212,240,203,236,200" alt="Ulm" title="Ulm">
<area href="/wiki/Landkreis_Waldshut" shape="poly" coords="42,311,55,276,59,275,71,288,77,285,79,276,96,277,103,285,93,301,110,301,107,310,96,313,83,313,73,306,48,317" alt="Landkreis Waldshut" title="Landkreis Waldshut">
<area href="/wiki/Zollernalbkreis" shape="poly" coords="126,197,113,215,124,231,130,227,139,240,143,239,148,233,163,234,163,224,156,218,164,216,167,208,161,202" alt="Zollernalbkreis" title="Zollernalbkreis"></map><img alt="Karte" src="http://upload.wikimedia.org/wikipedia/commons/thumb/9/9e/Baden-W%C3%BCrttemberg_UL_%28town%29.svg/300px-Baden-W%C3%BCrttemberg_UL_%28town%29.svg.png" width="300" height="329" srcset="http://upload.wikimedia.org/wikipedia/commons/thumb/9/9e/Baden-W%C3%BCrttemberg_UL_%28town%29.svg/450px-Baden-W%C3%BCrttemberg_UL_%28town%29.svg.png 1.5x, http://upload.wikimedia.org/wikipedia/commons/thumb/9/9e/Baden-W%C3%BCrttemberg_UL_%28town%29.svg/600px-Baden-W%C3%BCrttemberg_UL_%28town%29.svg.png 2x" data-file-width="1058" data-file-height="1161" usemap="#ImageMap_1_30407246">
<div style="margin-left: 280px; margin-top: -20px; text-align: left;"><a href="http://de.wikipedia.org/wiki/datei:Baden-W%C3%BCrttemberg_UL_(town).svg" title="Über dieses Bild"><img alt="Über dieses Bild" src="http://bits.wikimedia.org/static-1.25wmf12/extensions/ImageMap/desc-20.png" style="border: none;"></a></div>
In some dictionaries images from the network are not loaded
In some dictionaries images from the network are not loaded.
Dictionary examples are available here:
neolurk-org-20220821.lzma2.slob 364.8MiB
ru.gta.fandom.com-20220822.lzma2.slob 17.7MiB
Commands used to create dictionaries:
mwscrape -c http://admin:admin@localhost:5984 gta.fandom.com/ru --site-path=/ --speed 3
mw2slob scrape http://admin:admin@localhost:5984/gta-fandom-com/ru --no-math --ensure-ext-image-urls -a sklart
mwscrape -c http://admin:admin@localhost:5984 http://neolurk.org/ --speed 5
mw2slob scrape http://admin:admin@localhost:5984/neolurk-org -a sklart -f common wiki
Please help me with this problem. Thanks
Wrong link for slob
Hallo
on the main page under Installation the link for slob is wrong. It is github.org instead of github.com so it goes 404.
Thank.
frank
XPath lower-case not defined
Pastebin of the error (hit ctrl+c early so you wouldn't have to see miles of the same thing), which was run in a virtualenv dedicated to mwscrape2slob and the new format: https://ptpb.pw/QQle.txt
Possibly related issues: aarddict/tools#39 (comment)
How to reproduce:
- Grab the current list of OCG cards through the mediawiki api with this python script I whipped up. https://gist.github.com/ShadowKyogre/86881fe3e2b9ff00492b
- Run the following command to grab the test data:
mwscrape yugioh.wikia.com --site-path=/ [email protected]
This will take a while because card images. - Use the following filter set either in a new file or fed manually to the commandline
.cardtablespanrow:contains("sets")
.cardtablespanrow:contains("Card appearances")
tr:contains("Anime") + tr
tr:contains("Anime")
tr:contains("Video game statuses") + tr
tr:contains("Video game statuses")
table.collapsible.hslit:contains("pages")
- Run the following command to convert the test data to the *.slob format:
mwscrape2slob http://127.0.0.1:5984/yugioh-wikia-com -f ${yourfilterfilenamehere}
Expected results: No errors and a *.slob file filled with the cards.
Actual results: See the pastebin.
Switch to KateX
KaTeX is a lot faster than MathJax. The switch may be fairly easy.
Broken tables in aard
Hi,
while comparing two revisions of mwscrape2slob/mw2slob I found that slob archives created with mw2slob contain broken tables, only the right half of the tables are visible in aard 0.53:
working revision: https://github.com/itkach/mw2slob/tree/e01d3e92f0a372ebd0f57390e437a28f9d3c0438 - screenshot: 
broken revision: https://github.com/itkach/mw2slob/tree/0358818432fc1404ce2ef7d35da7a59d970b77ef - screenshot: 
The slob archives were created from a local CouchDB server using these options:
-f common wiki -c lzma2 -ll en,nl,pl
Broken external image URLs while parsing other MediaWikis
When dumping a database of a random MediaWiki site created with mwscrape (apart from wikipedia), the external image URLs generate will be broken, even with the --ensure-ext-image-urls flag enabled - thus, "Load remote content" on Aard2 clients won't work with the images included in those resulting slob files.
An example:
source: https://wiki.archlinux.org/title/File:Merge-arrows-2.png
target: https://wiki.archlinux.org/images/c/c9/Merge-arrows-2.png
broken result in generated slob: https://wiki.archlinux.org/title//c/c9/Merge-arrows-2.png
(In some other cases, the generated links are working fine, but "Load remote content" still won't work properly)
I suspect that it's related to
Lines 212 to 213 in 078ac83
I've created a .slob with this tool which reproduces this problem: archwiki-20221126.slob not this one but this: archwiki_image_link_test_not_working.slob
TOCs with enterprise dumps
Would it be possible to generate TOCs in the articles from the enterprise dumps? They are not included in these dumps. Or should the dumps themselves be fixed?
README.org typo (http://github.org/itkach/slob/ instead of https://github.com/itkach/slob/)
What is the license of this awesome project?
Cool project, thanks! : )
some simple math formulas won't render
It just displays small one line placeholder box containing icon and text '{\displaystyle '.
correct should be 'a+b i + c j + d k'
(i,j,k are in bold). Formula is from the top of article https://en.wikipedia.org/wiki/Quaternion:
"Quaternions are generally represented in the form 'a+b i + c j + d k' "
strangely it renders many other more complex formulas flawlessly.
DETAILS:
running under win 10 using command line
Java\bin\java.exe -Dslobber.host="localhost" -Dslobber.port=80 -jar aard2-web-0.8.jar ..*.slob
using portable java
java -version
java version "1.8.0_361"
Java(TM) SE Runtime Environment (build 1.8.0_361-b09)
Java HotSpot(TM) Client VM (build 25.361-b09, mixed mode)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.