GithubHelp home page GithubHelp logo

hardikvasa / wikipedia-crawler Goto Github PK

View Code? Open in Web Editor NEW
71.0 71.0 30.0 188 KB

This is a program to crawl entire 'Wikipedia' and extract & store information from the pages as required.

License: MIT License

Python 100.00%

wikipedia-crawler's People

Contributors

hardikvasa avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wikipedia-crawler's Issues

Title output broken

Title outputs:

Link = https://en.wikipedia.org/wiki/Minkowski_diagram
Title = \n<html class="client-nojs" lang="en" dir="ltr">\n<head>\n<meta charset="utf-8"/>\n<title>minkowski diagram - wikipedia</title>\n<script>document.documentelement.classname=document.documentelement.classname.replace(/(^|\\s)client-nojs(\\s|$)/,"$1client-js$2");rlconf={"wgcanonicalnamespace":"","wgcanonicalspecialpagename":!1,"wgnamespacenumber":0,"wgpagename":"minkowski_diagram","wgtitle":"minkowski diagram","wgcurrevisionid":904360354,"wgrevisionid":904360354,"wgarticleid":11647860,"wgisarticle":!0,"wgisredirect":!1,"wgaction":"view","wgusername":null,"wgusergroups":["*"],"wgcategories":["commons category link is on wikidata","special relativity","geometry","diagrams"],"wgbreakframes":!1,"wgpagecontentlanguage":"en","wgpagecontentmodel":"wikitext","wgseparatortransformtable":["",""],"wgdigittransformtable":["",""],"wgdefaultdateformat":"dmy","wgmonthnames":["","january","february","march","april","may","june","july","august","september","october","november","december"],"wgmonthnamesshort":["","jan","feb","mar","apr","may","jun","jul","aug","sep","oct","nov","dec"],"wgrelevantpagename":"minkowski_diagram",\n"wgrelevantarticleid":11647860,"wgrequestid":"xuwfyapamfwaahiw-gaaaabw","wgcspnonce":!1,"wgisprobablyeditable":!0,"wgrelevantpageisprobablyeditable":!0,"wgrestrictionedit":[],"wgrestrictionmove":[],"wgmediavieweronclick":!0,"wgmediaviewerenabledbydefault":!0,"wgpopupsreferencepreviews":!1,"wgpopupsconflictswithnavpopupgadget":!1,"wgvisualeditor":{"pagelanguagecode":"en","pagelanguagedir":"ltr","pagevariantfallbacks":"en"},"wgmfdisplaywikibasedescriptions":{"search":!0,"nearby":!0,"watchlist":!0,"tagline":!1},"wgwmeschemaeditattemptstepoversample":!1,"wgpoweredbyhhvm":!0,"wgulscurrentautonym":"english","wgnoticeproject":"wikipedia","wgwikibaseitemid":"q177596","wgcentralauthmobiledomain":!1,"wgeditsubmitbuttonlabelpublish":!0};rlstate={"ext.gadget.charinsert-styles":"ready","ext.globalcssjs.user.styles":"ready","ext.globalcssjs.site.styles":"ready","site.styles":"ready","noscript":"ready","user.styles":"ready","ext.globalcssjs.user":"ready",\n"ext.globalcssjs.site":"ready","user":"ready","user.options":"ready","user.tokens":"loading","ext.cite.styles":"ready","ext.math.styles":"ready","mediawiki.legacy.shared":"ready","mediawiki.legacy.commonprint":"ready","mediawiki.toc.styles":"ready","wikibase.client.init":"ready","ext.visualeditor.desktoparticletarget.noscript":"ready","ext.uls.interlanguage":"ready","ext.wikimediabadges":"ready","ext.3d.styles":"ready","mediawiki.skinning.interface":"ready","skins.vector.styles":"ready"};rlpagemodules=["ext.cite.ux-enhancements","ext.math.scripts","site","mediawiki.page.startup","mediawiki.page.ready","mediawiki.toc","mediawiki.searchsuggest","ext.gadget.teahouse","ext.gadget.referencetooltips","ext.gadget.watchlist-notice","ext.gadget.drn-wizard","ext.gadget.charinsert","ext.gadget.reftoolbar","ext.gadget.extra-toolbar-buttons","ext.gadget.switcher","ext.centralauth.centralautologin","mmv.head","mmv.bootstrap.autostart","ext.popups","ext.visualeditor.desktoparticletarget.init",\n"ext.visualeditor.targetloader","ext.eventlogging","ext.wikimediaevents","ext.navigationtiming","ext.uls.compactlinks","ext.uls.interface","ext.cx.eventlogging.campaigns","ext.quicksurveys.init","ext.centralnotice.geoip","ext.centralnotice.startup","skins.vector.js"];</script>\n<script>(rlq=window.rlq||[]).push(function(){mw.loader.implement("user.tokens@0tffind",function($,jquery,require,module){/*@nomin*/mw.user.tokens.set({"edittoken":"+\\\\","patroltoken":"+\\\\","watchtoken":"+\\\\","csrftoken":"+\\\\"});\n});});</script>\n<link rel="stylesheet" href="/w/load.php?lang=en&amp;modules=ext.3d.styles%7cext.cite.styles%7cext.math.styles%7cext.uls.interlanguage%7cext.visualeditor.desktoparticletarget.noscript%7cext.wikimediabadges%7cmediawiki.legacy.commonprint%2cshared%7cmediawiki.skinning.interface%7cmediawiki.toc.styles%7cskins.vector.styles%7cwikibase.client.init&amp;only=styles&amp;skin=vector"/>\n<script async="" src="/w/load.php?lang=en&amp;modules=startup&amp;only=scripts&amp;raw=1&amp;skin=vector"></script>\n<meta name="resourceloaderdynamicstyles" content=""/>\n<link rel="stylesheet" href="/w/load.php?lang=en&amp;modules=ext.gadget.charinsert-styles&amp;only=styles&amp;skin=vector"/>\n<link rel="stylesheet" href="/w/load.php?lang=en&amp;modules=site.styles&amp;only=styles&amp;skin=vector"/>\n<meta name="generator" content="mediawiki 1.34.0-wmf.16"/>\n<meta name="referrer" content="origin"/>\n<meta name="referrer" content="origin-when-crossorigin"/>\n<meta name="referrer"
content="origin-when-cross-origin"/>\n<meta property="og:image" content="https://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/minkowski_diagram_-_photon.svg/1200px-minkowski_diagram_-_photon.svg.png"/>\n<link rel="alternate" href="android-app://org.wikipedia/http/en.m.wikipedia.org/wiki/minkowski_diagram"/>\n<link rel="alternate" type="application/x-wiki" title="edit this page" href="/w/index.php?title=minkowski_diagram&amp;action=edit"/>\n<link rel="edit" title="edit this page" href="/w/index.php?title=minkowski_diagram&amp;action=edit"/>\n<link rel="apple-touch-icon" href="/static/apple-touch/wikipedia.png"/>\n<link rel="shortcut icon" href="/static/favicon/wikipedia.ico"/>\n<link rel="search" type="application/opensearchdescription+xml" href="/w/opensearch_desc.php" title="wikipedia (en)"/>\n<link rel="edituri" type="application/rsd+xml" href="//en.wikipedia.org/w/api.php?action=rsd"/>\n<link rel="license" href="//creativecommons.org/licenses/by-sa/3.0/"/>\n<link rel="canonical" href="https://en.wikipedia.org/wiki/minkowski_diagram"/>\n<link rel="dns-prefetch" href="//login.wikimedia.org"/>\n<link rel="dns-prefetch" href="//meta.wikimedia.org" />\n<!--[if lt ie 9]><script src="/w/resources/lib/html5shiv/html5shiv.js"></script><![endif]-->\n</head>\n<body class="mediawiki ltr sitedir-ltr mw-hide-empty-elt ns-0 ns-subject mw-editable page-minkowski_diagram rootpage-minkowski_diagram skin-vector action-view">\n<div id="mw-page-base" class="noprint"></div>\n<div id="mw-head-base" class="noprint"></div>\n<div id="content" class="mw-body" role="main">\n\t<a id="top"></a>\n\t<div id="sitenotice" class="mw-body-content"><!-- centralnotice --></div>\n\t<div class="mw-indicators mw-body-content">\n</div>\n\n\t<h1 id="firstheading" class="firstheading" lang="en">minkowski diagram</h1>\n\t\n\t<div id="bodycontent" class="mw-body-content">\n\t\t<div id="sitesub" class="noprint">from wikipedia, the free encyclopedia</div>\n\t\t<div id="contentsub"></div>\n\t\t\n\t\t\n\t\t\n\t\t<div id="jump-to-nav"></div>\n\t\t<a class="mw-jump-link" href="#mw-head">jump to navigation</a>\n\t\t<a class="mw-jump-link" href="#p-search">jump to search</a>\n\t\t<div id="mw-content-text" lang="en" dir="ltr" class="mw-content-ltr"><div class="mw-parser-output"><div class="thumb tright"><div class="thumbinner" style="width:258px;"><a href="/wiki/file:minkowski_diagram_-_photon.svg" class="image"><img alt="" src="//upload.wikimedia.org/wikipedia/commons/thumb/c/c2/minkowski_diagram_-_photon.svg/256px-minkowski_diagram_-_photon.svg.png" decoding="async" width="256" height="256" class="thumbimage" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/c/c2/minkowski_diagram_-_photon.svg/384px-minkowski_diagram_-_photon.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/c/c2/minkowski_diagram_-_photon.svg/512px-minkowski_diagram_-_photon.svg.png 2x" data-file-width="800" data-file-height="800" /></a>  <div class="thumbcaption"><div class="magnify"><a href="/wiki/file:minkowski_diagram_-_photon.svg" class="internal" title="enlarge"></a></div>the world line (yellow path) of a <a href="/wiki/photon" title="photon">photon</a>, which is at location <i>x</i> = 0 at time <i>ct</i> = 0.</div></div></div>\n<table class="vertical-navbox nowraplinks" style="float:right;clear:right;width:22.0em;margin:0 0 1.0em 1.0em;background:#f9f9f9;border:1px solid #aaa;padding:0.2em;border-spacing:0.4em 0;text-align:center;line-height:1.4em;font-size:88%"><tbody><tr><td style="padding-top:0.4em;line-height:1.2em">part of a series on</td></tr><tr><th style="padding:0.2em 0.4em 0.2em;padding-top:0;font-size:145%;line-height:1.2em"><a href="/wiki/spacetime" title="spacetime">spacetime</a></th></tr><tr><td style="padding:0.2em 0 0.4em"><a href="/wiki/file:gpb_circling_earth.jpg" class="image"><img alt="gpb circling earth.jpg" src="//upload.wikimedia.org/wikipedia/commons/thumb/d/d1/gpb_circling_earth.jpg/240px-gpb_circling_earth.jpg" decoding="async" width="240" height="180" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/d/d1/gpb_circling_earth.jpg/360px-gpb_circling_earth.jpg 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/d/d1/gpb_circling_earth.jpg/480px-gpb_circling_earth.jpg 2x" data-file-width="1200" data-file-height="900" /></a></td></tr><tr><td style="padding:0 0.1em 0.4em">\n<a href="/wiki/special_relativity" title="special relativity">special relativity</a><br /><a href="/wiki/general_relativity" title="general relativity">general relativity</a></td>\n</tr><tr><td style="padding:0 0.1em 0.4em">\n<div class="navframe collapsed" style="border:none;padding:0"><div class="navhead" style="font-size:105%;background:transparent;text-align:left">spacetime concepts</div><div class="navcontent" style="font-size:105%;padding:0.2em 0 0.4em;text-align:center"><a href="/wiki/spacetime" title="spacetime">spacetime manifold</a><br /><a href="/wiki/equivalence_principle" title="equivalence principle">equivalence
principle</a><br /><a href="/wiki/lorentz_transformation" title="lorentz transformation">lorentz transformations</a><br /><a href="/wiki/minkowski_space" title="minkowski space">minkowski space</a></div></div></td>\n</tr><tr><td style="padding:0 0.1em 0.4em">\n<div class="navframe collapsed" style="border:none;padding:0"><div class="navhead" style="font-size:105%;background:transparent;text-align:left">general relativity</div><div class="navcontent" style="font-size:105%;padding:0.2em 0 0.4em;text-align:center"><a href="/wiki/introduction_to_general_relativity" title="introduction to general relativity">introduction to general relativity</a><br /><a href="/wiki/introduction_to_the_mathematics_of_general_relativity" title="introduction to the mathematics of general relativity">mathematics of general relativity</a><br /><a href="/wiki/einstein_field_equations" title="einstein field equations">einstein field equations</a></div></div></td>\n</tr><tr><td style="padding:0 0.1em 0.4em">\n<div class="navframe collapsed" style="border:none;padding:0"><div class="navhead" style="font-size:105%;background:transparent;text-align:left">classical gravity</div><div class="navcontent" style="font-size:105%;padding:0.2em 0 0.4em;text-align:center"><a href="/wiki/gravity" title="gravity">introduction to gravitation</a><br /><a href="/wiki/newton%27s_law_of_universal_gravitation" title="newton&#39;s law of universal gravitation">newton\'s law of universal gravitation</a></div></div></td>\n</tr><tr><td style="padding:0 0.1em 0.4em">\n<div class="navframe collapsed" style="border:none;padding:0"><div class="navhead" style="font-size:105%;background:transparent;text-align:left">relevant mathematics</div><div class="navcontent" style="font-size:105%;padding:0.2em 0 0.4em;text-align:center"><a href="/wiki/four-vector" title="four-vector">four-vector</a><br /><a href="/wiki/derivations_of_the_lorentz_transformations" title="derivations of the lorentz transformations">derivations of relativity</a><br /><a class="mw-selflink selflink">spacetime diagrams</a><br /><a href="/wiki/differential_geometry" title="differential geometry">differential geometry</a><br /><a href="/wiki/curved_space" title="curved space">curved spacetime</a><br /><a href="/wiki/mathematics_of_general_relativity" title="mathematics of general relativity">mathematics of general relativity</a><br /><a href="/wiki/spacetime_topology" title="spacetime topology">spacetime topology</a></div></div></td>\n</tr><tr><td style="text-align:right;font-size:115%;padding-top: 0.6em;"><div class="plainlinks hlist navbar mini"><ul><li class="nv-view"><a href="/wiki/template:spacetime" title="template:spacetime"><abbr title="view this template">v</abbr></a></li><li class="nv-talk"><a href="/wiki/template_talk:spacetime" title="template talk:spacetime"><abbr title="discuss this template">t</abbr></a></li><li class="nv-edit"><a class="external text" href="https://en.wikipedia.org/w/index.php?title=template:spacetime&amp;action=edit"><abbr title="edit this template">e</abbr></a></li></ul></div></td></tr></tbody></table>\n<p>the <b>minkowski diagram</b>, also known as a <b>spacetime diagram</b>, was developed in 1908 by <a href="/wiki/hermann_minkowski" title="hermann minkowski">hermann minkowski</a> and provides an illustration of the properties of space and time in the <a href="/wiki/special_theory_of_relativity" class="mw-redirect" title="special theory of relativity">special theory of relativity</a>. it allows a qualitative understanding of the corresponding phenomena like <a href="/wiki/time_dilation" title="time dilation">time dilation</a> and <a href="/wiki/length_contraction" title="length contraction">length contraction</a> without mathematical equations.\n</p><p>minkowski diagrams are two-dimensional graphs that depict events as happening in
a <a href="/wiki/universe" title="universe">universe</a> consisting of one space dimension and one time dimension. unlike a
regular distance-time graph, the distance is displayed on the horizontal axis and time on the vertical axis. additionally, the time and space <a href="/wiki/units_of_measurement" class="mw-redirect" title="units of measurement">units of measurement</a> are chosen in such a way that an object moving at the speed of light is depicted as following a 45\xc2\xb0 angle to the diagram\'s axes.\n</p><p>in this way, each object, like an observer or a vehicle, traces a certain line in the diagram, which is called its <a href="/wiki/world_line" title="world line">world line</a>. also, each point in the diagram represents a
certain position in space and time, and is called an <b>event</b>, regardless of whether anything relevant happens there.\n</p>\n<div id="toc" class="toc"><input type="checkbox" role="button" id="toctogglecheckbox" class="toctogglecheckbox" style="display:none" /><div class="toctitle" lang="en" dir="ltr"><h2>contents</h2><span class="toctogglespan"><label class="toctogglelabel" for="toctogglecheckbox"></label>
Related Links = Minkowski space\nPenrose diagram\nRapidity\n\n
Introduction = The Minkowski diagram, also known as a spacetime diagram, was developed in 1908 by Hermann Minkowski and provides an illustration of the properties of space and time in the special theory of relativity. It allows a qualitative understanding of the corresponding phenomena like time dilation and length contraction without mathematical equations.\n

Expecting title to output:

minkowski diagram - wikipedia

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.