GithubHelp home page GithubHelp logo

kbnlwikimedia / glamoroustohtml Goto Github PK

View Code? Open in Web Editor NEW
2.0 1.0 0.0 33.73 MB

Creates a HTML page and a corresponding Excel file listing all Wikipedia articles (in all languages) in which (one or more) images from a given category on Wikimedia Commons are used.

Home Page: https://kbnlwikimedia.github.io/GLAMorousToHTML/

License: Creative Commons Zero v1.0 Universal

Python 0.09% HTML 99.91% CSS 0.01%

glamoroustohtml's Introduction

GLAMorousToHTML

Creates a HTML page and a corresponding Excel file listing all Wikipedia articles (in all languages) in which (one or more) images from a given category on Wikimedia Commons are used.

Latest update: 1 March 2024

What does it do?

The script GLAMorousToHTML.py creates a HTML page and a corresponding Excel file listing all Wikipedia articles (in all languages) in which (one or more) images/media from a given category on Wikimedia Commons are used. It does so by converting the XML output of the GLAMorous tool.

What problem does it solve?

The KB uses the GLAMorous tool to measure the use of KB media files (as stored in Wikimedia Commons) in Wikipedia articles. This tool rapports 4 things :

  • 1 The total number of KB media files in Category:Media contributed by Koninklijke Bibliotheek (Category "Media contributed by Koninklijke Bibliotheek" has XXXX files.)
  • 2 The total number of times that KB media files are used in WP articles (Total image usages).
  • 3 The number of Wikipedia language versions in which KB media files are used (length of the table)
  • 4 The number of unique KB media files that are used in Wikipedia articles in all those languages. (Distinct images used)

Please note: 'Total image usages' does NOT equal the number of unique WP articles! A single unique KB image can illustrate multiple unique WP articles, and/or the other way around, 1 unique WP article can contain multiple unique KB images. In other words: images-articles have many-to-many relationships.

What was still missing was the functionality to measure

  • 5 The number of unique WP articles in which KB media files are used,
  • 6 A manifest overview of those articles, grouped per WP language version,
  • 7 A structured output format that can be easily processed by tools, such as Excel.

That is why we made the GLAMorousToHTML tool. This script uses the XML-output of GLAMorous to make an HTML page listing unique WP articles (in which one or more KB media files are used), grouped by language.

Per 14-02-2024 it also delivers an Excel file with equivalent data.

Configuration of GLAMorous

The script relies on the XML output of GLAMorous, which needs to be configured so that it only lists pages from Wikipedia

  1. that are in the main namespace (a.k.a Wikipedia articles) (&ns0=1)

  2. and not pages from Wikimedia Commons, Wikidata or other Wikimedia projects (projects[wikipedia]=1)

The base URL looks like https://glamtools.toolforge.org/glamorous.php?doit=1&use_globalusage=1&ns0=1&projects[wikipedia]=1&format=xml&category=. The Commons category of interest needs to be added to the end, omitting the Category: prefix. It is defined (and can be adapted) in the xml_base_url variable in setup.py.

By default the depth of the GLAMorous output is set to 0, meaning no subcategories are read. If you want to include images from subcategories in your outputs, you can change the depth variable in setup.py.

Running the script yourself

If you want to run this script for your own Commons category and create HTML and Excel overviews for your own institution, you can clone/download the repo and run it on your own machine. You will need to make some simple adaptations to the existing code to make it work for the Commons category of your choice. These are:

  1. Adapt the category_logo_dict.json for your own needs, making sure the existing syntax is maintained.

    • If not yet available, make a new top level country key (similar to "Netherlands", "USA", "Norway" etc.) to include your country.

    • Under this country key, add a line with a syntax identical to the one starting with "Media contributed by Koninklijke Bibliotheek", but with modifications for three things:

      1. The exact name (without underscores '_') of the Wikimedia Commons category you want run the script for ("Media contributed by Koninklijke Bibliotheek")

      2. A shortname of the institution ("KoninklijkeBibliotheekNL"). This is used for the name of the sheet in the Excel file, so keep it shorter than 32 characters.

      3. Name of an institutional logo file, starting with "icon_", followed by a unique and descriptive letter code for the institution, and appended with a .png or .jpg extension at the end. This logo/icon is displayed at the top of the HTML page. Don't forget the next step!

  2. Add a small logo of the institution (256x256 px or so) as a .png of .jpg to the site/logos folder, and add the filename "icon_xxxxx.png/jpg" to the json file.

  3. In setup.py, change

    • the country_key variable to the new country key you added to the json file (default = "Netherlands")
    • the institute_index to the index of the line corresponding to your institution in the json file (default = 0; first line under a country key)

That's all, you should now be able to run the main GLAMorousToHTML script. The generated HTML page will be added to the site/ folder and the Excel to the data/ folder.

In case you can't get the script up and running, please open an issue in this repo.

Examples

KB, national library of the Netherlands

Media contributed by Koninklijke Bibliotheek

Atlas de Wit 1698

Atlas van der Hagen

Media from Atlas of Mutual Heritage - Koninklijke Bibliotheek

Nederlandsche vogelen van Nozeman en Sepp

Der naturen bloeme - KB KA 16

Catchpenny prints from Koninklijke Bibliotheek

Bookbindings from Koninklijke Bibliotheek

Other institutions

Netherlands

See also this LinkedIn post

USA

See also this LinkedIn post

Nordic countries

See also this LinkedIn post

Norway
Sweden
Finland
Denmark

Australia and New Zealand

See also this LinkedIn post

Australia
New Zealand

See also

Change log

14 March 2024

  • Included reports for 14 institutions from Australia and New Zealand.

29 February 2024

  • Included reports for institutions from Norway, Sweden, Finland and Sweden.
  • README.md: Added explanations how you can run the script yourself.

14 February 2024

  • Refactored all code into multiple separated modules: setup.py, general.py, buildHTML.py and buildExcel.py. This has reduced the complexity of the main script GLAMorousToHTML.py significantly and made the total suite of code much more modular and easier to understand, maintain and expand.
  • Moved all HTML report pages into a separate site/ folder. This has made the repo much cleaner, clearer and more maintainable.
  • Created five HTML files that redirect the old KB HTML pages (from 27-01-2022 to 16-01-2024) to the new equivalent ones in "/site" folder. Did not implement redirection for other institutions.
  • Per 14-02-2024 added Excel outputs in data/ folder, to be used as structured input for data applications, such as OpenRefine
  • In the proces of updating the data structure in category_logo_dict.json, where the new structure can be seen under the 'Netherlands' key.
  • Improved pagetemplate.html to be key based ({numarticles} Wikipedia articles) rather than index based ({0} Wikipedia articles)

Features to add

glamoroustohtml's People

Contributors

ookgezellig avatar

Stargazers

Rute avatar Sebastian Majstorovic avatar

Watchers

avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.