leibniz-hbi / smo-wiki Goto Github PK
View Code? Open in Web Editor NEWGenerates a Github Page from the Social Media Observatory Wiki with Bash, Python, Regexes and Jekyll.
Home Page: https://smo-wiki.leibniz-hbi.de
Generates a Github Page from the Social Media Observatory Wiki with Bash, Python, Regexes and Jekyll.
Home Page: https://smo-wiki.leibniz-hbi.de
Add the HBI logo via the respective theme function (see theme documentation of the 'Minimal Theme').
https://wiki.digitalmethods.net/Dmi/ToolDatabase
Add relevant tools to Wiki.
Not sure whether that's necessary, as the colours are quite neutral already, therefore this is low priority.
Presented during lunch lecture 11. Dec 2019:
https://github.com/adbar/trafilatura
Web scraping library: downloads web pages, finds main text and comments, converts to TXT, XML & TEI http://adrien.barbaresi.eu/research.html
https://github.com/adbar/htmldate
Find the creation date of web pages using common structural patterns, text-based heuristics and robust date extraction
https://arxiv.org/pdf/2005.14024.pdf
Even though it's about US media it might spark some ideas for a German dataset
Please update wiki accordingly (move to bottom and change details).
Also, apparently there's a mistake in that it does not support logging in. So that has to be changed in the table, and maybe in the text.
https://icon.colorado.edu/#!/networks
Especially the data in the subdomains communication and/or online is interesting to us. Maybe we need another category in the datasets wiki section for this, something like "Collections" – a meta-category of other collections, which contain open access and access on request data.
Regularly checking of the data collection and data anaylsis tools for up-to-dateness, functionality, possibly new features
Maybe this needs a new category longterm for location data tools: https://github.com/osm-search/Nominatim
Use PNG generated from vector-graphic instead of JPG
Aufgabe: automatisch von Zeit zu Zeit die Links anpingt und schaut, ob die noch online sind
Vorgehen 1:
Man könnte einen Test dafür schreiben, der bei jedem Build das testet
Mit pytest und requests
(oder unittest, aber pytest ist einfacher mE)
Braucht nur eine test.py irgendwo und dann pytest in dem build-script hier aufrufen: https://github.com/Leibniz-HBI/smo-wiki/blob/main/build.sh
Wo würde ich die test.py ablegen?: am besten in einem Ordner tests/
aber ist eigentlich egal
pytest such automatisch nach allem, was nach test_*.py aussieht
Dann mit requests in dem test halt für jeden Link in den Markdowns eine header-request machen und wenn irgendwo nicht 200 als Status-Code zurückkommt, Alarm schlagen
Vorgehen 2:
noch einfacher wäre es fast, einfach hier nochmal mit einer Regex alle Links rauszuziehen und einfach gleich zu prüfen
https://github.com/Leibniz-HBI/smo-wiki/blob/main/convert_links.py
Dadurch würde es halt bisschen langsam werden
das ginge wohl so in < 10 Zeilen
als Bonus kann man die Links dann auch gleich mit brackets versehen, so dass sie sicher funktionieren
Das kompliziertere (Vorgehen 1) wäre auch erstmal nicht schneller
aber man könnte es halt extra ausführen und nicht bei jedem build
eigentlich braucht man aber auch für das kompliziertere pytest erstmal nicht
Die Idee von ist nur, dass man auf Dauer halt noch mehr so Tests schreibt
Also wahrscheinlich die interessantere Lösung auf Dauer.
Dann führt man lokal halt nur pytest aus und das führt dann gleich ne ganze Menge an checks durch (also z.B. auch sind noch alle Bilder da)
Das könnte man dann auch wie in dem anderen Repo automatisch nochmal von Github laufen lassen bei jedem push
Copyright
Researchers Saftey
https://www.bellingcat.com/resources/2022/11/23/how-to-maintain-mental-hygiene-as-an-open-source-researcher/
https://neuemedienmacher.de/helpdesk-gegen-hass/
https://neuemedienmacher.de/fileadmin/dateien/PDF_Borschueren-Infomaterial-Flyer/NotfallKits_NHS.pdf
DIY Feminist Cybersecurity (hackblossom.org)
Data Protection
replace headings of single data analysis tools with dropdown headings
@kerosafr, @vomJupiter: ab einem gewissen Threshold auf der WoF sollten die Namen von da als Autoren auf der Startseite auftauchen (soweit nicht Pseudonym)
The rendered website version of the Wiki should have a "recent changes" page which is automatically created from the commit history of the wiki repository
When processing markdowns, convert
https://some-url.domain --> [[https://some-url.domain]]
frei zugänglicher Datensatz vom Standard u.a. mit Artikeln, Kommentaren und Moderationsentscheidung: https://ofai.github.io/million-post-corpus/
The Polly Corpus: online political debate in Germany:
http://www.organisms.be/downloads/polly.pdf
http://www.organisms.be/downloads/jaki2018.pdf
https://docs.google.com/spreadsheets/d/1c5peNMjt24U0FcEMSj8gD_JjzumqXTWbPWa_yb2nNt0/edit#gid=1445690638
https://drive.google.com/drive/folders/1uhx_NotkG3KTc2yU3-FjnlhBj5e07rcs
https://pub.uni-bielefeld.de/record/2909336
German corpus of tweets regarding refugees in Germany, annotated with hate speech ratings: https://github.com/UCSM-DUE/IWG_hatespeech_public
Data For Everyone by the Figure Eight platform: https://www.figure-eight.com/data-for-everyone/
ggf.: https://www.researchgate.net/publication/340618080_Classifying_Constructive_Comments
Außerdem ein paar Links zu deutschen Hatespeech-etc.-Diktionären:
deutschsprachige Begriffe auf hatebase.org (nur ca. 100): https://hatebase.org/search_results/language_id%3Ddeu
deutschsprachige Datensets auf http://hatespeechdata.com:
IWG_hatespeech_public: https://github.com/UCSM-DUE/IWG_hatespeech_public
GermEval-2018-Data: https://github.com/uds-lsv/GermEval-2018-Data
GermEval 2019: https://projects.fzai.h-da.de/iggsa/projekt/
Added to 'Twitter' Milestone, because some datasets contain tweets.
Later this issue should be split into several issues accordingly.
The wiki should be hosted on github pages, but reachable under the smo-wiki.leibniz-hbi.de subdomain
Either fix right away, or document here.
e.g.
Zonedo
GESIS
OSF
figshare
Github
At the moment the sidebar starts to act funny (with overlaps) when the screen/window gets smaller. This should be fixable in the default_layout.html and the css file in the assets folder.
At the moment the source/edge wiki lives in the Social-Media-Observatory Repo. It makes more sense to have it here (as a wiki again).
To do so, the wiki from the old repo would have to be cloned locally, then pushed to the wiki url of this repo.
It'll be important not to forget to change any hard-coded links in the Jekyll template and the build script.
This might then even be an opportunity to create a config file with those links/the domain.
https://github.com/digitalmethodsinitiative/zeeschuimer
Should go into cross-platform tool section
What:
Add a custom 404 page that tells the user that the page has not yet been created. Invite the user to contribute to this page with a link to the "How to edit this wiki" page.
Legislative text corpora inventory
Add this project: https://scicomm-support.de/
"Wir unterstützen und beraten Wissenschaftlerinnen und Wissenschaftskommunikatorinnen bei Angriffen und unsachlichen Konflikten in der Wissenschaftskommunikation"
https://journals.sagepub.com/doi/10.1177/02673231221112006
We present Meteor, a new inventory for European news sources (i.e. EU + UK, CH, NO, IL): https://wp3.opted.eu/. This inventory will facilitate researchers’ efforts to select sources across platforms and gather related textual data. It contains the names of print and online news sources, social media accounts, news blogs, and alternative news media sources, as well as rich meta-information for each entry (e.g. language, audience size, topical focus, ownership structures, access to full-text archives, secondary data, related research). Meteor accounts for the fuzziness of hybrid media systems through an interlinked knowledge graph. Entries are submitted by researchers, validated, continuously updated, and openly accessible to the public. Our inventory allows users to find various European news sources based on a wide range of criteria, putting scholars in a better position to navigate the European media landscape.
Change font of the wiki to Barlow, the official HBI font.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.