f4lco / om-parser-stw-potsdam-v2

Connect canteens of the Studentenwerk Potsdam to OpenMensa

OpenMensa Parser STW Potsdam

OpenMensa parser components query canteen websites for menus and transform them into OpenMensa's data format. This project came to life after the website of the canteens of the Studentenwerk Potsdam changed, and is therefore the successor to kaifabian/om-parser-potsdam (hence the "-v2").

Among others, OpenMensa powers the popular Mensa Uni Potsdam Android app.

The current application is built with Python, PyOpenMensa, and Flask. Learn more about the technical details at Read the Docs.

Contributions are always welcome, in particular if the response format of the canteens change. Feel free to file a PR with improvements.

Deployment If in need of a deployment, file a PR to this fork: kaifabian/om-parser-potsdam-v2. Kai is currently in charge of running an instance of the parser and the registration on the OpenMensa platform.

Where to go next maybe use this parser or the OpenMensa API to source a new dataset for training a predictor for your favorite lunch?

License Just assume this project is licensed in terms of WTFPL ;)

om-parser-stw-potsdam-v2's Issues

Occasional cron build failure due to API instability

The build health is also determined by the cron build which tests for parseable API output once per day. However, the API is a little unstable and at times the JSON endpoint returns outputs of the following kind:

<style type='text/css'>
		.extbase-debugger-tree{position:relative}
		.extbase-debugger-tree input{position:absolute !important;float: none !important;top:0;left:0;height:14px;width:14px;margin:0 !important;cursor:pointer;opacity:0;z-index:2}
		.extbase-debugger-tree input~.extbase-debug-content{display:none}
		.extbase-debugger-tree .extbase-debug-header:before{position:relative;top:3px;content:"";padding:0;line-height:10px;height:12px;width:12px;text-align:center;margin:0 3px 0 0;background-image:url(data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48c3ZnIHZlcnNpb249IjEuMSIgaWQ9IkViZW5lXzEiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgeG1sbnM6eGxpbms9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGxpbmsiIHg9IjBweCIgeT0iMHB4IiB2aWV3Qm94PSIwIDAgMTIgMTIiIHN0eWxlPSJlbmFibGUtYmFja2dyb3VuZDpuZXcgMCAwIDEyIDEyOyIgeG1sOnNwYWNlPSJwcmVzZXJ2ZSI+PHN0eWxlIHR5cGU9InRleHQvY3NzIj4uc3Qwe2ZpbGw6Izg4ODg4ODt9PC9zdHlsZT48cGF0aCBpZD0iQm9yZGVyIiBjbGFzcz0ic3QwIiBkPSJNMTEsMTFIMFYwaDExVjExeiBNMTAsMUgxdjloOVYxeiIvPjxnIGlkPSJJbm5lciI+PHJlY3QgeD0iMiIgeT0iNSIgY2xhc3M9InN0MCIgd2lkdGg9IjciIGhlaWdodD0iMSIvPjxyZWN0IHg9IjUiIHk9IjIiIGNsYXNzPSJzdDAiIHdpZHRoPSIxIiBoZWlnaHQ9IjciLz48L2c+PC9zdmc+);display:inline-block}
		.extbase-debugger-tree input:checked~.extbase-debug-content{display:inline}
		.extbase-debugger-tree input:checked~.extbase-debug-header:before{background-image:url(data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz48c3ZnIHZlcnNpb249IjEuMSIgaWQ9IkViZW5lXzEiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgeG1sbnM6eGxpbms9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGxpbmsiIHg9IjBweCIgeT0iMHB4IiB2aWV3Qm94PSIwIDAgMTIgMTIiIHN0eWxlPSJlbmFibGUtYmFja2dyb3VuZDpuZXcgMCAwIDEyIDEyOyIgeG1sOnNwYWNlPSJwcmVzZXJ2ZSI+PHN0eWxlIHR5cGU9InRleHQvY3NzIj4uc3Qwe2ZpbGw6Izg4ODg4ODt9PC9zdHlsZT48cGF0aCBpZD0iQm9yZGVyIiBjbGFzcz0ic3QwIiBkPSJNMTEsMTFIMFYwaDExVjExeiBNMTAsMUgxdjloOVYxeiIvPjxnIGlkPSJJbm5lciI+PHJlY3QgeD0iMiIgeT0iNSIgY2xhc3M9InN0MCIgd2lkdGg9IjciIGhlaWdodD0iMSIvPjwvZz48L3N2Zz4=)}
		.extbase-debugger{display:block;text-align:left;background:#2a2a2a;border:1px solid #2a2a2a;box-shadow:0 3px 0 rgba(0,0,0,.5);color:#000;margin:20px;overflow:hidden;border-radius:4px}
		.extbase-debugger-floating{position:relative;z-index:999}
		.extbase-debugger-top{background:#444;font-size:12px;font-family:monospace;color:#f1f1f1;padding:6px 15px}
		.extbase-debugger-center{padding:0 15px;margin:15px 0;background-image:repeating-linear-gradient(to bottom,transparent 0,transparent 20px,#252525 20px,#252525 40px)}
		.extbase-debugger-center,.extbase-debugger-center .extbase-debug-string,.extbase-debugger-center a,.extbase-debugger-center p,.extbase-debugger-center pre,.extbase-debugger-center strong{font-size:12px;font-weight:400;font-family:monospace;line-height:20px;color:#f1f1f1}
		.extbase-debugger-center pre{background-color:transparent;margin:0;padding:0;border:0;word-wrap:break-word;color:#999}
		.extbase-debugger-center .extbase-debug-string{color:#ce9178;white-space:normal}
		.extbase-debugger-center .extbase-debug-type{color:#569CD6;padding-right:4px}
		.extbase-debugger-center .extbase-debug-unregistered{background-color:#dce1e8}
		.extbase-debugger-center .extbase-debug-filtered,.extbase-debugger-center .extbase-debug-proxy,.extbase-debugger-center .extbase-debug-ptype,.extbase-debugger-center .extbase-debug-scope{color:#fff;font-size:10px;line-height:12px;padding:2px 4px;margin-right:2px;position:relative;top:-1px}
		.extbase-debugger-center .extbase-debug-scope{background-color:#497AA2}
		.extbase-debugger-center .extbase-debug-ptype{background-color:#698747}
		.extbase-debugger-center .extbase-debug-dirty{background-color:#FFFFB6}
		.extbase-debugger-center .extbase-debug-filtered{background-color:#4F4F4F}
		.extbase-debugger-center .extbase-debug-seeabove{text-decoration:none;font-style:italic}
		.extbase-debugger-center .extbase-debug-property{color:#f1f1f1}
	</style>
	<div class="extbase-debugger extbase-debugger-floating">
	<div class="extbase-debugger-top">xml error</div>
	<div class="extbase-debugger-center">
		<pre dir="ltr">NULL</pre>
	</div>
</div>
{"globalLaufschrift":"","wochentage":[]}

To my opinion, there is nothing we can really do about it. Option one - current state - is to fail loudly and to signal to clients that something went wrong. Option two - best-effort parsing of the API response - would recover exactly zero meals, which does not really justify the work required to implement the heuristic.

The parser seems to be down

And the linked repository from Kai is either private or deleted... Is there any information on the current situation? :)

add price parsing for "Salattheke"

prices are not correctly parsed for "Salattheke".
Someone can fix that, or reach out to Studentenwerk to make them use their API/field-values correctly.
This is purely optional.

BASE_URL environment handling is not functional

Just tested the new code handling the BASE_URL parameter in production. Some problems occured:

• Setting a hostname in BASE_URL leads (in some cases) to Flask failing to bind the socket, since it tries to resolve the hostname. It seems that Flask is not using the SERVER_NAME field for URL generation, but also for socket setup.
• Setting the PREFERRED_URL_SCHEME does not affect the resulting URLs. All URLs are now generated with 'http' scheme, despite being set to 'https' via BASE_URL/PREFERRED_URL_SCHEME.

New Canteens / Pickup Locations

We have discovered that we have new canteens - I assume at least some of which are not registered with openMensa, because they do not appear on the map on https://openmensa.org/#15/52.3923/13.1266. Can you @kaifabian please check and eventually link them to the parser?

• 'Abholung / Pick up: Wohnheim Stahnsdorfer Straße'
• 'Cafeteria Am Neuen Palais'
• 'Abholung / Pick up: Wohnheim Kaiser-Friedrich-Str.'

Eventually, when we have the code change ready, we would request a deployment, and then we would have menus for above canteens.

Builds fail, live system fails with error 500

https://travis-ci.org/f4lco/om-parser-stw-potsdam-v2:

    def _active_days(menu):
>       for container in menu['wochentage']:
E       TypeError: 'NoneType' object has no attribute '__getitem__'

Python 3

Eventually, Python 2 will encounter its EOL: https://pythonclock.org/
We should port to Python 3.

Cache deprecation warning

stw_potsdam/views.py:8
/home/travis/build/f4lco/om-parser-stw-potsdam-v2/stw_potsdam/views.py:8: DeprecationWarning: 'werkzeug.contrib.cache' is deprecated as of version 0.15 and will be removed in version 1.0. It has moved to https://github.com/pallets/cachelib.
from werkzeug.contrib.cache import SimpleCache

The warning should be fixed. In case the dependency is upgraded and the project fails to build, this is a friendly reminder.

CI: dependency resolution fails

As evident in https://travis-ci.org/f4lco/om-parser-stw-potsdam-v2/builds/447009338, make dependencies fails to execute.

Might consider switching to unofficial API

Instead of relying on https://www.studentenwerk-potsdam.de/essen/unsere-mensen-cafeterien/detailinfos/, it might be useful to switch to the unofficial API, which can be found at:
https://xml.stw-potsdam.de/xmldata/gs/xml.php
Future website changes shouldn't break the code!

URL and parameters have changed

The URL of the stw-potsdam API changed from /essen/unsere-mensen-cafeterien/detailinfos to /essen/unsere-mensen/detailinfos and the query parameters changed slightly. Currently (also in the latest build), requesting a menu feed causes the parser to crash with a JSONDecodeError.
The tests do not show this because of 0b630df.

Feature: add notes tag

The notes tag is optional and may contain additional information about food ingridients(pork, beef, vegan) or other information.
It could be discussed which information structures are needed by other apps handling such data.

Canteen Kiepenheuerallee failing due to variant

As of 2018-10-22, the JSON response of Kiepenheuerallee fails to parse. It appears that it is possible that the offers are not a list, but a dictionary mapping integer keys to offers. Second, there is an offer consisting of only one attribute at 'index' / key -1.

what is the current status?

what is the current status?
Are there still some problems that have to be solved to make it work again?

SWP partly switched to Webspeiseplan; current menu only contains redirect

The Studentenwerk seems to be switching to webspeiseplan.de (https://swp.webspeiseplan.de/Menu). So far e.g. Griebnitzsee still uses the "old" page, but e.g. Filmuniversität uses the new one.

Are you interested in building a new integration? Otherwise I might look into it :)