tables are not fetched? about laravel-wikipedia-grabber

[vesper8]

So I've been debugging.. I can confirm that all the data is returned from the API.

I can also confirm that the wikitables gets stripped as a result of this line

$sections = (new SectionsParser($title, $body))->sections();

That is, if I comment out this line then the tables are not removed. I'm still trying to understand why exactly this is happening

[dmitry-ivanov]

It looks like Wikipedia responding in such a way for that page (yeah, it happens for some pages).

You can check the pure Wiki response by this query:
https://en.wikipedia.org/w/api.php?action=query&format=json&formatversion=2&redirects=1&prop=extracts&exlimit=1&explaintext=1&exsectionformat=wiki&titles=2019_Australian_Open

Before the sections parsing there is no Matches info as well.

Maybe you see the "day info", which is above the table and consider it as a table contents?

Because I don't see it neither in response, nor in $body dump (which is the same).

[dmitry-ivanov]

PS: Actually I've seen that tables content is not present in the output from their API many times.

I'm requesting "wiki" format, which is the plain text. Maybe they consider that tables are some kind of rich content, which is NOT the plain text.

[vesper8]

Oops. I was running your package as part of a console command and using grep to determine if the wikitables content was included. If you comment out the line I pointed to above then you get an error of course since the $sections variable is missing. As part of the error message you can see the full body which does include the wikitables hence my grep was returning true

So it isn't necessarily that part of your code that strips away the wikitables

However, the api call IS returning the wikitables. Your grabber is somehow stripping it at some point or discarding it.

Here's the API call generated by your package (which I was able to get by adding debug=true to the guzzle params)

https://en.wikipedia.org/w/api.php?action=query&format=json&formatversion=2&redirects=1&prop=pageprops%7Cextracts%7Crevisions%7Cpageimages%7Cimages&titles=2019%20Australian%20Open&ppprop=disambiguation&exlimit=1&explaintext=1&exsectionformat=wiki&rvprop=content&rvcontentformat=text%2Fx-wiki&piprop=thumbnail%7Coriginal&pithumbsize=300&imlimit=max

You can search for the string "Matches on main courts" inside that response to see that the matches are included.. however they are included as part of a revisions which I'm not sure how that should affect it's availability as part of the Page object

[dmitry-ivanov]

I use revisions to link sections with the proper images.

The content of the page is taken from the plain "extract", and there is no table data there, unfortunately.

That's why you don't see it in output.