A universal web-util for PHP.
Home Page: https://phpscraper.de
License: GNU General Public License v3.0
I'm Peter, a developer with a passion for open source & InfoSec! 😎 I'm a fan of Linux, Bash and Tool builder 💻
🧰 Great developers & security researchers like Tom Hudson (tomnomnom) and Johan Carlsson (joaxcar) inspire my work. If you are keen you can check my tools & open-source projects too.
👷️ When it comes to developing web-apps, Laravel with Livewire and Filament are my stack ✨️🤩️
💬 You can find my writing on my dev blog and my general blog.
More to come soon!
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
The image URL should be returned, if this the only part of the anchor texts.
I am not sure if I'm simply missing something, but after installing phpscraper and simply writing this:
<?php require 'vendor/autoload.php'; $web = new \spekulatius\phpscraper; ?>
Brings this error.
Fatal error: Default value for parameters with a class type can only be NULL in .../vendor/symfony/browser-kit/AbstractBrowser.php on line 136
I'm using vanilla php with XAMPP.
spekulatius/phpscraper 0.5.2 requires symfony/dom-crawler ^5.0 -> found symfony/dom-crawler[v5.0.0, ..., v5.4.3] but the package is fixed to v6.0.3 (lock file version) by a partial update and that version does not match. Make sure you list it as an argument for the update command.
Details TBC
To benefit from simpler analysis it makes sense to provide a keyword extraction system out-of-the-box. Rake looks promising.
I saw some references to an API service in the documentation. I don't know if you plan to make the implementation open-source as well, but I really liked the idea of abstracting the extraction process.
Combined with the proxy feature, it provides an incredibly powerful tool for gathering data.
The idea is to implement a setApi method in the phpscraper class that supports various APIs. I went with the "namespace builder" approach, to load the API code as needed.
public $api = null;
...
public function setApi($api)
{
$apiClass = __NAMESPACE__ . '\apis\\' . $api;
$this->api = new $apiClass($this->core);
return $this;
}Then we will just need to create a file inside the new apis folder with the corresponding implementation, inside the namespace.
src/apis/example_api.php
namespace spekulatius\apis;
class example_api
{
protected $core = null;
public function __construct(core &$core)
{
$this->core = $core;
}
...
}I've implemented an API to scrap products from Mercado Libre (the biggest online marketplace in Latin America).
$web = new phpscraper;
$web->setApi('mercado_libre');
$web->go('https://mercadolivre.com.br/product_url');
$productData = $web->api->getProduct();More details about the api mercado libre API here.
What do you think?
#16 proposes adding support for JSONLD.
There isn't only JSONLD - structured data can be provided also in the microdata notation, and: good news - there is a project which parses microdata and converts it to the same data structure as JSONLD: https://github.com/yusufkandemir/microdata-parser
So it should be possible to use both and treat it just like an additional JSONLD block!
A first test:
$jsonlddata = \YusufKandemir\MicrodataParser\Microdata::fromHTML($web->client->getResponse()->getContent(), $web->currentUrl())->toArray();
Internally this project uses an own DOM document class derived from DOMDocument. It has a function to import a DOMDocument - but Symphonys response class doesn't allow to access the DOMDocument.
I did a small test, but didn't fiddle it out how to pass the DOMDocument without reparsing - my try which didn't work:
$dom = new \DOMDocument('1.0', 'UTF-8');
$dom->importNode($web->filterFirst('//*')->getNode(0), true);
$jsonlddata = \YusufKandemir\MicrodataParser\Microdata::fromDOMDocument($dom);
What makes sense?
Adding separate PHPScraper functions for JSONLD and microdata? Or mixing both automatically? (my opinion: mixing)
How should support for microdata look like? Adding the other project to PHPScraper? Extending the existing classes or porting the whole functionality to PHPScraper?
I'm working on a project that needs to be constantly changing proxies and taking a look in the Goute client, the underlying implementation already supports it.
HttpClientInterface (source code)...
class HttpBrowser extends AbstractBrowser
{
private $client;
public function __construct(HttpClientInterface $client = null, History $history = null, CookieJar $cookieJar = null)
...
$this->client = $client ?? HttpClient::create();
...HttpClient::create() supports proxy using the $defaultOptions parameter, that gets passed to the selected HttpClient (source code) public static function create(array $defaultOptions = [], int $maxHostConnections = 6, int $maxPendingPushes = 50): HttpClientInterfaceThe idea is to expose this functionality though a setProxy function in the core class. The library will continue to dynamically select the httpClient accordingly.
I just made a POC in my fork with all necessary code to make this work (a6589da).
public function setProxy(string $proxy)
{
$httpClient = HttpClient::create([
'proxy' => $proxy
]);
$this->client = new Client($httpClient);
return $this;
}$web = new phpscraper;
$web->__call('setProxy', [
'http://user:[email protected]:3128',
]);If this feature gets approved, I will open a PR with it. If anything needs to be changed, let me know.
Navigating through web pages works wonderful.
But i dont see a way for those ones
<a href="#" onclick="Postback('xxxx','2');return false;">2</a>
It would be beneficial to return the URL of the sitemap.xml file directly.
Implement basic CSV parsing functionality.
As usual, keep it simple:
$data = $web->parseCsv('https://test-pages.phpscraper.de/1.csv');$data = $web->go('https://test-pages.phpscraper.de/1.csv')->parseCsv()$web->go('https://test-pages.phpscraper.de/1.csv');
$data = $web->parseCsv();While the CSVs are flat by nature, they can be resolved into an associative array. This should be considered as an option. Suitable naming TBC.
This library is locked to Symfony 5, I'll submit a PR to allow Symfony 6 as well.
Hello,
How to make a condition if the timeout exceeds a certain number of seconds?
Thank
Hi
Its a great project, we use that library, its great 👍 thanks for all the work!
We are not sure what the versioning concept of this library looks like, also we could not find information about what has changed (a changelog), there are only tags: https://github.com/spekulatius/PHPScraper/tags. So it would be nice to read something about what is the versioning strategy and when are breaks expected.
Of course semver would be nice with an 1.0.0 release and a CHANGELOG.md and UPGRADE.md. My personal example would look like this https://github.com/nadar/quill-delta-parser/blob/master/CHANGELOG.md and this https://github.com/nadar/quill-delta-parser/blob/master/UPGRADE.md
Thanks and keep up to great work!
Symfony\Component\HttpClient\Exception\TransportException: SSL certificate problem: unable to get local issuer certificate for ***
Getting an error when trying to get data from a link that doesn't have ssl
Allow to select presets of common browsers (chrome-like, Firefox, Safari) presets with agent and maybe other headers using simple function calls:
$web->firefox();
$web->chrome();
$web->safari();These should be chainable as well.
I'm trying to integrate this into an app that is already locked to psr/log:^3.
jeremykendall/php-domain-parser version 5 is locked to psr/log:^1
There are 2 tests that fail when bumping to the new version. I can fix and submit a PR.
Allow to receive various feeds sites might provide:
rsssitemap.xmlindex.jsonInclusive the respective content.
Should we split the library and docs into 2 different repos?
Because when installing the library, we only use the phpscraper.php file, and the website directory is not necessary.
Is there data scraping using classes as a reference, or DOM structure? Thank you.
It would be great if the most common link types could be interpreted directly. To mind jump:
In my branch I've removed the magic __get and __call methods, and moved what was core into the phpscraper, so now there is only one class.
After a while I got tired of find/replace, so I created a rector rule to change the properties to method.
Is there a demo repository that uses I can use to test my branch? Tests are passing, except for the ones related to internal/external links, which I'll address in another issue.
I am using this example https://phpscraper.de/examples/extract-keywords.html with PHP 8.1.8
Deprecated: strpos(): Passing null to parameter #1 ($haystack) of type string is deprecated in /Users/khanakia/D1/www/php/scrap_php/vendor/spekulatius/phpscraper/src/phpscraper.php on line 858
PHP Deprecated: strpos(): Passing null to parameter #1 ($haystack) of type string is deprecated in /Users/khanakia/D1/www/php/scrap_php/vendor/spekulatius/phpscraper/src/phpscraper.php on line 859
Allow for returning of a lists set (unorder, order) with each list items below.
Currently unsure how to handle more HTML within the
Previously, there was an issue about exposing the Goutte/Client via core that was fixed but never released.
My code was depending on it via a dev branch.
In the latest version, this functionality was removed. I recommend that it be added back in.
In UsesGoutte.php,
/**
* Retrieve the client
*
* @param \Goutte\Client $client
*/
public function client(): GoutteClient
{
return $this->client;
}E.g.
// GET request
$web->get('https://...');
// POST request
$response = $web->post('https://...', [
'param' => 'first param',
]);
// ...This could be done either directly in PHPScraper or built upon another specialized lib such as Symfony HTTP. Exposing the functionality of the existing dependency sounds like a reasonable way to go, if the idea is of interest.
I wanted to know the response code of the url but could not get access.
Looking at Goutte under the hood, I think I could get it if I could do something like:
$web->getClient()->getInternalResponse()->getStatusCode()
So, I would like a function to get access to the $client, like:
public function getClient()
{
return $this->client;
}
It would make sense to parse the structured data JSON provided by some sites within the head-tag. This way the already accessed information from the meta tags could be made more robust and possibility extended later on.
Ref: https://developers.google.com/search/docs/data-types/article
How can I get all element with class for example "card_header_f3"?
Is this method used? Where is charset() defined?
/**
* Get the header collected as an array
*
* @return array
*/
public function headers()
{
return [
'charset' => $this->charset(),
'contentType' => $this->contentType(),
'viewport' => $this->viewport(),
'canonical' => $this->canonical(),
'csrfToken' => $this->csrfToken(),
];
}Hi
Im getting an
SSL connect error for "https://bbc.in/3cNMnkw".
when connecting to some websites. I'm writing a tweetdeck style system and so will scrape the site that the tweet links to to pick up the details so that a nicer link can be generated. Any ideas on how to solve this ? Most work fine, its just a few that fail.
Also Im sometimes getting a "This browser is no longer supported" coming through as a description from the page. I notice you are passing through 'Mozilla/5.0 (compatible; PHP Scraper/0.x; +https://phpscraper.de)' as the user agent. Is it worth altering the user agent ? I dont think I can pass it in as a parameter anywhere from what I can see?
Thanks.
Could be implemented similar to CSV (simple & header).
Greetings,
I would like to know how I can skip errors when the site does not load correctly or is not working, for example:
$scrap = $web->go('https://asdsadsadassd.com');
if ( $scrap )
{
echo 'ok';
}
Problem
Running composer require spekulatius/phpscraper fails on macOS 10.15.5 with the following:
[RuntimeException]
Failed to extract spekulatius/phpscraper: (50) '/usr/bin/unzip' -qq '/xxx/xxx/vendor/composer/tmp-123dcc14bbc3272649fdd489b0ecc9a3' -d '/xxx/xxx/vendor/composer/74030a62'
error: cannot create /xxx/xxx/vendor/composer/74030a62/spekulatius-PHPScraper-fb4f590/tests/resources/assets/katze-+?-++-+?.jpg
Illegal byte sequence
error: cannot create /xxx/xxx/vendor/composer/74030a62/spekulatius-PHPScraper-fb4f590/tests/resources/assets/???.jpg
Illegal byte sequence
error: cannot create /xxx/xxx/vendor/composer/74030a62/spekulatius-PHPScraper-fb4f590/websites/test-pages/assets/katze-+?-++-+?.jpg
Illegal byte sequence
error: cannot create /xxx/xxx/vendor/composer/74030a62/spekulatius-PHPScraper-fb4f590/websites/test-pages/assets/???.jpg
Illegal byte sequence
It seems like the unzip command within composer fails due to the strange characters.
How can I scrape a website that requires authentication?
That is, I want to start with at https://jardinado.herokuapp.com/login, fill in my credentials, and THEN start scraping the site.
That is, I want the $goutteClient to execute something like this first, then scrape:
if ($username) {
$crawler = $gouteClient->request('GET', $url = $baseUrl . "/login", [
]);
// select the form and fill in some values
$form = $crawler->selectButton('login-btn')->form();
$form['_username'] = 'user';
$form['_password'] = 'pass';
// submit that form
$crawler = $gouteClient->submit($form);
$response = $gouteClient->getResponse();Now that cookies are set, when I fetch a url that requires login I should get the page instead of the 302 (redirect to login).
I'm not sure how to implement this within the context of phpscraper. One idea would be to expose the goutte client.
Is there a way to click on links using CSS selectors instead of name or URL?
How do I check if the scraped page is an error page?
It would be helpful to have the http status for that - something like
$web->go('https://httpstat.us/404');
echo $web->status; // prints 404
Just before posting, I found the solution myself:
$web->client->getResponse()->getStatusCode()
It would be great to have this added to the documentation.
Or maybe add $web->status as shortcut?
Hello, I have a problem, I want to have all the links from // * [@ id = "content"] 'how do I do?
Sets to discover certain data (e.g. URLs, etc.) from markup and plain text content
When i run the sample code:
$web = new \Spekulatius\PHPScraper\PHPScraper();
$web->go('https://www.google.com/');
echo $web->title;It return:
Spekulatius\PHPScraper\Core::setHttpClient(): Argument #1 ($httpClient) must be of type Symfony\Component\HttpClient\CurlHttpClient, Symfony\Component\HttpClient\NativeHttpClient given, called in C:\www\web-crawer\vendor\spekulatius\phpscraper\src\PHPScraper.php on line 108
PHP: 8.1.13
PHPScraper: 1.0.1
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.