Skip to content

CViniciusSDias/google-crawler

Repository files navigation

Google Crawler

Latest Stable Version Build Status Code Coverage Scrutinizer Code Quality License

A simple Crawler for getting Google results.

This component can be used to retrieve the 100 first results for a search term.

Since google detects a crawler and blocks the IP when several requests are made, this component is prepared to use some online proxy services, such as hide.me.

Installation

Install the latest version with

$ composer require cviniciussdias/google-crawler

Usage

Crawler class constructor prototype

CViniciusSDias\GoogleCrawler\Crawler::__construct(
    SearchTermInterface $searchTerm, GoogleProxyInterface $proxy = null,
    string $googleDomain = 'google.com', string $countryCode = ''
)

Parameters

  • $searchTerm Term that will be searched on Google
  • $proxy Online proxy service that will be used to access Google [optional]
  • $googleDomain Your country specific google domain, like google.de, google.com.br, etc. [optional]
  • $countryCode Country code that will be added to gl parameter on Google's url, indicating the location of the search. E.g. 'BR', 'US', 'DE' [optional]

Examples

Without proxy

<?php
use CViniciusSDias\GoogleCrawler\{
    Crawler, SearchTerm
};

$searchTerm = new SearchTerm('Test');
$crawler = new Crawler($searchTerm); // or new Crawler($searchTerm, new NoProxy());

$resultList = $crawler->getResults();

With some proxy

<?php
use CViniciusSDias\GoogleCrawler\{
    Crawler, SearchTerm, Proxy\CommonProxy
};

$searchTerm = new SearchTerm('Test');
$commonProxy = new CommonProxy('https://us.hideproxy.me/includes/process.php?action=update');
$crawler = new Crawler($searchTerm, $commonProxy);

$resultList = $crawler->getResults();

More details on proxies

To know more details about which proxies are currently supported, see the files inside tests/Functional folder. There you'll see all the available proxies.

Iterating over results

foreach ($resultList as $result) {
    $title = $result->getTitle();
    $url = $result->getUrl();
    $description = $result->getDescription();
}

About

Requirements

  • This component works with PHP 7.2 or above
  • This component requires the extension php-ds to be installed

Author

Vinicius Dias (ZCE) - [email protected] - https://github.com/CViniciusSDias/ - http://www.zend.com/en/yellow-pages/ZEND030134

License

This component is licensed under the GPL v3.0 License - see the LICENSE file for details

About

A simple Crawler for getting Google results

Resources

License

Stars

Watchers

Forks

Packages

No packages published