v1.2.0

README.md

@@ -1,94 +1,144 @@
 # puppeteer-page-proxy <img src="https://i.ibb.co/kQrN9QJ/puppeteer-page-proxy-logo.png" align="right" width=150 height=150/>
-Additional Node.js module to use with '[puppeteer](https://www.npmjs.com/package/puppeteer)' for setting proxies per page basis.
+Additional Node.js module to use with **[puppeteer](https://www.npmjs.com/package/puppeteer)** for setting proxies per page basis.
 
-Uses **(http | https | socks)-proxy-agent** to achieve the desired results.
+Forwards intercepted requests from the browser to Node.js where it handles the request then returns the response to the browser, changing the proxy as a result.
 
-This small module consists of a class and a function taken from [ppspider](https://github.com/xiyuan-fengyu/ppspider).
+## Features
 
-All credit for the original code goes to the author [@xiyuan-fengyu](https://github.com/xiyuan-fengyu).
+- Proxy per page **and** per request
+- Supports **(** http, https, socks4, socks5 **)** proxies
+- Authentication
+- Cookie handling internally
 
 ## Installation
 ```
 npm i puppeteer-page-proxy
 ```
+## API
+#### PageProxy(pageOrReq, proxy)
 
-## Methods
-#### PageProxy(page, proxy[, enableCache])
-
-* `page` <[object](https://developer.mozilla.org/en-US/docs/Glossary/Object)> Page object to set a proxy for.
-* `proxy` <[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> Proxy to use in the current page. Must begin with a protocol e.g. **http://**, **https://**, **socks://**.
-* `enableCache` <[boolean](https://developer.mozilla.org/en-US/docs/Glossary/Boolean)> Whether to enable caching. Defaults to `true`.
+- `pageOrReq` <[object](https://developer.mozilla.org/en-US/docs/Glossary/Object)> 'Page' or 'Request' object to set a proxy for.
+- `proxy` <[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> Proxy to use in the current page.
+  * Begins with a protocol e.g. **http://**, **https://**, **socks://**
 
 #### PageProxy.lookup(page[, lookupService, isJSON, timeout])
 
-* `page` <[object](https://developer.mozilla.org/en-US/docs/Glossary/Object)> Page object to execute the request on.
-* `lookupService` <[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> External lookup service to request data from. Fetches data from `api.ipify.org` by default.
-* `isJSON` <[boolean](https://developer.mozilla.org/en-US/docs/Glossary/Boolean)> Whether to [JSON.parse](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse) the received response. Defaults to `true`.
-* `timeout` <[number](https://developer.mozilla.org/en-US/docs/Glossary/Number)|[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> Time in milliseconds after which the request times out. Defaults to `30000` ms.
-* returns: <[Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise)> Promise which resolves to the response of the lookup request.
+- `page` <[object](https://developer.mozilla.org/en-US/docs/Glossary/Object)> 'Page' object to execute the request on.
+- `lookupService` <[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> External lookup service to request data from.
+  * Fetches data from `api.ipify.org` by default.
+- `isJSON` <[boolean](https://developer.mozilla.org/en-US/docs/Glossary/Boolean)> Whether to [JSON.parse](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse) the received response.
+  * Defaults to `true`.
+- `timeout` <[number](https://developer.mozilla.org/en-US/docs/Glossary/Number)|[string](https://developer.mozilla.org/en-US/docs/Glossary/String)> Time in milliseconds after which the request times out.
+  * Defaults to `30000` ms.
+- returns: <[Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise)> Promise which resolves to the response of the lookup request.
 
 **NOTE:** By default this method expects a response in [JSON](https://en.wikipedia.org/wiki/JSON#Example) format and [JSON.parse](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse)'s it to a usable javascript object. To disable this functionality, set `isJSON` to `false`.
 
 ## Examples
-#### General usage:
+### Proxy per page:
 ```js
 const puppeteer = require('puppeteer');
 const useProxy = require('puppeteer-page-proxy');
 
 (async () => {
-    let site = 'https://example.com';
-    let proxy1 = 'http://host:port';
-    let proxy2 = 'https://host:port';
-    let proxy3 = 'socks://host:port';
-
-    // For proxies that require authentication
-    //  pass in the username and password in the format shown below
-    let proxy4 = 'https://login:pass@host:port';
+    const site = 'https://example.com';
+    const proxy = 'http://host:port';
+    const proxy2 = 'https://host:port';
 
     const browser = await puppeteer.launch({headless: false});
 
-    const page1 = await browser.newPage();
-    await useProxy(page1, proxy1);
-    await page1.goto(site);
+    const page = await browser.newPage();
+    await useProxy(page, proxy);
+    await page.goto(site);
 
     const page2 = await browser.newPage();
     await useProxy(page2, proxy2);
     await page2.goto(site);
+})();
+```
+#### Changing proxy if a page is already using one:
+```js
+await useProxy(page, proxy);
+await page.goto(site, {waitUntil: 'networkidle2'});
 
-    const page3 = await browser.newPage();
-    await useProxy(page3, proxy3);
-    await page3.goto(site);
-
-    const page4 = await browser.newPage();
-    await useProxy(page4, proxy4);
-    await page4.goto(site);
+await useProxy(page, proxy2);
+await page.reload();
+```
+#
+### Proxy per request:
+```js
+const puppeteer = require('puppeteer');
+const useProxy = require('puppeteer-page-proxy');
+
+(async () => {
+    const site = 'https://example.com';
+    const proxy = 'socks://host:port';
+
+    const browser = await puppeteer.launch({headless: false});
+    const page = await browser.newPage();
+
+    await page.setRequestInterception(true);
+    page.on('request', async req => {
+        await useProxy(req, proxy); // 'req' as argument
+    });
+    await page.goto(site);
 })();
 ```
+When changing proxies this way, the request object itself is passed as the first argument. Now 'proxy' can be changed every request.
+Leaving it as is will have the same effect as `useProxy(page, proxy)`, meaning that the same proxy will be used for all requests within the page.
+
+**NOTE:** It is necessary [page.setRequestInterception()](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#pagesetrequestinterceptionvalue) to true, otherwise the function will fail.
+#### Using with other request listeners:
+```js
+await page.setRequestInterception(true);
+page.on('request', async req => {
+	if (req.resourceType() === 'image') {
+		req.abort();
+	} else {
+		await useProxy(req, proxy);
+	}
+});
+```
+Internally `puppeteer-page-proxy` handles the request which was passed in, then responds back to the browser using the interception method [request.respond()](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestrespondresponse). Calling this method will fulfill the request with a given response and set the proxy.
+
+**NOTE:** Since all requests can be handled exactly once, it's not possible to call other interception methods e.g. [request.abort()](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestaborterrorcode), [request.continue()](https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#requestcontinueoverrides) after calling `useProxy()`, without getting a *'Request is already handled!'* error message.
+
+#
+#### Authentication:
+```js
+const proxy = 'https://login:pass@host:port';
+```
 #### Lookup IP used by proxy -> Useful in headless environment:
 ```js
 const puppeteer = require('puppeteer');
 const useProxy = require('puppeteer-page-proxy');
 
 (async () => {
-    let site = 'https://example.com';
-    let proxy1 = 'http://host:port';
-    let proxy2 = 'https://host:port';
+    const site = 'https://example.com';
+    const proxy1 = 'http://host:port';
+    const proxy2 = 'https://host:port';
 
     const browser = await puppeteer.launch({headless: false});
 
     /**1*/
     const page1 = await browser.newPage();
     await useProxy(page1, proxy1);
-    let data = await useProxy.lookup(page1); // Waits until done, "then" continues
+    let data = await useProxy.lookup(page1); // Waits until done, 'then' continues
         console.log(data.ip);
     await page1.goto(site);
 
     /**2*/
     const page2 = await browser.newPage();
     await useProxy(page2, proxy2);
-    useProxy.lookup(page2).then(data => {   // Executes and "comes back" once done
+    useProxy.lookup(page2).then(data => {   // Executes and 'comes back' once done
         console.log(data.ip);
     });
     await page2.goto(site);
 })();
 ```
+## Dependencies
+- [Got](https://github.com/sindresorhus/got)
+- [http-proxy-agent](https://github.com/TooTallNate/node-http-proxy-agent)
+- [https-proxy-agent](https://github.com/TooTallNate/node-https-proxy-agent)
+- [socks-proxy-agent](https://github.com/TooTallNate/node-socks-proxy-agent)
+- [tough-cookie](https://github.com/salesforce/tough-cookie)

changelog.md

@@ -0,0 +1,12 @@
+# Change log
+### [1.2.0] - 2020-02-09
+#### Major changes
+- Added capability to change proxy per request.
+- Added capability of changing the proxy if a page is already using one.
+- Replaced [Request](https://github.com/request/request) with [Got](https://github.com/sindresorhus/got) for forwarding requests.
+- Added [tough-cookie](https://github.com/salesforce/tough-cookie) for handling cookies
+- Now communicates directly with Chrome DevTools Protocol when getting cookies
+#### Minor changes
+- Added a simple type enforcer and proxy validator.
+- Removed some redundant code.
+- Removed obsolete 'cache' parameter

package.json

@@ -1,11 +1,11 @@
 {
   "name": "puppeteer-page-proxy",
-  "version": "1.0.9",
   "description": "Additional Node.js module to use with 'puppeteer' for setting proxies per page basis.",
+  "version": "1.2.0",
   "author": "Cuadrix <[email protected]> (https://github.com/Cuadrix)",
   "homepage": "https://github.com/Cuadrix/puppeteer-page-proxy",
-  "main": "puppeteer-page-proxy.js",
-  "types": "puppeteer-page-proxy.d.ts",
+  "main": "./src/index.js",
+  "types": "./src/index.d.ts",
   "scripts": {
     "test": "echo \"Error: no test specified\" && exit 1"
   },
@@ -15,17 +15,17 @@
   },
   "keywords": [
     "puppeteer",
-    "page",
-    "different",
     "proxy",
+    "page",
     "headless",
     "chromium"
   ],
   "license": "MIT",
   "dependencies": {
-    "request": "^2.88.0",
-    "http-proxy-agent": "^2.1.0",
-    "https-proxy-agent": "^2.2.1",
-    "socks-proxy-agent": "^4.0.2"
+    "got": "^10.5.2",
+    "http-proxy-agent": "^4.0.0",
+    "https-proxy-agent": "^5.0.0",
+    "socks-proxy-agent": "^5.0.0",
+    "tough-cookie": "^3.0.1"
   }
 }