{"id":6296,"date":"2023-10-18T14:47:43","date_gmt":"2023-10-18T14:47:43","guid":{"rendered":"https:\/\/royadata.io\/blog\/?p=6296"},"modified":"2023-10-18T14:47:43","modified_gmt":"2023-10-18T14:47:43","slug":"use-chrome-headless-and-dedicated-proxies-to-scrape-any-website","status":"publish","type":"post","link":"http:\/\/royadata.io\/blog\/use-chrome-headless-and-dedicated-proxies-to-scrape-any-website\/","title":{"rendered":"Use Chrome Headless and Dedicated Proxies to Scrape Any Website"},"content":{"rendered":"

\"Use\n<\/picture>\n

Everybody knows Google Chrome is the market leader when it comes to web browsing.<\/p>\n

\n

At least, that\u2019s what the statistics say: an overwhelming 61.20 percent of Internet users are browsing with Chrome as of 2018.<\/p>\n<\/blockquote>\n

If Chrome is the leading web browser, then it makes sense that Chrome Headless will be the leading browser for automated application testing<\/a>, web scraping<\/a>, and more. Google\u2019s release of Puppeteer<\/a><\/strong>, the Node.js API that makes automating web actions simple for Chrome users, sets the stage for easy, robust web scraping.<\/p>\n

\"Pyramid\n<\/picture>\n

The only problem is there is no built-in privacy or identity protection. You need to be able to use all the great features Chrome o\ufb00ers while being able to automate nearly anything you want and be able to do so anonymously through proxies<\/a>.<\/p>\n

\n

This can prove to be a challenge.<\/strong><\/p>\n<\/blockquote>\n

It turns out there are ways to use Chrome Headless with proxies to safely scrape website data without exposing your identity<\/strong>. To find out, we\u2019ll have to start with what makes Chrome what it is, and what makes a headless browser what it is.<\/p>\n\n

<\/span>What is Chrome Headless?<\/a><\/span><\/h2>\n

Google describes Chrome Headless<\/a><\/strong>\u00a0as, \u201c\u2026a way to run the Chrome browser in a headless environment. Essentially, running Chrome without chrome!\u201d<\/p>\n

Perfect. Clear as mud.<\/p>\n

A better way to think of Chrome Headless is running Chrome without the graphic interface.<\/p>\n

That\u2019s actually the definition of a headless browser \u2013 instead of clicking on things, you execute code through a command-line interface or network communication.<\/p>\n

There are plenty of reasons why you may want to do that. Automating Google Chrome is otherwise very di\ufb03cult to do, which is why web application developers have historically used other browsers like the now-defunct PhantomJS for the purpose. But if so much of the Internet uses Chrome, it just makes sense to have Headless Chrome in your toolkit.<\/p>\n

\"Phantom\n<\/picture>\n

If you need to scrape data from a variety of websites and need a browser that can handle all the HTML, CSS, and JavaScript without generating error after error, Headless Chrome is the solution for you. Once you learn how to use proxies with it, you\u2019ll be able to scrape just about any website on the Internet with ease and style.<\/p>\n\n

<\/span>How Chrome Headless Works<\/span><\/h2>\n

The basic environment o\ufb00ered by Chrome Headless should be familiar to anyone who has spent any time using another headless browser. You have a command-line interface perfectly suited to quickly scanning and scraping website data.<\/p>\n

What you have in addition to this is Google\u2019s fast, modern JavaScript engine and DevTools API. You have Google engineering ensuring you get support for every website on the Internet.<\/p>\n

But controlling Chrome Headless requires using specific libraries. There are lots of options here, and Puppeteer is one of the most popular, since it, too, is a Google product. Although there are libraries that use a variety of languages, NodeJS APIs are generally recommended because that\u2019s the same language as the data interpreted on the webpages you plan on scraping.<\/p>\n

If you want to use a simple, no-nonsense API that is designed for web scraping and not much else, you can use NickJS<\/a><\/strong> . If you want all-purpose mapping through a complete API that is somewhat similar to Google\u2019s DevTools, go with Puppeteer. You can even use a C++ API<\/a><\/strong> if you want.<\/p>\n

\"Working\n<\/picture>\n

But in order to fully leverage Chrome Headless for scraping, you need to be able to use it with proxies. As mentioned in the introduction to this article, that can be tricky.<\/p>\n\n

<\/span>How to Use Proxies with Chrome Headless<\/span><\/h2>\n

If you use a custom proxy that requires authentication, you may have found the Puppeteer library lacking full support. You need to use the page.authenticate() function to pass your log-in credentials into the Chrome login dialog box.<\/p>\n

Example code would look like this:<\/p>\n

\n

const puppeteerSmme=require(\u2018puppeteer’);<\/p>\n

(async() => {<\/p>\n

const proxyUrl = \u2018http:\/\/proxy.example.com:8000′;<\/p>\n

const username = \u2018bob’;
\nconst password = \u2018password123′;<\/p>\n

const browser = await puppeteer.launch({<\/p>\n

args: [\u2018\u2013proxy-server=${proxyUrl}’],<\/p>\n

headless: false,
\n});<\/p>\n

const page = await browser.newPage();<\/p>\n

await page.authenticate({ username, password });<\/p>\n

await page.goto(\u2018https:\/\/www.example.com’);<\/p>\n

await browser.close();
\n})();<\/p>\n<\/blockquote>\n

This is a simple way to use a proxy on Headless Chrome for web scraping. However, it can\u2019t do everything you may need your authenticated proxy browser to do. For instance, there is a chance it will hang up on a page that requires authentication since it\u2019s not clear from the code how the headless browser will handle multiple authentication requests.<\/p>\n

In that case, you can use Apify\u2019s proxy-chain<\/a><\/strong> package. Essentially, this package ensures that you can anonymize an authenticated proxy through Puppeteer by pushing it through a local proxy server first. It supports HTTP proxy forwarding and tunneling through HTTP CONNECT \u2013 so you can also use it when accessing HTTPS and FTP.<\/p>\n

\"HTTP\n<\/picture>\n

Here is what using Apify\u2019s proxy-chain package with Puppeteer looks like in Node.JS:<\/p>\n

\n

const puppeteer=require(\u2018puppeteer’);consl<\/p>\n

proxyChain = require(\u2018proxy-chain’);<\/p>\n

(async() => {<\/p>\n

const oldProxyUrl = \u2018http:\/\/bob:[email\u00a0protected]<\/a>:8000′;<\/p>\n

const newProxyUrl = await proxyChain.anonymizeProxy(oldProxyUrl);<\/p>\n

\/\/ Prints an IP address\u2028 console.log(newProxyUrl);<\/p>\n

const browser = await puppeteer.launch({<\/p>\n

args: [\u2018\u2013proxy-server=${newProxyUrl}’],<\/p>\n

});<\/p>\n

const page = await browser.newPage();<\/p>\n

await page.goto(\u2018https:\/\/www.example.com’);<\/p>\n

await page.screenshot({<\/p>\n

path: \u2018example.png’<\/p>\n

});<\/p>\n

await browser.close();<\/p>\n

})();<\/p>\n<\/blockquote>\n

This will cover most of the web scraping requests you\u2019re likely to make. There are additional, more advanced options<\/strong><\/a> for custom responses and connecting to external APIs, but in many cases, this code will do the job.<\/p>\n\n

You maybe like to read,<\/p>\n