{"id":5914,"date":"2023-10-18T14:47:43","date_gmt":"2023-10-18T14:47:43","guid":{"rendered":"https:\/\/royadata.io\/blog\/?p=5914"},"modified":"2023-10-18T14:47:43","modified_gmt":"2023-10-18T14:47:43","slug":"node-unblocker","status":"publish","type":"post","link":"http:\/\/royadata.io\/blog\/node-unblocker\/","title":{"rendered":"Node Unblocker \u2013 Create and Customize Your Own Proxy Network"},"content":{"rendered":"
\n

Are you looking to have control over your proxy network when you are surfing anonymously? Then you are in the right place. The information in this post will show you how to create and customize your own proxy network using Node Unblocker.<\/p>\n<\/blockquote>\n

\"Node\n<\/picture>\n

The Internet has several advantages, particularly in the fields of data collection and digital marketing. However, because there are so many cyberattacks, websites have learned to guard their domains relentlessly. However, there are some simple tasks that would necessitate browsing the internet anonymously. Some of which include web data scraping and accessing valuable geo-restricted content.<\/p>\n

As such, using a web proxy is one of the best ways to browse the internet more privately. Apart from securing your privacy, web proxies can also help you get around various restrictions and avoid blocks and bans. You can either buy an expensive off-the-shelf web proxy or create your own using a web proxy package like Node Unblocker. In order to route all incoming requests through a proxy, Node Unblocker will add an extra layer, regardless of the website you are attempting to scrape or access. This can be quite beneficial for freely browsing and scraping websites.<\/p>\n

In this article, we’ll explain what a Node Unblocker is and how you can use it to create your own web proxy in order to get around internet censorship and anti-web scraping tools. We would also examine its advantages and disadvantages further. Let’s begin, then.<\/p>\n\n

<\/span>What is Node Unblocker?<\/strong><\/span><\/h2>\n

\"What\n<\/picture>\n

Node Unblocker is essentially a web proxy that enables you to access blocked or restricted content on your target website. Node Unblocker is, in other words, a multipurpose library for proxying and modifying remote websites. It’s a Node.js package with an Express-compatible API that enables easy setup of your proxy. It has a very flexible JS interface and can be used in a variety of ways. Any Node.js developer who has prior experience with Express will find it simple to use the proxy server.<\/p>\n

The ability to use the Node Unblocker as a REST API is a key difference between a traditional HTTP or SOCKS5 proxy and a node unblocker proxy. As such, the REST API endpoints that the Node Unblocker exposes are how browsers and web scrapers communicate with it. You only need to add the page’s URL to the proxy endpoint if you want to retrieve it.<\/p>\n

The content is extracted by the Node Unblocker server using the IP address of the computer it is currently operating on. Node-Unblocker is a very fast and effective proxy choice since it makes sure that all the data is processed and transmitted to the client without needless buffering. Additionally, it is very adaptable and can be useful in a variety of ways thanks to its JS interface.<\/p>\n

In essence, a node unblocker server enables you to browse the internet anonymously, access geographically restricted content, and even scrape data from a website without being detected or banned. For instance, if you want to use AJAX calls to scrape data from a website or behind a login wall, this proxy can help.<\/p>\n

However, you should be aware that this wouldn’t be effective if your aim is a Google or Facebook-related website.\u00a0 That being said, let’s look at how to set up Node Unblocker.<\/p>\n\n

<\/span>Setting Up Node Unblocker<\/strong><\/span><\/h2>\n

\"Setting\n<\/picture>\n

Node unblocker server setup is simple and straightforward. You would however need to install all the necessary dependencies. As such, be sure to have Node.js and npm installed on your computer in order to set up the node-unblocker. You can do it by using a version management application like nvm or by referring to the official documentation from the Node.js website. Make sure to follow along with this tutorial as we go through the processes.<\/p>\n

<\/span>Step 1\uff1aintegrate Node Unblocker server\u00a0<\/strong><\/span><\/h3>\n

You need to integrate the Node Unblocker server with an API server like Express after installing Node.js on your computer. This can be done using the npm command:<\/p>\n

npm init -y\n\n\n\nnpm install unblocker express<\/pre>\n
Express is a straightforward web framework that can help Node.js developers quickly build servers. The use() method of the Express-compatible proxy enables you to establish a connection between the proxy instance and the Express server. On the other hand, the package name for Node Unblocker is Node Unblocker. In the code above, an Express app, a new unblocker instance, and npm are all initialized.<\/p>\n
<\/span>Step 2: build the Node Unblocker server<\/strong><\/span><\/h3>\n
Let’s build the Node Unblocker server by creating an app.js file. Observe the code below:<\/p>\n
\/\/ app.js<\/em>\n\n\n\nvar express = require('express')\n\nvar Unblocker = require('unblocker');\n\n\n\n\/\/ This Creates Express Server<\/em>\n\nvar app = express();\n\n\n\n\/\/ This Creates Unblocker Instance, and Configure your Express Server to Use It\n\n<\/em>var unblocker = new Unblocker({prefix: '\/proxy\/'});\n\napp.use(unblocker);\n\n\n\n\/\/ Launches Server on Port 8080<\/em>\n\n\n\napp.listen(process.env.PORT || 8080).on('upgrade', unblocker.onUpgrade);\n\nconsole.log(\"Node Unblocker Server Running On Port:\", process.env.PORT || 8080)<\/pre>\n
Your dependencies are first required in the code above. It then went ahead and started a new Unblocker instance and Express app. Additionally, only the prefix property is set, as you would see. This goes one step further by indicating the path, in this example, \u201c\/proxy\/,\u201d at which the proxy can be accessed. To connect the proxy instance with your Express server, all you have to do is call the use() method, thanks to the API’s compatibility with Express. This implies that you can use the listen() method to launch your Express server.<\/p>\n
If everything went according to plan, this basic server would be running at localhost:8080 or the PORT environment variable you specified when it was launched. Additionally, we have added the .on (upgrade handler) to the app. listen process.env.port 8080. This is helpful for informing the proxy if the connection protocol switches from HTTP to WebSocket.<\/p>\n
<\/span>Testing The Server.<\/strong><\/span><\/h3>\n
Now that you have setup the Node Unlocker proxy server, you will need to test it to be sure it is working properly. Implement this by first running the script using Node.js.<\/p>\n
node app.js<\/pre>\n
The console.log() message should appear in your terminal if everything is done right. Take a URL and attach localhost: [PORT]\/ proxy\/ to ensure the proxy is working. For instance:<\/p>\n
http:\/\/localhost:8080\/proxy\/https:\/\/www.amazon.com<\/pre>\n
Using the curl command, you can attempt this in your terminal as well. Even better, just type the aforementioned URL into your browser’s URL bar to test the Node Unblocker server. The request is made using the IP address of the computer executing the server when it is sent through the Node Unblocker server. The IP address won’t change if the Node Unblocker is being used locally. You must deploy the unblocker to a remote server if you wish to modify the IP address.<\/p>\n
If the proxy has any problems, be sure to set the DEBUG environment option so you can see the full details of each request:<\/p>\n
DEBUG=unblocker:* node index.js<\/pre>\n
<\/span>Deploying to Heroku.<\/strong><\/span><\/h3>\n
\"Deploying\n<\/picture>\n
Now that we are sure the server we created works properly, let’s look at how to deploy it to Heroku<\/a>. To use the Node Unblocker proxy server you just created, you simply need to deploy the server on a remote server and send requests to it. One such remote server is Heroku. You can deploy your proxy server anywhere; however, Heroku is one of the simplest options with a free plan.<\/p>\n
<\/span>Configuring The Server for Deployment.<\/strong><\/span><\/h3>\n
You must first change your package.json file to include the setup information for the server in order to deploy the proxy server to Heroku, or any other hosting provider for that matter. This will enable you to use the hosting provider’s machine’s IP address. In this situation, Heroku’s computer.<\/p>\n
{\n\n\n\n\"name\": \"proxy-server\",\n\n\n\n\"version\": \"1.0.0\",\n\n\n\n\"main\": \"app.js\",\n\n\n\n\"private\": true,\n\n\n\n\"engines\": {\n\n\n\n\"node\": \"16.x\"\n\n\n\n},\n\n\n\n\"dependencies\": {\n\n\n\n\"express\": \"^4.17.1\",\n\n\n\n\"unblocker\": \"^2.3.0\"\n\n\n\n},\n\n\n\n\"scripts\": {\n\n\n\n\"start\": \"node app.js\"\n\n\n\n}\n\n\n\n}<\/pre>\n
You’ve created a script in the code above to instruct Heroku on how to launch your app. It has an engine component that specifies the Node.js version that should be used if you look at it closely. Heroku is further instructed by the start script on how to launch the newly built proxy. In order to prevent pushing unused files and dependencies to your GitHub repositories and your Heroku server, it would also be helpful to create an\u00a0a.gitignore file.<\/p>\n
# Dependencies<\/em><\/p>\n
node_modules<\/pre>\n
# yarn error logs<\/em><\/p>\n
yarn-error.log<\/pre>\n
# Environment varibales<\/em><\/p>\n
.env*\n\n\n\n!.env*.example<\/pre>\n
# Code coverage<\/em><\/p>\n
coverage<\/pre>\n\n
<\/span>The Acceptable Use Policy<\/strong><\/span><\/h2>\n
The permitted usage policy is something we believe you should keep in mind while you try to install the Node Unblocker proxy application on a remote server. This kind of application cannot always be hosted on a provider’s servers. Even then, it would only be under very stringent restrictions.<\/p>\n
Regarding Heroku, the company policies prohibit hosting proxies for public use or engaging in web scraping activities without adhering to established robot exclusion guidelines, such as those found in the robots.txt file. You could also be required to supply a distinct user-agent string.<\/p>\n\n
<\/span>Deploying with Heroku CLI<\/strong><\/span><\/h2>\n
\"Deploying\n<\/picture>\n
So, in order to proceed, you would need a Heroku server account. Please visit the Heroku website right away to register for a free account if you don’t already have one.<\/p>\n
Step 1<\/strong><\/p>\n
If you already have an account, go ahead and install the Heroku CLI<\/a>\u00a0tool.<\/p>\n
Step 2<\/strong><\/p>\n
The next step is to create a new app using the Heroku dashboard. Use the login command in the CLI to log on to Heroku. This implies that in order to create a new SSH public key, you must provide your Heroku login information through the command line and continue to follow the instructions. On the Heroku dashboard, a new app would also need to be created. Use the following code:<\/p>\n
heroku login\n\n\n\nheroku apps:create<\/pre>\n
Step 3<\/strong><\/p>\n
If your login is successful, you’ll need to initialize Git in the directory the node unblocker created. The ID, URL, and Git URL for your app should all be visible in the console. Go ahead and set a remote origin for the just-established\u00a0Git repo using the ID.<\/p>\n
git init heroku\n\n\n\ngit:remote -a [the_app_ID]<\/pre>\n
Step 4<\/strong><\/p>\n
All that’s left to do is use git to deploy your code to Heroku and commit it.<\/p>\n
git add .\n\n\n\ngit commit -am \"Initial commit\"\n\n\n\ngit push heroku master<\/pre>\n
If you have done everything correctly, your Heroku app URL should allow you to access your proxy. Visit this link to see if it’s functioning properly: https:\/\/[the_app_ID].herokuapp.com\/proxy\/https:\/\/www.amazon.com<\/a>. This request would be forwarded through Heroku in this situation. On the other hand, as soon as it got the request, Amazon’s server would see Heroku’s IP address. Interestingly, you can use this Node Unblocker proxy server as a standalone service or a combination of it and a headless browser like Selenium or Puppeteer, especially in web scraping scenarios.<\/p>\n\n
<\/span>Generating the Node Unblocker Proxy Network.<\/strong><\/span><\/h2>\n
\"Node\n<\/picture>\n
It is also possible to scale the previous scenario and create your own proxy network or pool using it. If you distributed Node Unblocker servers on several devices, this would be quite useful. You can install the Node Unblocker proxy server on several Heroku machines as a means of scaling a proxy pool.<\/p>\n
Additionally, you can go ahead and rent relatively inexpensive remote servers, which start at around $2.00. After that, you would need to set up your web scrapers or bots to send each request to a different server from this proxy network. An illustration of a Python request scraper is shown below:<\/p>\n
import requests<\/p>\n
from itertools import cycle<\/p>\n
## These are the Node Unblocker Proxies<\/p>\n
list_proxy = [\n\n\n\n\"https:\/\/200.450.15.30:8080\",\n\n\n\n\"https:\/\/200.450.15.31:8080\",\n\n\n\n\"https:\/\/200.450.15.32:8080\",\n\n\n\n]\n\n\n\nproxy_cycle = cycle(list_proxy)\n\n\n\nproxy = next(proxy_cycle)\n\n\n\nfor i in range(1, 10):\n\n\n\nproxy = next(proxy_cycle)\n\n\n\nprint(proxy)\n\n\n\nproxies = {\n\n\n\n\"http\": proxy,\n\n\n\n\"https\":proxy\n\n\n\n}\n\n\n\nr = requests.get(url='https:\/\/www.amazon.com', proxies=proxies)\n\n\n\nprint(r.text)<\/pre>\n
Another approach may be that you want to scrape geo-restricted content, let’s say a France-restricted web page. By deploying Node Unblocker on this France-based server, you can use its proxy to access websites restricted to the French region. You can implement this with the following code:<\/p>\n
let FRonlyUrl = \"https:\/\/www.anexampleofwebsite.com\"\n\n\n\nfetch(`https:\/\/localhost:8080\/proxy\/${FRonlyUrl}`)\n\n\n\nlet proxyPool = [\n\n\n\n\"https:\/\/413.345.11.21:8080\",\n\n\n\n\"https:\/\/413.345.11.22:8080\",\n\n\n\n\"https:\/\/413.345.11.23:8080\",\n\n\n\n]\n\n\n\nlet proxy = proxyPool[Math.floor(Math.random()*proxyPool.length)];\n\n\n\nurl = \"https:\/\/www.anexampleofwebsite.com\"\n\n\n\nfetch(`${proxy}\/proxy\/${FRonlyUrl}`)<\/pre>\n
Let’s look at one more sophisticated usage of such a proxy: adding extra logic using request and response middlewares.<\/p>\n\n
<\/span>Node Unblocker Middleware<\/strong><\/span><\/h2>\n
\"NodeUnblocker\n<\/picture>\n
\u00a0<\/p>\n
So far, we have seen how to setup and deploy a Node Unblocker proxy server. However, to unlock more functionalities, you can use Node Unblocker middleware to make the proxy servers very powerful and efficient when using them.<\/p>\n
Node Unblocker middlewares are essentially little functions that let you alter requests and responses as they are being received by your proxy server. You can change requests and respond to them faster. Node Unblocker uses middleware to implement the majority of its features. Custom middleware can, however, also be used to take the place of the built-in debug activation.<\/p>\n
By including them in the requestMiddleware<\/em> and\/or responseMiddleware<\/em> arrays, you can alter both outgoing requests and incoming responses.<\/p>\n
\/\/ app.js<\/em>\n\n\n\nvar express = require('express')\n\n\n\nvar Unblocker = require('unblocker');\n\n\n\n\/\/ This Creates Express Server<\/strong>\n\n\n\nvar app = express();\n\n\n\n\/\/ This creates Unblocker Instance, and Configure the Express Server to Use It<\/em>\n\n\n\nvar unblocker = new Unblocker(\n\n\n\n{\n\n\n\nprefix: '\/proxy\/',\n\n\n\nrequestMiddleware: [] \/\/ Custom request middlewares go here\n\n\n\nresponseMiddleware: [] \/\/ Custom response middlewares go here\n\n\n\n});\n\n\n\napp.use(unblocker);\n\n\n\n\/\/ This launches Server on Port 8080<\/em>\n\n\n\napp.listen(process.env.PORT || 8080).on('upgrade', unblocker.onUpgrade);\n\n\n\nconsole.log(\"Node Unblocker Server Running On Port:\", process.env.PORT || 8080)\n\n\n\nYou have to set standardMiddleware to false (standardMiddleware: <\/em><\/strong>false<\/em>) in order to alter the middleware that Node Unblocker uses by default. Additionally, go ahead and turn on the desired default middleware.\n\n\n\nThe feature of Unblocker's Request Middlewares can be used frequently to add unique headers and user agents to your requests.\n\n\n\nYou can use the following code to add the user agent:\n\n\n\n\/\/ app.js<\/em>\n\n\n\nvar express = require('express')\n\n\n\nvar Unblocker = require('unblocker');\n\n\n\n\/\/ This Creates Express Server<\/em>\n\n\n\nvar app = express();\n\n\n\n\/\/ Your User Agent Middleware<\/em>\n\n\n\nfunction setUserAgent(data) {\n\n\n\ndata.headers[\"user_agent\"] = \"Mozilla\/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit\/605.1.15 (KHTML, like Gecko) Mobile\/15E148\";\n\n\n\n}\n\n\n\n\/\/ Create Unblocker Instance\n\n\n\nvar unblocker = new Unblocker(\n\n\n\n{\n\n\n\nprefix: '\/proxy\/',\n\n\n\nrequestMiddleware: [\n\n\n\nsetUserAgent\u00a0 \/\/ Enable User Agent Middleware\n\n\n\n]\n\n\n\n});\n\n\n\n\/\/ Configure Our Express Server to Use It\n\n\n\napp.use(unblocker);\n\n\n\n\/\/ Launches Server on Port 8080\n\n\n\napp.listen(process.env.PORT || 8080).on('upgrade', unblocker.onUpgrade);\n\n\n\nconsole.log(\"Node Unblocker Server Running On Port:\", process.env.PORT || 8080)\n\n\n\nAdditionally, you can set it up to request specific URLs using certain types of headers.\n\n\n\nfunction setUserAgent(data) {\n\n\n\nif (data.url.match(\/^https?:\\\/\\\/google.com\\\/\/)) {\n\n\n\ndata.headers[\"user-agent\"] = \"Mozilla\/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit\/605.1.15 (KHTML, like Gecko) Mobile\/15E148\";\n\n\n\n}\n\n\n\n}<\/pre>\n\n
<\/span>Benefit of Using Node Unblocker Middlewares<\/strong><\/span><\/h2>\n
Your infrastructure can become significantly more maintainable if you use Node Unblocker middleware. This is because you can extract a lot of information about your proxy setup from your proxy server using the requests<\/em> and responses <\/em>middleware with Node Unblocker. Hence, your proxy server would have this logic and would apply it to every request it receives from your scrapers, saving you from having to manage and maintain proxy header and user-agent configurations in each of your scrapers or bots.<\/p>\n\n
<\/span>Why You Should Use Node Unblocker?<\/strong><\/span><\/h2>\n
Node Unblocker has become one of the most popular and helpful tools for users who want to access restricted content. There are quite a number of benefits you stand to enjoy when you use Node Unblocker. Let’s get to see some of them.<\/p>\n