![\"Proxies](\"https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Proxies-with-Curl.jpg.webp)
\n<\/picture>\n<\/noscript><\/p>\nThere are plenty of legitimate uses for proxies<\/a>, though, and one of them is data mining<\/strong>. The web is packed full of awesome data resources, but a lot of them are hidden behind gates through which you need to make requests. Some of them, of course, put paywalls in these gates to make money. Others simply use them as a way to filter traffic to prevent excessive access.<\/p>\n These gates are the bane of every data miner. You want a large quantity of data from the publicly accessible database. You try to access it, and the first dozen or so requests work fine. Then you find your page timing out or your requests bouncing due to rate limits and excessive use bans.<\/p>\n When a server like this bans a user for excessive use<\/strong><\/a>, it\u2019s either an account ban or an IP ban. Account bans are di\ufb03cult to deal with. After all, you need to make a new account and jump through all of the hoops to verify it.<\/p>\n I don\u2019t recommend trying to use the proxy method to get around this, though if you have a script to fill out and verify profiles, you can do so easily enough. Excessive account creation and usage in the sequence are likely to lead to an IP ban anyways.<\/p>\n An IP ban, obviously enough, blocks the IP address you\u2019re using to connect to the Internet. If you want to get around it, you need to access the offending website from a different IP address. This is where web proxies come in.<\/p>\n The point of a web proxy, I\u2019m sure you know, is to anonymize your traffic by shuttling it through a third party server. It\u2019s like using a virtual assistant to call for take-out, or a chauffeur to drive you from place to place. You send your traffic request to the proxy, the proxy ferries it on, it gets the results and delivers them back to you.<\/p>\n There are a few problems with using proxies in this manner. For one thing, public free proxy lists<\/a> tend to have either rate limits or usage limits, typically alongside a bandwidth limit.<\/p>\n If you\u2019re making a lot of small requests, or several large requests, your operation is likely to time out partway through the process. This is extremely frustrating, as you might imagine, particularly when trying to complete single large requests that can\u2019t be resumed easily.<\/p>\n Some proxies also render your content in an iframe or use interstitial ads on every third or so request, in order to make some money<\/a>. Depending on the automatic system you\u2019re using to harvest data, this can add junk fields or break your script.<\/p>\n There are also all of the usual issues with using a proxy server, which is almost always located in a foreign country. This adds latency to the requests, making your operations take longer. Proxies are notoriously fickle and prone to dropping requests, all of which cause issues.<\/p>\n Related: How to avoid Proxies Get banned or blocked<\/a>.<\/p>\n Backconnect Proxy API & Dedicated Proxies<\/p>\n<\/blockquote>\n Now more and more the rotating proxy providers<\/strong><\/a> are coming to proxies market, lots of those backconnect proxy providers offer the residential proxy network, It’s a little like Tor network that gives your p2p internet connection, the proxy providers give you a Gateway server or API to access their proxy network.<\/p>\n Read more: Guide to Backconnect Proxy: How Backconnect Proxies Work?<\/a><\/p>\n Learn Brightdata formerly Luminati proxies scaping performance From our review<\/a>.<\/p>\n Learn Oxylabs proxies scaping performance from our review<\/a>.<\/p>\n This is where CURL<\/a><\/strong> and PHP come together to solve the problem. What you do, in this case, is set up a script that rotates through a variety of proxies in a list. You have a list of requests, a list of proxies, and a target site. You make request A from proxy A, then request B from proxy B, and so on down the list. At some point, say request L, you loop back to proxy A because request A has been done for a while.<\/p>\n Though those are dedicated proxies (A list of Fixed IP:Port proxies). The idea is to have a rotating list of proxies to handle your requests in a more organic, low rate manner. To the database site, it looks as though a dozen or so users are just making normal requests. Not only does this distribute the load of the requests you\u2019re putting on the database; it also cuts the time it takes to process your requests significantly.<\/p>\n This distributes the load throughout numerous IP addresses, so it\u2019s harder if not impossible to ban them all. If one does receive a ban, you just add another proxy to the list to replace it. In no time, your harvesting will be complete and you can leave the poor database alone to contemplate its role in life.<\/p>\n Private proxy lists alleviate many of these issues. For one thing, you\u2019re typically paying to access the proxies, so those proxies are clean \u2013 that is, they don\u2019t have to add advertising to your data stream to make money. They have lower volume and can support a more consistent throughput. You\u2019ll have fewer timeouts, fewer lost requests, and less latency.<\/p>\n Of course, that only alleviates the issues with the proxy; you still have the issues with the IP bans from the site you\u2019re harvesting. You can jump from proxy to proxy each time one gets banned, but that\u2019s tedious. Data mining requires as much automation as possible because the real fun begins with analysis rather than the harvest.<\/p>\n I like this particular script<\/strong><\/a>, as I said, in PHP and CURL. The script allows you to fill a file with proxy addresses and submit your requests. It also automatically includes retry logic, so when a request fails, it tries again rather than hanging or skipping to the next request. It\u2019s surprisingly efficient the way it\u2019s programmed, so it doesn\u2019t overload your CPU as well.<\/p>\n If you\u2019re using the Mac OS or Linux \u2013 a Posix system \u2013 you can run the script directly. Windows systems will need a PHP interpreter to run it. You can see examples of the specific usage for downloading files in the readme on the GitHub page.<\/p>\n If you want to do something more advanced than downloading sample files, you\u2019ll need to add a small script to run to make that request. You\u2019re on your own for that one.<\/p>\n When many people think of web proxies, they think of borderline illegal activities. There\u2019s something of a bad reputation generated by spambots, black hat SEOs, and 4chan hackers. When proxies are associated with DDOS attacks and spam, it\u2019s hard to talk about them openly. There are plenty of legitimate uses for proxies, though, and one … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":682,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/posts\/6506"}],"collection":[{"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/comments?post=6506"}],"version-history":[{"count":0,"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/posts\/6506\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/media\/682"}],"wp:attachment":[{"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/media?parent=6506"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/categories?post=6506"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/tags?post=6506"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}<\/span>Avoiding IP and Account Bans<\/span><\/h2>\n
<\/span>Rate Limits and Usage Limits<\/span><\/h2>\n
\n
<\/span>Using IP rotation API for data mining with CURL<\/span><\/h2>\n
<\/span>API with CURL Examples<\/span><\/h3>\n
\n
<\/span>Sample use Luminati.io<\/a><\/strong><\/span><\/h5>\n<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/luminati-API-Examples.jpg.webp)
\n<\/picture>\n<\/noscript><\/p>\n[email\u00a0protected]<\/a>';\n\n$username = 'lum-customer-estvpnserve-zone-zone4-route_err-pass_dyn';\n\n$password = 'qigxme6kr73n';\n\n$port = 22225;\n\n$user_agent = 'Mozilla\/5.0 (Windows NT 6.1) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/41.0.2228.0 Safari\/537.36';\n\n$session = mt_rand();\n\n$super_proxy = 'servercountry-CA.zproxy.lum-superproxy.io';\n\n$curl = curl_init('http:\/\/lumtest.com\/myip.json');\n\ncurl_setopt($curl, CURLOPT_USERAGENT, $user_agent);\n\ncurl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);\n\ncurl_setopt($curl, CURLOPT_PROXY, \"http:\/\/$super_proxy:$port\");\n\ncurl_setopt($curl, CURLOPT_PROXYUSERPWD, \"$username-country-us-dns-local-session-$session:$password\");\n\n$result = curl_exec($curl);\n\ncurl_close($curl);\n\nif ($result)\n\n echo $result;\n\n?>\n\n<\/code><\/pre>\n\n
\n
<\/span>Sample use oxylabs.io<\/a><\/strong><\/span><\/h5>\n<\/li>\n<\/ul>\n
# In this example a query to `example.com` is performed from a random IP address from Munich, Germany.\n\n\n\n\n\n$username<\/span> =<\/span> 'USERNAME'<\/span>;<\/span>\n\n$password<\/span> =<\/span> 'PASSWORD'<\/span>;<\/span>\n\n$country<\/span> =<\/span> 'DE'<\/span>;<\/span>\n\n$city<\/span> =<\/span> 'munich'<\/span>;<\/span>\n\n$proxy<\/span> =<\/span> 'pr.oxylabs.io:7777'<\/span>;<\/span>\n\n$query<\/span> =<\/span> curl_init<\/span>(<\/span>'https:\/\/ipinfo.io'<\/span>);<\/span>\n\ncurl_setopt<\/span>(<\/span>$query<\/span>,<\/span> CURLOPT_RETURNTRANSFER<\/span>,<\/span> 1<\/span>);<\/span>\n\ncurl_setopt<\/span>(<\/span>$query<\/span>,<\/span> CURLOPT_PROXY<\/span>,<\/span> \"http:\/\/<\/span>$proxy<\/span>\"<\/span>);<\/span>\n\ncurl_setopt<\/span>(<\/span>$query<\/span>,<\/span> CURLOPT_PROXYUSERPWD<\/span>,<\/span> \"customer-<\/span>$username<\/span>-cc-<\/span>$country<\/span>-city-<\/span>$city<\/span>:<\/span>$password<\/span>\"<\/span>);<\/span>\n\n$output<\/span> =<\/span> curl_exec<\/span>(<\/span>$query<\/span>);<\/span>\n\ncurl_close<\/span>(<\/span>$query<\/span>);<\/span>\n\nif<\/span> (<\/span>$output<\/span>)<\/span>\n\n echo<\/span> $output<\/span>;<\/span>\n\n?><\/span><\/code><\/pre>\n\n
<\/span>Using dedicated proxies list to\u00a0retrieve data with CURL<\/span><\/h2>\n
<\/span>Benefits of Private Proxies (Versus Shared Ones\/public)<\/span><\/h3>\n
<\/span>Example Script to Try<\/span><\/h3>\n
![\"php](\"https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/php-parallel-downloader.jpg.webp)
\n<\/picture>\n<\/noscript><\/p>\n\n