{"id":6309,"date":"2023-10-18T14:47:43","date_gmt":"2023-10-18T14:47:43","guid":{"rendered":"https:\/\/royadata.io\/blog\/?p=6309"},"modified":"2023-10-18T14:47:43","modified_gmt":"2023-10-18T14:47:43","slug":"scrape-amazon","status":"publish","type":"post","link":"http:\/\/royadata.io\/blog\/scrape-amazon\/","title":{"rendered":"7 Things to Know Before Scraping Amazon Product Results"},"content":{"rendered":"

\"scrape\n<\/picture>\n

There are a lot of reasons you might want to scrape data from Amazon. As a competing retailer, you might want to keep a database of their pricing data, so you can try to match them. You might want to keep an eye on competitors selling through the Amazon Marketplace. Maybe you want to aggregate review scores from around the Internet, and Amazon is one of the sources you\u2019ll want to use. You could even be selling on Amazon yourself, and using the scraper to keep ahead of others doing the same.<\/p>\n

I don\u2019t recommend some of the more black hat uses for data scraping. If you\u2019re scraping product descriptions to use for your own site, all you\u2019re doing is shooting yourself in the foot as far as SEO is concerned. You should avoid basing your business model on scraped Amazon data; more on that later.<\/p>\n

There are a lot of pieces of software out there designed to help you scrape Amazon data, as well as some that are general use screen scraping tools. You can get a lot of mileage out of them, but always exercise caution. You\u2019ll want to be very sure of the validity of a piece of software before you drop $400 on it. Oh, and if the product you\u2019re researching is primarily marketed through low-view YouTube videos with a\ufb03liate links in the descriptions, I recommend staying away.<\/p>\n

On the other hand, you can use scripts rather than software. Scripts have the benefit of being infinitely configurable, as well as open-source by definition. As long as you have some idea of what you\u2019re doing with the code, you can read it to make sure there\u2019s nothing tricky going on, and you can change it to work exactly the way you want it to. Of course, that relies on you having enough code knowledge to be able to create and change a script. Knowing PHP<\/a>, CURL<\/a>, XML, JS and other languages is a good idea.<\/p>\n

If you\u2019ve looked into this before, you might ask yourself why you would want to use some third party scraper or a script you barely understand when you could just use Amazon\u2019s API<\/a>. It\u2019s true that, for some purposes, Amazon\u2019s API is a good alternative. However, it doesn\u2019t provide you with all of the data you might want. The API is primarily designed for a\ufb03liates to use it for advertising in a custom way, using some method that isn\u2019t covered by a\ufb03liate links or the product box widgets Amazon provides. There\u2019s a lot of data it won\u2019t let you harvest.<\/p>\n

Technically, scraping data has been against Amazon\u2019s policy for a long time. It wasn\u2019t until 2012 that they really started enforcing it, however, so a lot of people got away with scraping data for a long time. Finally, when they did, many people considered it an insulting disruption<\/strong><\/a> to their business model, never considering how they were never in the right from the start.<\/p>\n

\"Know\n<\/picture>\n

The point I\u2019m trying to make here is that if you\u2019re scraping data from Amazon, what you\u2019re doing is against their terms of service. That means you\u2019re always at risk of numerous penalties. Usually, Amazon just shrugs and bans your IP. However, if you\u2019ve been an especially tenacious pest or are using their data in a way they don\u2019t approve of, they are perfectly within their rights to take you to court over it. This is, obviously, something to be avoided.<\/p>\n

All of that said, Amazon seems to have slackened up in recent years. This thread<\/a>\u00a0from 2014 indicates that Amazon doesn\u2019t bother with enforcing low-scale scraping blocks. They have automated systems that will slap you with a ban if you cross their path, but they aren\u2019t actively and persistently seeking out and banning all data scrapers. It makes sense; a retailer of their size has so much data to filter through on an hourly basis that it would be impossible to ban every single data scraper.<\/p>\n

Before you continue, here are seven things you should know about making Amazon the target of your data scraping. By keeping them in mind, you should be able to keep yourself safe from both automated bans and legal action.<\/p>\n\n

<\/span>1. Amazon is Very Liberal with IP Bans<\/span><\/h2>\n

\"Amazon\n<\/picture>\n

The first thing to keep in mind if you\u2019re going to be harvesting data from Amazon is that Amazon very much is liberal with their bans. You won\u2019t be harvesting data while logged into an account, at least, not if you\u2019re smart. That means the only way you\u2019ll be able to be banned is through an IP ban.<\/p>\n

The nice thing is that IP bans can be circumvented. The bad thing is that an IP ban from Amazon is not a punishment to be taken lightly. As far as I know, such bans are permanent.<\/p>\n

There\u2019s a prevailing attitude across the web, at least among tech circles, that IP bans are ine\ufb00ectual and ine\ufb03cient. They can also be detrimental to the site that uses them, depending on how broad an IP ban they use. You don\u2019t want to ban a whole IP block; you might be eliminating an entire neighborhood from your potential customer base. At the same time, anyone dedicated to getting around an IP ban can do it with relative ease.<\/p>\n

This is, more or less, true. IP bans are easy enough to get around, specifically if you\u2019re expecting and preparing for a ban in the first place. That\u2019s what proxy servers are for.<\/p>\n

A proxy server, in case you aren\u2019t aware, is a way to filter your IP address. The website, in this case Amazon, will see your connection as coming from the proxy server rather than your home connection. If they ban you, they ban the proxy<\/a>, and you can just use a di\ufb00erent proxy.<\/p>\n

Therefore, in order to maximize your chances of successfully harvesting all of the data you need on an ongoing basis, you\u2019ll need numerous proxy servers. You want to be able to cycle through them to avoid any one IP being flagged for bot-like activity. You also want to have backups in case any of your proxies are banned, so you can keep harvesting without issues.<\/p>\n\n

<\/span>2. Amazon is Very Good at Detecting Bots<\/span><\/h2>\n

The number one mistake that scrapers make when harvesting data from Amazon, or any other site with a high profile and a plan to ban scrapers, is using scraper software<\/a> without configuring it properly.<\/p>\n

Think about it. If you were tasked with detecting bots and filtering them out from legitimate tra\ufb03c, what would you look for? There are simple things, like the user agent and whether or not it identifies itself as a bot. Those are easily spoofed, though.<\/p>\n

A more accurate way of detecting bots is by their behavior. A poorly programmed bot will try to make as many requests as possible as quickly as possible or will make them on a fixed timer. Bots are, by definition, robotic. They repeat actions, they make the same set of actions in the same order with the same timing again and again.<\/p>\n

Amazon is very good at distinguishing between bot actions and human actions. Therefore, to avoid your bots being banned, you need to mimic human behavior as much as possible. Don\u2019t be repetitive. Don\u2019t be predictable. Vary your actions, your timing, and your IP. It\u2019s harder to identify a bot when it only accesses a couple of pages. From your end, you have an unbroken stream of data; from their end, a hundred di\ufb00erent users came and performed in a similar way.<\/p>\n

Safer for you, harder for them to handle.<\/p>\n\n

<\/span>3. You Absolutely Must Follow Laws and Keep a Low Profile<\/span><\/h2>\n

There are some regulations that apply to bots of all sorts, including content scrapers. I hesitate to call them laws, because there\u2019s very little actual legal precedent about all of this. One of the big high profile cases, though, is Craigslist versus Padmapper<\/strong><\/a>. Padmapper took Craigslist data for real estate listings and laid it over a Google Maps interface. This is a decent business model, and Padmapper still exists today, but Craigslist took o\ufb00ense to the use of their data in a way that didn\u2019t benefit them.<\/p>\n

In the end, Craigslist won that case, though it didn\u2019t go to a court judgment. Instead, it was settled out of court. This is a good lesson to take to heart. If you step on the wrong toes \u2013 and don\u2019t comply with cease and desist letters immediately \u2013 you can be the subject of legal action, and you\u2019re usually in the wrong. You can read about that case and others here<\/strong><\/a>.<\/p>\n

Data scrapers on their own do not violate any laws unless you\u2019re harvesting private data, or you\u2019re harvesting at a rate that is disruptive to the operation of the site, such as a DDoS attack would do. Your scraper must act as a public visitor and cannot access internal data or data that an account is required to access.<\/p>\n

Otherwise, all of the restrictions placed upon you are more about the way you use that data, rather than the way you obtain it.<\/p>\n