{"id":5897,"date":"2023-10-18T14:47:43","date_gmt":"2023-10-18T14:47:43","guid":{"rendered":"https:\/\/royadata.io\/blog\/?p=5897"},"modified":"2023-10-18T14:47:43","modified_gmt":"2023-10-18T14:47:43","slug":"python-requests-retry","status":"publish","type":"post","link":"http:\/\/royadata.io\/blog\/python-requests-retry\/","title":{"rendered":"Python Requests Retry: How to Retry Failed Requests in Python"},"content":{"rendered":"<blockquote>\n<p>Looking to learn how to integrate retries in the right way in your Python requests scraping script? Then you are on the right page, as the article below provides you the step-by-step guide on how to get that done.<\/p>\n<\/blockquote>\n<p><picture class=\"aligncenter size-full wp-image-23284 perfmatters-lazy\" loading=\"lazy\"><source type=\"image\/webp\" data-srcset=\"https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry.jpg.webp 1000w, https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry-300x167.jpg.webp 300w, https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry-768x426.jpg.webp 768w\" srcset=\"data:image\/svg+xml,%3Csvg%20xmlns='http:\/\/www.w3.org\/2000\/svg'%20viewBox='0%200%201000%20555'%3E%3C\/svg%3E\" data-sizes=\"(max-width: 1000px) 100vw, 1000px\"\/><img decoding=\"async\" src=\"data:image\/svg+xml,%3Csvg%20xmlns='http:\/\/www.w3.org\/2000\/svg'%20viewBox='0%200%201000%20555'%3E%3C\/svg%3E\" alt=\"Python Requests Retry\" width=\"1000\" height=\"555\" data-src=\"https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry.jpg\" data-srcset=\"https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry.jpg 1000w, https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry-300x167.jpg 300w, https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry-768x426.jpg 768w\" data-sizes=\"(max-width: 1000px) 100vw, 1000px\" loading=\"lazy\"\/>\n<\/picture>\n<noscript><picture class=\"aligncenter size-full wp-image-23284\"><source type=\"image\/webp\" srcset=\"https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry.jpg.webp 1000w, https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry-300x167.jpg.webp 300w, https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry-768x426.jpg.webp 768w\" sizes=\"(max-width: 1000px) 100vw, 1000px\"\/><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry.jpg\" alt=\"Python Requests Retry\" width=\"1000\" height=\"555\" srcset=\"https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry.jpg 1000w, https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry-300x167.jpg 300w, https:\/\/royadata.io\/blog\/wp-content\/uploads\/2023\/10\/Python-Requests-Retry-768x426.jpg 768w\" sizes=\"(max-width: 1000px) 100vw, 1000px\"\/>\n<\/picture>\n<\/noscript><\/p>\n<p>Whenever you design the logic of your Python requests-based web scraper, you need to keep in mind that things will not always go your own way. And one of the problems that may occur is that your requests will fail. This could be a result of a connection error or even your target blocking you. If this happens, your code to break and throw an exception.<\/p>\n<p>While throwing an exception is not a bad thing, we will want a code that is robust and can even retry some of its actions until it is sure it can\u2019t proceed before calling up on you. Fortunately for us, Python\u2019s requests module does have support for retrying requests for the number of times you set it. In this guide, I will show you how to make use of this to retry failed requests in your web scrapers.<\/p>\n<hr\/>\n<h2 style=\"text-align: center;\"><span class=\"ez-toc-section\" id=\"Python_Requests_Retry_%E2%80%94_An_Overview\"><\/span><strong>Python Requests Retry \u2014 An Overview<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<div class=\"su-youtube su-u-responsive-media-yes\">\n<div class=\"perfmatters-lazy-youtube\" data-src=\"https:\/\/www.youtube.com\/embed\/KVxnvAAM0CQ\" data-id=\"KVxnvAAM0CQ\" data-query=\"\" onclick=\"if (!window.__cfRLUnblockHandlers) return false; perfmattersLazyLoadYouTube(this);\" data-cf-modified-9ada457200d03c200afcd002-=\"\">\n<div><img loading=\"lazy\" decoding=\"async\" class=\"perfmatters-lazy\" src=\"data:image\/svg+xml,%3Csvg%20xmlns='http:\/\/www.w3.org\/2000\/svg'%20viewBox='0%200%20480%20360%3E%3C\/svg%3E\" data-src=\"https:\/\/i.ytimg.com\/vi\/KVxnvAAM0CQ\/hqdefault.jpg\" alt=\"YouTube video\" width=\"480\" height=\"360\" data-pin-nopin=\"true\"\/><\/p>\n<div class=\"play\"\/><\/div>\n<\/div>\n<p><noscript><iframe loading=\"lazy\" width=\"600\" height=\"400\" src=\"https:\/\/www.youtube.com\/embed\/KVxnvAAM0CQ?\" frameborder=\"0\" allowfullscreen=\"\" allow=\"autoplay; encrypted-media; picture-in-picture\" title=\"\"\/><\/noscript><\/div>\n<p>Requests have been designed to be robust in themselves and make a lot of your work easier for you. However, it does have one aspect you just have to do things from your own end. Requests is designed in a way that there is no retry function or logic embedded into it.<\/p>\n<p>If you need your code to retry requests that failed, you will need to code the logic for that, and for the most part, you will either have to use the Retry Object in urllib or simply code the retry function yourself. Fortunately for us, doing this is not difficult, as you can do that with just a few lines of code.<\/p>\n<hr\/>\n<h2 style=\"text-align: center;\"><span class=\"ez-toc-section\" id=\"How_to_Retry_Requests_Using_Custom_Retry_Logic\"><\/span><strong>How to Retry Requests Using Custom Retry Logic<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<div class=\"su-youtube su-u-responsive-media-yes\">\n<div class=\"perfmatters-lazy-youtube\" data-src=\"https:\/\/www.youtube.com\/embed\/gzQUzslnfeU\" data-id=\"gzQUzslnfeU\" data-query=\"\" onclick=\"if (!window.__cfRLUnblockHandlers) return false; perfmattersLazyLoadYouTube(this);\" data-cf-modified-9ada457200d03c200afcd002-=\"\">\n<div><img loading=\"lazy\" decoding=\"async\" class=\"perfmatters-lazy\" src=\"data:image\/svg+xml,%3Csvg%20xmlns='http:\/\/www.w3.org\/2000\/svg'%20viewBox='0%200%20480%20360%3E%3C\/svg%3E\" data-src=\"https:\/\/i.ytimg.com\/vi\/gzQUzslnfeU\/hqdefault.jpg\" alt=\"YouTube video\" width=\"480\" height=\"360\" data-pin-nopin=\"true\"\/><\/p>\n<div class=\"play\"\/><\/div>\n<\/div>\n<p><noscript><iframe loading=\"lazy\" width=\"600\" height=\"400\" src=\"https:\/\/www.youtube.com\/embed\/gzQUzslnfeU?\" frameborder=\"0\" allowfullscreen=\"\" allow=\"autoplay; encrypted-media; picture-in-picture\" title=\"\"\/><\/noscript><\/div>\n<p>The easiest method to retry requests with Python requests is by developing your own custom retry logic. All you basically have to do is have a variable that holds the number of tries you want and then loop through it in a range as you send the web requests. Once your requests is successful, you break out of the for loop. If it does not, you keep retrying until you reach the maximum allowed retries before throwing an exception. Below is an example of how to get that done.<\/p>\n<pre>import requests\n\n\n\nNUMBER_OF_RETRIES = 5\n\n\n\nfor i in range(NUMBER_OF_RETRIES):\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 try:\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 response = requests.get(\u201cYOUR_TARGET_URL\u201d)\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if response.status_code in [200, 404]:\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 break\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 except requests.exceptions.ConnectionError:\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pass\n\n\n\nIf response is not None and response.status_code == 200:\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 print(response.content)<\/pre>\n<p>As you can see above, I check the status code to see whether it is 200 (successful) or 404 (page not found). This is because, in either of these cases, there is no need to retry the request. The last if statement is to check if the request was successful and then carry out the task you want. In my own case above, all I did was to print the content to the screen.<\/p>\n<p>In another way, you can just create a reusable retry function that wraps Python\u2019s requests. Below is how to get that done.<\/p>\n<pre>import requests \n\n\n\ndef retry_requests(url, num_of_retries=5,\u00a0 **kwargs):\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 for i in range(num_of_retries):\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 try:\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 response = requests.get(url, **kwargs)\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if response.status_code in [200, 404]:\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 return response\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 except requests.exception.ConnectionError:\n\n\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 pass\n\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 return None\n\n\n\nx = retry_requests(\u201chttps:\/\/google.com\/search\u201d, number_of_retries=3)\n\n\n\nprint(x,content)<\/pre>\n<hr\/>\n<h2 style=\"text-align: center;\"><span class=\"ez-toc-section\" id=\"How_to_Retry_Requests_Using_Sessions_HTTPAdapter\"><\/span><strong>How to Retry Requests Using Sessions &#038; HTTPAdapter<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<div class=\"su-youtube su-u-responsive-media-yes\">\n<div class=\"perfmatters-lazy-youtube\" data-src=\"https:\/\/www.youtube.com\/embed\/Hui1IvaIAfM\" data-id=\"Hui1IvaIAfM\" data-query=\"\" onclick=\"if (!window.__cfRLUnblockHandlers) return false; perfmattersLazyLoadYouTube(this);\" data-cf-modified-9ada457200d03c200afcd002-=\"\">\n<div><img loading=\"lazy\" decoding=\"async\" class=\"perfmatters-lazy\" src=\"data:image\/svg+xml,%3Csvg%20xmlns='http:\/\/www.w3.org\/2000\/svg'%20viewBox='0%200%20480%20360%3E%3C\/svg%3E\" data-src=\"https:\/\/i.ytimg.com\/vi\/Hui1IvaIAfM\/hqdefault.jpg\" alt=\"YouTube video\" width=\"480\" height=\"360\" data-pin-nopin=\"true\"\/><\/p>\n<div class=\"play\"\/><\/div>\n<\/div>\n<p><noscript><iframe loading=\"lazy\" width=\"600\" height=\"400\" src=\"https:\/\/www.youtube.com\/embed\/Hui1IvaIAfM?\" frameborder=\"0\" allowfullscreen=\"\" allow=\"autoplay; encrypted-media; picture-in-picture\" title=\"\"\/><\/noscript><\/div>\n<p>Another way to integrate the request logic into your code is to use the sessions and HTTPAdapter. This is a little more complex compared to just creating the logic yourself, as done above. However, if you use the Requests.Session object often, there is no better option for you to retry file requests than to make use of this. Let&#8217;s first take a look at the code then I explain.<\/p>\n<pre><em>import requests<\/em>\n\n\n\n<em>from requests.adapters import HTTPAdapter<\/em>\n\n\n\n<em>from requests.packages.urllib3.util.retry import Retry<\/em>\n\n\n\n<em>s = requests.Session()<\/em>\n\n\n\n<em>retries = Retry(total=5, status_forcelist=[429, 500, 502, 503, 504])<\/em>\n\n\n\n<em>s.mount('http:\/\/', HTTPAdapter(max_retries=retries))<\/em>\n\n\n\n<em>x = s.get(TARGET_URL)<\/em><\/pre>\n<p>As you can see above, the code depends on the HTTPAdapter\u00a0 and Session Class in the requests module as well as the Retry Object.<\/p>\n<hr\/>\n<h2 style=\"text-align: center;\"><span class=\"ez-toc-section\" id=\"FAQs_About_Python_Requests_Retry\"><\/span><strong>FAQs About Python Requests Retry<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<h3><span class=\"ez-toc-section\" id=\"Q_Does_Python_Requests_Support_Out_of_the_Box\"><\/span><strong>Q. Does Python Requests Support Out of the Box?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Python\u2019s requests does come with support for retry out of the box. But it is not obvious to most beginners. If you check the last code above, you will see that the retry function is present in the requests module. However, its usage is quite advanced. If you find it difficult to comprehend, you can still create your own retry function that you understand so you can understand the logic of your script better.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Q_Why_Do_Python_Requests_Fail\"><\/span><strong>Q. Why Do Python Requests Fail?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Requests failing has nothing to do with Python requests. Once you try to access a remote resource, many things could go wrong that will cause your request to fail. You might be accessing a URL that\u2019s taking too long to respond until timeout, or even your connection not going through. In some other cases, it could just be as a request of bad request from the way the request is sent.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Q_Is_Python_Requests_Good_at_Handling_Retries\"><\/span><strong>Q. Is Python Requests Good at Handling Retries?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>Yes, Python request is good at handling retries, provided you instruct it to do so. As stated earlier, it does have support for that, and that is what was discussed in the second method of handling retries. For the first method, you aren\u2019t depending on the support of requests but coding your own logic to prevent your script from throwing an exception when your request fails but retry it a number of times first.<\/p>\n<hr\/>\n<pre style=\"text-align: center;\"><strong>Conclusion<\/strong><\/pre>\n<p>Having your code retry request makes your code robust and protects it from occasional breaking that isn\u2019t really a reason for the code to break. I do web scraping a lot, and because of the region I am with bad Internet, your script can just break because of low Internet connectivity. But with the help of retries, I did not have to interface until a real issue that needed my attention occurred.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Looking to learn how to integrate retries in the right way in your Python requests scraping script? Then you are on the right page, as the article below provides you the step-by-step guide on how to get that done. Whenever you design the logic of your Python requests-based web scraper, you need to keep in &#8230; <a title=\"Python Requests Retry: How to Retry Failed Requests in Python\" class=\"read-more\" href=\"http:\/\/royadata.io\/blog\/python-requests-retry\/\" aria-label=\"More on Python Requests Retry: How to Retry Failed Requests in Python\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":84,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/posts\/5897"}],"collection":[{"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/comments?post=5897"}],"version-history":[{"count":0,"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/posts\/5897\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/media\/84"}],"wp:attachment":[{"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/media?parent=5897"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/categories?post=5897"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/royadata.io\/blog\/wp-json\/wp\/v2\/tags?post=5897"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}