{"id":15609,"date":"2022-10-12T07:20:03","date_gmt":"2022-10-12T01:50:03","guid":{"rendered":"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/"},"modified":"2022-10-12T07:20:03","modified_gmt":"2022-10-12T01:50:03","slug":"solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it","status":"publish","type":"post","link":"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/","title":{"rendered":"[Solved] BeautifulSoup &#8211; Amazon and Google identify me as a robot; how can i fix it?"},"content":{"rendered":"<p> [ad_1]<br \/>\n<\/p>\n<div id=\"answer-69902839\" class=\"answer js-answer accepted-answer js-accepted-answer\" data-answerid=\"69902839\" data-parentid=\"69875028\" data-score=\"0\" data-position-on-page=\"2\" data-highest-scored=\"0\" data-question-has-accepted-highest-score=\"0\" itemprop=\"suggestedAnswer\" itemscope itemtype=\"https:\/\/schema.org\/Answer\">\n<div class=\"post-layout\">\n<div class=\"votecell post-layout--left\"><\/div>\n<div class=\"answercell post-layout--right\">\n<div class=\"s-prose js-post-body\" itemprop=\"text\">\n<ul>\n<li>Rotating proxies<\/li>\n<li>Delays<\/li>\n<li>Avoid the same pattern<\/li>\n<li>IP rate limit (<em>probably your issue<\/em>)<\/li>\n<\/ul>\n<p>IP rate limit. It&#8217;s a basic security system that can ban or block incoming requests from the same IP. It means that a regular user would not make 100 requests to the same domain in a few seconds with the exact same pattern (<em>scroll, click, scroll, click, open. As an example<\/em>).<\/p>\n<p><a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/dev.to\/dimitryzub\/how-to-reduce-chance-being-blocked-while-web-scraping-search-engines-1o46#ip\">How to reduce the chance of being blocked while web scraping search engines<\/a>.<\/p>\n<hr>\n<p>Alternatively, you can use <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/serpapi.com\/shopping-results\">Google Shopping Results API<\/a> from SerpApi. It&#8217;s a paid API with a free plan.<\/p>\n<p>The difference in your case is that you don&#8217;t have to spend time figuring out how to bypass blocks from Google since it&#8217;s already done for the end-user.<\/p>\n<p>Example code to integrate to parse data from Google Shopping and <a rel=\"nofollow noopener\" target=\"_blank\" href=\"https:\/\/replit.com\/@DimitryZub1\/Scrape-Google-Shopping-Product-Result-python#serpapi_get_product_results.py\">example in the online IDE<\/a>:<\/p>\n<pre class=\"lang-py prettyprint-override\"><code>import os\nfrom serpapi import GoogleSearch\n\n\nparams = {\n    \"api_key\": os.getenv(\"API_KEY\"),\n    \"engine\": \"google_product\",\n    \"product_id\": \"14506091995175728218\", # can be iterated over multiple product ids\n    \"gl\": \"us\",                           # country to search from\n    \"hl\": \"en\"                            # language\n}\n\nsearch = GoogleSearch(params)\nresults = search.get_dict()\n\ntitle = results['product_results']['title']\nprices = results['product_results']['prices']\nreviews = results['product_results']['reviews']\nrating = results['product_results']['rating']\nextensions = results['product_results']['extensions']\ndescription = results['product_results']['description']\nuser_reviews = results['product_results']['reviews']\nreviews_results = results['reviews_results']['ratings']\n\nprint(f'{title}\\n'\n    f'{prices}\\n'\n    f'{reviews}\\n'\n    f'{rating}\\n'\n    f'{extensions}\\n'\n    f'{description}\\n'\n    f'{user_reviews}\\n'\n    f'{reviews_results}')\n\n\n'''\nGoogle Pixel 4 White 64 GB, Unlocked\n['$247.79', '$245.00', '$439.00']\n526\n3.7\n['October 2019', 'Google', 'Pixel Family', 'Pixel 4', 'Android', '5.7\u2033', 'Facial Recognition', '8 MP front camera', 'Smartphone', 'With Wireless Charging']\nPoint and shoot for the perfect photo. Capture brilliant color and control the exposure balance of different parts of your photos. Get the shot without the flash. Night Sight is now faster and easier to use it can even take photos of the Milky Way. Get more done with your voice. The new Google Assistant is the easiest way to send texts, share photos, and more. A new way to control your phone. Quick Gestures let you skip songs and silence calls \u2013 just by waving your hand above the screen. End the robocalls. With Call Screen, the Google Assistant helps you proactively filter our spam before your phone ever rings.\n526\n[{'stars': 1, 'amount': 101}, {'stars': 2, 'amount': 43}, {'stars': 3, 'amount': 39}, {'stars': 4, 'amount': 73}, {'stars': 5, 'amount': 270}]\n'''\n<\/code><\/pre>\n<p>Example to iterate over multiple item ID&#8217;s:<\/p>\n<pre class=\"lang-py prettyprint-override\"><code># import os\n# from serpapi import GoogleSearch\n\n\n# random numbers except the first one\nproducts = ['14506091995175728218', '1450609199517512118', '145129895175728218']\n\n\nfor product in products:\n    params = {\n        \"api_key\": os.getenv(\"API_KEY\"),\n        \"engine\": \"google_product\",\n        \"product_id\": product,\n        \"gl\": \"us\",\n        \"hl\": \"en\"   \n    }\n\n    search = GoogleSearch(params)\n    results = search.get_dict()\n\n    title = results['product_results']['title']\n\n    print(title, sep='\\n')  # prints 3 titles from 3 different products\n<\/code><\/pre>\n<blockquote>\n<p>Disclaimer, I work for SerpApi.<\/p>\n<\/blockquote><\/div>\n<div class=\"mt24\"><\/div>\n<\/div>\n<p>            <span class=\"d-none\" itemprop=\"commentCount\"><\/span> <\/p><\/div>\n<\/div>\n<p>[ad_2]<\/p>\n<p>solved BeautifulSoup &#8211; Amazon and Google identify me as a robot; how can i fix it? <\/p>\n","protected":false},"excerpt":{"rendered":"<p>[ad_1] Rotating proxies Delays Avoid the same pattern IP rate limit (probably your issue) IP rate limit. It&#8217;s a basic security system that can ban or block incoming requests from the same IP. It means that a regular user would not make 100 requests to the same domain in a few seconds with the exact &#8230; <a title=\"[Solved] BeautifulSoup &#8211; Amazon and Google identify me as a robot; how can i fix it?\" class=\"read-more\" href=\"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/\" aria-label=\"More on [Solved] BeautifulSoup &#8211; Amazon and Google identify me as a robot; how can i fix it?\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[320],"tags":[622,349,760],"class_list":["post-15609","post","type-post","status-publish","format-standard","hentry","category-solved","tag-beautifulsoup","tag-python","tag-web-scraping"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>[Solved] BeautifulSoup - Amazon and Google identify me as a robot; how can i fix it? - JassWeb<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"[Solved] BeautifulSoup - Amazon and Google identify me as a robot; how can i fix it? - JassWeb\" \/>\n<meta property=\"og:description\" content=\"[ad_1] Rotating proxies Delays Avoid the same pattern IP rate limit (probably your issue) IP rate limit. It&#8217;s a basic security system that can ban or block incoming requests from the same IP. It means that a regular user would not make 100 requests to the same domain in a few seconds with the exact ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/\" \/>\n<meta property=\"og:site_name\" content=\"JassWeb\" \/>\n<meta property=\"article:published_time\" content=\"2022-10-12T01:50:03+00:00\" \/>\n<meta name=\"author\" content=\"Kirat\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kirat\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/\"},\"author\":{\"name\":\"Kirat\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31\"},\"headline\":\"[Solved] BeautifulSoup &#8211; Amazon and Google identify me as a robot; how can i fix it?\",\"datePublished\":\"2022-10-12T01:50:03+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/\"},\"wordCount\":189,\"publisher\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\"},\"keywords\":[\"beautifulsoup\",\"python\",\"web-scraping\"],\"articleSection\":[\"Solved\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/\",\"url\":\"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/\",\"name\":\"[Solved] BeautifulSoup - Amazon and Google identify me as a robot; how can i fix it? - JassWeb\",\"isPartOf\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#website\"},\"datePublished\":\"2022-10-12T01:50:03+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/jassweb.com\/solved\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"[Solved] BeautifulSoup &#8211; Amazon and Google identify me as a robot; how can i fix it?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/jassweb.com\/solved\/#website\",\"url\":\"https:\/\/jassweb.com\/solved\/\",\"name\":\"JassWeb\",\"description\":\"Build High-quality Websites\",\"publisher\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/jassweb.com\/solved\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\",\"name\":\"Jass Web\",\"url\":\"https:\/\/jassweb.com\/solved\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png\",\"contentUrl\":\"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png\",\"width\":693,\"height\":132,\"caption\":\"Jass Web\"},\"image\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31\",\"name\":\"Kirat\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586\",\"contentUrl\":\"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586\",\"caption\":\"Kirat\"},\"sameAs\":[\"http:\/\/jassweb.com\"],\"url\":\"https:\/\/jassweb.com\/solved\/author\/jaspritsinghghumangmail-com\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"[Solved] BeautifulSoup - Amazon and Google identify me as a robot; how can i fix it? - JassWeb","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/","og_locale":"en_US","og_type":"article","og_title":"[Solved] BeautifulSoup - Amazon and Google identify me as a robot; how can i fix it? - JassWeb","og_description":"[ad_1] Rotating proxies Delays Avoid the same pattern IP rate limit (probably your issue) IP rate limit. It&#8217;s a basic security system that can ban or block incoming requests from the same IP. It means that a regular user would not make 100 requests to the same domain in a few seconds with the exact ... Read more","og_url":"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/","og_site_name":"JassWeb","article_published_time":"2022-10-12T01:50:03+00:00","author":"Kirat","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Kirat","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/#article","isPartOf":{"@id":"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/"},"author":{"name":"Kirat","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31"},"headline":"[Solved] BeautifulSoup &#8211; Amazon and Google identify me as a robot; how can i fix it?","datePublished":"2022-10-12T01:50:03+00:00","mainEntityOfPage":{"@id":"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/"},"wordCount":189,"publisher":{"@id":"https:\/\/jassweb.com\/solved\/#organization"},"keywords":["beautifulsoup","python","web-scraping"],"articleSection":["Solved"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/","url":"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/","name":"[Solved] BeautifulSoup - Amazon and Google identify me as a robot; how can i fix it? - JassWeb","isPartOf":{"@id":"https:\/\/jassweb.com\/solved\/#website"},"datePublished":"2022-10-12T01:50:03+00:00","breadcrumb":{"@id":"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/jassweb.com\/solved\/solved-beautifulsoup-amazon-and-google-identify-me-as-a-robot-how-can-i-fix-it\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/jassweb.com\/solved\/"},{"@type":"ListItem","position":2,"name":"[Solved] BeautifulSoup &#8211; Amazon and Google identify me as a robot; how can i fix it?"}]},{"@type":"WebSite","@id":"https:\/\/jassweb.com\/solved\/#website","url":"https:\/\/jassweb.com\/solved\/","name":"JassWeb","description":"Build High-quality Websites","publisher":{"@id":"https:\/\/jassweb.com\/solved\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/jassweb.com\/solved\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/jassweb.com\/solved\/#organization","name":"Jass Web","url":"https:\/\/jassweb.com\/solved\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/","url":"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png","contentUrl":"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png","width":693,"height":132,"caption":"Jass Web"},"image":{"@id":"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31","name":"Kirat","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/image\/","url":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586","contentUrl":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586","caption":"Kirat"},"sameAs":["http:\/\/jassweb.com"],"url":"https:\/\/jassweb.com\/solved\/author\/jaspritsinghghumangmail-com\/"}]}},"_links":{"self":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts\/15609","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/comments?post=15609"}],"version-history":[{"count":0,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts\/15609\/revisions"}],"wp:attachment":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/media?parent=15609"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/categories?post=15609"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/tags?post=15609"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}