{"id":16755,"date":"2022-10-22T02:14:02","date_gmt":"2022-10-21T20:44:02","guid":{"rendered":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/"},"modified":"2022-10-22T02:14:02","modified_gmt":"2022-10-21T20:44:02","slug":"solved-unable-to-communicate-with-api","status":"publish","type":"post","link":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/","title":{"rendered":"[Solved] Unable to communicate with API"},"content":{"rendered":"

[ad_1]
\n<\/p>\n

\n
\n
<\/div>\n
\n
\n

CONCLUSION – 07-25-2021<\/h2>\n
\n

After looking at this problem in more detail, I believe that it is NOT technically possible to use Python Requests<\/code> to scrape the website and table in your question.<\/strong><\/em><\/p>\n

Which means that your question cannot be solved in the manner that you would prefer.<\/strong><\/em><\/p>\n

\n

Why?<\/strong><\/p>\n

The website employs anti-scraping mechanisms.<\/p>\n

The GBK<\/code> values are only one part of these mechanisms. The table that you’re trying to scrape has 1504 pages. A new unique GBK<\/code> value is created each time you navigate from page 1 to the next page and so forth. Thus there are 1503 unique GBK<\/code> value.<\/p>\n

The site also uses a unique session management cookie for each page.<\/p>\n

# page 1 cookie\nJSESSIONID=0AC56294FE6857A236F0E68A9106E1AE.7; neCYtZEjo8GmS=5El51n08q7nzOG_bzzqhGWyfW_Lx9tCv1uZA6QjBcUq0rH0d1XYIvTKzN3MfNn2cZasqfZoM8Yo5NTpuq9gM.IG; neCYtZEjo8GmT=53HTPEbke3aQqqqm_6QLwIaUKu0tMygss.En464jhvNz1mMzbOatzmLLtv9x_xiCP6JaO_JzcbvHqtsnQYydBa6B_YjSg6sFm7cVBBOhB35_.TZuwDsbOnDinJkNwMs3AaMPtM83dP9YnogFKHpNJo5.RHMTKT6_XNPr0mxebR6stRrQ7LFfACcWqHHhbc.j6gZfZzxsgwnPE3RGP6aT9nYuMJbvK2EGrdAv0O12G03KTk_BMk.xLeEwrQq5VjyH1tB7t4wQ.jQ1geshvbDPCs8_VHCkd2.6uIag5Md.lngzeDshhSjMrmBjyy0HTqAXQ3; acw_tc=276aedd816272186939626726e424a5dd554d4b095225e2cac90fc6d2da583\n\n# page 2 cookie\nJSESSIONID=651AD12FD349FFB1842E08CA578EA37D.7; neCYtZEjo8GmS=5El51n08q7nzOG_bzzqhGWyfW_Lx9tCv1uZA6QjBcUq0rH0d1XYIvTKzN3MfNn2cZasqfZoM8Yo5NTpuq9gM.IG; neCYtZEjo8GmT=53HTPeKke3e7qqqm_6Q_YEqK9dBPNnJQF00YvHDMLHlJeb.4rrpTsgfwZxU0S5OXIAB2aduoOTmj7RuKIL.LUXRaRqfh5ZByuTFX3LxK1Ia3sr3V45c.PPx6Eas5EF5EkQztquzrX78QIbjrJUcQoKoOKcqgX5UuRIN0gCyGDyI6FFj.JbPhwYf65Hcx9BzDQnrlGAPHM3WGvmKf7OJnLY1SGIuxtdyVUE359Ll2lr0QJxUq1Dacqz_WsFa_ZantBbP7MklHX6J21wmDnyo6s4xCeeTYwsGq.kGUbE74Dx.QjQBCM_SiLKccTog8_EdBDg; acw_tc=276aedd816272186939626726e424a5dd554d4b095225e2cac90fc6d2da583\n\n# page 3 cookie\nJSESSIONID=2121D74E0EFCEC3BE104DAA2791481B6.7; neCYtZEjo8GmS=5El51n08q7nzOG_bzzqhGWyfW_Lx9tCv1uZA6QjBcUq0rH0d1XYIvTKzN3MfNn2cZasqfZoM8Yo5NTpuq9gM.IG; neCYtZEjo8GmT=53HTPeKke3e7qqqm_6wBfEGBZsTF9_uGtgepzPXNOzFh0RNtGcE1Cf4hEQNppVywcI5mk3SlLkzvNll6ovr4XmfL2Ujy3AFZR5leVY2H3_584At3GmIwmnsEjOx5v5e_lMon3AbX9t2W8UiLoK.9SBX0vgNRfkqdpyPjWKk3Zs8gQG0k3_6UwxGTvEwWkaWL8vquJgCGlvLEFTjNvd07eHiR482UfpLPFP6yAkx8Wi9pM79cL.26KE3U2L79hgBKLHyOdNyj3VKOkDsaXefNdPXd.YqT4kevShGxzMM2PuzqnuuQnW.GQ5mr9Rx8VxUjEa; acw_tc=276aedd816272186939626726e424a5dd554d4b095225e2cac90fc6d2da583\n\n<\/code><\/pre>\n
So you would need to acquire both the unique GBK<\/code> value and the unique session management cookie for each page between 2 and 1504.<\/p>\n
I also noted that site employs some type of latency.  The first page can take some time to fully load.  If you attempt to navigate to another page before this page load is complete you will get this message “\u8bf7\u52ff\u9891\u7e41\u64cd\u4f5c!”<\/p>\n
It’s worth nothing that some pages took up to 2 minutes to load.  When they didn’t load the message above was displayed.<\/p>\n
Like I previously mentioned you should consider scraping this site with selenium,<\/code> which might work to bypass the anti-scraping mechanisms.<\/strong><\/em><\/p>\n
UPDATE POST (SELENIUM)- 07-27-2021<\/h2>\n
\n
I have attempted to scrape your target website with selenium<\/code>. The chromedriver<\/code> continually fails to connect to the site. Even if these switches are used:<\/p>\n
chrome_options.add_experimental_option(\"useAutomationExtension\", False)\nchrome_options.add_experimental_option(\"excludeSwitches\", ['enable-automation'])\nchrome_options.add_argument(\"--disable-blink-features=AutomationControlled\")\n<\/code><\/pre>\n
I also tried using the undetected-chromedriver<\/a>, but that also failed to property connect.<\/p>\n
Even when I set a high timeout period with either driver.set_page_load_timeout()<\/code> or driver.implicitly_wait()<\/code> the session still fails with chromedriver<\/code>.<\/p>\n
I was able to access the website when I used selenium<\/code> with the geckodriver.<\/code> There are still time out issues, but adding a WebDriverWait<\/code> couple with an expected_conditions<\/code> seems to overcome some of the timeout issues.<\/p>\n
Scraping this website will be a long and arduous process, because of the anti-scraping mechanisms being used.<\/p>\n
ORIGINAL POST – 07-24-2021<\/h2>\n
\n
First let me say, that your question doesn’t have enough details to make a solid recommendation for solving your issue.<\/em><\/p>\n
I looked into your problem.  I found that your target website uses some Javascript to dynamically create the GBK<\/code> value that is used each post request.<\/p>\n
6SQk6G2z:GBK-5lkb7acLMDDxywZsCHoJagJlT50f1gw4.jaVgaBpBcGZDs1T_pcR_OPFgvOm_6oM8PfyL3L6xDPxFqgIqgwbVAEw8y4jd0P5yTWo3dx1cNLnCOYTa4mVr7azAXa9YiDEhOz7M1Qsw6BJIOSq0QVp.Ng.NWri7ByAK6dwme99ZEOnjraxZex1xLVGakyVVCoOEhFGfphV8D1GDFKLt1dG.4_XuCPDIoLNGmy4Dzd92SxlNWCQ707A8tvqP7jQq2wyRBV0M3y0moSs8I03rIXeYNKE3AkMmI8Xp4M6GZd0seJqGvGrN7vA8lJbiBfmEgtcSvPZF0hrfkVRvQGq9uHRx9JOLtdkujsYHk6TW7rYBVsQ\n<\/code><\/pre>\n
\"enter<\/a><\/p>\n
This GBK<\/code> value is used when navigating between the pages 1 through 1504.  I noted that the value changes for each page.<\/p>\n
\"enter<\/a><\/p>\n
import difflib\n\n# page 2\na = \"6SQk6G2z:GBK-5lkb7acLMDDxywZsCHoJagJlT50f1gw4.jaVgaBpBcGZDs1T_pcR_OPFgvOm_6oM8PfyL3L6xDPxFqgIqgwbVAEw8y4jd0P5yTWo3dx1cNLnCOYTa4mVr7azAXa9YiDEhOz7M1Qsw6BJIOSq0QVp.Ng.NWri7ByAK6dwme99ZEOnjraxZex1xLVGakyVVCoOEhFGfphV8D1GDFKLt1dG.4_XuCPDIoLNGmy4Dzd92SxlNWCQ707A8tvqP7jQq2wyRBV0M3y0moSs8I03rIXeYNKE3AkMmI8Xp4M6GZd0seJqGvGrN7vA8lJbiBfmEgtcSvPZF0hrfkVRvQGq9uHRx9JOLtdkujsYHk6TW7rYBVsQ\"\n\n# page 1504\nb = \"6SQk6G2z:GBK-59tY9cXfYPiYfpgB1rj16jFZNwQuke.NUV5ZljqD6daOH4pxgaFcRE7bERjrvfoY4OTl5PAWUo70VNRIqnYOi_TQCSWzvrcCgfTtEFl_ZdMHRVLhosJLSFwHiPdVn4cXZ7VnF5xahstqJHD6EBfd71iZT8HQBmx1dssd7RWA2Gdv8lGhJbS0ZeaxIVkfK5qaO.lxHVvG_9cq4weBdHeUQlGlIWhxKFYePkTr9Jp0eN2yDTZljeX0XWWOxIjEkdj89FOqaNDB2slUE.54oC96baGe7lttoz_2AoTbjHSTjfDh.eSyT6vA6.5dP5X.4XsFVYSnYKIznIdkjTURmm3kjvGM_iQoYT3V5gAKs1c6r6cE\"\n\n\ns = difflib.SequenceMatcher(None, a, b, autojunk=False)\nfor tag, i1, i2, j1, j2 in s.get_opcodes():\n    if tag != 'equal':\n        print('{:7}   a[{}:{}] --> b[{}:{}] {!r:>8} --> {!r}'.format(\n            tag, i1, i2, j1, j2, a[i1:i2], b[j1:j2]))\n\n        # output \n        insert    a[14:14] --> b[14:49]       '' --> '9tY9cXfYPiYfpgB1rj16jFZNwQuke.NUV5Z'\n        replace   a[15:18] --> b[50:55]    'kb7' --> 'jqD6d'\n        replace   a[19:24] --> b[56:60]  'cLMDD' --> 'OH4p'\n        delete    a[25:49] --> b[61:61] 'ywZsCHoJagJlT50f1gw4.jaV' --> ''\n        insert    a[51:51] --> b[63:158]       '' --> 'FcRE7bERjrvfoY4OTl5PAWUo70VNRIqnYOi_TQCSWzvrcCgfTtEFl_ZdMHRVLhosJLSFwHiPdVn4cXZ7VnF5xahstqJHD6E'\n        insert    a[52:52] --> b[159:243]       '' --> 'fd71iZT8HQBmx1dssd7RWA2Gdv8lGhJbS0ZeaxIVkfK5qaO.lxHVvG_9cq4weBdHeUQlGlIWhxKFYePkTr9J'\n        insert    a[53:53] --> b[244:276]       '' --> '0eN2yDTZljeX0XWWOxIjEkdj89FOqaND'\n        replace   a[54:55] --> b[277:291]      'c' --> '2slUE.54oC96ba'\n        replace   a[56:57] --> b[292:311]      'Z' --> 'e7lttoz_2AoTbjHSTjf'\n        insert    a[58:58] --> b[312:369]       '' --> 'h.eSyT6vA6.5dP5X.4XsFVYSnYKIznIdkjTURmm3kjvGM_iQoYT3V5gAK'\n        delete    a[60:63] --> b[371:371]    'T_p' --> ''\n        delete    a[64:74] --> b[372:372] 'R_OPFgvOm_' --> ''\n        replace   a[75:84] --> b[373:374] 'oM8PfyL3L' --> 'r'\n        replace   a[85:99] --> b[375:376] 'xDPxFqgIqgwbVA' --> 'c'\n        delete    a[100:377] --> b[377:377] 'w8y4jd0P5yTWo3dx1cNLnCOYTa4mVr7azAXa9YiDEhOz7M1Qsw6BJIOSq0QVp.Ng.NWri7ByAK6dwme99ZEOnjraxZex1xLVGakyVVCoOEhFGfphV8D1GDFKLt1dG.4_XuCPDIoLNGmy4Dzd92SxlNWCQ707A8tvqP7jQq2wyRBV0M3y0moSs8I03rIXeYNKE3AkMmI8Xp4M6GZd0seJqGvGrN7vA8lJbiBfmEgtcSvPZF0hrfkVRvQGq9uHRx9JOLtdkujsYHk6TW7rYBVsQ' --> ''\n<\/code><\/pre>\n
The GBK<\/code> value is created with this call in the HTML of the page.<\/p>\n
javascript:commitForECMA(callbackC,\"content.jsp?tableId=27&tableName=TABLE27&tableView=\u6769\u6d98\u5f5b\u9356\u8364\u679f\u9363\u3126\ue76b\u6d5c\u0443\u6427\u951b\u581f\u655e\u9350?&Id=60456\",null)\n<\/code><\/pre>\n
This is the Javascript that is called.<\/p>\n
function commitForECMA($_17, $_12, $_19) {\n    request = createXMLHttp();\n    request.onreadystatechange = $_17;\n    if ($_19 == null) {\n        _$b6(request, _$JI('ZM6r2MG'), _$JI(\"Op0YV\"), $_12);\n        request.setRequestHeader(_$JI(\"RACeXwDYXwcTV8Ur2\"), _$JI(\"9wDYgwceLwDT7iCYX3Ce9FKyvHKwPFa\"));\n    } else {\n        var $_16 = \"\";\n        var $_11 = $_19.elements;\n        var $_14 = $_11.length;\n        for (var $_4 = 0; $_4 < $_14; $_4++) {\n            var $_6 = _$kH($_11, $_4);\n            if ($_6.type != _$JI(\"aQ6YPMK20\") && _$kH($_6, _$JI('Cwbm7wKV')) != \"\") {\n                if ($_16.length > 0) {\n                    $_16 += \"&\" + $_6.name + \"=\" + _$kH($_6, _$JI('swbm7wKV'));\n                } else {\n                    $_16 += $_6.name + \"=\" + _$kH($_6, _$JI('8wbm7wKV'));\n                }\n                $_16 += _$JI(\"xx2J03Up2Hsl\");\n            }\n        }\n        _$b6(request, _$JI('iM6r2MG'), _$JI(\"IVlesYq\"), $_12);\n        $_16 = encodeURI($_16);\n        $_16 = encodeURI($_16);\n        request.setRequestHeader(_$JI(\"53CmOFDVz3CeXwoxBMq\"), _$JI(\"wMbZz3CmOFDV\"));\n        request.setRequestHeader(_$JI(\"ZACeXwDYXwcTV8Ur2\"), _$JI(\"F3UraMD2O3UpNMCgB8cT6w6QzRbenM1TTQbS2MbJBRDY9\"));\n    }\n    request.send($_16);\n    if ($_19 != null) {\n        $_19.reset();\n    }\n}\n\n\ntruncated....\n\n\nfunction createXMLHttp() {\n    if (window.XMLHttpRequest) {\n        return new XMLHttpRequest();\n    } else if (window.ActiveXObject) {\n        var $_17 = [_$JI(\"5sYJ3sVanh2fJslf0woqXJ1ga\"), _$JI(\"osYJ3sVanh2fJslf0woqXJcga\"), _$JI(\"ZsYJ3sVanh2fJslf0woqXWnga\"), _$JI(\"fsYJ3sVanh2fJslf0woq\"), _$JI(\"3sK2OQbeuMCR0h2fJslf0woq\")];\n        for (var $_16 = 0; $_16 < $_17.length; $_16++) {\n            try {\n                return new ActiveXObject(_$kH($_17, $_16));\n            } catch ($_19) {}\n        }\n        throw new Error(\"\u60a8\u7684\u6d4f\u89c8\u5668\u4e0d\u652f\u6301\u8bbf\u95ee\u6b64\u7f51\u9875\");\n    }\n}\n\n\ntruncated....\n\n\nfunction callback() {\n    if (request.readyState == 1) {\n        _$_J(document.getElementById(_$JI(\"x3CeXwDYXwq\")), '=', _$JI('3FKyXRUxEYlTW'), _$JI(\"EHDxnHOaB3vE5HDxnHOSNMKQGQ6xOHK2z3Kw2Qne7MCm9FKyvhbwNROg\"));\n    }\n    if (request.readyState == 4) {\n        if (request.status == 200) {\n            oldContent.length = 0;\n            oldContent[0] = request.responseText;\n            _$_J(document.getElementById(_$JI(\"H3CeXwDYXwq\")), '=', _$JI('OFKyXRUxEYlTW'), request.responseText);\n            request = null;\n        } else {\n            _$_J(document.getElementById(_$JI(\"w3CeXwDYXwq\")), '=', _$JI('eFKyXRUxEYlTW'), \"<br><br><br><span style=font-size:x-large;color:#215add>\u8bf7\u52ff\u9891\u7e41\u64cd\u4f5c\uff01<\/span>\");\n        }\n    }\n}\n\n<\/code><\/pre>\n
I’m looking to see if it is possible to obtain the GBK<\/code> value and pass it back via Python Requests<\/code> in some way.<\/p>\n
For instance this code gives me a Status code of 202.<\/p>\n
import requests\n\nheaders = {\"User-Agent\": \"Mozilla\/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko\/20100101 Firefox\/89.0\",\n           }\n\nconn_timeout = 30\nread_timeout = 60\ntimeouts = (conn_timeout, read_timeout)\n\nurl=\"http:\/\/app1.nmpa.gov.cn\/data_nmpa\/face3\/base.jsp?tableId=27&tableName=TABLE27&title=%E8%BF%9B%E5%8F%A3%E5%8C%BB%E7%96%97%E5%99%A8%E6%A2%B0%E4%BA%A7%E5%93%81%EF%BC%88%E6%B3%A8%E5%86%8C&bcId=152904442584853439006654836900\"\nresponse = requests.get(url, headers=headers, timeout=timeouts)\nprint(response.status_code)\n# output\n202\n\nprint(response.cookies)\n# output\n<RequestsCookieJar[<Cookie acw_tc=3ccdc15616274084596245338e08543386ba17e301ed12362cb2860b0af57f for app1.nmpa.gov.cn\/>, <Cookie neCYtZEjo8GmS=5oWYI0i1mRB70b.XyuRJwTdiW_WgqfsIoOM8LNI8nfdTGyX4kfKTl0TDpV5HSMj2KIpgl8ircG4c9uAz_u50UkG for app1.nmpa.gov.cn\/>]>\n\n\nfor key, value in response.headers.items():\n    print(f'Key: {key} -- Value: {value}')\n    # output\n    Key: Date -- Value: Tue, 24 Jul 2021 17:54:19 GMT\n    Key: Content-Type -- Value: text\/html; charset=utf-8\n    Key: Transfer-Encoding -- Value: chunked\n    Key: Connection -- Value: keep-alive\n    Key: Set-Cookie -- Value: acw_tc=3ccdc15616274084596245338e08543386ba17e301ed12362cb2860b0af57f;path=\/;HttpOnly;Max-Age=1800, neCYtZEjo8GmS=5oWYI0i1mRB70b.XyuRJwTdiW_WgqfsIoOM8LNI8nfdTGyX4kfKTl0TDpV5HSMj2KIpgl8ircG4c9uAz_u50UkG; Path=\/; expires=Fri, 25 Jul 2031 17:50:29 GMT; HttpOnly\n    Key: Server -- Value: ******\n    Key: Pragma -- Value: no-cache\n    Key: Cache-Control -- Value: no-store\n    Key: Expires -- Value: Tue, 24 Jul 2021 17:50:29 GMT\n\n<\/code><\/pre>\n
Receiving a status code of 202 is an issue when trying to access the website using Python Requests<\/code>, because the request isn’t being fully processed before the connection closes.<\/p>\n
\n
202 Accepted<\/p>\n
The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be  disallowed when processing actually takes place. There is no facility for re-sending a status code from an asynchronous operation such as this.<\/p>\n
The 202 response is intentionally non-committal. Its purpose is to allow a server to accept a request for some other process (perhaps a batch-oriented process that is only run once per day) without requiring that the user agent’s connection to the server persist until the process is completed. The entity returned with this response SHOULD include an indication of the request’s current status and either a pointer to a status monitor or some estimate of when the user can expect the request to be fulfilled.<\/p>\n<\/blockquote>\n
I noted in the browser I get these items.<\/p>\n
\"enter<\/a><\/p>\n
I’m still exploring how to pass these items in a Python Requests<\/code> POST to get something useful back.<\/p>\n
I haven’t looked at using selenium,<\/code> but that Package seems to be the best option for extracting dynamically created content from this website in your question.<\/strong><\/em><\/p>\n<\/p><\/div>\n
<\/div>\n<\/div>\n
            8<\/span> <\/p><\/div>\n<\/div>\n
[ad_2]<\/p>\n
solved Unable to communicate with API <\/p>\n","protected":false},"excerpt":{"rendered":"
[ad_1] CONCLUSION – 07-25-2021 After looking at this problem in more detail, I believe that it is NOT technically possible to use Python Requests to scrape the website and table in your question. Which means that your question cannot be solved in the manner that you would prefer. Why? The website employs anti-scraping mechanisms. The … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[320],"tags":[622,349,2258],"class_list":["post-16755","post","type-post","status-publish","format-standard","hentry","category-solved","tag-beautifulsoup","tag-python","tag-python-requests"],"yoast_head":"\n[Solved] Unable to communicate with API - JassWeb<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"[Solved] Unable to communicate with API - JassWeb\" \/>\n<meta property=\"og:description\" content=\"[ad_1] CONCLUSION – 07-25-2021 After looking at this problem in more detail, I believe that it is NOT technically possible to use Python Requests to scrape the website and table in your question. Which means that your question cannot be solved in the manner that you would prefer. Why? The website employs anti-scraping mechanisms. The ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/\" \/>\n<meta property=\"og:site_name\" content=\"JassWeb\" \/>\n<meta property=\"article:published_time\" content=\"2022-10-21T20:44:02+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/jassweb.com\/solved\/wp-content\/uploads\/2022\/10\/Solved-Unable-to-communicate-with-API.jpg\" \/>\n<meta name=\"author\" content=\"Kirat\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kirat\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"10 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/\"},\"author\":{\"name\":\"Kirat\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31\"},\"headline\":\"[Solved] Unable to communicate with API\",\"datePublished\":\"2022-10-21T20:44:02+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/\"},\"wordCount\":671,\"publisher\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\"},\"image\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/jassweb.com\/solved\/wp-content\/uploads\/2022\/10\/Solved-Unable-to-communicate-with-API.jpg\",\"keywords\":[\"beautifulsoup\",\"python\",\"python-requests\"],\"articleSection\":[\"Solved\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/\",\"url\":\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/\",\"name\":\"[Solved] Unable to communicate with API - JassWeb\",\"isPartOf\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/jassweb.com\/solved\/wp-content\/uploads\/2022\/10\/Solved-Unable-to-communicate-with-API.jpg\",\"datePublished\":\"2022-10-21T20:44:02+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#primaryimage\",\"url\":\"https:\/\/jassweb.com\/solved\/wp-content\/uploads\/2022\/10\/Solved-Unable-to-communicate-with-API.jpg\",\"contentUrl\":\"https:\/\/jassweb.com\/solved\/wp-content\/uploads\/2022\/10\/Solved-Unable-to-communicate-with-API.jpg\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/jassweb.com\/solved\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"[Solved] Unable to communicate with API\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/jassweb.com\/solved\/#website\",\"url\":\"https:\/\/jassweb.com\/solved\/\",\"name\":\"JassWeb\",\"description\":\"Build High-quality Websites\",\"publisher\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/jassweb.com\/solved\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/jassweb.com\/solved\/#organization\",\"name\":\"Jass Web\",\"url\":\"https:\/\/jassweb.com\/solved\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png\",\"contentUrl\":\"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png\",\"width\":693,\"height\":132,\"caption\":\"Jass Web\"},\"image\":{\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31\",\"name\":\"Kirat\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/jassweb.com\/solved\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586\",\"contentUrl\":\"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586\",\"caption\":\"Kirat\"},\"sameAs\":[\"http:\/\/jassweb.com\"],\"url\":\"https:\/\/jassweb.com\/solved\/author\/jaspritsinghghumangmail-com\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"[Solved] Unable to communicate with API - JassWeb","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/","og_locale":"en_US","og_type":"article","og_title":"[Solved] Unable to communicate with API - JassWeb","og_description":"[ad_1] CONCLUSION – 07-25-2021 After looking at this problem in more detail, I believe that it is NOT technically possible to use Python Requests to scrape the website and table in your question. Which means that your question cannot be solved in the manner that you would prefer. Why? The website employs anti-scraping mechanisms. The ... Read more","og_url":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/","og_site_name":"JassWeb","article_published_time":"2022-10-21T20:44:02+00:00","og_image":[{"url":"https:\/\/jassweb.com\/solved\/wp-content\/uploads\/2022\/10\/Solved-Unable-to-communicate-with-API.jpg","type":"","width":"","height":""}],"author":"Kirat","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Kirat","Est. reading time":"10 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#article","isPartOf":{"@id":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/"},"author":{"name":"Kirat","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31"},"headline":"[Solved] Unable to communicate with API","datePublished":"2022-10-21T20:44:02+00:00","mainEntityOfPage":{"@id":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/"},"wordCount":671,"publisher":{"@id":"https:\/\/jassweb.com\/solved\/#organization"},"image":{"@id":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#primaryimage"},"thumbnailUrl":"https:\/\/jassweb.com\/solved\/wp-content\/uploads\/2022\/10\/Solved-Unable-to-communicate-with-API.jpg","keywords":["beautifulsoup","python","python-requests"],"articleSection":["Solved"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/","url":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/","name":"[Solved] Unable to communicate with API - JassWeb","isPartOf":{"@id":"https:\/\/jassweb.com\/solved\/#website"},"primaryImageOfPage":{"@id":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#primaryimage"},"image":{"@id":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#primaryimage"},"thumbnailUrl":"https:\/\/jassweb.com\/solved\/wp-content\/uploads\/2022\/10\/Solved-Unable-to-communicate-with-API.jpg","datePublished":"2022-10-21T20:44:02+00:00","breadcrumb":{"@id":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#primaryimage","url":"https:\/\/jassweb.com\/solved\/wp-content\/uploads\/2022\/10\/Solved-Unable-to-communicate-with-API.jpg","contentUrl":"https:\/\/jassweb.com\/solved\/wp-content\/uploads\/2022\/10\/Solved-Unable-to-communicate-with-API.jpg"},{"@type":"BreadcrumbList","@id":"https:\/\/jassweb.com\/solved\/solved-unable-to-communicate-with-api\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/jassweb.com\/solved\/"},{"@type":"ListItem","position":2,"name":"[Solved] Unable to communicate with API"}]},{"@type":"WebSite","@id":"https:\/\/jassweb.com\/solved\/#website","url":"https:\/\/jassweb.com\/solved\/","name":"JassWeb","description":"Build High-quality Websites","publisher":{"@id":"https:\/\/jassweb.com\/solved\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/jassweb.com\/solved\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/jassweb.com\/solved\/#organization","name":"Jass Web","url":"https:\/\/jassweb.com\/solved\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/","url":"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png","contentUrl":"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png","width":693,"height":132,"caption":"Jass Web"},"image":{"@id":"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31","name":"Kirat","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/image\/","url":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586","contentUrl":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1776403586","caption":"Kirat"},"sameAs":["http:\/\/jassweb.com"],"url":"https:\/\/jassweb.com\/solved\/author\/jaspritsinghghumangmail-com\/"}]}},"_links":{"self":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts\/16755","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/comments?post=16755"}],"version-history":[{"count":0,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts\/16755\/revisions"}],"wp:attachment":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/media?parent=16755"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/categories?post=16755"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/tags?post=16755"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}