{"id":13039,"date":"2022-10-02T20:29:13","date_gmt":"2022-10-02T14:59:13","guid":{"rendered":"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/"},"modified":"2022-10-02T20:29:13","modified_gmt":"2022-10-02T14:59:13","slug":"solved-web-scraping-from-asp-urls","status":"publish","type":"post","link":"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/","title":{"rendered":"[Solved] Web Scraping From .asp URLs"},"content":{"rendered":"<p> [ad_1]<br \/>\n<\/p>\n<div id=\"answer-49412483\" class=\"answer js-answer accepted-answer js-accepted-answer\" data-answerid=\"49412483\" data-parentid=\"48574638\" data-score=\"10\" data-position-on-page=\"1\" data-highest-scored=\"1\" data-question-has-accepted-highest-score=\"1\" itemprop=\"acceptedAnswer\" itemscope itemtype=\"https:\/\/schema.org\/Answer\">\n<div class=\"post-layout\">\n<div class=\"votecell post-layout--left\"><\/div>\n<div class=\"answercell post-layout--right\">\n<div class=\"s-prose js-post-body\" itemprop=\"text\">\n<p>I would recommend using JSoup for this. To do so add below to <code>pom.xml<\/code><\/p>\n<pre><code>&lt;dependency&gt;\n  &lt;groupId&gt;org.jsoup&lt;\/groupId&gt;\n  &lt;artifactId&gt;jsoup&lt;\/artifactId&gt;\n  &lt;version&gt;1.11.2&lt;\/version&gt;\n&lt;\/dependency&gt;\n<\/code><\/pre>\n<p>Then you fire a first request to just get cookied<\/p>\n<pre><code>    Connection.Response initialPage = Jsoup.connect(\"https:\/\/www.flightview.com\/flighttracker\/\")\n            .headers(headers)\n            .method(Connection.Method.GET)\n            .userAgent(userAgent)\n            .execute();\n    Map&lt;String, String&gt; initialCookies = initialPage.cookies();\n<\/code><\/pre>\n<p>Then you fire the next request with these cookies<\/p>\n<pre><code>    Connection.Response flights = Jsoup.connect(\"https:\/\/www.flightview.com\/TravelTools\/FlightTrackerQueryResults.asp\")\n            .userAgent(userAgent)\n            .headers(headers)\n            .data(postData)\n            .cookies(initialCookies)\n            .method(Connection.Method.POST)\n            .execute();\n<\/code><\/pre>\n<p>The <code>postData<\/code> and <code>headers<\/code> in this case is <\/p>\n<pre><code>    HashMap&lt;String, String&gt; postData = new HashMap&lt;String, String&gt;();\n    HashMap&lt;String, String&gt; headers = new HashMap&lt;String, String&gt;();\n\n    headers.put(\"Accept\", \"text\/html,application\/xhtml+xml,application\/xml;q=0.9,image\/webp,image\/apng,*\/*;q=0.8\");\n    headers.put(\"Accept-Encoding\", \"gzip, deflate, br\");\n    headers.put(\"Accept-Language\", \"en-US,en;q=0.9\");\n    headers.put(\"Cache-Control\", \"no-cache\");\n    headers.put(\"DNT\", \"1\");\n    headers.put(\"Pragma\", \"no-cache\");\n    headers.put(\"Upgrade-Insecure-Requests\", \"1\");\n\n    postData.put(\"qtype\", \"cpi\");\n    postData.put(\"sfw\", \"\/FV\/FlightTracker\/Main\");\n    postData.put(\"namdep\", \"DFW Dallas, TX (Dallas\/Ft Worth) - Dallas Fort Worth International\");\n    postData.put(\"depap\", \"DFW\");\n    postData.put(\"namarr\", \"JFK New York, NY (Kennedy) - John F Kennedy International\");\n    postData.put(\"arrap\", \"JFK\");\n    postData.put(\"namal2\", \"Enter name or code\");\n    postData.put(\"al\", \"\");\n    postData.put(\"whenArrDep\", \"dep\");\n    postData.put(\"whenHour\", \"all\");\n    postData.put(\"whenDate\", \"20180321\");\n    postData.put(\"input\", \"Track Flight\");\n<\/code><\/pre>\n<p>Now when you have got the data, you can parse and print stuff out of it<\/p>\n<pre><code>    String page = flights.body();\n    System.out.println(page);\n    Document doc = Jsoup.parse(page);\n    Elements elems = doc.select(\"tr.FlightTrackerListRowOdd, tr.FlightTrackerListRowEven\");\n\n    for(Element element : elems) {\n        Elements childElems = element.select(\"td\");\n        String text1 =  childElems.get(0).text();\n        String text2 =  childElems.get(1).text();\n        System.out.println(text1 + \" \" + text2);\n    }\n<\/code><\/pre>\n<p>The output of the same is<\/p>\n<pre><code>Aeroflot Airlines 3453\nAeroflot Airlines 3455\nAeroMexico 4966\nAeroMexico 4935\nAir France 2535\nAlitalia 3403\nAmerican Airlines 1294\nBritish Airways 1880\nChina Eastern Airlines 8804\nDelta Air Lines 3869\nDelta Air Lines 3789\nEtihad Airways 3040\nFinnair 5726\nGulf Air 4139\nIberia Airlines 4043\nJet Airways 7692\nKLM Royal Dutch Airlines 6597\nKLM Royal Dutch Airlines 8117\nKorean Air 7326\nMalaysia Airlines 9442\nQatar Airways 5107\nTAM Brazilian Airlines 8379\nVirgin Atlantic 4620\nVirgin Atlantic 3471\n<\/code><\/pre>\n<p>Rest you can start altering the same based on your needs. This shows you can example of how to do it<\/p>\n<\/p><\/div>\n<div class=\"mt24\"><\/div>\n<\/div>\n<p>            <span class=\"d-none\" itemprop=\"commentCount\">1<\/span> <\/p><\/div>\n<\/div>\n<p>[ad_2]<\/p>\n<p>solved Web Scraping From .asp URLs <\/p>\n","protected":false},"excerpt":{"rendered":"<p>[ad_1] I would recommend using JSoup for this. To do so add below to pom.xml &lt;dependency&gt; &lt;groupId&gt;org.jsoup&lt;\/groupId&gt; &lt;artifactId&gt;jsoup&lt;\/artifactId&gt; &lt;version&gt;1.11.2&lt;\/version&gt; &lt;\/dependency&gt; Then you fire a first request to just get cookied Connection.Response initialPage = Jsoup.connect(&#8220;https:\/\/www.flightview.com\/flighttracker\/&#8221;) .headers(headers) .method(Connection.Method.GET) .userAgent(userAgent) .execute(); Map&lt;String, String&gt; initialCookies = initialPage.cookies(); Then you fire the next request with these cookies Connection.Response flights = &#8230; <a title=\"[Solved] Web Scraping From .asp URLs\" class=\"read-more\" href=\"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/\" aria-label=\"More on [Solved] Web Scraping From .asp URLs\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[320],"tags":[751,323,939,760],"class_list":["post-13039","post","type-post","status-publish","format-standard","hentry","category-solved","tag-asp-net","tag-java","tag-url","tag-web-scraping"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>[Solved] Web Scraping From .asp URLs - JassWeb<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"[Solved] Web Scraping From .asp URLs - JassWeb\" \/>\n<meta property=\"og:description\" content=\"[ad_1] I would recommend using JSoup for this. To do so add below to pom.xml &lt;dependency&gt; &lt;groupId&gt;org.jsoup&lt;\/groupId&gt; &lt;artifactId&gt;jsoup&lt;\/artifactId&gt; &lt;version&gt;1.11.2&lt;\/version&gt; &lt;\/dependency&gt; Then you fire a first request to just get cookied Connection.Response initialPage = Jsoup.connect(&quot;https:\/\/www.flightview.com\/flighttracker\/&quot;) .headers(headers) .method(Connection.Method.GET) .userAgent(userAgent) .execute(); Map&lt;String, String&gt; initialCookies = initialPage.cookies(); Then you fire the next request with these cookies Connection.Response flights = ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/\" \/>\n<meta property=\"og:site_name\" content=\"JassWeb\" \/>\n<meta property=\"article:published_time\" content=\"2022-10-02T14:59:13+00:00\" \/>\n<meta name=\"author\" content=\"Kirat\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kirat\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/solved-web-scraping-from-asp-urls\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/solved-web-scraping-from-asp-urls\\\/\"},\"author\":{\"name\":\"Kirat\",\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/#\\\/schema\\\/person\\\/65c9c7b7958150c0dc8371fa35dd7c31\"},\"headline\":\"[Solved] Web Scraping From .asp URLs\",\"datePublished\":\"2022-10-02T14:59:13+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/solved-web-scraping-from-asp-urls\\\/\"},\"wordCount\":95,\"publisher\":{\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/#organization\"},\"keywords\":[\"asp.net\",\"java\",\"url\",\"web-scraping\"],\"articleSection\":[\"Solved\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/solved-web-scraping-from-asp-urls\\\/\",\"url\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/solved-web-scraping-from-asp-urls\\\/\",\"name\":\"[Solved] Web Scraping From .asp URLs - JassWeb\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/#website\"},\"datePublished\":\"2022-10-02T14:59:13+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/solved-web-scraping-from-asp-urls\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/jassweb.com\\\/solved\\\/solved-web-scraping-from-asp-urls\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/solved-web-scraping-from-asp-urls\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"[Solved] Web Scraping From .asp URLs\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/#website\",\"url\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/\",\"name\":\"JassWeb\",\"description\":\"Build High-quality Websites\",\"publisher\":{\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/#organization\",\"name\":\"Jass Web\",\"url\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/jassweb.com\\\/wp-content\\\/uploads\\\/2021\\\/02\\\/jass-website-logo-1.png\",\"contentUrl\":\"https:\\\/\\\/jassweb.com\\\/wp-content\\\/uploads\\\/2021\\\/02\\\/jass-website-logo-1.png\",\"width\":693,\"height\":132,\"caption\":\"Jass Web\"},\"image\":{\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/#\\\/schema\\\/person\\\/65c9c7b7958150c0dc8371fa35dd7c31\",\"name\":\"Kirat\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/wp-content\\\/litespeed\\\/avatar\\\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1777613206\",\"url\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/wp-content\\\/litespeed\\\/avatar\\\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1777613206\",\"contentUrl\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/wp-content\\\/litespeed\\\/avatar\\\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1777613206\",\"caption\":\"Kirat\"},\"sameAs\":[\"http:\\\/\\\/jassweb.com\"],\"url\":\"https:\\\/\\\/jassweb.com\\\/solved\\\/author\\\/jaspritsinghghumangmail-com\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"[Solved] Web Scraping From .asp URLs - JassWeb","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/","og_locale":"en_US","og_type":"article","og_title":"[Solved] Web Scraping From .asp URLs - JassWeb","og_description":"[ad_1] I would recommend using JSoup for this. To do so add below to pom.xml &lt;dependency&gt; &lt;groupId&gt;org.jsoup&lt;\/groupId&gt; &lt;artifactId&gt;jsoup&lt;\/artifactId&gt; &lt;version&gt;1.11.2&lt;\/version&gt; &lt;\/dependency&gt; Then you fire a first request to just get cookied Connection.Response initialPage = Jsoup.connect(\"https:\/\/www.flightview.com\/flighttracker\/\") .headers(headers) .method(Connection.Method.GET) .userAgent(userAgent) .execute(); Map&lt;String, String&gt; initialCookies = initialPage.cookies(); Then you fire the next request with these cookies Connection.Response flights = ... Read more","og_url":"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/","og_site_name":"JassWeb","article_published_time":"2022-10-02T14:59:13+00:00","author":"Kirat","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Kirat","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/#article","isPartOf":{"@id":"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/"},"author":{"name":"Kirat","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31"},"headline":"[Solved] Web Scraping From .asp URLs","datePublished":"2022-10-02T14:59:13+00:00","mainEntityOfPage":{"@id":"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/"},"wordCount":95,"publisher":{"@id":"https:\/\/jassweb.com\/solved\/#organization"},"keywords":["asp.net","java","url","web-scraping"],"articleSection":["Solved"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/","url":"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/","name":"[Solved] Web Scraping From .asp URLs - JassWeb","isPartOf":{"@id":"https:\/\/jassweb.com\/solved\/#website"},"datePublished":"2022-10-02T14:59:13+00:00","breadcrumb":{"@id":"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/jassweb.com\/solved\/solved-web-scraping-from-asp-urls\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/jassweb.com\/solved\/"},{"@type":"ListItem","position":2,"name":"[Solved] Web Scraping From .asp URLs"}]},{"@type":"WebSite","@id":"https:\/\/jassweb.com\/solved\/#website","url":"https:\/\/jassweb.com\/solved\/","name":"JassWeb","description":"Build High-quality Websites","publisher":{"@id":"https:\/\/jassweb.com\/solved\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/jassweb.com\/solved\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/jassweb.com\/solved\/#organization","name":"Jass Web","url":"https:\/\/jassweb.com\/solved\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/","url":"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png","contentUrl":"https:\/\/jassweb.com\/wp-content\/uploads\/2021\/02\/jass-website-logo-1.png","width":693,"height":132,"caption":"Jass Web"},"image":{"@id":"https:\/\/jassweb.com\/solved\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/jassweb.com\/solved\/#\/schema\/person\/65c9c7b7958150c0dc8371fa35dd7c31","name":"Kirat","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1777613206","url":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1777613206","contentUrl":"https:\/\/jassweb.com\/solved\/wp-content\/litespeed\/avatar\/1261af3c9451399fa1336d28b98ea3bb.jpg?ver=1777613206","caption":"Kirat"},"sameAs":["http:\/\/jassweb.com"],"url":"https:\/\/jassweb.com\/solved\/author\/jaspritsinghghumangmail-com\/"}]}},"_links":{"self":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts\/13039","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/comments?post=13039"}],"version-history":[{"count":0,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/posts\/13039\/revisions"}],"wp:attachment":[{"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/media?parent=13039"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/categories?post=13039"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jassweb.com\/solved\/wp-json\/wp\/v2\/tags?post=13039"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}