[Solved] Selenium jSoup get data from Javascript Webpage

by


If you are going to scrape / datamine someone’s site, here are some considerations:

  1. Get permission from the site’s owner! If you do not, you will piss off the owner and get blacklisted in the best case, or be served with a lawsuit in the worst case.
  2. Find out if the site exposes an api. This is always the better way of scraping a site.
  3. Research tools / libraries that are more appropriate for this task. Some of these include curl, wget, httpbuilder, ….. Depending on your level of comfort / knowledge, you may need to research the underlying technologies: http, rest, …..
  4. selenium is a functional test library for browser applications, which makes it a poor choice for this task.

PS: I am fully expecting for this to get downvoted / closed, because discussions / opinions are off-topic for SO.

solved Selenium jSoup get data from Javascript Webpage