Comments for screen-scrapeable

Comment on How to surf and screen-scrape anonymously by J.K.

J.K. — Tue, 06 Dec 2011 19:38:24 +0000

Anonymizers are great, but you traffic can still be read. A VPN is better in this regard, it will encrypt everything that leaves your computer. I use a OpenVPN service that encrypts with AES and 256 bits keys. Site: http://www.sunvpn.com/.

Comment on Techniques for Scraping Large Datasets by Stu

Stu — Mon, 12 Sep 2011 17:24:26 +0000

I figured out there was something wrong with the “for (i=0;” portion because there was no closing bracket in your example, BUT being a sysadmin and not a programmer makes life a little difficult sometimes. Hopefully the posted information will assist someone else with the head scratching…. Thanks Jason.

Comment on Techniques for Scraping Large Datasets by jason

jason — Mon, 12 Sep 2011 15:05:04 +0000

Stu,

You seem to be missing a bit. Look at this:

/*


Script gets all page numbers from the Pages extractor pattern, and iterates through them
*/
// Get variable

pages = Integer.parseInt(session.getVariable(”Pages”));
// Clear session variable so it doesn’t linger

session.setVariable(”Pages”, null);

// Loop through pages for (i=0; i>pages; i++) { // Since the page list appears twice, use only a number larger than that just used if (i>session.getVariable(”PAGE”)) { session.setVariable(”PAGE”, i); session.log(”+++Scraping page #” + i); session.scrapeFile(”Next search results”); } else { session.log(”+++Already have page #” + i + ” so not scraping”); } }

Comment on Techniques for Scraping Large Datasets by Stu

Stu — Mon, 12 Sep 2011 13:47:07 +0000

OK, Figured out the above problem ””, Replace ” with ” if you cut and paste the above, BUT I have been presented with another… argh!! An error occurred while processing the script: The error message was: class bsh.ParseException (line 23): if– Encountered “if” at line 23, column 5.

Line 23 reads: if (i>session.getVariable(”PAGE”))

I do hope all this pain will help someone else…. If anyone can shed some light on the “New” problem, it would be appreciated..

Comment on Techniques for Scraping Large Datasets by Stu

Stu — Mon, 12 Sep 2011 11:04:11 +0000

I invoke the first script “When I get the to the search results page, the first extractor runs as always and drills into the details of each result as usual. The second extractor pattern grabs all the pages listed so I get a dataSet named “Pages,” containing links to pages 2 through 20, and I save the dataSet as a session variable. On the scripts tab, I then run this script after the file is scraped:”

…. da da da Script Runs with result: The error message was: class bsh.EvalError (line 9): session .getVariable ( Pages ) — Error in method invocation: Attempt to pass void argument (position 0) to method: getVariable

Line Nine reads “pages = session.getVariable(”Pages”);”

On my Search results page I extract “~@PAGE@~”, “~@Pages@~” and “~@Next@~” all are stored as a Session. My gut is telling me that the Session Requested “session.getVariable(”Pages”);” is empty.

Comment on Screen-Scraping for iPhone, Andriod, Blackberry, and Most Any Other Mobile Device by Todd Wilson

Todd Wilson — Mon, 15 Aug 2011 14:47:44 +0000

You can actually think of screen-scraper much like you would a web or database server in this respect. Just as a web server can handle many requests at once, screen-scraper can run multiple scrapes simultaneously. But also like a web server, screen-scraper can get overwhelmed if it’s asked to run too many scrapes at once. As this occurs it would probably make sense to either add more screen-scraper instances on the current machine, or add more machines running screen-scraper.

Comment on Screen-Scraping for iPhone, Andriod, Blackberry, and Most Any Other Mobile Device by smol84

smol84 — Sat, 13 Aug 2011 22:56:44 +0000

great article and very helpful blog but I am curious as to how you would configure the screen scraper to run multiple instance at once (or multiple screen scraper threads). Is it best to have it across multiple computers (similar to distributed computing) or is it best to have one computer running multiple screen scrape threads or sessions, so that when one session is not scraping another can operate. What’s the best way to approach this?

Comment on Scraping CAPTCHA forms (you know, those HTML forms with the wavy text) by scottw

scottw — Tue, 17 May 2011 23:49:22 +0000

Just added a sample scraping session that downloads CAPTCHA image from Google’s recaptcha.com, passes image to decaptcher.com service and receives response as TEXT.

Check it out here.

Comment on Version 5.5 of screen-scraper Released! by Josh

Josh — Tue, 19 Apr 2011 19:14:54 +0000

A size-customizable box would be great!

Another useful customizable option would be the regarding how the regular expressions are now automatically filled in based on where you put a pattern token. In general I find this feature very useful. But in some cases I need to manually remove the regex that is automatically applied.

Thanks for the wrappable text in the last request/response windows. I also like how the sub extractor Test Pattern returns only its relevant data!

Comment on Version 5.5 of screen-scraper Released! by Todd Wilson

Todd Wilson — Tue, 19 Apr 2011 18:51:54 +0000

Thanks for the feedback, Josh. We took an internal poll here which concluded that the sub-extractor patterns box should be shrunk. The vast majority of our sub-extractor patterns tend to be one-liners, so the large box ended up just consuming unnecessary space. That said, your point is well-taken; perhaps we could add a setting that allows the size of that box to be adjusted according to preference. I’ll add it to our to-do list!