For those interested, the bug fix addresses an issue where line numbers reported in error messages will be offset by two lines.  That is, if your script has an error on line 100, the error message will report line 102 instead.  The bug fix simply corrects this so that the correct line number of the error is reported.

Along with this release we’ve also done some significant revamping of our docs, which we’ll continue to do.  We want to make sure screen-scraper is easier to learn for the beginner and quicker to use for the seasoned veteran.

And on top of all of that it’s on sale!  Until August 15 of this year you can get the Enterprise Edition for $800 off and the Professional Edition for $150 off.  We’ll be raising prices back up after that date, so get it while it’s hot.  Interested in purchasing?  Click here to be taken to our registration page.

View full-size

An astute screen-scraper user yesterday suggested a solution to this that is both simple and effective.  In the case described in the blog posting you end up with a big stack of scripts, all of which have references to objects, which causes the OutOfMemoryError.  The number of scripts on the stack can be viewed in the breakpoint window, and in version 4.5.45a we added a method that will allow you to see how many scripts are on the stack from within a script:

session.getNumScriptsOnStack()

You can check this number as often as you’d like.  As it grows it could mean trouble, so you can respond appropriately in your scraping session.  We’ve also added a failsafe mechanism inside of screen-scraper that will hopefully save you from an OutOfMemoryError.  If too many scripts are pushed on the stack your scraping session will be stopped and the following message will be output to the log:

ERROR–halting the scraping session because the maximum number of scripts allowed on the stack was reached.

You can control the maximum number of scripts allowed on the stack by invoking this method at any time:

session.setMaxScriptsOnStack( 50 )

Set that number to whatever you’d like.

By design screen-scraper provides a lot of flexibility and power in the data extraction process, but this same power can also result in our shooting ourselves in the foot on occasion.  The inclusion of this new mechanism will hopefully help some to avoid this problem down the road.

I’m going to make a script that runs at the beginning of the scrape to check for a file that contains the last scraping state.

inputFile = "resume_" + session.getName() + ".txt";
File file = new File(inputFile);
if (file.exists())
{
session.loadVariables(inputFile); // This may set several variables, but we'll pay no attention to most
session.log("==============================================================");
session.log("Scrape is set to resume at " + session.getVariable("RESUME_AT_LOCATION") + " and " + session.getVariable("RESUME_AT_COMPANY");
session.log("==============================================================");
}

If the file isn’t there, that’s good.  I’ll just scrape the list of LOCATIONS, and go through all of them.

If the file is there, then I change a couple of variable names so I can throw them away when I’m done with them.

Now I get to the list of companies, and I run this script on each pattern application:

// Function to write the restore point and scrape
scrapeLocation(location)
{
// Set RESUME_AT_LOCATION to write out
session.setVariable("RESUME_AT_LOCATION", location);
session.saveVariables("resume_" + session.getName() + ".txt");
session.setVariable("RESUME_AT_LOCATION", null); // We don't want this set next time the loop is run
session.scrapeFile("Single location");
}
if (session.getVariable("RESUME_AT_LOCATION")==null)
{
// No resume at location set, so scraping
scrapeLocation(dataRecord.get("LOCATION"));
}
else
{
// There is a resume at location set, so compare
if (dataRecord.get("LOCATION").equals(session.getVariable("RESUME_AT_LOCATION")))
{
// Matches, so scrape
scrapeLocation(dataRecord.get("LOCATION"));
}
else
{
session.log("---Skipping this location: " + dataRecord.get("LOCATION"));
session.log("-----Looking to resume at: " + session.getVariable("RESUME_AT_LOCATION"));
}
}

Now when the “RESUME_AT_LOCATION” is set at the beginning of a scrape, it will skip all locations until it finds a match.  It will then scrape that location, and clear the variable so it will get the rest of the locations.

You could copy this same logic into the COMPANIES.

Finally, when the scrape completes naturally, we should delete the file with the resume point since 1) you’re now at the last location, and 2) you’re done … nothing to resume.  Without that file, the scrape will start fresh next time.

// Delete the file with the resume point
outputFile = "resume_" + session.getName() + ".txt";
File file = new File(outputFile);
file.delete();
session.log("Resume point deleted");

You can see more on the loadVariables and saveVariable on the API Page.

In order to address the issues above we’ve been trying out a few other tidiers–NekoHTML and Jericho.  We’ve actually already found issues with NekoHTML, so Jericho looks to be the favorite as of right now.  Both will still require some experimentation, though, so please use them at your own risk for now.  Once we’ve put them both through the paces we’ll likely settle on one as the recommended default.  And not to worry about any scrapeable files that are already using JTidy–they’ll stay just as they are.  At some point, though, for any new scrapeable files, you might notice a different tidier as the default.

scrapeableFile.setMaxResponseLength( int maxKBytes )

This tells screen-scraper to only download a given number of kilobytes at the beginning of the file.  You would want to run this method in a script that gets invoked before a file is scraped.  For example, if your script contained this line:

scrapeableFile.setMaxResponseLength( 50 );

screen-scraper would download the first 50K of the file, cut it off, then continue on.

If the speed of a scraping session is especially critical this can also be a great way to trim off quite a bit of download time.

As is, the scraping session is intended to run on Linux.  However, it is possible to run both dependent programs under Windows either directly or using Cygwin.

To use:

Download and import the following scraping session.

http://community.screen-scraper.com/samples/ocr

Each developer on a project will have his own instance of screen-scraper, but may be using some scraping sessions and scripts that are also used by other developers.  Generally speaking, though, a given developer is in charge of a certain set of scraping sessions, and we have a series of general scripts that might be used by all developers.  These general scripts can be edited by anyone, but when edits are made everyone needs to be notified so that they can update their own instances of screen-scraper with the latest scripts.  Each time a new scraping session is created or an existing scraping session is modified, it gets exported then committed to the repository.  This isn’t quite as automated as some IDE’s allow, so developers need to be conscientious of their work so that the export and commit at the appropriate times.

We often also make use of debug scripts, which each developer will generally cater to his own work.  It’s likely that he won’t want these scripts overwritten by  those of other developers, so for each of these scripts he need only un-check the “Overwrite this script on import” box in the workbench to protect a such a script.

We also typically keep a separate folder in our version control repository for the scripts that are general to a series of scraping sessions.  It’s possible that a particular developer has a slightly out-dated script, and when he exports that script may go with the scraping session.  To keep it from getting imported into a production environment we’ll copy all of the general scripts (which are always kept current) into screen-scraper’s “import” folder along with the scraping session(s) to be deployed.  screen-scraper will always import scraping sessions first, then scripts.  That way you can guarantee that the current scripts don’t get overwritten.

Because screen-scraper doesn’t use a purely file-based approach to persist its objects, version control can require another step or two beyond what you’d normally find in a modern-day IDE.  Our experience has been, though, that once developers get accustomed to it it’s not too burdensome.  That said, we have plans in the near future to add features that will make working with version control systems even easier with screen-scraper.