Thoughts, tips, and updates on screen-scraping
Yesterday I opened a fortune cookie that said, “Do something unusual tomorrow.” I thought about sky-diving or going the whole day blind-folded, but instead opted for something even crazier–sell screen-scraper for half price! If you’re on the fence about purchasing now might be a good time to take the plunge. I don’t see us doing this again any time soon. The sale will last until April 11, 2009 at 11:00 a.m. Mountain time.
We’ve had people asking for this for quite a while, and have finally gotten to it. We now have a video version of our first tutorial, accessible from the tutorial itself:
http://community.screen-scraper.com/Tutorial_1_Page_1
It isn’t perfect, but I think it’s a pretty good first version (and definitely better than what we had previously). We’re hoping to get some feedback, then will likely do another version soon based on that feedback. Feel free to give it a try and let us know what you think.
Well, we finally did it. Break out the party hats and the sparkling apple juice (yes, we live in Utah).
We invite everyone and anyone to download or update to version 4.5 of screen-scraper. It is by far the most feature-rich and stable version to date. If you’re interested in checking out what’s new, take a look at the release notes.
Also, for anyone listening, keep an eye on the site if you’re considering purchasing in the near future. We’re about to do a little sale to celebrate the release of the new version…
Here at screen-scraper we’re on pins and needles as we’re about to release another public version of screen-scraper (we’re anticipating calling it 4.5). Our current alpha release is looking to be pretty solid, and we’re planning on giving it just a bit more testing to ensure that there aren’t any bugs left. If you’re interested in helping us test, feel free to upgrade to the latest alpha version. Here’s a FAQ that might help on that: http://community.screen-scraper.com/faq#80n867.
One of our eagle-eyed developers recently spotted a couple of blog postings by Bronwyn Mauldin (here and here) wherein she discusses Iowa Workforce Development’s use of our screen-scraping technology in building out their job board. Bronwyn is a great writer and a consultant in the workforce development industry. After reading Bronwyn’s postings we decided to contact the Iowa office ourselves to catch up on how things have been going for them. It makes a great story as to how screen-scraping technology is being used in a very effective way. We decided to make a press release on it, which you can find here:
Iowa Workforce Development Uses Screen-Scraper to Enhance Job Search
Some of the sites we aspire to scrape contain vast, huge amounts of data. In such cases, an attempt to scrape data from it may run fine for a time, but eventually stop prematurely with the following message printed to the log:
The error message was: The application script threw an exception: java.lang.OutOfMemoryError: Java heap space BSF info: null at line: 0 column: columnNo
There can be a variety of causes, but most of the time it is caused by memory use in page iteration. Turning up the memory allocation for screen-scraper may take care of it, but it doesn’t address the root cause.
In a typical site structure, we input search parameters and are presented with a page of results and a link to view subsequent pages. If there are ten to twenty pages of results, it’s easiest to just scrape the “next page” link and run a script after the pattern is applied that scrapes the next page. The problem lies in the fact that this is recursive. When we’ve requested the search results, and 2 subsequent “next pages” the scrapeable files are still open in memory thusly:
Every “Next search results” opens a new scrapable file while the previous is still open. While you can run the script on the scripts tab after the file is scraped to prevent the dataSets from remaining in scope, the scrapeable files remain in memory—the scrape may get further, but the memory still fills up with scrapable files, and it mayn’t be enough to get all the data.
The solution is to use an iterative approach.
If the site we’re scraping shows the total number of pages, using an iterative method easy. For my example, I’ll describe a site that has a link for pages 1 through 20, and a “>>” indicator to show there are pages beyond 20.
On first page of search results, I have 3 extractor patterns to extract the following information:
When I get the to the search results page, the first extractor runs as always and drills into the details of each result as usual. The second extractor pattern grabs all the pages listed so I get a dataSet named “Pages,” containing links to pages 2 through 20, and I save the dataSet as a session variable. On the scripts tab, I then run this script after the file is scraped:
/*
Script gets all page numbers from the Pages extractor pattern, and iterates through them
*/
// Get variable
pages = session.getVariable(”Pages”);
// Clear session variable so it doesn’t linger
session.setVariable(”Pages”, null);
// Loop through pages
for (i=0; i
{
// Since the page list appears twice, use only a number larger than that just used
if (i>session.getVariable(”PAGE”))
{
session.setVariable(”PAGE”, i);
session.log(”+++Scraping page #” + i);
session.scrapeFile(”Next search results”);
}
else
{
session.log(”+++Already have page #” + i + ” so not scraping”);
}
}
The “for” loop will have the first page of search results in memory, but when it calls the “Next search results” scrapeable file to go to page 2, it only gets the results, and doesn’t try to look for a next page. The loop closes out the second page before it starts the third, and closes the third before starting the forth, etc.
The last extractor on “Search results” looks for “>>”. I save the that dataSet as a session variable named “Next batch pages”, and put this as the last script to run on the scripts tab:
import com.screenscraper.common.*;
/*
Script that checks if there is a next batch of pages
*/
if (session.getVariable(”Next batch pages”)!=null)
{
pageSet = session.getVariable(”Next batch pages”);
session.setVariable(”Next batch pages”, null);
pages = pageSet.getDataRecord(0);
page = Integer.parseInt(pages.get(”PAGE”));
if (page>session.getVariable(”PAGE”))
{
session.setVariable(”PAGE”, page);
session.log(”+++Scraping page #” + page);
session.scrapeFile(”Next batch search results”);
}
else
{
session.log(”+++Already have page #” + page + ” so not scraping”);
}
}
Now the “Next batch search results” scrapable file must do all the things the first page of search results did; get each result, look for next page links, and look for a next batch of results. Using the iterative approach to cycle through pages enables you request many more pages without keeping as many in memory, and without unnecessary pages in memory, the scrape will run far longer.
Microsoft ASP.NET sites have consistently proven to be some of the most difficult to scrape. This is due to their unconventional nature and cryptic information passed between your browser and the server. You’ll know you’re at an ASP.NET site when your URLs end in .aspx, your links look like this:
javascript:__doPostBack('gvLicensing','Select$0')
And your POSTs look like this*:
/wEPDwUJNDczODExNjY1D2QWAgIFD2QWAgIXDzwrAA0CAA8WBB4LXyFEYXRhQm91 bmRnHgtfIUl0ZW1Db3VudGZkDBQrAABkGAIFHl9fQ29udHJvbHNSZXF1aXJlUG9zdEJh Y2tLZXlfXxYBBQdidG5JbmZvBQtndkxpY2Vuc2luZw88KwAJAgYVAQ1saWNfc2VyaWFsX 2lkCGZk/kjEfRuqcTBAeylGENOP9dFkERc=
If you’re at all familiar with conventional HTTP transactions, prepare to forgo what you’ve come to expect. Once again, Microsoft manages to defy many standard practices that, in this case, has gone unnoticed by everyone but your tireless browser tasked with making sense of it all. But now, it is your job to pick apart what’s going on and to try to reconstruct the mixed up conversation your poor browser’s been having with the server. In this blog entry I’ll attempt to cover the more common (and not so common) characteristics of ASP.NET sites and offer techniques for how best you can play the role of your submissive browser to an unforgiving taskmaster.
If you’ve already been down this road before, please post your own stories of things you’ve encountered and how you went about slaying the dragon.
As you begin the process of scraping data from a website we recommend that you start by using screen-scraper’s proxy to record the HTTP transactions while you navigate the site. You’ll then need to identify which of the proxy transactions should be made in to scrapeable files, add extractor patterns to your scrapeable files to be used as session variables for other scrapeable files**, and tie the whole thing together with scripts to run recursively and in the proper sequence while it traverses the site scraping the data you need.
Here are some general rules and recommendations
* If a page’s VIEWSTATE is too large, screen-scraper can hang when you click on the offending proxy transaction. Wait for a while and it should recover.
**As you’re converting proxy transactions into scrapeable files, a good approach is to replace the values of parameters that look like they’re generated dynamically with session variables containing values extracted from the referring page, test it and compare side-by-side the raw request from your proxy session to that of your test run. And, repeat until you’ve successfully given the server what it wants in order to give you back what you want.
The internet can be thought of as the world’s largest database. This is so, because it is comprised of inter-connected databases, files, and computer systems. By simply typing in some keywords, one can access hundreds to millions of websites containing treasure troves of facts, statistics, and other formats of information on an endless array of topics. Because the internet is such a valuable resource, we should seek new and innovative ways to mine the data using ethical means.
You may have never heard of screen-scraping, web-fetching, or web-data extraction, but if you’ve ever surfed the internet, you’ve quite likely been a beneficiary of the method of retrieving information on the web described by these terms. They refer to the increasingly popular method of methodically retrieving information with specialized tools. Numerous programs utilize many computer languages for the purpose of mining data. Software often assists users in intercepting HTTP requests and responses by incorporating proxy servers. The software then displays the pages’ source code (HTML, JavaScript, etc.) for users to extract the desired information. In addition, such software can aid iteration through pages (sometimes thousands of them) all the while gleaning valuable data in various forms.
The goal of scraping websites is to access information, but the uses of that information can vary. Users may wish to store the information in their own databases or manipulate the data within a spreadsheet. Other users may utilize data extraction techniques as means of obtaining the most recent data possible, particularly when working with information subject to frequent changes. Investors analyzing stock prices, realtors researching home listings, meteorologists studying weather, or insurance salespeople following insurance prices are a few individuals who might fit this category of users of frequently updated data.
Access to certain information may also provide users with strategic advantage in business. Attorneys might wish to scrape arrest records from county courthouses in search of potential clients. Businesses, such as restaurants or video-rental stores that know the locations of competitors can make better decisions about where to focus further growth. Companies that provide complementary (not to be confused with complimentary) products, like software, may wish to know the make, model, cost, and market share of hardware that are compatible with their software.
Another common, but controversial use of information taken from websites is reposting scraped data to other sites. Scrapers may wish to consolidate data from a myriad of websites and then create a new website containing all of the information in one convenient location. In some cases, the new site’s owner may benefit from ads placed on his or her site or from fees charged to access the site. Companies usually go to great lengths to disseminate information about their products or services. So, why would a website owner not wish to have his or her website’s information scraped?
Several reasons exist for why website owners may not wish to have their site’s scraped by others (excluding search engines). Some people feel that data that is reposted to other sites is plagiarized, if not stolen. These individuals may feel that they made the effort to gather information and make it available on their websites only to have it copied to other sites. Are individuals justified in feeling that they have been taken advantage of, even if their websites are posted publicly?
Interpretation of what exactly “republish” means is widely disputed. One of the most authoritative explanations may be found in the 1991 supreme-court case of Feist Publications v. Rural Telephone Service. This case involved Rural Telephone Service suing Feist Publications for copyright infringement when Feist copied telephone listings after Rural denied Feist’s request to license the information. While information has never been copyrightable under U.S. law, a collection of information, defined mostly in terms of creative arrangement or original ideas, can be copyrighted. The Supreme Court’s ruling in Feist Publications v. Rural Telephone Service stated that “information contained in Rural’s phone directory was not copyrightable, and that therefore no infringement existed.” Justice O’ Conner focused on the need for information to have a “creative” element in order to be termed a “collection” (1). Similarly, information, taken from publicly available websites should not be considered plagiarism or even theft if only the information (numbers, statistics, etc.) is reposted to new sites or used for other purposes.
Scraped websites also experience an increase in used bandwidth as a result of being scraped. Some scrapes take place once, but many scrapes must be performed over and over to achieve the desired results. In such cases, the servers that host the pages being scraped inevitably experience an increased load. Site owners may not wish to have the increased bandwidth, but more importantly, excessive page requests can cause a web server to function slowly or even fail. Rarely, however, do most scrapes cause such strain on a server on their own. Accessing a page through scraping is no different from visiting a page manually, except that scraping allows more pages to be visited over a shorter period. Additionally, scrapes can be adjusted to run more slowly, so as to minimize the strain on the server. Scraping is usually slowed when more than a few scraping sessions are being run against a single server at one time.
Interestingly, having one’s website scraped can have positive effects. Of course the recipient of the scraped data is pleased to have desired data, but owners of scraped sites may also benefit. Think of the case mentioned above in which home listings are scraped from a site. Whether the information is reposted or stored in a database for later querying to match homebuyer’s needs, the purpose of the original site is met—to get the home-listing information into the hands of potential buyers.
Individuals who scrape websites can do so, while still following guidelines for ethical data extraction. Perhaps it would be helpful to review a list of tips for maintaining ethical scraping. One website I consulted gave the following suggestions:
· Obey robots.txt.
· Don’t flood a site.
· Don’t republish, especially not anything that might be copyrighted.
· Abide by the site terms of service (2).
Occasionally, individuals who scrape websites have paid for access to the material being scraped. Many job- and résumé-posting websites fall into this category. Employers must pay a monthly fee for an account which provides access to the résumés of potential new hirers. Certainly, the fact that employers pay for the service entitles them to use whatever means are necessary to sort through and record the desired data. The only exception would be where the site’s terms of service specifically prohibit scraping.
While republishing images, artwork, and other original content without permission is unethical and in many cases illegal, using scraped data for personal purposes is certainly within the limits of ethical behavior. Nevertheless, page scrapers should always avoid taking copyrighted materials. Use of bandwidth is no more deserved by any one person than another. Even making scraped data available to others online can be argued as ethical, especially when the scraped website is posted on public space and the data taken doesn’t include any creative content. After all, the purpose of hosting a website in the first place is to provide information.
(1) http://en.wikipedia.org/wiki/Feist_Publications_v._Rural_Telephone_Service
Well, it’s now official. It’s been just over a full year in development, and we’re now happy to release it to the world. Thanks to all who have helped in testing alpha versions and provided feedback.
In order to upgrade an existing instance, you’ll need to un-install and re-install. Take a look at this FAQ for details as to the whys and wherefores.
We’re anticipating releasing version 4.0 of screen-scraper quite soon. Perhaps as soon as this week. There will be quite a few changes that come along with this. Aside from the usual new features and bug fixes, we’ll be adding a new edition–screen-scraper Enterprise Edition. Essentially what is now the latest pre-release version of screen-scraper Professional Edition will become screen-scraper Enterprise Edition. The new screen-scraper Professional Edition will simply be the Enterprise Edition with a number of features stripped out. Additionally, those who license the Enterprise Edition will get phone support, as well as a few other non-tangibles.
Along with all of this there will be a pricing change. The Professional Edition will be available for $399 USD, and the Enterprise Edition will cost $2,499. Those who licensed screen-scraper Professional Edition before the release of the Enterprise Edition will be eligible for a free upgrade to it (though they will not get the phone support that subsequent licensees will get). In the interest of fairness, I thought it would be a good idea to point this out prior to the release of 4.0. Those considering licensing screen-scraper Professional Edition right now might want to consider it a bit more seriously, given the price increase that will take place with the new version. As always, don’t hesitate to drop us a line with any questions.
| Subscribe: |
 |
 |
 |
 |
 |
 |
 |