01.14.10
Posted in Tips at 4:07 pm by Todd Wilson
Many have probably noticed that when a scraping session is exported from screen-scraper all of the scripts invoked from within that scraping session get exported along with it. All of the scripts, that is, except those that get invoked via the session.executeScript method. The exporter isn’t quite smart enough to actually parse the text of scripts to look for scripts that should be exported because they’re invoked via that method.
Fortunately, there’s an easy workaround. For scripts that get invoked via session.executeScript simply associate them with the scraping session itself, but then disable them. That is, on the “General” tab for a scraping session add the scripts via the “Add Script” button, then under the “Enabled?” column in the scripts table un-check the box. This way the scripts won’t get executed at the beginning of the scraping session, but they will get exported.
Permalink
11.11.09
Posted in Uncategorized at 1:22 pm by Todd Wilson
One of the primary design goals of screen-scraper from the very beginning has been to emphasize extensibility. We’ve tried to build in a number of features and tools to make screen-scraping easier, but we also realize that we can’t fit it all in. Features such as the internal scripting engine and the ability to invoke screen-scraper from external applications allow it to be extended according to the whims of the developer.
Recently astute scraper Rodney Aiglstorfer came up with an excellent way to link data extracted within screen-scraper to custom-built classes. He’s dubbed it “Screen-Scraper Annotations for Java”, and you can find it here: http://code.google.com/p/ssa4j/. Rodney’s been good enough to release the library under an open source license, so others can benefit as well.
Permalink
09.02.09
Posted in Updates at 5:38 pm by Todd Wilson
We’re constantly updating screen-scraper with bug fixes and new features, but haven’t always been good about documenting changes. These newer features are typically only available in our alpha versions. Whereas previously you were on your own to figure out what was new, we’re now going to do our best to document new features here:
Alpha documentation
These docs might not be quite as neat and clean as the others, but if you’re using our alpha versions and want to see what’s new, this is a good page to watch.
Permalink
08.28.09
Posted in Updates at 5:27 pm by Todd Wilson
Actually, I should probably call it a REST-like API. I have no doubt the purists will point out that it isn’t a REST API at all. How about we’ll call it an “API accessible via GET requests”.
With that loquacious introduction, I’m happy to announce that, as of version 4.5.18a, you can access screen-scraper via GET requests. Let me just state right here and now that this is alpha functionality and may very well change before the next public release. Use it at your own risk. As with any of our alpha features the documentation is scant, so I’ll simply provide a long list of examples as to how you might use it. Hopefully you’ll get the idea.
You’ll first need to start up screen-scraper in server mode. Once that’s done you can then access a slew of features you’d normally only be able to access via the web interface. Here they are:
http://localhost:8779/ss/rest?action=get_runnable_scraping_sessions
http://localhost:8779/ss/rest?action=get_scrapeable_sessions
http://localhost:8779/ss/rest?action=run_scraping_session&scraping_session_name=Shopping+Site
http://localhost:8779/ss/rest?action=stop_running_scraping_session&scrapeable_session_id=43
http://localhost:8779/ss/rest?action=stop_all_running_scraping_session
http://localhost:8779/ss/rest?action=remove_scrapeable_session&scrapeable_session_id=29
http://localhost:8779/ss/rest?action=reload_settings
http://localhost:8779/ss/rest?action=peek_scrapeable_session_log&scrapeable_session_id=42&num_lines=50
http://localhost:8779/ss/rest?action=get_scheduled_scraping_sessions
http://localhost:8779/ss/rest?action=disable_enable_scheduled_scraping_session&scheduled_scraping_session_id=110&enable=false
http://localhost:8779/ss/rest?action=remove_scheduled_scraping_session&scheduled_scraping_session_id=0
http://localhost:8779/ss/rest?action=set_scheduled_scraping_session&scheduled_scraping_session_id=3&scraping_session_name=Shopping+Site&timeout=123&schedule_date=08%2F20%2F2009&schedule_time=11:22:33&repeat_days=4&repeat_hours=3&repeat_minutes=2&repeat_seconds=1&threshold_time=21&threshold_record_count=43&settable_session_variables=this%3Dthatx%26foo%3Dbar
http://localhost:8779/ss/rest?action=save_settings&default_timeout=89&default_repeat_days=9&default_repeat_hours=8&default_repeat_minutes=7&default_repeat_seconds=6&default_threshold_time=4&default_threshold_record_count=3
http://localhost:8779/ss/rest?action=set_session_variable_on_scrapeable_session&scrapeable_session_id=3&key=foo&value=bap
http://localhost:8779/ss/rest?action=get_session_variable_from_scrapeable_session&scrapeable_session_id=3&key=foo
http://localhost:8779/ss/rest?action=get_memory_usage
As with any alpha feature we appreciate bug reports and feedback. Please don’t hesitate to drop us a line.
Permalink
08.17.09
Posted in Thoughts, Tips at 4:36 pm by jason
We previously listed some means to try to stop screen-scraping, but since it is an ongoing topic for us, it bears revisiting. Any site can be scraped, but some require such an influx of time and resources as to make it prohibitively expensive. Some of the common methods to do so are:
Turing tests
The most common implementation of the Turning Test is the old CAPTCHA that tries to ensure a human reads the text in an image, and feeds it into a form.
We have found a large number of sites that implement a very weak CAPTCHA that takes only a few minutes to get around. On the other hand, there are some very good implementations of Turing Tests that we would opt not to deal with given the choice, but a sophisticated OCR can sometimes overcome those, or many bulletin board spammers have some clever tricks to get past these.
Data as images
Sometimes you know which parts of your data are valuable. In that case it becomes reasonable to replace such text with an image. As with the Turing Test, there is ORC software that can read it, and there’s no reason we can’t save the image and have someone read it later.
Often times, however, listing data as an image without a text alternate is in violation of the Americans with Disabilities Act (ADA), and can be overcome with a couple of phone calls to a company’s legal department.
Code obfuscation
Using something like a JavaScript function to show data on the page though it’s not anywhere in the HTML source is a good trick. Other examples include putting prolific, extraneous comments through the page or having an interactive page that orders things in an unpredictable way (and the example I think of used CSS to make the display the same no matter the arrangement of the code.)
CSS Sprites
Recently we’ve encountered some instances where a page has one images containing numbers and letters, and used CSS to display only the characters they desired. This is in effect a combination of the previous 2 methods. First we have to get that master-image and read what characters are there, then we’d need to read the CSS in the site and determine to what character each tag was pointing.
While this is very clever, I suspect this too would run afoul the ADA, though I’ve not tested that yet.
Limit search results
Most of the data we want to get at is behind some sort of form. Some are easy, and submitting a blank form will yield all of the results. Some need an asterisk or percent put in the form. The hardest ones are those that will give you only so many results per query. Sometimes we just make a loop that will submit the letters of the alphabet to the form, but if that’s too general, we must make a loop to submit all combination of 2 or 3 letters–that’s 17,576 page requests.
IP Filtering
On occasion, a diligent webmaster will notice a large number of page requests coming from a particular IP address, and block requests from that domain. There are a number of methods to pass requests through alternate domains, however, so this method isn’t generally very effective.
Site Tinkering
Scraping always keys off of certain things in the HTML. Some sites have the resources to constantly tweak their HTML so that any scrapes are constantly out of date. Therefore it becomes cost ineffective to continually update the scrape for the constantly changing conditions.
Permalink
04.10.09
Posted in Miscellaneous at 12:03 pm by Todd Wilson
Yesterday I opened a fortune cookie that said, “Do something unusual tomorrow.” I thought about sky-diving or going the whole day blind-folded, but instead opted for something even crazier–sell screen-scraper for half price! If you’re on the fence about purchasing now might be a good time to take the plunge. I don’t see us doing this again any time soon. The sale will last until April 11, 2009 at 11:00 a.m. Mountain time.
Permalink
03.25.09
Posted in Miscellaneous at 10:24 am by Todd Wilson
We’ve had people asking for this for quite a while, and have finally gotten to it. We now have a video version of our first tutorial, accessible from the tutorial itself:
http://community.screen-scraper.com/Tutorial_1_Page_1
It isn’t perfect, but I think it’s a pretty good first version (and definitely better than what we had previously). We’re hoping to get some feedback, then will likely do another version soon based on that feedback. Feel free to give it a try and let us know what you think.
Permalink
03.09.09
Posted in Updates at 3:53 pm by Todd Wilson
Well, we finally did it. Break out the party hats and the sparkling apple juice (yes, we live in Utah).
We invite everyone and anyone to download or update to version 4.5 of screen-scraper. It is by far the most feature-rich and stable version to date. If you’re interested in checking out what’s new, take a look at the release notes.
Also, for anyone listening, keep an eye on the site if you’re considering purchasing in the near future. We’re about to do a little sale to celebrate the release of the new version…
Permalink
02.02.09
Posted in Updates at 12:00 pm by Todd Wilson
Here at screen-scraper we’re on pins and needles as we’re about to release another public version of screen-scraper (we’re anticipating calling it 4.5). Our current alpha release is looking to be pretty solid, and we’re planning on giving it just a bit more testing to ensure that there aren’t any bugs left. If you’re interested in helping us test, feel free to upgrade to the latest alpha version. Here’s a FAQ that might help on that: http://community.screen-scraper.com/faq#80n867.
Permalink
10.27.08
Posted in Miscellaneous at 11:44 am by Todd Wilson
One of our eagle-eyed developers recently spotted a couple of blog postings by Bronwyn Mauldin (here and here) wherein she discusses Iowa Workforce Development’s use of our screen-scraping technology in building out their job board. Bronwyn is a great writer and a consultant in the workforce development industry. After reading Bronwyn’s postings we decided to contact the Iowa office ourselves to catch up on how things have been going for them. It makes a great story as to how screen-scraping technology is being used in a very effective way. We decided to make a press release on it, which you can find here:
Iowa Workforce Development Uses Screen-Scraper to Enhance Job Search
Permalink
« Previous entries · Next entries »