screen scraping : Java Glossary

Local Screen Scraping

The Screen scraper program has to fool the host into thinking it is talking to one of its usual hardware terminals with an operator sitting at it. It must compose queries in the format the usual hardware would produce, and interpret the formatted data coming back, parsing it to extract the data and leave behind the formatting.

Before you leap into writing a screen scraper, investigate thoroughly all the possible terminals you might emulate that will work with the existing app. You might find some simpler to emulate than others.

Some of the old terminals had quite complex protocols, e.g. SDLC, so you usually you don’t want to write that part from scratch. Look for a third party libary to handle the low-level protocol details.

Screen scraping can also refer to capturing a bit image off the screen the program is running on using Robot.createScreenCapture.

To convert the pixels back to text is not quite as difficult as you might think. You can do a primitive OCR that just compares clip regions with a cast of prototype characters set in the same font and size looking for an exact match. You might want to adjust colours to pure black and white before you start. This is quite a bit easier than real OCR where you have to deal with imprecisely formed characters.

To separate characters you have to look for a vertical strip of white. To rapidly find the matching character you could use several methods:

• You might normalise your comparisons by creating a tight bounding box around each character. If the bounding box sizes don’t match exactly there is no point in comparing the bit contents. You presort letters by bounding box size chain them off that size and only compare against potential matches.
• Do a hash on the bits of the character, and use a Hashtable lookup.
• Create a set of probe points on the character and see if you can arrange the probe points such that every character has a unique pattern using a minimum number of probe points. Just look up the pattern (bits considered as int) in an array to get the corresponding character. To get your set, compute for each probe point calculate how many letters it can discriminate. Take the #1 discriminator as your first probe point. Then calculate how many letters each of the remaining probe points can discriminate that the #1 point can’t. This is your #2 point, etc. Then manually tweak interfering at each stage.

Web-Based Screen Scraping

Even if you screen-scrape for non-commercial purposes, even if you don’t repost the data, and even if you don’t put much load on their server, they can still get irate, block you and send lawyer letters. I think the main reason is they put up their site primarily to serve ads, (the data offered are just bait), and you obviously are not reading the ads if you are screenscraping. If you stick to government sources, you will likely be safe.

Before you launch on a screen scraping project, do a thorough search of the site for a downloadable version of the data, sometimes in CSV format, or spreadsheet format or XML. This is ever so much more convenient and stable. If there is no such download, it never hurts to ask the source to provide one. It somehow never occurs to data providers that data are almost useless unless they are in computer-friendly format, i.e. not HTML.

Once you have your downloaded page, String.indexOf and regexes are useful tools to extract the data. Usually the data are too malformed to use a straightforward HTML parser. TagSoup can be useful to tidy up mangled HTML syntax prior to simple-minded programs sifting through the data.