"The Road Ahead" 10 years on

Handwriting recognition

Some users will find it convenient to handwrite instructions to a computer, rather than speaking or typing them. Many companies, including Microsoft, have spent some years working on what we call “pen-based computers” capable of reading handwriting. I was overly optimistic about how quickly we would be able to create software that would recognise the handwriting of a broad range of people. The difficulties turned out to be quite subtle.

When we tested the system ourselves it worked quite well, but new users continued to have trouble with it. We discovered we were unconsciously making our handwriting neater and more recognisable than usual. We were adapting to the machine rather than the other way around. Another time, when the team thought they had created a program that worked, they came proudly to demonstrate their achievement to me. It didn’t work at the demonstration. Everyone on the project happened to be right-handed, and the computer, which was programmed to look at the strokes in the writing, couldn’t interpret the very different ones in my left-handed penmanship.

It turned out that getting a computer to recognise handwriting is as difficult as getting one to recognise speech. But I remain optimistic that as computer performance increases we’ll have computers able to do this too.

This one wasn't exactly a huge leap. Apple launched their pen-based Newton MessagePad two years before Gates wrote the above prediction. The Newton featured the now ubiquitous touchscreen-stylus combination and introduced handwriting recognition to the masses. Regrettably, those first impressions were less than favourable.

The recognition software had been developed by a third party and required substantial training to help it learn your writing style. It was so inept at this task that the Newton rapidly became the butt of jokes worldwide as it mangled perfectly straightforward phrases into complete nonsense. Given its 20MHz CPU and limited 640KB of RAM, it was a case of a nice idea that was before ahead of its time.

The next evolution was the PalmPilot, launched in 1996. Having witnessed the Newton disaster, Palm eschewed the challenge of recognising natural handwriting in favour of developing a custom input method they called Graffiti. This required users to input characters by following pre-defined motions. By streamlining the required strokes, Palm made the job of accurate character recognition far easier - an crucial move given the original PalmPilot's modest Motorola DragonBall / 68000 CPU.

While techheads quickly adapted to Graffiti, inputting characters at speed, it was always seen as a barrier to the widespread adoption of Palm-based devices as novice users struggled to learn the unique input language. The Palm was still the best of the bunch, but it took until Microsoft enter the market before things went really mainstream.

Is that a PC in your Pocket or are you just pleased to see me?

In 2000, Microsoft launched Pocket PC - essentially Windows for mobile devices. This included a few different options for input, designed to make these new devices easy to use. Pocket PC devices offer a Graffiti-esque input method, allowing for an easy transition for Palm users to the new platform.

Most users would clearly prefer to use their regular handwriting and Pocket PC devices support that using the in-built Transcriber software. It is now possible to scribble everyday cursive handwriting on the screen at the software will turn it into neatly typed text. This is certain faster than tapping the on-screen keyboard but transcription speeds still lag behind an experienced user on a mini keyboard.

As a result, the market as fragmented somewhat: heavy keyboard users favour devices such as a BlackBerry or Treo, while many more prefer the larger screen of the pen-based devices such as the Dell Axim Pocket PC. At the end of the day, you choose the device best suited to your needs.