Scanner Design
==============

Lookahead
---------
I am expecting that the xml grammer only demands one token of lookahead.  In this
case a token is not a single character but a recognisable string such as "<!", "<?"
or a name token.

I anticipate that, depending on the context, the scanner will be asked for the
next lookahead string - and this function will be qualified with the types of
characters that can form that string.  This means that the string cannot be removed 
from the input buffer unless some form of back-tracking is provided.

Perhaps the characters can be removed from the input if a string matching the
specified characterists exists.  The complication with this is that the
characteristcs could represent a complex regular expression.  

For complex regular expression tokens a custom procedure is probably the most
effective way of obtaining the next lookahead token.

I assume that leading whitespace can always be ignored, but I remember that there is something
about this in the spec.


Parser Design
=============

If scanner lookahead is restricted to one token, then we are constrained in the
way we write the parser.  We want to write a recursive descent parser, so the
recursive routines will be forced to return a positive result if a token has been 
eaten from the input stream.


This is more a constraint on the Grammer than the parser itself.  If the grammer
allows more than one lookahead token then this approach is doomed.  We will be on
the lookout for this!

If back-tracking becomes necessary then it will have an impact on the design of the
input buffer.  A naive approach would be to load the entire file into the input
buffer - but this may prove too memory intensive, so a design that allows a small
portion of the input file to be kept in buffer is preferable.

Error Recovery
==============