XMLLexer

Description

XMLLexer is a transform

• from a Readable stream representing the content of an XML (HTML and so on) document;
• to an object mode stream of strings representing its lexemes,

where by a lexeme we mean either

• a text node
• or some pointy brackets delimited tag
  • including comments, processing instructions and all weird <! stuff inherited from SGML
    • and, yes <![CDATA[ belongs here.

Usage

const lexer = XMLLexer ({...options})

someInputStream.pipe (lexer).pipe (someObjectOutputStream)

// or

lexer.on  ('data', s => console.log (s))
lexed.end ('<root>...</root>')

Options

Name	Default	Description
maxLength	1 << 20	Maximum lexeme length (in bytes or characters)
encoding	utf8	Encoding for output lexemes, used with binary input

Lexeme types

XMLLexer splits the incoming stream into strings matching following patterns:

Template	Description	Note
<? ... ?>	Processing instruction	or prolog <?xml ... ?>
<!-- ... -->	Comment
<! ... [ ... [ ... ]]>	Conditional, complex doctype	or <![CDATA[ ... ]]>
<! ... [ ... ]>	Medium doctype
<! ... >	Doctype, atomic
< ... >	Element tag (opening, closing or both)
...	Text node

Implementation

Members

Name	Type	Description
body	XMLLexerBodyBuffer or XMLLexerBodyString	What's left unparsed
beforeBody	BigInt	Amount of chars or bytes left past current body contents
state	int	Current state
poosition	int	Last position analyzed for [ in state ST_TAG_X
awaited	Buffer	Enclosing sequence

Constants

States (state values)

XMLLexer is basically a finite state machine with the following states:

Constant	Body from start	Description
ST_TEXT	unknown or anyting but <	find the nearest <, slice the text
ST_LT	<	Look ahead for the next byte to switch to ST_LT_X (if !) or ST_TAG (otherwise)
ST_LT_X	<!	Look ahead for the next byte to switch to ST_TAG with awaited --> (if -) or ST_TAG_X (otherwise)
ST_TAG	<...	Scan for '>', find the one preceded with awaited sequence, slice the tag
ST_TAG_X	<!...	Scan through all bytes, adjust awaited when [ occur, watch for '>', finally slice the tag

Following transitions are possible:

From	To	Awaited	Explanation
ST_TEXT	ST_LT		< occured, buffered text is given out (if any)
ST_TEXT	ST_TEXT		some other char occured
ST_LT	ST_LT_X		! occured
ST_LT	ST_TAG	?>	! occured
ST_LT	ST_TAG	>	not !, normal tag
ST_LT_X	ST_TAG	-->	- occured, must be a comment (unless broken)
ST_LT_X	ST_TAG_X	> / ]> / ]]>	counting [s
ST_TAG	ST_TEXT		> occured, previous characters match -- giving out the tag
ST_TAG_X	ST_TEXT		> occured, previous characters match -- giving out the tag

Enclosing sequences (awaited values)

Possible enclosing sequences (> chopped) are stored as Buffer instances:

Name	Value	Description
CL_DEAFULT	''	default
CL_PI	?	for <?
CL_COMMENT	--	for <!-
CL_SQ_1	]	if [ occured in ST_TAG state and CL_DEAFULT awaited
CL_SQ_2	]]	if [ occured in ST_TAG state and CL_SQ_1 awaited

Methods

Name, Params	Type	Description
append (chunk)		append this chunk to the body
setState (state, awaited)		set the state and awaited members
isClosing (pos)	boolean	do bytes preceding pos match awaited
publishTo (pos)		push the the slice from start to pos in the output stream, then start after pos
parse ()		scan the body from start, publish lexemes found, move start after the last one
checkMaxLength ()		Throws an error if body.size () > maxLength
_transform (chunk, encoding, callback)		append the incoming chunk, parse the body, then trim, reset start to 0 it and invoke callback
_flush (callback)		publish the rest of the body, invoke callback