class interface for flex

[kroening]

This adds an abstract class that serves as interface for flex-generated tokenizers.

[tautschnig]

1. What about the C++ interface provided by flex? I think we should at least say why we don't use it (portability across system-provided flex versions could be one reason).
2. If we go down this route, shouldn't this be done in the CBMC repository?

[kroening]

1. What about the C++ interface provided by flex? I think we should at least say why we don't use it (portability across system-provided flex versions could be one reason).

Yes; the flex C++ interface was the first thing I tried, and I wasn't able to make the build work reliably, owing to the header file. I've added a comment saying so.

2. If we go down this route, shouldn't this be done in the CBMC repository?

Yes. I'll need to find a way to deliver the line number, which is done by the macro in util/parser.h.