Tokenizer Class Reference

Tokenizer converts text string into tokens. More...

#include <Tokenizer.h>

Collaboration diagram for Tokenizer:

Classes
class	Error
	Error. More...

Public Member Functions
	Tokenizer (Ustring &)
	Creates tokenizer from Unicode string.
virtual	~Tokenizer ()
	Destroys tokenizer.
std::string	toString ()
	Converts token list to string.

Detailed Description

Tokenizer converts text string into tokens.

Definition at line 29 of file Tokenizer.h.

Member Function Documentation

std::string Tokenizer::toString

(

)

Converts token list to string.

Returns

string

Definition at line 345 of file Tokenizer.cc.

Here is the call graph for this function:

---

The documentation for this class was generated from the following files:

• src/language/Tokenizer.h
• src/language/Tokenizer.cc