Class Tokenizer

Namespace

flx.SILQ.Core

Assembly

flx.SILQ.dll

public class Tokenizer

Inheritance

object

Tokenizer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Methods

Tokenize(string[])

Tokenizes the input lines into a list of tokens. Handles keywords, operators, numbers, identifiers, strings, and comments. Adds error tokens for unexpected characters.

public List<Token> Tokenize(string[] input)

Parameters

input string[]

The input lines to tokenize.

Returns

List<Token>

A list of Token objects representing the tokenized input.