interpreter.lexer package

Submodules

interpreter.lexer.tokens module

exception interpreter.lexer.tokens.BuildTreeException(message='Error building parse tree')

Bases: Exception

Exception to throw when there is an error in building the parse tree.

exception interpreter.lexer.tokens.BuiltTreeException(message='Error building parse tree')

Bases: Exception

Exception to throw the parser has finished parsing.

class interpreter.lexer.tokens.CommaToken(line, col)

Bases: Token

Represents a comma token.

regex()

class interpreter.lexer.tokens.CommentToken(line, col)

Bases: Token

Represents a comment.

regex()

class interpreter.lexer.tokens.IdentifierToken(value, line, col)

Bases: Token

Represents an identifier token.

regex()

class interpreter.lexer.tokens.IntegerToken(value, line, col)

Bases: Token

Represents an integer token.

regex()

exception interpreter.lexer.tokens.InvalidTokenException

Bases: Exception

Exception to throw when an invalid token is encountered.

classmethod fromLine(line, col)

classmethod fromToken(token: Token)

class interpreter.lexer.tokens.KeywordToken(value, line, col)

Bases: Token

Represents a keyword token.

type_keywords()

values()

class interpreter.lexer.tokens.LParenToken(line, col)

Bases: Token

Represents a left parenthesis token.

regex()

class interpreter.lexer.tokens.OperatorToken(value, line, col)

Bases: Token

Represents an operator token.

regex()

class interpreter.lexer.tokens.RParenToken(line, col)

Bases: Token

Represents a right parenthesis token.

regex()

class interpreter.lexer.tokens.SemiColonToken(line, col)

Bases: Token

Represents a semicolon token in the interpreter.

regex()

class interpreter.lexer.tokens.SpacesToken(line, col)

Bases: Token

Represents a space token.

regex()

class interpreter.lexer.tokens.StringToken(value, line, col)

Bases: Token

Represents a string token.

regex()

class interpreter.lexer.tokens.Token(type, value, line=None, col=None)

Bases: object

Represents a token in the interpreter.

Attributes:

type (str): The type of the token. value (str): The value of the token. line (int): The line number where the token appears. col (int): The column number where the token appears.

Methods:

isType(t) -> bool: Checks whether the token is of a certain type. isValue(value) -> bool: Checks whether the token has a certain value. getValue() -> str: Returns the value of the token. getType() -> str: Returns the type of the token.

classmethod fromValue(value)

getType()

getValue()

classmethod instance()

isType(t)

isValue(value)

regex()

class interpreter.lexer.tokens.TokenRegex

Bases: object

CloseParen = re.compile('\\)')

Comma = re.compile(',')

Comment = re.compile('\\/\\/[\\"\\(\\);,\\\\ a-zA-Z0-9+\\-*<>&.@/:=~|$!#%^_\\[\\]\\{\\}\\"\\`?]*\\n')

Identifier = re.compile('[a-zA-Z][a-zA-Z0-9_]*')

Integer = re.compile('[0-9]+')

OpenParen = re.compile('\\(')

Operator = re.compile('[+\\-*<>&.@/:=~|$!#%^_\\[\\]\\{\\}\\"\\`?]+')

SemiColon = re.compile(';')

Spaces = re.compile('[\\s\\n]+')

String = re.compile('\\\'[(\\\\t)(\\\\n)(\\\\)(\\\\\\")\\(\\);,\\sa-zA-Z0-9+\\-*<>&.@/:=~|$!#%^_\\[\\]\\{\\}\\"\\`?]*\\\'')

Module contents

class interpreter.lexer.Lexer(program)

Bases: object

The Lexer class is responsible for tokenizing a program string into a list of tokens.

Attributes:

program (str): The program string to be tokenized. tokens (list): The list of tokens generated from the program string. position (int): The current position in the program string. line_no (int): The current line number. char_pos (int): The current character position.

Methods:

reset(): Resets the lexer to its initial state. tokenize(count=-1): Tokenizes the program string up to a specified count. nextToken(): Retrieves the next token from the program string. __nextToken(): Retrieves the next token from the program string (internal method).

Raises:

InvalidTokenException: If the next token is invalid.

lookAhead()

Retrieves the next token from the program string without consuming the token

Returns: Next Token or None if there are no more tokens.

nextToken()

Retrieves the next token from the program string. This consumes the token. Returns only if there are no more tokens.

Returns:

Token: The next token from the program string | None if there are no more tokens.

reset()

Resets the lexer to its initial state.

tokenize(count: int = -1)

Tokenizes the program string up to a specified count and returns the list of tokens from the look ahead queue.

Args:

count (int, optional): The number of tokens to tokenize. Defaults to -1.

Returns:

list: The list of tokens generated from the program string.