Class CToken

• Defined in File token.h

Class Documentation

class CToken

Token within the source string.

Unnamed Group

operator const std::string&() const

Returns the code chunk that the token points to. When a length is part of the token the string is limited to the length. Otherwise the chunk contains the complete string starting at the token.

Returns:

Returns the string chunk.

const char *c_str() const

Returns the code chunk that the token points to. When a length is part of the token the string is limited to the length. Otherwise the chunk contains the complete string starting at the token.

Returns:

Returns the string chunk.

Unnamed Group

bool IsLiteral() const

Information functions.

Returns:

Return the information.

bool IsInteger() const

Information functions.

Returns:

Return the information.

bool IsSigned() const

Information functions.

Returns:

Return the information.

bool IsUnsigned() const

Information functions.

Returns:

Return the information.

bool IsLong() const

Information functions.

Returns:

Return the information.

bool IsLongLong() const

Information functions.

Returns:

Return the information.

bool IsDecimal() const

Information functions.

Returns:

Return the information.

bool IsHexadecimal() const

Information functions.

Returns:

Return the information.

bool IsOctal() const

Information functions.

Returns:

Return the information.

bool IsBinary() const

Information functions.

Returns:

Return the information.

bool IsFloatingPoint() const

Information functions.

Returns:

Return the information.

bool IsFixedPoint() const

Information functions.

Returns:

Return the information.

bool IsCharacter() const

Information functions.

Returns:

Return the information.

bool IsCharacterSequence() const

Information functions.

Returns:

Return the information.

bool IsString() const

Information functions.

Returns:

Return the information.

bool IsAscii() const

Information functions.

Returns:

Return the information.

bool IsUtf8() const

Information functions.

Returns:

Return the information.

bool IsUtf16() const

Information functions.

Returns:

Return the information.

bool IsUtf32() const

Information functions.

Returns:

Return the information.

bool IsWide() const

Information functions.

Returns:

Return the information.

bool IsRawString() const

Information functions.

Returns:

Return the information.

bool IsBoolean() const

Information functions.

Returns:

Return the information.

bool IsNullptr() const

Information functions.

Returns:

Return the information.

Public Functions

CToken() = default

Default constructor.

CToken(const std::string &rssText, ETokenType eType = ETokenType::token_identifier)

Specific constructor for code insertion.

During processing, it sometimes is necessary to add created tokens to token lists.

Attention

The constructor doesn’t check for validity of the token text and types.

Parameters:

• rssText – [in] The text the token represents.

• eType – [in] The token type (default identifier).

CToken(const std::string &rssText, ETokenLiteralType eLiteralType)

Specific constructor for code insertion.

During processing, it sometimes is necessary to add created tokens to token lists.

Attention

The constructor doesn’t check for validity of the token text and types.

Parameters:

• rssText – [in] The text the token represents.

• eLiteralType – [in] The token literal type (default undefined).

CToken(const std::string &rssText, ETokenMetaType eMetaType)

Specific constructor for code insertion.

During processing, it sometimes is necessary to add created tokens to token lists.

Attention

The constructor doesn’t check for validity of the token text and types.

Parameters:

• rssText – [in] The text the token represents.

• eMetaType – [in] The token meta type (default undefined).

void MoveStart(int32_t iOffset)

Move start; adjusts the start position of the token. Does not adjust the end.

Attention

Does move before the beginning of the line.

Attention

Does not move past the end of the token.

Parameters:

iOffset – [in] The amount of characters to adjust. A negative offset moves to the left.

void MoveEnd(int32_t iOffset)

Move end; adjusts the end position of the token. Does not adjust the start.

Attention

Does move before the start of the token.

Parameters:

iOffset – [in] The amount of characters to adjust. A negative offset moves to the left.

bool operator==(const CToken &rtoken) const

Equality operator.

Parameters:

rtoken – [in] The token to compare with.

Returns:

Returns ‘true’ if the provided token and this token are identical. Otherwise returns ‘false’.

bool operator!=(const CToken &rtoken) const

Inequality operator.

Parameters:

rtoken – [in] The token to compare with.

Returns:

Returns ‘true’ if the provided token and this token are not identical. Otherwise returns ‘false’.

uint32_t GetLine() const

Return the line number.

Returns:

The line number.

uint32_t GetCol() const

Return the column number.

Returns:

Get column number.

uint32_t GetEndLine() const

Get the last line this token on.

Returns:

The line number.

uint32_t GetEndCol() const

Get the last column this token is on.

Returns:

The column number.

uint32_t GetLength() const

Return the length of the current token snippet. Or 0xFFFFFFFF when the rest of the code is part of this snippet.

Returns:

Returns the length of the token snippet.

ETokenType GetType() const

Get the token type.

Returns:

The token type of this token.

ETokenLiteralType GetLiteralType() const

Get the token’s literal type (if the token is a literal token).

Returns:

The literal type of this token when the token is a literal token. ETokenLiteralType::token_undefined otherwise.

ETokenMetaType GetMetaType() const

Get the token’s meta type (if the token is a meta token).

Returns:

The meta type of this token when the token is a meta token. ETokenMetaType::token_undefined otherwise.

operator bool() const

Validity operator.

Returns:

Returns ‘true’ when the token is valid. Otherwise returns ‘false’.

std::string Left(uint32_t uiLen) const

Get the left part of the string.

Parameters:

uiLen – [in] The amount of characters to return.

Returns:

Returns the left part of the string chunk.

std::string Right(uint32_t uiLen) const

Get the right part of the string.

Parameters:

uiLen – [in] The amount of characters to return.

Returns:

Returns the right part of the string chunk.

bool operator==(const char *szString) const

String equality operator.

Parameters:

szString – [in] Zero terminated string to use for the comparison.

Returns:

Returns ‘true’ if the string chunk of this token is identical with the provided string. Otherwise returns ‘false’.

bool operator!=(const char *szString) const

String inequality operator.

Parameters:

szString – [in] Zero terminated string to use for the comparison.

Returns:

Returns ‘true’ if the string chunk of this token is not identical with the provided string. Otherwise returns ‘false’.

bool operator==(const std::string &rssString) const

String equality operator.

Parameters:

rssString – [in] String object to use for the comparison.

Returns:

Returns ‘true’ if the string chunk of this token is identical with the provided string. Otherwise returns ‘false’.

bool operator!=(const std::string &rssString) const

String inequality operator.

Parameters:

rssString – [in] String object to use for the comparison.

Returns:

Returns ‘true’ if the string chunk of this token is identical with the provided string. Otherwise returns ‘false’.

CConstVariant ValueRef() const

Get the value of the token in the smallest fitting data type.

Throws:

The – function throws an exception in case the assignment is not possible.

Returns:

Returns the value as const variant.

inline void SetContext(const CContextPtr &rptrContext)

Set source code context.

The parser can attach source code context. The context contains all the allocated memory this token is pointing to and holding the context in the token prevents unexpected disappearance of the token during post-processing.

Parameters:

rptrContext – [in] Reference to the smart pointer to the source code context.

inline CContextPtr GetContext() const

Get source code context.

Returns:

Pointer to the source code context.