PpToken¶

Represents a preprocessing Token in C/C++ source code.

cpip.core.PpToken.ENUM_NAME = {0: 'header-name', 1: 'identifier', 2: 'pp-number', 3: 'character-literal', 4: 'string-literal', 5: 'preprocessing-op-or-punc', 6: 'non-whitespace', 7: 'whitespace', 8: 'concat'}¶

Map of {integer : PREPROCESS_TOKEN_TYPE, ...} So this can be used thus:

if ENUM_NAME[token_type] == 'header-name':

exception cpip.core.PpToken.ExceptionCpipToken¶: Used by PpToken.

exception cpip.core.PpToken.ExceptionCpipTokenIllegalMerge¶: Used by PpToken when PpToken.merge() is called illegally.

exception cpip.core.PpToken.ExceptionCpipTokenIllegalOperation¶: Used by PpToken when an illegal operation is performed.

exception cpip.core.PpToken.ExceptionCpipTokenReopenForExpansion¶: Used by PpToken when a non-expandable token is made available for expansion.

exception cpip.core.PpToken.ExceptionCpipTokenUnknownType¶: Used by PpToken when the token type is out of range.

cpip.core.PpToken.LEX_PPTOKEN_TYPES = ['header-name', 'identifier', 'pp-number', 'character-literal', 'string-literal', 'preprocessing-op-or-punc', 'non-whitespace', 'whitespace', 'concat']¶

Types of preprocessing-token From: ISO/IEC 14882:1998(E) 2.4 Preprocessing tokens [lex.pptoken] and ISO/IEC 9899:1999 (E) 6.4.7 Header names .. note:

Para 3 of the latter says that: "A header name preprocessing token is
recognized only within a ``#include`` preprocessing directive."

So in other contexts a header-name that is a q-char-sequence should be treated
as a string-literal

This produces interesting issues in this case:

#define str(s) # s
#include str(foo.h)

The stringise operator creates a string-literal token but the #include directive expects a header-name. So in certain contexts (macro stringising followed by #include instruction) we need to ‘downcast’ a string-literal to a header-name.

See cpip.core.PpLexer.PpLexer for how this is done

cpip.core.PpToken.LEX_PPTOKEN_TYPE_ENUM_RANGE = range(0, 9)¶: Range of allowable enum values

cpip.core.PpToken.NAME_ENUM = {'header-name': 0, 'whitespace': 7, 'preprocessing-op-or-punc': 5, 'pp-number': 2, 'string-literal': 4, 'non-whitespace': 6, 'concat': 8, 'identifier': 1, 'character-literal': 3}¶

Map of {PREPROCESS_TOKEN_TYPE : integer, ...} So this can be used thus:

self._cppTokType = NAME_ENUM['header-name']

class cpip.core.PpToken.PpToken(t, tt, lineNum=0, colNum=0, isReplacement=False)¶

Holds a preprocessor token, its type and whether the token can be replaced.

t is the token (a string) and tt is either an enumerated integer or a string. Internally tt is stored as an enumerated integer. If the token is an identifier then it is eligible for replacement unless marked otherwise.

SINGLE_SPACE = ' '¶: Representation of a single whitespace

WORD_REPLACE_MAP = {'/': '//', '||': ' or ', 'false': 'False', 'true': 'True', '&&': ' and '}¶: Operators that are replaced directly by Python equivalents for constant evaluation

canReplace¶: Flag to control whether this token is eligible for replacement

colNum¶: Returns the column number of the start of the token as an integer.

copy()¶: Returns a shallow copy of self. This is useful where the same token is added to multiple lists and then a merge() operation on one list will be seen by the others. To avoid this insert self.copy() in all but one of the lists.

evalConstExpr()¶

Returns an string value suitable for eval’ing in a constant expression. For numbers this removes such tiresome trivia as 'u', 'L' etc. For others it replaces '&&' with 'and' and so on.

See ISO/IEC ISO/IEC 14882:1998(E) 16.1 Conditional inclusion sub-section 4 i.e. section 16.1-4

And: ISO/IEC 9899:1999 (E) 6.10.1 Conditional inclusion sub-section 3 i.e. section 6.10.1-3

Returns:	`str` – String for `eval()`.
Raises:	`KeyError`

getIsReplacement()¶: Gets the flag that records that this token is the result of macro replacement

getPrevWs()¶: Gets the flag that records prior whitespace.

getReplace()¶

Gets the flag that controls whether this can be replaced.

Returns:	`bool` – Flag.

isCond¶

Flag that if True indicates that the token appeared within a section that was conditionally compiled. This is False on construction and can only be set True by setIsCond()

Returns:	`bool` – Flag.

isIdentifier()¶

Returns:	`bool` – `True` if the token type is ‘identifier’.

isReplacement¶: Flag that records that this token is the result of macro replacement

isUnCond¶

Flag that if True indicates that the token appeared within a section that was un-conditionally compiled. This is the negation of isCond.

Returns:	`bool` – Flag.

isWs()¶

Returns:	`bool` – `True` if the token type is ‘whitespace’.

lineNum¶: Returns the line number of the start of the token as an integer.

merge(other)¶: This will merge by appending the other token if they are different token types the type becomes ‘concat’.

prevWs¶: Flag to indicate whether this token is preceded by whitespace

replaceNewLine()¶

Replace any newline with a single whitespace character in-place.

See: ISO/IEC 9899:1999(E) 6.10-3 and C++ ISO/IEC 14882:1998(E) 16.3-9

Returns:	`NoneType`
Raises:	`ExceptionCpipTokenIllegalOperation` if I am not a whitespace token.

setIsCond()¶

Sets self._isCond to be True.

Returns:	`NoneType`

setIsReplacement(val)¶

Sets the flag that records that this token is the result of macro replacement.

Parameters:	val (`bool`) – Flag.
Returns:	`NoneType`

setPrevWs(val)¶: Sets the flag that records prior whitespace.

setReplace(val)¶: Setter, will raise if I am not an identifier or val is True and if I am otherwise not expandable.

shrinkWs()¶

Replace all whitespace with a single ‘ ‘

Returns:	`NoneType`
Raises:	`ExceptionCpipTokenIllegalOperation` if I am not a whitespace token.

subst(t, tt)¶: Substitutes token value and type.

t¶

Returns the token as a string.

Returns:	`str` – Token.

tokEnumToktype¶: Returns the token and the enumerated token type as a tuple.

tokToktype¶: Returns the token and the token type (as a string) as a tuple.

tt¶

Returns the token type as a string.

Returns:	`str` – Token type.

cpip.core.PpToken.tokensStr(theTokens, shortForm=True)¶

Given a list of tokens this returns them as a string.

Parameters:	theTokens (`list([cpip.core.PpToken.PpToken])`) – List of tokens. shortForm (`bool`) – If `shortForm` is `True` then the lexical string is returned. If `False` then the `PpToken` representations separated by ‘ \| ‘ is returned. e.g. `PpToken(t="f", tt=identifier, line=True, prev=False, ?=False) \| ...`
Returns:	`str` – Tokens as a string.