PpLexer¶

Generates tokens from a C or C++ translation unit.

TODO: Fix accidental token pasting. See: TestFromCppInternalsTokenspacing and, connected is: TODO: Set setPrevWs flag on the token where necessary.

TODO: Preprocessor statements in arguments of function like macros. Sect. 3.9 of cpp.pdf and existing MacroEnv tests.

exception cpip.core.PpLexer.ExceptionConditionalExpression¶: Exception when eval() conditional expressions.

exception cpip.core.PpLexer.ExceptionPpLexer¶: Exception when handling PpLexer object.

exception cpip.core.PpLexer.ExceptionPpLexerAlreadyGenerating¶: Exception when two generators are created then the internal state will become inconsistent.

exception cpip.core.PpLexer.ExceptionPpLexerCallStack¶: Exception when finding issues with the call stack or nested includes.

exception cpip.core.PpLexer.ExceptionPpLexerCallStackTooSmall¶: Exception when sys.getrecursionlimit() is too small.

exception cpip.core.PpLexer.ExceptionPpLexerCondLevelOutOfRange¶: Exception when handling a conditional token generation level.

exception cpip.core.PpLexer.ExceptionPpLexerDefine¶: Exception when loading predefined macro definitions.

exception cpip.core.PpLexer.ExceptionPpLexerNestedInclueLimit¶: Exception when nested #include limit exceeded.

exception cpip.core.PpLexer.ExceptionPpLexerNoFile¶: Exception when can not find file.

exception cpip.core.PpLexer.ExceptionPpLexerPreInclude¶: Exception when loading pre-include files.

exception cpip.core.PpLexer.ExceptionPpLexerPreIncludeIncNoCp¶: Exception when loading a pre-include file that has no current place (e.g. a StringIO object) and the pre-include then has an #include statement.

exception cpip.core.PpLexer.ExceptionPpLexerPredefine¶: Exception when loading predefined macro definitions.

cpip.core.PpLexer.PREPROCESSING_DIRECTIVES = ['if', 'ifdef', 'ifndef', 'elif', 'else', 'endif', 'include', 'define', 'undef', 'line', 'error', 'pragma']¶: Allowable preprocessing directives

class cpip.core.PpLexer.PpLexer(tuFileId, includeHandler, preIncFiles=None, diagnostic=None, pragmaHandler=None, stdPredefMacros=None, autoDefineDateTime=True, gccExtensions=False, annotateLineFile=False)¶

Create a translation unit tokeniser that applies ISO/IEC 9899:1999(E) Section 6 and/or ISO/IEC 14882:1998(E) section 16.

tuFileId

A file ID that will be given to the include handler to find the translation unit. Typically this will be the file path (as a string) to the file that is the Initial Translation Unit (ITU) i.e. the file being preprocessed.

includeHandler

A handler to file #includ‘d files typically a IncludeHandler.IncludeHandlerStd. This might have user and system include path information and a means of resolving file references.

preIncFiles

An ordered list of file like objects that are pre-include files. These are processed in order before the ITU is processed. Macro redefinition rules apply.

diagnostic

A diagnostic object, defaults to a CppDiagnostic.PreprocessDiagnosticStd.

pragmaHandler

A handler for #pragma statements.

This must have the attribute replaceTokens is to be implemented, if True then the tokens stream will be be macro replaced before being passed to the pragma handler.

This must have a function pragma() defined that takes a non-zero length list of PpToken.PpToken the last of which will be a newline token. The tokens returned will be yielded.

stdPredefMacros

A dictionary of Standard pre-defined macros. See for example: ISO/IEC 9899:1999 (E) 6.10.8 Predefined macro names ISO/IEC 14882:1998 (E) 16.8 Predefined macro names N2800=08-0310 16.8 Predefined macro names

The macros __DATE__ and __TIME__ will be automatically updated to current locale date/time (see autoDefineDateTime).

autoDefineDateTime

If True then the macros __DATE__ and __TIME__ will be automatically updated to current locale date/time. Mostly this is used for testing.

gccExtensions

Support GCC extensions. Currently just #include_next is supported.

annotateLineFile - if True then PpToken will output line number and file as cpp.

For example:

# 22 "/usr/include/stdio.h" 3 4
# 59 "/usr/include/stdio.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4

TODO: Set flags here rather than supplying them to a generator? This would make the API simply the ctor and ppTokens/next(). Flags would be: incWs - Include whitespace tokens. condLevel - (0, 1, 2) thus:

0: No conditionally compiled tokens. The fileIncludeGraphRoot will

not have any information about conditionally included files.

1: Conditionally compiled tokens are generated but not from

conditionally included files. The fileIncludeGraphRoot will have a reference to a conditionally included file but not that included file’s includes.

2: Conditionally compiled tokens including tokens from conditionally

included files. The fileIncludeGraphRoot will have all the information about conditionally included files recursively.

CALL_STACK_DEPTH_ASSUMED_PPTOKENS = 10¶: Each include The call stack depth, D = A + B + C*L Where L is the number of levels of nested includes and A is the call stack A above:

CALL_STACK_DEPTH_FIRST_INCLUDE = 3¶: B above:

CALL_STACK_DEPTH_PER_INCLUDE = 3¶: C above:

COND_LEVEL_DEFAULT = 0¶: Conditianlity settings for token generation

COND_LEVEL_OPTIONS = range(0, 3)¶: Conditionality level (0, 1, 2)

MAX_INCLUDE_DEPTH = 200¶: The maximum value of nested #include’s

colNum¶: Returns the current column number as an integer during processing.

condCompGraph¶: The conditional compilation graph as a CppCond.CppCondGraph object.

condState¶: The conditional state as (boolean, string).

currentFile¶: Returns the file ID on the top of the file stack.

definedMacros¶: Returns a string representing the currently defined macros.

fileIncludeGraphRoot¶: Returns the FileIncludeGraph.FileIncludeGraphRoot object.

fileLineCol¶: Returns a FileLineCol object or None

fileName¶: Returns the current file name during processing.

fileStack¶: Returns the file stack.

finalise()¶: Finalisation, may raise any Exception.

includeDepth¶: Returns the integer depth of the include stack.

lineNum¶: Returns the current line number as an integer during processing or None.

macroEnvironment¶: The current Macro environment as a MacroEnv.MacroEnv object.

Caution

Write to this at your own risk. Your write might be ignored or cause undefined behaviour.

ppTokens(incWs=True, minWs=False, condLevel=0)¶

A generator for providing a sequence of PpToken.PpToken in accordance with section 16 of ISO/IEC 14882:1998(E).

incWs - if True than whitespace tokens are included (i.e. tok.isWs() == True).

minWs - if True then whitespace runs will be minimised to a single space or, if newline is in the whitespace run, a single newline

condLevel - if !=0 then conditionally compiled tokens will be yielded and they will have have tok.isCond == True. The fileIncludeGraphRoot will be marked up with the appropriate conditionality. Levels are:

0: No conditionally compiled tokens. The fileIncludeGraphRoot will
not have any information about conditionally included files.

1: Conditionally compiled tokens are generated but not from 
conditionally included files. The fileIncludeGraphRoot will have
a reference to a conditionally included file but not that
included file's includes.

2: Conditionally compiled tokens including tokens from conditionally
included files. The fileIncludeGraphRoot will have all the
information about conditionally included files recursively.

(see _cppInclude where we check if self._condStack.isTrue():).

tuFileId¶: Returns the user supplied ID of the translation unit.

cpip.core.PpLexer.UNNAMED_FILE_NAME = 'Unnamed Pre-include'¶: Used when file objects have no name