ItuToTokens¶

Converts an ITU (i.e. a file like object and tokenises it into extended preprocessor tokens. This does not act on any preprocessing directives.

class cpip.core.ItuToTokens.ItuToTokens(theFileObj=None, theFileId=None, theDiagnostic=None)¶

Tokensises a file like object.

__init__(theFileObj=None, theFileId=None, theDiagnostic=None)¶

Constructor.

Parameters:	theFileObj (`_io.TextIOWrapper`) – File object. theFileId (`str`) – File ID such as the path. theDiagnostic (`NoneType`) – A diagnostic for processing messages.
Returns:	`NoneType`

_translatePhase_1()¶

Performs translation phase one. Note: We do not (yet) support universal-character-name conversion so this only does trigraphs.

Returns:	`NoneType`
Raises:	`IndexError`

_translatePhase_2()¶

Performs translation phase two. This does line continuation markers Note: We do not (yet) test for accidental UCN creation.

Returns:	`NoneType`
Raises:	`IndexError`

_translatePhase_3()¶

Performs translation phase three. Replaces comments and decomposes stream into preprocessing tokens.

Returns:	`NoneType`
Raises:	`IndexError`

genTokensKeywordPpDirective()¶

Process the file and generate tokens. This changes the type to a keyword or preprocessing-directive if it can do so.

Returns:	`tuple([str, str])` – Token value, token type.
Raises:	`StopIteration`

multiPassString¶

Returns:	`cpip.util.MultiPassString.MultiPassString` – The multi-pass string object.

translatePhases123()¶

Translate phase 1, 2, 3.

Returns:	`NoneType`