CPIPMain - The main command line tool for preprocessing.¶

CPIPMain.py – Preprocess the file or the files in a directory.

(CPIP36) $ python src/cpip/CPIPMain.py --help
usage: CPIPMain.py [-h] [-c] [-d DUMP] [-g GLOB] [--heap] [-j JOBS] [-k]
                   [-l LOGLEVEL] [-o OUTPUT] [-p] [-r] [-t] [-G]
                   [-S PREDEFINES] [-C] [-D DEFINES] [-P PREINC] [-I INCUSR]
                   [-J INCSYS]
                   path

CPIPMain.py - Preprocess the file or the files in a directory.
  Created by Paul Ross on 2011-07-10.
  Copyright 2008-2017. All rights reserved.
  Version: v0.9.8rc0
  Licensed under GPL 2.0
USAGE

positional arguments:
  path                  Path to source file or directory.

optional arguments:
  -h, --help            show this help message and exit
  -c                    Add conditionally included files to the plots.
                        [default: False]
  -d DUMP, --dump DUMP  Dump output, additive. Can be: C - Conditional
                        compilation graph. F - File names encountered and
                        their count. I - Include graph. M - Macro environment.
                        T - Token count. R - Macro dependencies as an input to
                        DOT. [default: []]
  -g GLOB, --glob GLOB  Pattern match to use when processing directories.
                        [default: []] i.e. every file.
  --heap                Profile memory usage. [default: False]
  -j JOBS, --jobs JOBS  Max simultaneous processes when pre-processing
                        directories. Zero uses number of native CPUs [4]. 1
                        means no multiprocessing. [default: 0]
  -k, --keep-going      Keep going. [default: False]
  -l LOGLEVEL, --loglevel LOGLEVEL
                        Log Level (debug=10, info=20, warning=30, error=40,
                        critical=50) [default: 30]
  -o OUTPUT, --output OUTPUT
                        Output directory. [default: out]
  -p                    Ignore pragma statements. [default: False]
  -r, --recursive       Recursively process directories. [default: False]
  -t, --dot             Write an DOT include dependency table and execute DOT
                        on it to create a SVG file. [default: False]
  -G                    Support GCC extensions. Currently only #include_next.
                        [default: False]
  -S PREDEFINES, --predefine PREDEFINES
                        Add standard predefined macro definitions of the form
                        name<=definition>. They are introduced into the
                        environment before anything else. They can not be
                        redefined. __DATE__ and __TIME__ will be automatically
                        allocated in here. __FILE__ and __LINE__ are defined
                        dynamically. See ISO/IEC 9899:1999 (E) 6.10.8
                        Predefined macro names. [default: []]
  -C, --CPP             Sys call 'cpp -dM' to extract and use platform
                        specific macros. These are inserted after -S option
                        and before the -D option. [default: False]
  -D DEFINES, --define DEFINES
                        Add macro definitions of the form name<=definition>.
                        These are introduced into the environment before any
                        pre-include. [default: []]
  -P PREINC, --pre PREINC
                        Add pre-include file path, this file precedes the
                        initial translation unit. [default: []]
  -I INCUSR, --usr INCUSR
                        Add user include search path. [default: []]
  -J INCSYS, --sys INCSYS
                        Add system include search path. [default: []]

class cpip.CPIPMain.FigVisitorDot(lenPrefix=0)¶

Simple visitor that collects parent/child links for plotting the graph with dot.

_fileName(theFigNode)¶: Treat the file name consistently.

visitGraph(theFigNode, theDepth, theLine)¶: .

class cpip.CPIPMain.FigVisitorLargestCommanPrefix¶

Simple visitor that walks the tree and finds the largest common file name prefix.

visitGraph(theFigNode, theDepth, theLine)¶: Capture the file name.

class cpip.CPIPMain.MainJobSpec(incHandler, preDefMacros, preIncFiles, diagnostic, pragmaHandler, keepGoing, conditionalLevel, dumpList, helpMap, includeDOT, cmdLine, gccExtensions)¶

__getnewargs__()¶: Return self as a plain tuple. Used by copy and pickle.

static __new__(_cls, incHandler, preDefMacros, preIncFiles, diagnostic, pragmaHandler, keepGoing, conditionalLevel, dumpList, helpMap, includeDOT, cmdLine, gccExtensions)¶: Create new instance of MainJobSpec(incHandler, preDefMacros, preIncFiles, diagnostic, pragmaHandler, keepGoing, conditionalLevel, dumpList, helpMap, includeDOT, cmdLine, gccExtensions)

__repr__()¶: Return a nicely formatted representation string

_asdict()¶: Return a new OrderedDict which maps field names to their values.

classmethod _make(iterable, new=<built-in method __new__ of type object at 0xa385c0>, len=<built-in function len>)¶: Make a new MainJobSpec object from a sequence or iterable

_replace(_self, **kwds)¶: Return a new MainJobSpec object replacing specified fields with new values

cmdLine¶: Alias for field number 10

conditionalLevel¶: Alias for field number 6

diagnostic¶: Alias for field number 3

dumpList¶: Alias for field number 7

gccExtensions¶: Alias for field number 11

helpMap¶: Alias for field number 8

incHandler¶: Alias for field number 0

includeDOT¶: Alias for field number 9

keepGoing¶: Alias for field number 5

pragmaHandler¶: Alias for field number 4

preDefMacros¶: Alias for field number 1

preIncFiles¶: Alias for field number 2

class cpip.CPIPMain.PpProcessResult(ituPath, indexPath, tuIndexFileName, total_files, total_lines, total_bytes)¶

__getnewargs__()¶: Return self as a plain tuple. Used by copy and pickle.

static __new__(_cls, ituPath, indexPath, tuIndexFileName, total_files, total_lines, total_bytes)¶: Create new instance of PpProcessResult(ituPath, indexPath, tuIndexFileName, total_files, total_lines, total_bytes)

__repr__()¶: Return a nicely formatted representation string

_asdict()¶: Return a new OrderedDict which maps field names to their values.

classmethod _make(iterable, new=<built-in method __new__ of type object at 0xa385c0>, len=<built-in function len>)¶: Make a new PpProcessResult object from a sequence or iterable

_replace(_self, **kwds)¶: Return a new PpProcessResult object replacing specified fields with new values

indexPath¶: Alias for field number 1

ituPath¶: Alias for field number 0

total_bytes¶: Alias for field number 5

total_files¶: Alias for field number 3

total_lines¶: Alias for field number 4

tuIndexFileName¶: Alias for field number 2

cpip.CPIPMain._removeCommonPrefixFromResults(titlePathTupleS)¶: Given a list of: PpProcessResult(ituPath, indexPath, tuIndexFileName(ituPath), total_files, total_lines, total_bytes) This prunes the commmon prefix from the ituPath.

cpip.CPIPMain._tdCallback(theS, attrs, _k, href_nav_text_file_data)¶

Callback function for the file count table.

Parameters:	theS (`cpip.util.XmlWrite.XhtmlStream`) – HTML stream. attrs (`dict({str : [str]})`) – Attributes. _k (`list([str])`) – <insert documentation for argument> href_nav_text_file_data (`tuple([str, str, tuple([int, int, int])])`) – <insert documentation for argument>
Returns:	`NoneType`

cpip.CPIPMain._trThCallback(theS, theDepth)¶

Create the table header:

<tr>
  <th class="filetable" colspan="9">File Path&nbsp;</th>
  <th class="filetable">Include Count</th>
  <th class="filetable">Lines</th>
  <th class="filetable">Bytes</th>
  <th class="filetable">Total Lines</th>
  <th class="filetable">Total Bytes</th>
</tr>

Parameters:	theS (`cpip.util.XmlWrite.XhtmlStream`) – HTML stream. theDepth (`int`) – <insert documentation for argument>
Returns:	`NoneType`

cpip.CPIPMain._writeCommandLineInvocationToHTML(theS, theJobSpec)¶

Writes the command line to the index page.

Parameters:

theS (cpip.util.XmlWrite.XhtmlStream) – HTML stream to write to.
theJobSpec (__main__.MainJobSpec([cpip.core.IncludeHandler.CppIncludeStdOs, dict({str : [str]}), list([_io.StringIO]), NoneType, <class 'NoneType'>, bool, int, list([]), dict({str : [tuple([<class 'bool'>, str]), tuple([<class 'list'>, str]), tuple([int, str]), tuple([list([]), str]), tuple([list([str]), str]), tuple([str, str])]}), <class 'bool'>, str, <class 'bool'>])) – Job specification.

Returns:

NoneType

cpip.CPIPMain._writeDirectoryIndexHTML(theInDir, theOutDir, titlePathTupleS, theJobSpec, time_start)¶: Writes a super index.html when a directory has been processed. titlePathTuples is a list of: PpProcessResult(ituPath, indexPath, tuIndexFileName, total_files, total_lines, total_bytes)

cpip.CPIPMain._writeIndexHtmlTrailer(theS, time_start)¶

Write a trailer to the index.html page with the start/finish time and version. If time_start is None then only the current time is written.

Parameters:	theS (`cpip.util.XmlWrite.XhtmlStream`) – HTML stream. time_start (`NoneType, float`) – Start time.
Returns:	`NoneType`

cpip.CPIPMain._writeParagraphWithBreaks(theS, theParas)¶

Writes the paragraphs with page breaks.

Parameters:	theS (`cpip.util.XmlWrite.XhtmlStream`) – HTML stream. theParas (`list([str])`) – Paragraphs.
Returns:	`NoneType`

cpip.CPIPMain.includeGraphFileNameCcg(theItu)¶

Returns the path to the CCG output file.

Parameters:	theItu (`str`) – Path to the ITU.
Returns:	`str` – path.

cpip.CPIPMain.includeGraphFileNameDotSVG(theItu)¶

Returns the path to the SVG DOT output file.

Parameters:	theItu (`str`) – Path to the ITU.
Returns:	`str` – path.

cpip.CPIPMain.includeGraphFileNameDotTxt(theItu)¶

Returns the path to the DOT output file.

Parameters:	theItu (`str`) – Path to the ITU.
Returns:	`str` – path.

cpip.CPIPMain.includeGraphFileNameSVG(theItu)¶

Returns the path to the SVG output file.

Parameters:	theItu (`str`) – Path to the ITU.
Returns:	`str` – path.

cpip.CPIPMain.includeGraphFileNameText(theItu)¶

Returns the path to the include graph output file.

Parameters:	theItu (`str`) – Path to the ITU.
Returns:	`str` – path.

cpip.CPIPMain.main()¶

Processes command line to preprocess a file or a directory.

Returns:	`int` – status, 0 is success.

cpip.CPIPMain.preProcessFilesMP(dIn, dOut, jobSpec, glob, recursive, jobs)¶: Multiprocessing code to preprocess directories. Returns a count of ITUs processed.

cpip.CPIPMain.preprocessDirToOutput(inDir, outDir, jobSpec, globMatch, recursive, numJobs)¶: Pre-process all the files in a directory. Returns a count of the TUs. This uses multiprocessing where possible. Any Exception (such as a KeyboardInterupt) will terminate this function but write out an index of what has been achieved so far.

cpip.CPIPMain.preprocessFileToOutput(ituPath, outDir, jobSpec)¶

Preprocess a single file. May raise ExceptionCpip (or worse!). Returns a: PpProcessResult(ituPath, indexPath, tuIndexFileName(ituPath) total_files, total_lines, total_bytes)

Parameters:

ituPath (str) – Path to the initial translation unit (ITU).
outDir (str) – Output directory.
jobSpec (__main__.MainJobSpec([cpip.core.IncludeHandler.CppIncludeStdOs, dict({}), list([_io.StringIO]), NoneType, <class 'NoneType'>, bool, int, list([]), dict({str : [tuple([<class 'bool'>, str]), tuple([<class 'list'>, str]), tuple([int, str]), tuple([list([]), str]), tuple([list([str]), str]), tuple([str, str])]}), <class 'bool'>, str, <class 'bool'>])) – Job specification.

Returns:

__main__.PpProcessResult([str, str, str, int, int, int]) – Details of the result.

cpip.CPIPMain.preprocessFileToOutputNoExcept(ituPath, *args, **kwargs)¶: Preprocess a single file and catch all ExceptionCpip exceptions and log them.

cpip.CPIPMain.retFileCountMap(theLexer)¶

Visits the Lexers file include graph and returns a dict of:

{file_name : (inclusion_count, line_count, bytes_count).

The line_count, bytes_count are obtained by (re)reading the file.

Parameters:	theLexer (`cpip.core.PpLexer.PpLexer`) – The Lexer
Returns:	`dict({str : tuple([int, int, int])})` – The file count map.

cpip.CPIPMain.retOptionMap(theOptParser, theOpts)¶

Returns map of {opt_name : (value, help), ...} from the current options.

Parameters:	theOptParser (`argparse.ArgumentParser`) – The option parser. theOpts (`argparse.Namespace`) – Parsed options.
Returns:	`dict({str : [tuple([<class 'bool'>, str]), tuple([bool, str]), tuple([int, str]), tuple([list([]), str]), tuple([list([str]), str]), tuple([str, str])]})` – Option name to value and help text.
Raises:	`KeyError`

cpip.CPIPMain.tuFileName(theTu)¶

Returns the path to the translation unit output file.

Parameters:	theItu (`str`) – Path to the ITU.
Returns:	`str` – path.

cpip.CPIPMain.tuIndexFileName(theTu)¶

Returns the path to the index output file.

Parameters:	theItu (`str`) – Path to the ITU.
Returns:	`str` – path.

cpip.CPIPMain.writeIncludeGraphAsText(theOutDir, theItu, theLexer)¶

Writes out the include graph as plain text.

Parameters:	theOutDir (`str`) – Output directory. theItu (`str`) – Path to ITU. theLexer (`cpip.core.PpLexer.PpLexer`) – The lexer.
Returns:	`NoneType`

cpip.CPIPMain.writeIndexHtml(theItuS, theOutDir, theJobSpec, time_start, total_files, total_lines, total_bytes)¶

Writes the top level index.html page for a pre-processed file.

Parameters:

theItuS (list([str])) – The list of translation units processed.
theOutDir (str) – The output directory.
theJobSpec (tuple([cpip.core.IncludeHandler.CppIncludeStdOs, dict({str : [str]}), list([_io.StringIO]), NoneType, <class 'NoneType'>, bool, int, list([]), dict({str : [tuple([<class 'bool'>, str]), tuple([<class 'list'>, str]), tuple([int, str]), tuple([list([]), str]), tuple([list([str]), str]), tuple([str, str])]}), <class 'bool'>, str, <class 'bool'>])) – The job specification.
time_start (float) – Time that the process started.
total_files (int) – Total number of files processed.
total_lines (int) – Total number of lines processed.
total_bytes (int) – Total number of bytes processed.

Returns:

str – The path to the index.html file that has been written.

cpip.CPIPMain.writeTuIndexHtml(theOutDir, theTuPath, theLexer, theFileCountMap, theTokenCntr, hasIncDot, macroHistoryIndexName)¶

Write the index.html for a single TU.

Parameters:	theOutDir (`str`) – The output directory to write to. theTuPath (`str`) – The path to the original ITU. theLexer (`cpip.core.PpLexer.PpLexer`) – The pre-processing Lexer that has pre-processed the ITU/TU. theFileCountMap (`dict({str : [tuple([<class 'int'>, int, int]), tuple([int, int, <class 'int'>])]})`) – dict of `{file_path : data, ...}` where data is things like inclusion count, lines, bytes and so on. theTokenCntr (`cpip.core.PpTokenCount.PpTokenCount`) – `cpip.core.PpTokenCount.PpTokenCount` containing the token counts. hasIncDot (`bool`) – bool to emit graphviz .dot files. macroHistoryIndexName (`str`) – String of the filename of the macro history.
Returns:	`tuple([int, int, int])` – (total_files, total_lines, total_bytes) as integers.
Raises:	`StopIteration`