Non-critical deficiencies in Tools/c-globals/c-globals.py. #49

(see #33)
(blocked by #47)

In #47 we are addressing critical deficiencies in the check-c-globals.py (and adding c-globals.py). Here we are focusing on nice-to-haves for c-globals.py (at the least identifying them). While they are not the focus of #47, some of them might be partially addressed there.

Here are some non-critical deficiencies:

• only identifies statics for a single set of defines (i.e. for applicable ifdefs)
• isn't great at auto-detecting the correct runtime level of statics (requiring extra info in ignored-globals.txt)
• ignored-globals.txt isn't great
  • could provide more info to the script (e.g. filename, scope, reason)
• imperfect statics detection
  • could identify static const, etc.
  • bpo-36876: Avoid static locals. python/cpython#13372
  • bpo-36876: Use a consistent variable name for kwlist. python/cpython#13531
• doesn't identify thread-safety of variables
• detection relies on having a built binary (hence post-preprocessor)

...
<TBD>
...

(Everything past here is just dumped WIP text...)

Directory structure:

Tools/c-analyzer/
    c_statics/
        README (including "Resolving Failures" section)
        __main__.py
        find.py
        show.py
        known.py
        supported.py
        ignored.tsv (temporary)
    c_parser/
        files.py
        info.py
        statics.py
        declarations.py
        (pycparser / ply)  <--------------- new
    c_symbols/  <--------------- new
        symbols.py (wrapper around "nm" command, etc.)
    c-statics.py
    check-c-statics.py
Lib/test/
    test_tools/test_tool_c_analyzer/
        testdata/...
        test_c_statics/...
        test_c_parser/...
        test_c_symbols/...  <--------------- new
    test_check_c_statics.py

TODO:

Formatting:

• --formatter FORMATTER,... CLI arg

• "single-line" formatter

• "basic" formatter

• "table" formatter (columns, sort, group-by)

• "json" formatter

• "tsv" formatter

• show.py: add show_table(rows, columns)

• __main__.py: add "formatter" arg to main() and command funcs

• __main__.py: add "--formatter" CLI arg to "check" & "show" command

• info.py: add StaticVar.level and StaticVar.reason

• known.py: add "resolveVar(StaticVar)"

• add more functionality (e.g. better info) to c-globals.py

  + identify static vars, including their scope, type, and runtime level
  + identify macros that produce static vars
  + identify types, including their runtime level
  + 
  

  • _cg/__main__: show-known (table of statics, table of macros, table of types)
  • _cg/info.py: StaticVar, Macro, Struct
  • _cg/__main__: show-found (table of statics, table of macros, table of types)
  • _cg/__main__: structs (show table)
  • parsing: find statics (w/scope), macros, structs
  • identify per-interpreter struct defs (e.g. PyObject), macros, and statics

Here are things we can do later:

• add StaticVar.macro
• _cg/info.py: add Macro, Struct

• _cg/__main__: show-known (table of statics, table of macros, table of types)

  • _cg/__main__: show-found (table of statics, table of macros, table of types)

  • _cg/__main__: structs (show table)

  • parsing: find statics (w/scope), macros, structs

  • identify per-interpreter struct defs (e.g. PyObject), macros, and statics

• CLI: [-v|--verbose] [-q|--quiet]

• CLI: show [--all] [--filter FILTER] [--columns COLUMNS] [--sort COL[,COL,...]] [--group-by COL[,COL,...]]

• track ifdefs

• unfold where ifdefs split a block (e.g. func, struct)

• stop using preprocessor (or compare with results with preprocessor)

• option to skip self check?

bpo-36876: "Global C variables are a problem."

1. update check-c-globals.py to identify static locals (and thread-locals)
2. deal with any identified globals
   * move them to _PyRuntimeState (or thread-locals to PyThreadState, etc.)
   * ignore them by adding them to Tools/c-globals/ignored-globals.txt
3. add check-c-globals.py to "make check"
4. (if "make check" isn't already there), ensure check-c-globals.py is run at
    some point in CI
...
Also, Tools/c-globals/ignored-globals.txt is a bit out of date (some vars have
been removed, renamed, or moved to another file).  That should get cleaned
up.  It might also make sense to update check-c-globals.py to verify that all
variables in ignored-globals.txt actually exist.