Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
204 changes: 204 additions & 0 deletions .github/workflows/qch.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
# Builds the Qt Help (.qch) archive, and publishes it when a version is tagged.
#
# The raw copy of the website is not in this repository and is far too large to
# be. Rather than crawling en.cppreference.com on every run -- which the README
# asks us not to do -- the site copy is taken out of the source archive of an
# earlier release: `make release` packs `reference/` into
# `cppreference-doc-<version>.tar.xz`, which is 13 MB and downloads in about a
# minute. Point `site_release` at the tag whose content you want, or refresh
# the content for a real release by running `make source` locally and publishing
# a new source archive.

name: Qt Help archive

on:
push:
branches: ['**']
tags: ['v*']
workflow_dispatch:
inputs:
site_release:
description: 'Release to take the reference/ site copy from (blank = latest)'
type: string
default: ''
site_repository:
description: 'Repository to take it from (blank = this one, then the ones it was forked from)'
type: string
default: ''

# A run that has been superseded is of no interest; a tag builds on its own.
concurrency:
group: qch-${{ github.ref }}
cancel-in-progress: ${{ github.ref_type != 'tag' }}

jobs:
build:
runs-on: ubuntu-latest
timeout-minutes: 60

permissions:
contents: write # the publishing step attaches the archives

steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: '3.12'
cache: pip
cache-dependency-path: |
requirements.txt
requirements-ci.txt

- name: Install the Python dependencies
run: pip install -r requirements.txt -r requirements-ci.txt

- name: Install qhelpgenerator
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends qt6-documentation-tools
# Qt keeps it out of the way under /usr/lib/qt6/libexec, and the
# package is free to move it, so ask dpkg where it landed.
qhg=$(dpkg -L qt6-documentation-tools | grep -E '/qhelpgenerator$' | head -1)
test -x "$qhg" || { echo "::error::qhelpgenerator not found"; exit 1; }
echo "QHELPGENERATOR=$qhg" >> "$GITHUB_ENV"

- name: Check the style
# pylint is left out on purpose: `config/pylintrc` predates pylint 3 and
# is rejected across the whole repository, so `test.sh` does not run
# clean here. Fixing that is its own change.
run: >
flake8 --config=config/flake8rc
*.py commands index_transform
gadgets/standard_revisions_tests/*.py tests

- name: Run the tests
run: python -m unittest discover

- name: Work out the version
run: |
version=$(sed -n 's/^VERSION=//p' Makefile)
test -n "$version" || { echo "::error::no VERSION in the Makefile"; exit 1; }
echo "VERSION=$version" >> "$GITHUB_ENV"
echo "Building version $version"

# A tag that disagrees with the Makefile would publish archives whose
# names do not match the release they hang under.
if [ "$GITHUB_REF_TYPE" = tag ] && [ "$GITHUB_REF_NAME" != "v$version" ]; then
echo "::error::tag $GITHUB_REF_NAME does not match VERSION=$version in the Makefile"
exit 1
fi

- name: Fetch the copy of the website
env:
GH_TOKEN: ${{ github.token }}
SITE_TAG: ${{ inputs.site_release }}
SITE_REPO: ${{ inputs.site_repository }}
run: |
# Where to look, in order: what the run asked for, this repository,
# and the ones it was forked from. A fork has no releases of its
# own until it makes one, and the archive is the same either way.
candidates=()
if [ -n "$SITE_REPO" ]; then
candidates+=("$SITE_REPO")
fi
candidates+=("$GITHUB_REPOSITORY")
while read -r parent; do
if [ -n "$parent" ]; then
candidates+=("$parent")
fi
done < <(gh api "repos/$GITHUB_REPOSITORY" \
-q '.parent.full_name // empty, .source.full_name // empty' \
2>/dev/null)

for repo in "${candidates[@]}"; do
echo "Looking for the site copy in $repo"
# No tag asks gh for the latest release.
if gh release download ${SITE_TAG:+"$SITE_TAG"} \
--repo "$repo" \
--pattern 'cppreference-doc-*.tar.xz' \
--clobber; then
site_repo="$repo"
break
fi
done

if [ -z "${site_repo:-}" ]; then
echo "::error::no cppreference-doc-*.tar.xz release asset in ${candidates[*]}"
exit 1
fi
echo "Site copy from $site_repo"

tar xJf cppreference-doc-*.tar.xz
rm -rf reference
mv cppreference-doc-*/reference reference
rm -rf cppreference-doc-*/ cppreference-doc-*.tar.xz

echo "Pages in the site copy: $(find reference -name '*.html' | wc -l)"

- name: Build the archive
run: make doc_qch qhelpgenerator="$QHELPGENERATOR"

- name: Check the archive
# The .qch is a SQLite database, so it can say for itself whether the
# pages, the images and both stylesheets arrived in it.
run: |
python - <<'EOF'
import sqlite3
import sys

db = sqlite3.connect('output/cppreference-doc-en-cpp.qch')
count, = db.execute('select count(*) from FileNameTable').fetchone()
sheets = sorted(name for (name,) in db.execute(
"select Name from FileNameTable where Name like 'common/offline%'"))
images, = db.execute(
"select count(*) from FileNameTable where Name like '%.svg'"
" or Name like '%.png'").fetchone()

print('files: %d, images: %d' % (count, images))
print('stylesheets: %s' % ', '.join(sheets))

if count < 6000:
sys.exit('only %d files in the archive' % count)
if len(sheets) != 3:
sys.exit('expected three stylesheets, found %d' % len(sheets))
if images < 50:
sys.exit('only %d images in the archive' % images)
EOF

# Handed to this, GitHub zips whatever it is given and serves the result
# under the artifact's name, so it gets the .qch itself rather than the
# release archives: those would come down as a zip holding an archive
# holding the one file anybody wanted, compressed twice over.
- uses: actions/upload-artifact@v4
with:
name: cppreference-doc-en-cpp-${{ env.VERSION }}
path: output/cppreference-doc-en-cpp.qch
if-no-files-found: error

- name: Package it as the release archives
# Release assets are served as they are, so these are the real thing:
# the same two names and the same format `make release` writes.
if: github.ref_type == 'tag'
run: |
mkdir -p release
pushd output > /dev/null
tar cJf "../release/qch-book-$VERSION.tar.xz" "cppreference-doc-en-cpp.qch"
zip -qr "../release/qch-book-$VERSION.zip" "cppreference-doc-en-cpp.qch"
popd > /dev/null
ls -lh release

- name: Publish the release
if: github.ref_type == 'tag'
env:
GH_TOKEN: ${{ github.token }}
run: |
if gh release view "$GITHUB_REF_NAME" --repo "$GITHUB_REPOSITORY" > /dev/null 2>&1; then
# a release made by hand, or an earlier run of this workflow
gh release upload "$GITHUB_REF_NAME" --repo "$GITHUB_REPOSITORY" \
--clobber release/*
else
gh release create "$GITHUB_REF_NAME" --repo "$GITHUB_REPOSITORY" \
--title "$GITHUB_REF_NAME" --generate-notes \
release/*
fi
35 changes: 24 additions & 11 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ DISTFILES= \
devhelp2qch.py \
export.py \
fix_devhelp-links.py \
fixup_source.py \
index2autolinker.py \
index2browser.py \
index2ddg.py \
Expand All @@ -65,6 +66,8 @@ DISTFILES= \
preprocess.py \
preprocess-css.css \
preprocess_qch.py \
qch-dark.css \
qch-viewer.css \
test.sh \
xml_utils.py \
requirements.txt \
Expand Down Expand Up @@ -179,19 +182,19 @@ output/cppreference-doc-en-cpp.devhelp2: output/reference output/link-map.xml
#build the .qch (QT help) file
output/cppreference-doc-en-cpp.qch: output/qch-help-project-cpp.xml
#qhelpgenerator only works if the project file is in the same directory as the documentation
cp "output/qch-help-project-cpp.xml" "output/reference_cssless/qch.qhp"
cp "output/qch-help-project-cpp.xml" "output/reference_qch/qch.qhp"

pushd "output/reference_cssless" > /dev/null; \
pushd "output/reference_qch" > /dev/null; \
$(qhelpgenerator) "qch.qhp" -o "../cppreference-doc-en-cpp.qch"; \
popd > /dev/null

rm -f "output/reference_cssless/qch.qhp"
rm -f "output/reference_qch/qch.qhp"

output/qch-help-project-cpp.xml: output/cppreference-doc-en-cpp.devhelp2 output/reference_cssless
output/qch-help-project-cpp.xml: output/cppreference-doc-en-cpp.devhelp2 output/reference_qch
#build the file list
echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?><files>" > "output/qch-files.xml"

pushd "output/reference_cssless" > /dev/null; \
pushd "output/reference_qch" > /dev/null; \
find . -type f -not -iname "*.ttf" \
-exec echo "<file>"'{}'"</file>" \; | LC_ALL=C sort >> "../qch-files.xml" ; \
popd > /dev/null
Expand Down Expand Up @@ -221,8 +224,8 @@ output/reference:
mkdir -p output
./preprocess.py --src reference --dst output/reference

output/reference_cssless: output/reference
./preprocess_qch.py --src output/reference --dst output/reference_cssless
output/reference_qch: output/reference
./preprocess_qch.py --src output/reference --dst output/reference_qch

# create indexes for the wiki
indexes:
Expand All @@ -241,16 +244,26 @@ source:
mkdir "reference"

pushd "reference" > /dev/null; \
regex=".*index\\.php.*|.*action=.*|.*printable=.*|.*en.cppreference.com/book.*" \
regex+="|.*/(User|Special|Talk|Help|File|Cppreference|WhatLinksHere|Template|Category|MediaWiki)(_talk)?(:|%253[aA]).*" ; \
regex=".*index\\.php.*|.*rest\\.php.*|.*action=.*|.*printable=.*|.*en.cppreference.com/book.*" \
regex+="|.*/(User|Special|Talk|Help|File|Cppreference|WhatLinksHere|Template|Category|MediaWiki)(_talk)?(:|%253[aA]).*" \
regex+="|.*\\.cppreference\\.com/mwiki/.*" ; \
echo $$regex ; \
wget --adjust-extension --page-requisites --convert-links \
--force-directories --recursive --level=15 \
--span-hosts --domains=en.cppreference.com,upload.cppreference.com \
--reject-regex $$regex \
--timeout=5 --tries=50 --no-verbose \
--retry-connrefused --waitretry=10 --read-timeout=20 \
https://en.cppreference.com/w/ ; \
https://en.cppreference.com/ ; \
popd > /dev/null

./export.py --url=https://en.cppreference.com/mwiki reference/cppreference-export-ns0,4,8,10.xml 0 4 8 10
# the site still points at the pre-2026 locations for images and skin
# files; fetch those from where they live now
./fixup_source.py --src reference

# The wiki source of the pages is no longer part of the copy: since the
# 2026 upgrade the site answers 403 to any request that names more than
# one page -- Special:Export and the API alike, which is what export.py
# asks for -- and a copy worth having is not worth 19000 requests.
# Nothing in the build reads it; preprocess.py threw it away.
#./export.py --url=https://en.cppreference.com reference/cppreference-export-ns0,4,8,10.xml 0 4 8 10
11 changes: 9 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,13 @@ documentation formats are currently supported:
`make install` installs the documentation into proper locations.

3) QT Help documentation format (.qch). Can be generated using `make doc_qch`.
`make install` installs the documentation into proper locations.
`make install` installs the documentation into proper locations. The pages
keep the site's own markup and stylesheet, which the help viewers that read
the archive -- Qt Assistant and Qt Creator, both of which render with
litehtml -- lay out much as a browser does. A dark stylesheet is shipped
beside the light one, and either viewer serves it when its own theme is
dark. 'commands/qch_css.py' builds both; 'qch-viewer.css' and 'qch-dark.css'
are the parts written by hand.

Simply running `make all` will generate documentation in all three formats.

Expand All @@ -43,7 +49,8 @@ Dependencies
------------

The package depends on 'wget' (>=1.15), 'python3', 'python3-lxml',
and 'qhelpgenerator' for the generation of the documentation.
'python3-cssutils' and 'qhelpgenerator' for the generation of the
documentation.

See also
--------
Expand Down
5 changes: 4 additions & 1 deletion build_link_map.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,10 @@ def build_link_map(directory):
text = f.read()
f.close()

m = re.search(r'<script>[^<]*mw\.config\.set([^<]*wgPageName[^<]*)</script>', text) # noqa
# MediaWiki used to hand the page configuration to `mw.config.set`
# directly; since 1.43 it assigns it to `RLCONF` for the resource
# loader to pick up
m = re.search(r'<script>[^<]*(?:mw\.config\.set|RLCONF\s*=)([^<]*wgPageName[^<]*)</script>', text) # noqa
if not m:
continue
text = m.group(1)
Expand Down
Loading