> From 8cc66fa0c3f4d7b33b42ba9ff44c9cb634c30c61 Mon Sep 17 00:00:00 2001 > From: Jiri Pirko > Date: Thu, 15 Oct 2026 16:50:34 +0200 > Subject: [PATCH 01/20] ctlv: Add the schema language and generator > > Define the YAML schema and append-only family registry. Add ctlvgen to > validate specifications and emit kernel and UAPI outputs. The commit message undersells and misdescribes the patch. Besides "validate specifications and emit kernel and UAPI outputs" it adds an ABI snapshot compatibility checker (`compat.py`), a registry baseline, whole-tree coverage checks (`tree.py`), an output-manifest reconciler (`outputs.py`), a Makefile, a regeneration script and what looks like a userspace-binding name table. None of the design (why a new schema language, the inheritance model, the ABI rules the checker enforces) is explained, and there is no Documentation/ .rst at all: the YAML meta-schema is the only documentation. Bisectability: this patch adds a Makefile and `ctlv-regen.sh` that operate on `Documentation/ctlv/specs/*.yaml`, `Documentation/ctlv/families.yaml`, `Documentation/ctlv/abi/` and `Documentation/ctlv/generated-files.yaml`. None of those exist until a later patch (git ls-files at HEAD shows them added by the root-family patch). With only this patch applied "make -C `tools/ctlv`" silently does nothing (empty $(wildcard)) and `ctlv-regen.sh` fails on a missing `ctlv.yaml`. The tool cannot be exercised, let alone tested, at this commit. Nothing in this series adds a test: `outputs.py` refers to `tools/ctlv/tests/specs` and `tests/generated-files.yaml`, and the docstrings in `errors.py`, `sizes.py` and `cgen.py` say "a test checks that" / "the test suite matches diagnostics", but `tools/ctlv/tests` does not exist at HEAD. Either add the tests or drop the references (see inline comments). > > Assisted-by: Cursor: GPT-5.6 Sol > Assisted-by: Cursor: Claude Opus 5.5 > Signed-off-by: Jiri Pirko The author/SoB address is jiri@nvidia.com while MAINTAINERS (below) and pyproject.toml use jiri@resnulli.us. Please make these consistent. > --- > Documentation/ctlv/ctlv-registry-schema.yaml | 38 + > Documentation/ctlv/ctlv-schema.yaml | 359 +++++++ > .../ctlv/generated-files-schema.yaml | 64 ++ > MAINTAINERS | 7 + > tools/ctlv/Makefile | 59 ++ > tools/ctlv/ctlv-regen.sh | 55 ++ > tools/ctlv/ctlvgen/__init__.py | 1 + > tools/ctlv/ctlvgen/cli.py | 172 ++++ > tools/ctlv/ctlvgen/lib/__init__.py | 1 + > tools/ctlv/ctlvgen/lib/cgen.py | 825 ++++++++++++++++ > tools/ctlv/ctlvgen/lib/compat.py | 821 ++++++++++++++++ > tools/ctlv/ctlvgen/lib/errors.py | 14 + > tools/ctlv/ctlvgen/lib/gen_family.py | 347 +++++++ > tools/ctlv/ctlvgen/lib/gen_kernel.py | 517 ++++++++++ > tools/ctlv/ctlvgen/lib/gen_uapi.py | 223 +++++ > tools/ctlv/ctlvgen/lib/outputs.py | 329 +++++++ > tools/ctlv/ctlvgen/lib/semantics.py | 893 ++++++++++++++++++ > tools/ctlv/ctlvgen/lib/sizes.py | 238 +++++ > tools/ctlv/ctlvgen/lib/spec.py | 583 ++++++++++++ > tools/ctlv/ctlvgen/lib/tree.py | 71 ++ > tools/ctlv/ctlvgen/lib/yamlload.py | 83 ++ > tools/ctlv/pyproject.toml | 22 + > tools/ctlv/requirements.txt | 2 + > 23 files changed, 5724 insertions(+) > create mode 100644 Documentation/ctlv/ctlv-registry-schema.yaml > create mode 100644 Documentation/ctlv/ctlv-schema.yaml > create mode 100644 Documentation/ctlv/generated-files-schema.yaml > create mode 100644 tools/ctlv/Makefile > create mode 100755 tools/ctlv/ctlv-regen.sh > create mode 100644 tools/ctlv/ctlvgen/__init__.py > create mode 100755 tools/ctlv/ctlvgen/cli.py > create mode 100644 tools/ctlv/ctlvgen/lib/__init__.py > create mode 100644 tools/ctlv/ctlvgen/lib/cgen.py > create mode 100644 tools/ctlv/ctlvgen/lib/compat.py > create mode 100644 tools/ctlv/ctlvgen/lib/errors.py > create mode 100644 tools/ctlv/ctlvgen/lib/gen_family.py > create mode 100644 tools/ctlv/ctlvgen/lib/gen_kernel.py > create mode 100644 tools/ctlv/ctlvgen/lib/gen_uapi.py > create mode 100644 tools/ctlv/ctlvgen/lib/outputs.py > create mode 100644 tools/ctlv/ctlvgen/lib/semantics.py > create mode 100644 tools/ctlv/ctlvgen/lib/sizes.py > create mode 100644 tools/ctlv/ctlvgen/lib/spec.py > create mode 100644 tools/ctlv/ctlvgen/lib/tree.py > create mode 100644 tools/ctlv/ctlvgen/lib/yamlload.py > create mode 100644 tools/ctlv/pyproject.toml > create mode 100644 tools/ctlv/requirements.txt > > diff --git a/Documentation/ctlv/ctlv-registry-schema.yaml b/Documentation/ctlv/ctlv-registry-schema.yaml > new file mode 100644 > index 000000000000..d9a2840fc230 > --- /dev/null > +++ b/Documentation/ctlv/ctlv-registry-schema.yaml > @@ -0,0 +1,38 @@ > +# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) > +%YAML 1.2 > +--- > +$id: http://kernel.org/schemas/ctlv/ctlv-registry-schema.yaml# > +$schema: https://json-schema.org/draft-07/schema > + > +$defs: > + name: > + type: string > + maxLength: 63 > + pattern: ^[a-z0-9]+(-[a-z0-9]+)*$ > + > +title: CTLV family registry > +description: Append-only list of CTLV family names, indexed by family ID > +type: object > +required: [ families ] > +additionalProperties: false > +properties: > + families: > + description: | > + Family names in family ID order: the entry at index N has family ID N. > + The root family owns ID 0 and so is the first entry, and cannot be > + deprecated. Every other entry comes after the entry of the family it > + inherits. A deprecated entry keeps its ID reserved. > + type: array > + minItems: 1 > + maxItems: 65536 > + items: > + oneOf: > + - $ref: '#/$defs/name' > + - type: object > + required: [ name, deprecated ] > + additionalProperties: false > + properties: > + name: > + $ref: '#/$defs/name' > + deprecated: > + const: true > diff --git a/Documentation/ctlv/ctlv-schema.yaml b/Documentation/ctlv/ctlv-schema.yaml > new file mode 100644 > index 000000000000..38fd56e6627c > --- /dev/null > +++ b/Documentation/ctlv/ctlv-schema.yaml > @@ -0,0 +1,359 @@ > +# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) > +%YAML 1.2 > +--- > +$id: http://kernel.org/schemas/ctlv/ctlv-schema.yaml# > +$schema: https://json-schema.org/draft-07/schema > + > +# The syntax gate for a CTLV family specification: permitted keys, their shapes > +# and the closed sets of keywords. Anything needing more than one declaration at > +# a time is checked by tools/ctlv/ctlvgen/lib/semantics.py. > + > +$defs: > + name: > + description: Kebab-case identifier. A dot is reserved for selectors. > + type: string > + pattern: ^[a-z0-9]+(-[a-z0-9]+)*$ > + > + family-name: > + description: Name carried by family discovery. > + allOf: > + - $ref: '#/$defs/name' > + - maxLength: 63 > + > + selector: > + description: | > + A named attr set, a whole attr group, or one exact attr as > + "group.attr". There are no tuple selectors and no cross-family > + qualification. > + type: string > + pattern: ^[a-z0-9]+(-[a-z0-9]+)*(\.[a-z0-9]+(-[a-z0-9]+)*)?$ > + > + selectors: > + oneOf: > + - $ref: '#/$defs/selector' > + - type: array > + minItems: 1 > + items: > + $ref: '#/$defs/selector' > + > + uint: > + type: integer > + minimum: 0 > + maximum: 4294967295 > + > + integer-bound: > + type: integer > + minimum: -9223372036854775808 > + maximum: 18446744073709551615 > + > + max-occurrences: > + description: | > + A finite positive count, or unlimited, which is reachable only from > + query replies. > + oneOf: > + - type: integer > + minimum: 1 > + maximum: 4294967295 > + - const: unlimited > + > + unit: > + description: CTLV-owned unit, matching enum ctlv_unit. > + enum: [ seconds, hertz, bytes, celsius ] > + > + value-list: > + description: An enum or a flags definition. > + type: object > + required: [ name, values ] > + additionalProperties: false > + properties: > + name: > + $ref: '#/$defs/name' > + values: > + description: | > + Value names in declaration order. Enum values are numbered from 1 > + because 0 is reserved; flag bits are numbered from bit 0. Both are > + append-only. > + type: array > + minItems: 1 > + items: > + $ref: '#/$defs/name' > + > + enum-list: > + allOf: > + - $ref: '#/$defs/value-list' > + - properties: > + values: > + maxItems: 4294967295 > + > + flags-list: > + allOf: > + - $ref: '#/$defs/value-list' > + - properties: > + values: > + maxItems: 64 > + > + blob: > + type: object > + required: [ direction, allowed-backings, max-length ] > + additionalProperties: false > + properties: > + direction: > + description: How the family may use the described memory. > + enum: [ input, output, inout ] > + allowed-backings: > + type: array > + minItems: 1 > + uniqueItems: true > + items: > + enum: [ va ] > + max-length: > + description: | > + Authoritative maximum external payload length for this placement, > + independent of the inline message limit. > + type: integer > + minimum: 1 > + maximum: 18446744073709551615 > + alignment: > + description: Required alignment of the described address or offset. > + type: integer > + minimum: 1 > + maximum: 4294967295 > + > + attr: > + description: | > + One attr declaration. Exactly one primary form out of type, enum, > + flags, attrs and blob must be present. optional-attrs and recursive > + may only accompany attrs. > + type: object > + required: [ name ] > + additionalProperties: false > + properties: > + name: > + $ref: '#/$defs/name' > + type: > + description: | > + Primitive payload. Integers are always 64-bit, and a bool is one > + byte holding zero or one. > + enum: [ uint, sint, bool, string, binary ] > + enum: > + description: Name of an enum definition; implies an unsigned payload. > + $ref: '#/$defs/name' > + flags: > + description: Name of a flags definition; implies an unsigned payload. > + $ref: '#/$defs/name' > + attrs: > + description: Nested payload, given as selectors. > + $ref: '#/$defs/selectors' > + optional-attrs: > + description: Further nested payload whose placements are optional. > + $ref: '#/$defs/selectors' > + recursive: > + description: | > + The nested payload is one attr set that may contain this attr > + again, describing a structure of any depth. Only a query reply can > + hold one, since such a nest has no maximum length. > + const: true > + blob: > + $ref: '#/$defs/blob' > + min: > + description: Inclusive minimum of an integer payload. > + $ref: '#/$defs/integer-bound' > + max: > + description: Inclusive maximum of an integer payload. > + $ref: '#/$defs/integer-bound' > + length: > + description: Exact payload length of a binary attr. > + $ref: '#/$defs/uint' > + min-length: > + description: | > + Minimum payload length, excluding the terminator of a string. > + $ref: '#/$defs/uint' > + max-length: > + description: | > + Maximum payload length, excluding the terminator of a string. > + $ref: '#/$defs/uint' > + unit: > + $ref: '#/$defs/unit' > + scale: > + description: Decimal exponent applied to the payload. > + type: integer > + minimum: -128 > + maximum: 127 > + max-occurrences: > + $ref: '#/$defs/max-occurrences' > + > + attr-group: > + description: | > + A namespace of attr declarations. The group name is also the generated > + symbol prefix of its attrs. > + type: object > + required: [ name, attrs ] > + additionalProperties: false > + properties: > + name: > + $ref: '#/$defs/name' > + attrs: > + type: array > + minItems: 1 > + maxItems: 4294967295 > + items: > + $ref: '#/$defs/attr' > + > + attr-set: > + description: | > + A compositional schema unit. Either a stub, which may be filled once > + by one descendant family, or a set of members. After fills are composed, > + every attr ID at one nesting level must still have one placement. > + type: object > + required: [ name ] > + additionalProperties: false > + properties: > + name: > + $ref: '#/$defs/name' > + stub: > + const: true > + attrs: > + $ref: '#/$defs/selectors' > + optional-attrs: > + $ref: '#/$defs/selectors' > + > + placements: > + description: The required and optional attr placements at one level. > + type: object > + additionalProperties: false > + minProperties: 1 > + properties: > + attrs: > + $ref: '#/$defs/selectors' > + optional-attrs: > + $ref: '#/$defs/selectors' > + > + op: > + type: object > + required: [ name, type ] > + additionalProperties: false > + properties: > + name: > + $ref: '#/$defs/name' > + type: > + description: | > + action is invoked once and never fragments, query may fragment its > + reply, and event is an unsolicited reply-only schema. > + enum: [ action, query, event ] > + fd-access: > + description: | > + Authority needed to execute an action or query. read permits an > + observation fd; read-write, the default, requires a full-control fd. > + Events are always readable and cannot declare this property. > + enum: [ read, read-write ] > + availability: > + description: | > + Whether a family may enable and disable this op per device. fixed, > + the default, is always enabled. dynamic is action and query only, > + and is a permission rather than a state: every device starts with > + the op enabled. > + enum: [ fixed, dynamic ] > + optional: > + description: | > + Whether family initialization must opt in before publication for > + this op to exist on the device. Action and query only, and false by > + default. An op not opted into is absent from introspection. > + type: boolean > + pre-post: > + description: | > + Whether this op gets a generated pre/post hook pair of its own, > + which its declaring family implements. It runs inside the pair of > + the device family, or the core family for a core op. > + Action and query only, and false by default. Kernel implementation > + metadata: it reaches neither UAPI nor op-schema introspection. > + type: boolean This says pre-post "reaches neither UAPI nor op-schema introspection", but `compat.py` records "pre-post" in the ABI snapshot (`_ops`/`_compact_op`/ `_expand_ops`) and `_compare_ops()` reports a change of it as an ABI incompatibility. Flipping a purely kernel-internal hook flag therefore forces an "ABI" snapshot update and is reported as a break. Either the property is ABI or it is not; the two documents disagree. > + request: > + description: | > + Omitted means a request root with no children. An event has no > + request. > + $ref: '#/$defs/placements' > + reply: > + description: | > + Required on a query. Omitted on an action means ACK-only, while an > + event always has a reply root, so omitting it there means a root > + with no children. > + $ref: '#/$defs/placements' > + suggested-reply-length: > + description: | > + Query only. An omitted value is CTLV_DEFAULT_QUERY_REPLY_LEN raised > + to the computed minimum. A declared value must be between that > + minimum and CTLV_MAX_INLINE_MESSAGE_LEN. > + type: integer > + minimum: 1 > + maximum: 1048576 The upper bound 1048576 hard-codes `CTLV_MAX_INLINE_MESSAGE_LEN`. The same limit is then enforced a second time in `spec.py` (`_u32_value()`) and a third time in `sizes._suggested_len()`. Three copies of one constant (plus `sizes.MAX_INLINE_MESSAGE_LEN`) will drift. > + > +title: CTLV family > +description: Specification of one CTLV family schema > +type: object > +required: [ family ] > +additionalProperties: false > +properties: > + family: > + type: object > + required: [ name ] > + additionalProperties: false > + properties: > + name: > + description: | > + Registered family name. It carries no numeric ID; the ID comes > + from the position in Documentation/ctlv/families.yaml. > + $ref: '#/$defs/family-name' > + inherits: > + description: | > + Name of the single direct parent family, which every family but > + the root one names. The complete parent schema is inherited and > + its definitions keep the parent's IDs. The name above must be > + this name, one dash, and a local suffix. > + $ref: '#/$defs/family-name' > + pre-init: > + description: | > + Whether this family implements the generated pre-init callback > + which translates its init info for its parent. False by default; > + without one, the parent receives NULL. > + type: boolean > + post-init: > + description: | > + Whether this family implements the generated post-init callback > + after every family priv has been initialized. False by default. > + type: boolean > + pre-fini: > + description: | > + Whether this family implements the generated pre-fini callback > + before any family priv is finalized. False by default. > + type: boolean > + enums: > + type: array > + minItems: 1 > + items: > + $ref: '#/$defs/enum-list' > + > + flags: > + type: array > + minItems: 1 > + items: > + $ref: '#/$defs/flags-list' > + > + attr-groups: > + type: array > + minItems: 1 > + maxItems: 65535 > + items: > + $ref: '#/$defs/attr-group' > + > + attr-sets: > + type: array > + minItems: 1 > + items: > + $ref: '#/$defs/attr-set' > + > + ops: > + type: array > + minItems: 1 > + maxItems: 4294967295 > + items: > + $ref: '#/$defs/op' General: many limits are enforced three times - here (minimum/maximum/ maxItems/maxLength), again in `spec.py` (`_max_occurrences`, `_u32_value`, `_integer_bounds` scale range, `_blob` ranges, `_family_name` length, registry size) and again in `semantics.py` (`_check_count` against `U16_MAX`/`U32_MAX`, `MAX_FLAG_BITS`). After jsonschema validation succeeds, e.g. `value < 0` in `_u32_value()`, "value < 1 or value > `U32_MAX`" in `_max_occurrences()` and `_family_name()`'s length test cannot fire. Pick one place per rule. (The per-kind min/max narrowing in `_integer_bounds()` is not redundant: the schema accepts the full -2^63..2^64-1 range for both kinds.) > diff --git a/Documentation/ctlv/generated-files-schema.yaml b/Documentation/ctlv/generated-files-schema.yaml > new file mode 100644 > index 000000000000..d3529ce6e8e7 > --- /dev/null > +++ b/Documentation/ctlv/generated-files-schema.yaml > @@ -0,0 +1,64 @@ > +# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) > +%YAML 1.2 > +--- > +$schema: https://json-schema.org/draft-07/schema > + > +$defs: > + path: > + type: string > + minLength: 1 > + pattern: ^[^/].*$ > + > + outputs: > + type: object > + minProperties: 1 > + additionalProperties: false > + properties: > + uapi-header: > + $ref: '#/$defs/path' > + kernel-header: > + $ref: '#/$defs/path' > + kernel-source: > + $ref: '#/$defs/path' > + > + registry: > + type: object > + required: [ source, outputs ] > + additionalProperties: false > + properties: > + source: > + $ref: '#/$defs/path' > + outputs: > + type: object > + required: [ uapi-header ] > + additionalProperties: false > + properties: > + uapi-header: > + $ref: '#/$defs/path' > + > + family: > + type: object > + required: [ name, source, outputs ] > + additionalProperties: false > + properties: > + name: > + type: string > + source: > + $ref: '#/$defs/path' > + outputs: > + $ref: '#/$defs/outputs' > + > +title: CTLV generated-file manifest > +type: object > +required: [ version, registry, families ] > +additionalProperties: false > +properties: > + version: > + const: 1 > + registry: > + $ref: '#/$defs/registry' > + families: > + type: array > + minItems: 1 > + items: > + $ref: '#/$defs/family' `generated-files-schema.yaml` has no $id, unlike the other two meta-schemas, and its path pattern ^[^/].*$ accepts "../x", ".hidden" and non-normalized paths; `outputs._relative()` re-validates all of that in Python. Either tighten the pattern or drop one of the two checks. > diff --git a/MAINTAINERS b/MAINTAINERS > index 782389bd5cd3..2c758d2607b6 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -7003,6 +7003,13 @@ M: Jaya Kumar > S: Maintained > F: sound/pci/cs5535audio/ > > +CTLV SUBSYSTEM > +M: Jiri Pirko Address differs from the Signed-off-by (jiri@nvidia.com). Also note that at this commit F: covers only `Documentation/ctlv/` and `tools/ctlv/`; the entry grows later in the series, which is fine, but then "S: Maintained" with only L: linux-kernel and no T:/W: is thin for a new UAPI-bearing subsystem. > +L: linux-kernel@vger.kernel.org > +S: Maintained > +F: Documentation/ctlv/ > +F: tools/ctlv/ > + > CTU CAN FD DRIVER > M: Pavel Pisa > M: Ondrej Ille > diff --git a/tools/ctlv/Makefile b/tools/ctlv/Makefile > new file mode 100644 > index 000000000000..bb3129405fb5 > --- /dev/null > +++ b/tools/ctlv/Makefile > @@ -0,0 +1,59 @@ > +# SPDX-License-Identifier: GPL-2.0 > + > +KDIR = ../.. > +DOCDIR = $(KDIR)/Documentation/ctlv > +SPECDIR = $(DOCDIR)/specs > +SPECS = $(wildcard $(SPECDIR)/*.yaml) SPECS expands to nothing at this commit (see commit message comment), so "all" is a silent no-op. Even once specs exist, "all" (validate + abi-check) only visits spec files that exist: it does not detect a spec missing from the registry, a registry entry with no spec file, or a stale generated file. Only "check-regen" (`--outputs-check`, which runs `tree.check()`) does, and it is not part of "all". > +TOOL = ./ctlvgen/cli.py > + > +all: validate abi-check > + > +validate: > + @for spec in $(SPECS) ; do \ > + echo -e "\tVALIDATE $$(basename $$spec)" ; \ "echo -e" is not portable: make runs recipes with /bin/sh, and under dash (Debian/Ubuntu) this prints a literal "-e" and no tab. Use printf, or "echo" with a literal tab, as other tools/ Makefiles do (they use "@echo ' CC $@'" style or `quiet_cmd`). > + $(TOOL) --spec $$spec --validate || exit 1 ; \ > + done > + > +# Fails when a specification no longer matches its checked-in ABI snapshot. > +abi-check: > + @for spec in $(SPECS) ; do \ > + echo -e "\tABICHECK $$(basename $$spec)" ; \ > + $(TOOL) --spec $$spec --abi-check || exit 1 ; \ > + done > + > +# Deliberate: an updated snapshot is how a UAPI change gets reviewed. > +abi-update: > + @for spec in $(SPECS) ; do \ > + $(TOOL) --spec $$spec --abi-update || exit 1 ; \ > + done abi-update only rewrites per-family snapshots. The registry baseline (`Documentation/ctlv/abi/families.registry.yaml`) is rewritten by `cli.py --registry-abi-update`, for which there is no Makefile target and no script, although `compat.REGISTRY_HEADER` claims it is "Updated deliberately by make -C `tools/ctlv` abi-update". Also `--abi-update` does not run `compat.check()` first, so any ABI-breaking change (removed attr, renumbered ID, narrowed bound) is simply blessed by running the target; the HEADER text "is an ABI break, whatever this file says afterwards" is not enforced by anything in-tree. > + > +# Style checks, needing yamllint, pycodestyle and pyflakes, which the tool > +# itself does not use. > +# > +# A flow sequence is written "[ one, two ]" here, as in the tree's other > +# meta-schemas, which stock yamllint rejects; the same relaxation is in > +# Documentation/devicetree/bindings/.yamllint. > +YAMLLINT_CONF = {extends: default, rules: {brackets: {max-spaces-inside: 1}}} > + > +PYSRC = ctlvgen > + > +lint: > + yamllint -d "$(YAMLLINT_CONF)" $(DOCDIR) > + python3 -m pycodestyle $(PYSRC) > + python3 -m pyflakes $(PYSRC) "lint" is not hooked into "all" and no CI/doc mentions it; it needs yamllint, pycodestyle and pyflakes which are not in `requirements.txt`. Fine as a developer aid but then say so where requirements are listed. > + > +regen: > + @./ctlv-regen.sh > + > +# Fails when a checked-in generated file does not match what the generator > +# produces from the current specifications. Writes nothing. > +check-regen: > + @./ctlv-regen.sh -c > + > +clean: > + rm -rf ctlvgen/__pycache__ > + rm -rf ctlvgen/lib/__pycache__ > + rm -rf ctlvgen.egg-info > + rm -rf build Nothing in this Makefile (or the series) creates ctlvgen.egg-info or build/; only an out-of-tree "pip install ." does. Drop or explain. > + > +.PHONY: all validate abi-check abi-update lint regen check-regen clean > diff --git a/tools/ctlv/ctlv-regen.sh b/tools/ctlv/ctlv-regen.sh > new file mode 100755 > index 000000000000..575114ba93e7 > --- /dev/null > +++ b/tools/ctlv/ctlv-regen.sh > @@ -0,0 +1,55 @@ > +#!/bin/bash > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +# > +# Reconcile the checked-in files against the independent output manifests. > +# > +# -f accepted for compatibility; reconciliation always renders everything `-f` mirrors `-f` of `tools/net/ynl/ynl-regen.sh` (where it forces regeneration), but here it is a documented no-op and nothing uses it. Drop it and its case arm, or make it do something. > +# -c do not write anything, fail if any output differs from what the > +# generator produces now > +# -p search the given directory instead of the kernel tree > + > +TOOL=$(dirname "$(realpath "$0")")/ctlvgen/cli.py TOOL is derived from realpath "$0" but KDIR (line 672) from plain dirname "$0" through realpath; both work, but pick one. Also "-p" is documented as "search the given directory instead of the kernel tree", while it is passed as `--output-root`, i.e. it selects where outputs are written/compared, not where anything is searched for. > +check= > +search= > + > +while getopts ':fcp:' option; do > + case "$option" in > + f) > + ;; > + c) > + check=yes > + ;; > + p) > + if [ -z "$OPTARG" ]; then > + echo "Option -p needs a nonempty directory" >&2 > + exit 1 > + fi > + search=$OPTARG > + ;; > + :) > + echo "Option -$OPTARG needs an argument" >&2 > + exit 1 > + ;; > + ?) > + echo "Unrecognized option '-$OPTARG'" >&2 > + exit 1 > + ;; > + esac > +done > +shift $((OPTIND - 1)) > +if [ "$#" -ne 0 ]; then > + echo "Unexpected argument '$1'" >&2 > + exit 1 > +fi > + > +KDIR=$(realpath "$(dirname "$0")/../..") > +args=(--spec "$KDIR/Documentation/ctlv/specs/ctlv.yaml") The entry point is hard-wired to `specs/ctlv.yaml` (the root family) only to locate `Documentation/ctlv`; everything else comes from the manifests. That makes `--spec` a meaningless argument for `--outputs-check`/`--outputs-update`. > +if [ -n "$search" ]; then > + args+=(--output-root "$search") > +fi > +if [ -n "$check" ]; then > + args+=(--outputs-check) > +else > + args+=(--outputs-update) > +fi > +exec "$TOOL" "${args[@]}" > diff --git a/tools/ctlv/ctlvgen/__init__.py b/tools/ctlv/ctlvgen/__init__.py > new file mode 100644 > index 000000000000..f84069cb2c3d > --- /dev/null > +++ b/tools/ctlv/ctlvgen/__init__.py > @@ -0,0 +1 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > diff --git a/tools/ctlv/ctlvgen/cli.py b/tools/ctlv/ctlvgen/cli.py > new file mode 100755 > index 000000000000..c6bfafeb2852 > --- /dev/null > +++ b/tools/ctlv/ctlvgen/cli.py > @@ -0,0 +1,172 @@ > +#!/usr/bin/env python3 > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""Command line front end of the CTLV schema tool.""" > + > +import argparse > +import pathlib > +import sys > + > +# So that this file works both as a module of the ctlvgen package and as the > +# script the Makefile and ctlv-regen.sh run. The imports below have to follow. > +sys.path.append(pathlib.Path(__file__).resolve().parent.as_posix()) Importing PyYAML/jsonschema happens at module import (lib.spec, lib.yamlload, lib.compat); on a machine without them the user gets a raw ImportError traceback from this script and from `ctlv-regen.sh` with no hint to install `requirements.txt`. A one-line friendly check would help. > + > +from lib import cgen # noqa: E402 > +from lib import compat # noqa: E402 > +from lib import gen_kernel # noqa: E402 > +from lib import gen_uapi # noqa: E402 > +from lib import outputs # noqa: E402 > +from lib import semantics # noqa: E402 > +from lib import spec # noqa: E402 > +from lib import tree # noqa: E402 > +from lib.errors import CtlvError # noqa: E402 > + > + > +# What each non-kernel --mode and file kind generates. > +GENERATORS = { > + ('uapi', 'header'): gen_uapi.render, > +} GENERATORS is a one-entry table keyed by (mode, kind) and only ever holds ('uapi', 'header'). Argument validation (lines 777-783) accepts "`--mode` uapi `--source`", which then does GENERATORS[('uapi', 'source')] on line 853 and dies with an uncaught KeyError traceback instead of a parser error. Reject that combination in the parser (or drop the table). > + > + > +def main(): Undocumented and non-trivial: It interleaves argument-combination validation with a long if/elif dispatch in which some actions return early and others fall through to a shared load-the-family step and a single CtlvError handler. Suggested docstring (format used for every suggestion in this review: short summary, blank line, longer description, then Args:/Returns:/Throws: sections, one "NAME: DESCRIPTION" or "TYPE: DESCRIPTION" entry each): """Run the CTLV schema tool and return its process exit status. Parses the command line, rejects invalid flag combinations through parser.error() (which exits with status 2 by itself), builds the spec.Paths for --spec and dispatches on exactly one action. --spec is always required; --schema, --registry and --abi are derived from it when not given. The actions fall in two groups. The whole-tree actions (--tree-check, --tree-check-update, --registry-abi-update, --outputs-check and --outputs-update) work from the family registry and the output manifests and return as soon as they are done, without loading the family named by --spec. --outputs-update runs tree.check_for_update() first and then rewrites the generated files; it prints one "ctlvgen: ..." line per change to standard error. --registry-abi-update requires complete ABI snapshots (tree.finish_update()) before it overwrites the registry ABI snapshot. The remaining actions load the single family with semantics.load() and then validate it (--validate), compare it with its ABI snapshot and the registry baseline (--abi-check), rewrite its ABI snapshot (--abi-update), or generate one uapi or kernel file (--mode with --header or --source). Generated text goes to standard output unless -o/--output is given. Only CtlvError is caught: it is printed as "ctlvgen: " on standard error and turned into status 1. Any other exception propagates as a crash. The spec.Paths constructor runs outside the try block. Returns: int: 0 on success, 1 when a CtlvError was reported. Throws: SystemExit: through argparse, when the arguments are malformed or the combination of options is invalid (status 2). """ > + parser = argparse.ArgumentParser(description='CTLV schema tool') > + parser.add_argument('--spec', type=str, required=True, > + help='path to a CTLV family specification') > + parser.add_argument('--schema', type=str, > + help='path to the meta-schema, derived from --spec ' > + 'when not given') > + parser.add_argument('--registry', type=str, > + help='path to the family registry, derived from ' > + '--spec when not given') > + parser.add_argument('--abi', type=str, > + help='directory of the ABI snapshots, derived from ' > + '--spec when not given') > + action = parser.add_mutually_exclusive_group(required=True) > + action.add_argument('--validate', action='store_true', > + help='validate the specification and exit') > + action.add_argument('--abi-check', action='store_true', > + help='check the specification against its ABI ' > + 'snapshot') > + action.add_argument('--abi-update', action='store_true', > + help='rewrite the ABI snapshot from the ' > + 'specification') > + action.add_argument('--tree-check', action='store_true', > + help='check registry, spec, snapshot and output ' > + 'coverage') > + action.add_argument('--tree-check-update', action='store_true', > + help='check whole-tree inputs before an ABI update') > + action.add_argument('--registry-abi-update', action='store_true', > + help='update the ordered family registry baseline') > + action.add_argument('--outputs-check', action='store_true', > + help='check every file in the output manifests') > + action.add_argument('--outputs-update', action='store_true', > + help='regenerate every file in the output manifests') `--tree-check` and `--tree-check-update` duplicate what `--outputs-check` and `--outputs-update` already do (`tree.check()`/`check_for_update()`). The manual `--mode/--header/--source/-o/--cmp-out/--schema-only` generation path has no caller in the Makefile or `ctlv-regen.sh`, which only use `--validate`, `--abi-check`, `--abi-update`, `--outputs-check` and `--outputs-update`. `--registry-abi-update` is the only writer of the registry baseline and the only user of `tree.finish_update()`, but has no Makefile target. Add a wrapper or drop it. Speculative surface. > + action.add_argument('--mode', > + choices=('uapi', 'kernel'), > + help='generate the files of this mode') > + parser.add_argument('--schema-only', action='store_true', > + help='omit the family API from kernel output') > + kind = parser.add_mutually_exclusive_group() > + kind.add_argument('--header', action='store_true', > + help='generate the header of --mode') > + kind.add_argument('--source', action='store_true', > + help='generate the source of --mode') > + parser.add_argument('-o', '--output', type=str, > + help='write to this file instead of standard output') > + parser.add_argument('--cmp-out', action='store_true', > + help='leave the output file alone when its contents ' > + 'would not change') > + parser.add_argument('--output-root', type=str, > + help='alternate destination root for generated files') > + args = parser.parse_args() > + > + kinds = (args.header, args.source) > + if args.mode and not any(kinds): > + parser.error('--mode needs --header or --source') > + if any(kinds) and not args.mode: > + parser.error('--header and --source need --mode') > + if args.schema_only and args.mode != 'kernel': > + parser.error('--schema-only needs --mode kernel') > + output_action = args.outputs_check or args.outputs_update > + if args.output is not None and not args.mode: > + parser.error('-o/--output needs --mode') > + if args.cmp_out and (not args.mode or args.output is None): > + parser.error('--cmp-out needs --mode and --output') > + if args.output_root is not None: > + if not output_action: > + parser.error('--output-root needs an outputs action') > + if not args.output_root: > + parser.error('--output-root cannot be empty') > + > + paths = spec.Paths(args.spec, schema=args.schema, registry=args.registry, > + abi=args.abi) > + try: > + if args.tree_check: > + tree.check(paths) > + return 0 > + elif args.tree_check_update: > + tree.check_for_update(paths) > + return 0 > + elif args.registry_abi_update: > + registry = tree.finish_update(paths) > + paths.registry_abi.write_text( > + compat.registry_render(registry), encoding='utf-8') > + print(f'ctlvgen: wrote {paths.registry_abi}', file=sys.stderr) `paths.registry_abi.write_text()` is neither atomic nor does it create the parent directory (unlike `_abi_update()` which does mkdir). Either use the same helper `outputs._replace()` or at least mkdir here. > + return 0 > + elif args.outputs_check: > + tree.check(paths) > + outputs.reconcile(paths, args.output_root, check=True) > + return 0 > + elif args.outputs_update: > + tree.check_for_update(paths) > + changed = outputs.reconcile(paths, args.output_root) > + for item in changed: > + print(f'ctlvgen: {item}', file=sys.stderr) > + return 0 > + family = semantics.load(paths) > + if args.validate: > + cgen.symbols(family) > + elif args.abi_check: > + compat.registry_check(paths, spec.load_registry(paths)) > + compat.check(family, paths.abi(family.name)) > + elif args.abi_update: > + _abi_update(paths, family) > + else: > + _generate(args, family) > + except CtlvError as exc: > + print(f'ctlvgen: {exc}', file=sys.stderr) > + return 1 > + return 0 > + > + > +def _abi_update(paths, family): > + path = paths.abi(family.name) > + path.parent.mkdir(parents=True, exist_ok=True) > + path.write_text(compat.render(family), encoding='utf-8') > + print(f'ctlvgen: wrote {path}', file=sys.stderr) > + > + > +def _generate(args, family): Undocumented and non-trivial: It hides a lookup that only works for some (mode, kind) pairs, an inverted kernel-versus-other renderer choice, and an output rule (--cmp-out) that exists to preserve file timestamps for a build script. Suggested docstring: """Render one generated file of a family and write it out. Called by main() for --mode with --header or --source. The kernel mode calls gen_kernel.render_header() or render_source(); --schema-only leaves the family API out of the kernel output (family_api=False). Every other mode is looked up in the GENERATORS table by (mode, kind), where kind is "header" or "source"; at present only ('uapi', 'header') is there. With no -o/--output the text goes to standard output. Otherwise it is written to that file. With --cmp-out, an existing file whose contents are already identical is not rewritten, so its modification time stays put. The write is not atomic and does not create parent directories, unlike outputs.reconcile(), which is what the manifest actions use. Args: args: argparse.Namespace from main(), already validated. Uses mode, header, schema_only, output and cmp_out. Not modified. family: the resolved family returned by semantics.load(). Returns: None: the result is the side effect of writing the text. Throws: KeyError: for --mode uapi with --source, since GENERATORS has no such entry and main() does not reject the combination. It is not a CtlvError, so it is not caught there. CtlvError: when a generator rejects the family. main() reports it. OSError: when the output file cannot be read or written. """ > + if args.header: > + kind = 'header' > + else: > + kind = 'source' > + if args.mode == 'kernel': > + renderer = (gen_kernel.render_header if args.header > + else gen_kernel.render_source) > + text = renderer(family, family_api=not args.schema_only) > + else: > + text = GENERATORS[(args.mode, kind)](family) > + if args.output is None: > + sys.stdout.write(text) > + return > + path = pathlib.Path(args.output) > + # Leaving an unchanged file untouched keeps its timestamp, which is what > + # ctlv-regen.sh compares against the specification. > + if args.cmp_out and path.exists() and \ > + path.read_text(encoding='utf-8') == text: This comment is stale: `ctlv-regen.sh` does not compare timestamps of anything (`outputs.reconcile()` compares contents). `--cmp-out` has no user in the tree. > + return > + path.write_text(text, encoding='utf-8') > + > + > +if __name__ == '__main__': > + sys.exit(main()) > diff --git a/tools/ctlv/ctlvgen/lib/__init__.py b/tools/ctlv/ctlvgen/lib/__init__.py > new file mode 100644 > index 000000000000..f84069cb2c3d > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/__init__.py > @@ -0,0 +1 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > diff --git a/tools/ctlv/ctlvgen/lib/cgen.py b/tools/ctlv/ctlvgen/lib/cgen.py > new file mode 100644 > index 000000000000..2cb754b42106 > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/cgen.py > @@ -0,0 +1,825 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""C output scaffolding shared by the CTLV generators. This docstring (and the one for REGEN) is stale: "the preamble `ctlv-regen.sh` recovers a file's specification and mode from" - `ctlv-regen.sh` does no such thing; it is the Python `outputs._identity()` that parses the preamble. Also nothing in the preamble requires "the specification stays directly above the marker" other than `_identity()`'s fixed line positions. > + > +The preamble ctlv-regen.sh recovers a file's specification and mode from, which > +is why the specification stays directly above the marker. > +""" > + > +from __future__ import annotations > + > +import os > + > +from . import semantics > +from . import spec > +from .errors import CtlvError > + > +REGEN = 'tools/ctlv/ctlv-regen.sh' > + > +LINE_LIMIT = 100 > +HARD_LINE_LIMIT = 100 `LINE_LIMIT` and `HARD_LINE_LIMIT` are both 100. One of them is redundant (`HARD_LINE_LIMIT` is only used in `Writer.signature()`). > +TAB_WIDTH = 8 > + > +# C spelling of each primary form. A test checks these against > +# enum ctlv_schema_kind. "A test checks these against `enum ctlv_schema_kind`": no such test exists in the series, and `include/linux/ctlv_family.h` does not exist at this commit either. If KINDS and the C enum drift, the generated kernel metadata silently refers to a missing/renumbered enumerator only at C compile time. > +KINDS = { > + 'uint': 'CTLV_SCHEMA_UINT', > + 'sint': 'CTLV_SCHEMA_SINT', > + 'bool': 'CTLV_SCHEMA_BOOL', > + 'string': 'CTLV_SCHEMA_STRING', > + 'binary': 'CTLV_SCHEMA_BINARY', > + 'enum': 'CTLV_SCHEMA_ENUM', > + 'flags': 'CTLV_SCHEMA_FLAGS', > + 'nested': 'CTLV_SCHEMA_NEST', > + 'blob': 'CTLV_SCHEMA_BLOB', > +} > + > +LIFECYCLE_API_SYMBOLS = ( > + 'ctlv_create', > + 'ctlv_destroy', > + 'ctlv_family_priv_size_fn', > + 'ctlv_family_pre_init_fn', > + 'ctlv_family_init_fn', > + 'ctlv_family_post_init_fn', > + 'ctlv_family_pre_fini_fn', > + 'ctlv_family_fini_fn', > + 'ctlv_priv_parent', > + 'ctlv_priv_child', > +) `LIFECYCLE_API_SYMBOLS` is a hand-maintained mirror of C function names that do not exist in this patch (`ctlv_create`, `ctlv_family_priv_size_fn`, ...). Nothing verifies it against the headers that later define them. > + > + > +def width(line): > + """Return the display width of one line, a tab being eight columns.""" > + return len(line.expandtabs(TAB_WIDTH)) > + > + > +def ident(*names): > + """Return the lower case C spelling of one or more schema names.""" > + return '_'.join(name.replace('-', '_') for name in names) > + > + > +def macro(*names): > + """Return the upper case C spelling of one or more schema names.""" > + return ident(*names).upper() > + > + > +def enum_tag(*names): > + return f'_{ident(*names)}' > + > + > +def family_suffix(family_name): > + suffix = family_name > + if family_name.startswith('ctlv-'): > + suffix = family_name.removeprefix('ctlv-') The conditional is redundant: `str.removeprefix()` already returns the string unchanged when the prefix is absent. This whole function is `return macro(family_name.removeprefix('ctlv-'))`. > + return macro(suffix) > + > + > +def family_const(family_name): > + if family_name == 'ctlv': > + return 'CTLV_FAMILY_ROOT' > + return f'CTLV_FAMILY_{family_suffix(family_name)}' > + > + > +def uapi_header(family_name, in_kernel=False): > + """Return the include of the generated UAPI header of family_name.""" > + prefix = 'uapi/' if in_kernel else '' > + return f'<{prefix}linux/{ident(family_name)}_gen.h>' > + > + > +def families_uapi_header(in_kernel=False): > + prefix = 'uapi/' if in_kernel else '' > + return f'<{prefix}linux/ctlv_families_gen.h>' > + > + > +def kernel_header(family_name): > + """Return the include of the generated kernel header of family_name. > + > + The root family's interface is installed beside ; any other > + family's is private to the family implementation compiling it. > + """ > + name = f'{ident(family_name)}_gen.h' > + if family_name == spec.CORE_FAMILY_NAME: > + return f'' > + return f'"{name}"' > + > + > +def spec_path(path): > + """Return path relative to the kernel tree that holds it. > + > + ctlv-regen.sh resolves it from the tree root, so it must not depend on how > + the generator was invoked. Outside a kernel tree it is recorded as given. > + """ > + full = os.path.abspath(path) > + relative = '' > + while True: > + relative = os.path.join(os.path.basename(full), relative) > + parent = os.path.dirname(full) > + if os.path.exists(os.path.join(parent, 'MAINTAINERS')): > + return relative[:-1] > + if parent == full: > + return str(path) > + full = parent `spec_path()`: if no MAINTAINERS file is found the (possibly absolute, host-specific) path is embedded verbatim into the generated file's preamble, making the output depend on where the generator was invoked, which defeats the "reproducible/check-regen" goal. Prefer failing, or require `--spec` to be inside a kernel tree. > + > + > +def wrapped(opening, items, closing, trailer='', adaptive=False): > + """Return the lines of one parenthesized list, wrapped at the limit. > + > + trailer is what a continued line ends with beyond the item: a backslash > + inside a macro definition, nothing elsewhere. > + """ > + parts = [f'{item}, ' for item in items[:-1]] + [items[-1] + closing] `wrapped()` does items[-1] on an empty sequence (IndexError) - e.g. `define_call(name, fn)` with no args that happens to exceed the line limit. Callers presently never do this, but nothing enforces it. > + column = width(opening) > + part_width = max(width(part.rstrip() + trailer) for part in parts) > + indent_column = column > + if adaptive: > + indent_column = min(column, max(0, LINE_LIMIT - part_width)) > + indent = '\t' * (indent_column // TAB_WIDTH) + \ > + ' ' * (indent_column % TAB_WIDTH) > + lines = [] > + line = opening > + for part in parts: > + if width(line + part.rstrip() + trailer) > LINE_LIMIT: > + if line != opening: > + lines.append(line.rstrip()) > + elif adaptive and opening.rstrip(): > + lines.append(opening.rstrip()) > + else: > + line += part > + continue > + line = indent > + line += part > + lines.append(line.rstrip()) > + return lines > + > + > +class Writer: > + """The lines of one generated file.""" > + > + def __init__(self): > + self._lines = [] > + > + def line(self, text=''): > + self._lines.append(text) > + > + def comment(self, *lines): > + """One comment block, one input line per output line.""" > + if len(lines) == 1: > + self.line(f'/* {lines[0]} */') > + return > + self.line('/*') > + for text in lines: > + self.line(f' * {text}'.rstrip()) > + self.line(' */') > + > + def define_block(self, name, *lines): > + """One #define whose value takes more than the one line.""" > + header = f'#define {name} \\' > + if width(header) > LINE_LIMIT and '(' in name: > + macro, params = name.removesuffix(')').split('(', 1) > + for text in wrapped(f'#define {macro}(', "macro, params = ..." shadows the module-level `macro()` function within `define_block()`; harmless today, a trap for the next edit. > + tuple(params.split(', ')), ')', ' \\', > + adaptive=True): > + self.line(f'{text} \\') > + else: > + self.line(header) > + for text in lines[:-1]: > + self.line(f'{text} \\') > + self.line(lines[-1]) > + > + def define(self, name, value): > + """One #define, continued when it would run past the line limit.""" > + text = f'#define {name} {value}' > + if width(text) <= LINE_LIMIT: > + self.line(text) > + return > + self.define_block(name, f'\t{value}') > + > + def define_call(self, name, function, *args): > + """One #define whose value is a call, wrapped where it has to be.""" > + call = f'{function}({", ".join(args)})' > + if width(f'#define {name} {call}') <= LINE_LIMIT: > + self.line(f'#define {name} {call}') > + return > + if width(f'\t{call}') <= LINE_LIMIT: > + self.define_block(name, f'\t{call}') > + return > + self.define_block( > + name, *wrapped(f'\t{function}(', args, ')', ' \\', > + adaptive=True)) > + > + def declaration(self, prefix, declarator): > + """One declaration, continued when it would run past the limit.""" > + text = f'{prefix} {declarator}' > + if width(text) <= LINE_LIMIT: > + self.line(text) > + return > + indent = prefix[:len(prefix) - len(prefix.lstrip('\t'))] > + self.line(prefix) > + continuation = f'{indent}\t' > + if width(f'{continuation}{declarator}') > LINE_LIMIT: > + continuation = '\t' > + self.line(f'{continuation}{declarator}') > + > + def parenthesized(self, opening, items, closing): > + """One parenthesized list, wrapped at the line limit.""" > + for text in wrapped(opening, items, closing, adaptive=True): > + self.line(text) > + > + def assertion(self, left, operator, right): > + """One static_assert, broken after its operator when it must be.""" > + opening = 'static_assert(' > + text = f'{opening}{left} {operator} {right});' > + if width(text) <= LINE_LIMIT: > + self.line(text) > + return > + column = width(opening) > + indent = '\t' * (column // TAB_WIDTH) + ' ' * (column % TAB_WIDTH) > + self.line(f'{opening}{left} {operator}') > + self.line(f'{indent}{right});') > + > + def signature(self, prefix, name, params, closing=');'): > + """One function signature, wrapped at the line limit.""" > + lines = wrapped(f'{prefix} {name}(', params, closing) > + one_line = f'{prefix} {name}({", ".join(params)}{closing}' > + if lines[0].endswith('(') and width(one_line) <= HARD_LINE_LIMIT: > + self.line(one_line) > + return > + if max(width(line) for line in lines) <= LINE_LIMIT: > + for text in lines: > + self.line(text) > + return > + if name.startswith('*'): > + self.line(f'{prefix} *') > + name = name.removeprefix('*') > + else: > + self.line(prefix) > + declarator = f'{name}({", ".join(params)}{closing}' > + if width(declarator) <= HARD_LINE_LIMIT: > + self.line(declarator) > + return > + opening = f'{name}(' > + lines = wrapped(opening, params, closing) > + if max(width(line) for line in lines) > LINE_LIMIT: > + self.line(opening) > + self.parenthesized('\t', params, closing) > + else: > + for text in lines: > + self.line(text) > + > + def open_guard(self, guard): > + self.line(f'#ifndef {guard}') > + self.line(f'#define {guard}') > + self.line() > + > + def close_guard(self, guard): > + self.line(f'#endif /* {guard} */') > + > + def includes(self, *headers): > + """One include block, an empty entry separating two groups.""" > + for header in headers: > + self.line(f'#include {header}' if header else '') > + self.line() > + > + def text(self): > + while self._lines and not self._lines[-1]: > + self._lines.pop() > + return '\n'.join(self._lines) + '\n' > + > + > +def preamble(origin, mode, kind, source=None): > + """Return a Writer holding the preamble of one generated file.""" > + writer = Writer() > + if kind == 'source': > + writer.line(f'// SPDX-License-Identifier: {origin.license}') > + else: > + writer.line(f'/* SPDX-License-Identifier: {origin.license} */') > + writer.line(f'/* Do not edit, regenerate with {REGEN}, from: */') > + writer.line(f'/*\t{spec_path(source or origin.path)} */') > + writer.line(f'/* CTLV-GEN {mode} {kind} */') > + writer.line() > + return writer > + > + > +class Names: > + """The C names of one effective family's declarations. > + > + A declaration is named after the family that owns it, so an inherited attr > + keeps the name its own family's header gives it. > + """ > + > + def __init__(self, family): > + self.family = family > + self.prefix = macro(family.name) > + self.ident = ident(family.name) > + > + def owner_prefix(self, family_id): > + return macro(self.family.owners[family_id]) > + > + def owner_ident(self, family_id): > + return ident(self.family.owners[family_id]) > + > + # Names that a family's UAPI header defines. > + > + def family_const(self): > + return family_const(self.family.name) > + > + def group(self, group): > + return f'{self.owner_prefix(group.family_id)}_GROUP_' \ > + f'{macro(group.name)}' > + > + def group_enum(self, group): > + return enum_tag(self.family.owners[group.family_id], 'group') > + > + def group_attr_enum(self, group): > + return enum_tag(self.family.owners[group.family_id], > + 'attr', group.name) > + > + def group_metadata_enum(self, group): > + return enum_tag(self.family.owners[group.family_id], > + 'attr', group.name, 'metadata') > + > + def group_attr(self, group): > + return f'_{self.owner_prefix(group.family_id)}_ATTR_' \ > + f'{macro(group.name)}' > + > + def family_attr(self, group): > + return f'_{self.owner_prefix(group.family_id)}_ATTR' > + > + def local_attr_prefix(self, group): > + return f'_{self.owner_prefix(group.family_id)}_ATTR_' \ > + f'{macro(group.name)}_' > + > + def local_attr(self, attr): > + group = self.family.groups[attr.group] > + return f'{self.local_attr_prefix(group)}{macro(attr.name)}' > + > + def attr(self, attr): > + return f'{self.owner_prefix(attr.family_id)}_ATTR_' \ > + f'{macro(attr.group, attr.name)}' > + > + def attr_unit(self, attr): > + return f'{self.attr(attr)}_UNIT' > + > + def attr_scale(self, attr): > + return f'{self.attr(attr)}_SCALE' > + > + def attr_set(self, attr_set): > + return f'{self.owner_prefix(attr_set.family_id)}_SET_' \ > + f'{macro(attr_set.name)}' > + > + def set_enum(self, attr_set): > + return enum_tag(self.family.owners[attr_set.family_id], 'set') > + > + def op_prefix(self, op): > + return f'_{self.owner_prefix(op.family_id)}_OP_' > + > + def op(self, op): > + return f'{self.op_prefix(op)}{macro(op.name)}' > + > + def operation(self, op): > + return f'_{self.owner_prefix(op.family_id)}_OP' > + > + def op_enum(self, op): > + return enum_tag(self.family.owners[op.family_id], 'op') > + > + def op_id(self, op): > + return f'{self.owner_prefix(op.family_id)}_OP_{macro(op.name)}' > + > + def definition(self, definition): > + """The C type name of an enum definition.""" > + return enum_tag(self.family.owners[definition.family_id], > + definition.name) > + > + def value(self, definition, value_name): > + """One enum value or flag bit.""" > + return f'{self.owner_prefix(definition.family_id)}_' \ > + f'{macro(definition.name, value_name)}' > + > + def value_bit(self, definition, value_name): > + return f'{self.value(definition, value_name)}_BIT' > + > + def bit_enum(self, definition): > + return enum_tag(self.family.owners[definition.family_id], > + definition.name, 'bit') > + > + def value_mask(self, definition): > + """The mask of every bit a flags definition declares.""" > + return f'{self.owner_prefix(definition.family_id)}_' \ > + f'{macro(definition.name)}_MASK' > + > + # Names that a family's kernel metadata defines: its own ops and the > + # stubs it fills. > + > + def op_schemas(self): > + return f'{self.ident}_op_schemas' > + > + def schema(self): > + return f'{self.ident}_schema' > + > + def stub_fill_schemas(self): > + return f'{self.ident}_stub_fill_schemas' > + > + def node_schemas(self, op, which): > + # File scope, so no family prefix: an op name is unique within its > + # effective family. > + return f'{ident(op.name)}_{which}_node_schemas' > + > + def stub_schemas(self, op, which): > + return f'{ident(op.name)}_{which}_stub_schemas' > + > + def fill_node_schemas(self, fill): > + return f'{ident(fill.name)}_fill_node_schemas' > + > + def message_node_schemas(self, message): > + return f'{ident(message.name)}_node_schemas' > + > + def message_schema(self, message): > + """The tree the core builds one of its own messages against.""" > + return f'{self.ident}_{ident(message.name)}_schema' > + > + def message_max_len(self, message): > + return f'{self.prefix}_{macro(message.name)}_MAX_LEN' > + > + # Names that a family's device descriptor defines. Named after the family > + # that declared what they answer: an inherited op is answered by that > + # family implementation, on whatever device it arrived at. > + > + def descriptor(self): > + return f'{self.ident}_family' > + > + def ops(self): > + return f'{self.ident}_ops' > + > + def stub_ops(self): > + return f'{self.ident}_stub_ops' > + > + def stub_ident(self, fill): > + return f'{self.ident}_{ident(fill.name)}' > + > + def stub_callback(self, fill, which): > + """One half of what this family does with an extension point.""" > + return f'{self.stub_ident(fill)}_{which}' > + > + def lifecycle_hook(self, which): > + """One operation in the lifecycle of this family's priv.""" > + return f'{self.ident}_{which}' > + > + def op_ident(self, op): > + return f'{self.owner_ident(op.family_id)}_{ident(op.name)}' > + > + def callback(self, op): > + return f'{self.op_ident(op)}_{op.type}' > + > + def op_helper(self, op, what): > + """What one device decides about one op.""" > + return f'{self.op_ident(op)}_{what}' > + > + def attr_helper(self, attr, what): > + """One helper for one attr: reading it, or appending it. > + > + Named after the family that declares the attr, like the constant that > + names it on the wire. > + """ > + return f'{self.owner_ident(attr.family_id)}_' \ > + f'{ident(attr.group, attr.name)}_{what}' > + > + def attr_iterator(self, attr): > + """What walks every occurrence of one repeated attr.""" > + return f'{self.owner_ident(attr.family_id)}_{ident(attr.group)}_' \ > + f'for_each_{ident(attr.name)}' > + > + def input_name(self, scope, attr): > + """The stem of one request-placement-specific reader.""" > + owner_parts = self.family.name.split('-') > + group = () > + if attr.group not in (*scope.name, *owner_parts): > + group = (attr.group,) > + parts = (*scope.name, *group, attr.name) > + parts = tuple(part for index, part in enumerate(parts) > + if not index or part != parts[index - 1]) > + return f'{self.ident}_{ident(*parts)}' > + > + def input_helper(self, scope, attr, what): > + return f'{self.input_name(scope, attr)}_{what}' > + > + def input_iterator(self, scope, attr): > + return self.input_helper(scope, attr, 'for_each') > + > + def input_ref(self, scope, attr): > + return self.input_helper(scope, attr, 'ref') > + > + def family_hook(self, which): > + """One half of the pair that wraps every op of this family.""" > + return f'{self.ident}_op_{which}' > + > + def op_hook(self, op, which): > + """One half of the pair that wraps this op alone.""" > + return f'{self.op_ident(op)}_{which}' > + > + # Where a generated file lives: the installed constants, the schema > + # metadata a family compiles, and the descriptor it registers. > + > + def uapi_guard(self): > + return f'_UAPI_LINUX_{self.prefix}_GEN_H' > + > + def uapi_header(self, in_kernel=False): > + return uapi_header(self.family.name, in_kernel) > + > + def kernel_guard(self): > + return f'_LINUX_{self.prefix}_GEN_H' > + > + def kernel_header(self): > + return kernel_header(self.family.name) > + > + def user_guard(self): `Names.user_guard`/`user_header`/`user_op`/`user_op_helper`/`user_op_len`/ `user_node_schema`/`user_spec`/`user_op_specs` (through line 1441) are dead code: there is no userspace binding generator in this series (`cli.py` has only 'uapi' and 'kernel' modes), and a grep over `tools/ctlv` shows `user_guard`, `user_header`, `user_node_schema`, `user_spec` and `user_op_specs` are never called at all, and the others are only called from `user_symbols()` below. > + return f'_CTLV_{self.prefix}_USER_GEN_H' > + > + def user_header(self): > + return f'"{self.ident}_user_gen.h"' > + > + # Names that a family's userspace binding defines, after the family that > + # declared the op, as a per-op helper is in the kernel. > + > + def user_op(self, op): > + return f'{self.owner_ident(op.family_id)}_{ident(op.name)}' > + > + def user_op_helper(self, op, what): > + return f'{self.user_op(op)}_{what}' > + > + def user_op_len(self, op, what): > + """One length of one op, as its specification computed it.""" > + return f'{self.op_id(op)}_{macro(what)}' > + > + def user_node_schema(self, op, which): > + # File scope, so no family prefix, as on the kernel side. > + return f'{ident(op.name)}_{which}_node_schema' > + > + def user_spec(self): > + """What this family's specification says, to compare a device to.""" > + return f'{self.ident}_spec' > + > + def user_op_specs(self): > + return f'{self.ident}_op_specs' > + > + > +def attr_getter_value(attr): > + """Whether an attr has a getter beside the reader that locates it.""" > + return attr.decl.kind != 'blob' and \ > + not (attr.decl.kind == 'nested' and attr.decl.max_occurrences != 1) > + > + > +def input_readers(names, scope, attr): > + readers = [names.input_ref(scope, attr)] > + readers.append(names.input_iterator(scope, attr) > + if attr.decl.max_occurrences != 1 > + else names.input_helper(scope, attr, 'present')) > + if attr.decl.kind == 'blob': > + readers.append(names.input_helper(scope, attr, 'resolve')) > + elif attr_getter_value(attr): > + readers.append(names.input_helper(scope, attr, 'get')) > + return tuple(readers) > + > + > +def attr_writers(names, attr): > + """The names of what appends one attr, as the family API declares them. > + > + A blob has none: a request carries one and a reply never does. > + """ > + if attr.decl.kind == 'blob': > + return () > + what = 'start' if attr.decl.kind == 'nested' else 'put' > + return (names.attr_helper(attr, what),) > + > + > +# What a blob attr may be backed by, and the libctlv call describing one. A > +# putter exists per backing the placement permits, so what a caller may > +# describe is decided by the specification. > +BLOB_PUTTERS = { > + 'va': ('va', ('const void *address', '__u64 length')), > +} > + > +# What an event gets in the place of a wrapper. > +EVENT_HELPERS = ('is', 'subscribe', 'unsubscribe') > + > +# The lengths a user-mode header states for one op, by sizes.OpSizes member. > +USER_LENGTHS = { > + 'request_max_len': 'request-max-len', > + 'action_reply_max_len': 'action-reply-max-len', > + 'query_reply_min_len': 'query-reply-min-len', > + 'query_reply_suggested_len': 'query-reply-suggested-len', > + 'event_record_max_len': 'event-record-max-len', > +} > + > + > +def user_attr_readers(names, attr): > + """The names of what reads one attr in the userspace binding.""" > + if attr.decl.kind == 'blob': > + return () > + readers = [names.attr_iterator(attr) if attr.decl.max_occurrences != 1 > + else names.attr_helper(attr, 'present')] > + if attr_getter_value(attr): > + readers.append(names.attr_helper(attr, 'get')) > + return tuple(readers) > + > + > +def user_attr_writers(names, attr): > + """The names of what appends one attr in the userspace binding. > + > + A blob has one putter per backing the placement allows, where the kernel > + side has none. > + """ > + if attr.decl.kind == 'blob': > + return tuple(names.attr_helper(attr, f'put_{BLOB_PUTTERS[backing][0]}') > + for backing in attr.decl.blob.backings) > + what = 'start' if attr.decl.kind == 'nested' else 'put' > + return (names.attr_helper(attr, what),) > + > + > +def user_symbols(family): > + """Return every C name of family's binding, mapped to its origin. > + > + The binding covers this family's own attrs and ops, reaching the rest > + through the parent's header. > + """ > + names = Names(family) > + table = {} > + > + def add(name, origin): > + previous = table.get(name) > + if previous is not None: > + raise CtlvError(f'{family.path}: {origin}', > + f'generated name {name} is already used by ' > + f'{previous}') > + table[name] = origin > + > + for group in family.local_groups: > + for attr in group.attrs: > + what = f'attr {attr.full_name}' > + for helper in user_attr_readers(names, attr) + \ > + user_attr_writers(names, attr): > + add(helper, what) > + for op in family.local_ops: > + what = f'op {op.name}' > + # An event has those helpers and no wrapper of its own name. > + if op.type == 'event': > + for helper in EVENT_HELPERS: > + add(names.user_op_helper(op, helper), what) > + else: > + add(names.user_op(op), what) > + if op.request is not None: > + add(names.user_op_helper(op, 'request_new'), what) > + for field, what_len in USER_LENGTHS.items(): > + if getattr(op.sizes, field): > + add(names.user_op_len(op, what_len), what) > + return table Dead: `user_symbols()` (together with `BLOB_PUTTERS`, `EVENT_HELPERS`, `USER_LENGTHS`, `user_attr_readers()` and `user_attr_writers()` above, and the five `op.sizes` fields read only through `USER_LENGTHS`) has no caller anywhere in `tools/ctlv`. This is speculative generality for a "userspace binding" that is not part of this series; remove it until the generator that needs it is submitted. > + > + > +def _stubs_anywhere(node): > + return bool(node.stubs) or any(_stubs_anywhere(child) > + for child in node.children()) Duplicate of `semantics._stubs_anywhere()` (same name, same body; `semantics.py` even imports nothing from here). Likewise `sizes._reenters()` duplicates `semantics._reenters_anywhere()`. Export one and reuse it. > + > + > +def symbols(family): > + """Return every generated C name of family, mapped to its declaration. > + > + Dashes become underscores, so two YAML names can spell one C name. That > + has to be an error: one of the two constants would silently not exist. > + """ > + names = Names(family) > + table = {} > + > + def add(name, origin): > + previous = table.get(name) > + if previous is not None: > + raise CtlvError(f'{family.path}: {origin}', > + f'generated name {name} is already used by ' > + f'{previous}') > + table[name] = origin > + > + for name in LIFECYCLE_API_SYMBOLS: > + add(name, 'the CTLV lifecycle API') > + add(names.family_const(), f'family {family.name}') > + # A child header may include every ancestor's family API, and built-in > + # families also share one linker namespace with the root. > + for family_id, owner in family.owners.items(): > + owner_ident = ident(owner) > + if any(op.family_id == family_id for op in family.ops): > + add(f'{owner_ident}_op_schemas', f'family {owner}') > + add(f'{owner_ident}_schema', f'family {owner}') > + add(f'{owner_ident}_family', f'family {owner}') > + for which in ('pre', 'post'): > + add(f'{owner_ident}_op_{which}', f'family {owner}') > + for which in ('priv_size', 'pre_init', 'init', 'post_init', > + 'pre_fini', 'fini'): > + add(f'{owner_ident}_device_{which}', f'family {owner}') > + for fill in family.fills: > + for which in ('fill', 'read'): > + add(names.stub_callback(fill, which), f'attr set {fill.name}') 'read' reserves a symbol `__read` that nothing generates: `gen_family.STUB_HOOKS` declares only 'fill'. Either generate it or drop it from the collision table (and then `STUB_HOOKS`, a one-element table iterated by a loop in `_stub_hooks()`, can be simplified too). > + for definition in family.definitions.values(): > + what = f'{definition.kind} {definition.name}' > + if definition.kind == 'enum': > + add(f'enum {names.definition(definition)}', what) > + else: > + add(f'enum {names.bit_enum(definition)}', what) > + add(names.value_mask(definition), what) > + for value_name, _ in definition.values: > + if definition.kind == 'flags': > + add(names.value_bit(definition, value_name), what) > + add(names.value(definition, value_name), what) > + group_enums = set() > + family_attrs = set() > + for group in family.groups.values(): > + group_enum = f'enum {names.group_enum(group)}' > + if group_enum not in group_enums: > + add(group_enum, f'family {family.owners[group.family_id]} groups') > + group_enums.add(group_enum) > + family_attr = names.family_attr(group) > + if family_attr not in family_attrs: > + add(family_attr, f'family {family.owners[group.family_id]} attrs') > + family_attrs.add(family_attr) > + add(names.group(group), f'attr group {group.name}') > + add(f'enum {names.group_attr_enum(group)}', > + f'attr group {group.name}') > + if any(attr.decl.unit is not None for attr in group.attrs): > + add(f'enum {names.group_metadata_enum(group)}', > + f'attr group {group.name} metadata') > + add(names.group_attr(group), f'attr group {group.name}') > + for attr in group.attrs: > + what = f'attr {attr.full_name}' > + add(names.local_attr(attr), what) > + add(names.attr(attr), what) > + if attr.decl.unit is not None: > + add(names.attr_unit(attr), what) > + add(names.attr_scale(attr), what) > + for helper in attr_writers(names, attr): > + add(helper, what) > + for ancestor in family.chain[:-1]: > + ancestor_names = Names(ancestor) > + owner_family_id = next( > + family_id for family_id, owner in family.owners.items() > + if owner == ancestor.name) > + > + for scope in semantics.input_scopes(family, owner_family_id): > + for child in scope.node.children(): > + what = f'attr {child.attr.full_name} in ancestor ' \ > + f'{ancestor.name}' > + for helper in input_readers(ancestor_names, scope, > + child.attr): > + add(helper, what) > + for scope in semantics.input_scopes(family): > + for child in scope.node.children(): > + what = f'attr {child.attr.full_name} in {".".join(scope.name)}' > + for helper in input_readers(names, scope, child.attr): > + add(helper, what) > + set_enums = set() > + for attr_set in family.sets.values(): > + set_enum = f'enum {names.set_enum(attr_set)}' > + if set_enum not in set_enums: > + add(set_enum, f'family {family.owners[attr_set.family_id]} sets') > + set_enums.add(set_enum) > + add(names.attr_set(attr_set), f'attr set {attr_set.name}') > + operation_enums = set() > + for op in family.ops: > + what = f'op {op.name}' > + operation_enum = f'enum {names.op_enum(op)}' > + if operation_enum not in operation_enums: > + add(operation_enum, > + f'family {family.owners[op.family_id]} operations') > + add(names.operation(op), > + f'family {family.owners[op.family_id]} operations') > + operation_enums.add(operation_enum) > + add(names.op(op), what) > + add(names.op_id(op), what) > + if op.type == 'event': > + add(names.op_helper(op, 'create'), what) > + continue > + add(names.callback(op), what) > + if op.optional: > + add(names.op_helper(op, 'provide'), what) > + if op.availability == 'dynamic': > + add(names.op_helper(op, 'enable'), what) > + add(names.op_helper(op, 'disable'), what) > + if op.pre_post: > + add(names.op_hook(op, 'pre'), what) > + add(names.op_hook(op, 'post'), what) > + for message in family.messages: > + what = f'attr set {message.name}' > + add(names.message_node_schemas(message), what) > + add(names.message_schema(message), what) > + add(names.message_max_len(message), what) > + if family.fills: > + add(names.stub_fill_schemas(), f'family {family.name}') > + add(names.stub_ops(), f'family {family.name}') > + if any(op.type != 'event' for op in family.local_ops): > + add(names.ops(), f'family {family.name}') > + for op in family.local_ops: > + what = f'op {op.name}' > + for which in ('request', 'reply'): > + root = getattr(op, which) > + if root is None: > + continue > + add(names.node_schemas(op, which), what) > + if _stubs_anywhere(root): > + add(names.stub_schemas(op, which), what) > + for fill in family.fills: > + add(names.fill_node_schemas(fill), f'attr set {fill.name}') > + return table `symbols()` is used as a validation pass whose returned table is thrown away by every caller, and it is re-run from `gen_uapi.render()`, `gen_kernel.render_header()` and `gen_kernel.render_source()` (so three times per family on `--outputs-`*), plus once more via `--validate`. The name is misleading for what is really `check_symbol_collisions()`. It also does not cover collisions between different families that end up in one kernel image (only the "pre/post", `_schema`, `_family` names of the chain), though a comment above claims built-in families share one namespace. > diff --git a/tools/ctlv/ctlvgen/lib/compat.py b/tools/ctlv/ctlvgen/lib/compat.py > new file mode 100644 > index 000000000000..4cf93af06ea9 > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/compat.py > @@ -0,0 +1,821 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""ABI snapshots and the compatibility checker. > + > +A change is compatible only if the new schema still contains everything the > +checked-in snapshot describes, with the same IDs, no narrowed bound and no > +placement that became required. A new placement has to be optional. Updating a > +snapshot is deliberate: it is how the change gets reviewed. > +""" > + > +from __future__ import annotations > + > +import yaml > + > +from . import spec > +from . import yamlload > +from .errors import CtlvError > + > +HEADER = '''\ > +# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) > +%YAML 1.2 > +--- > +# Checked-in ABI snapshot of the {name} family, written by > +# "make -C tools/ctlv abi-update". > +# > +# The checker compares this family's declarations and inherited stub fills > +# against this file. Updating it is how a UAPI change gets reviewed; an update > +# that removes or renumbers anything, narrows a bound, or makes a placement > +# required is an ABI break, whatever this file says afterwards. > +''' > + > +REGISTRY_HEADER = '''\ > +# SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) > +%YAML 1.2 > +--- > +# Checked-in ABI baseline of the ordered CTLV family registry. > +# Updated deliberately by "make -C tools/ctlv abi-update". This is wrong: "make -C `tools/ctlv` abi-update" does not update the registry baseline (see the Makefile comment above). > +''' > + > + > +class _Dumper(yaml.SafeDumper): > + """Dumper that never emits anchors, which the strict loader rejects.""" > + > + def ignore_aliases(self, data): > + return True > + > + def increase_indent(self, flow=False, indentless=False): > + # Indent a sequence under its key, not flush left as pyyaml does. > + return super().increase_indent(flow, False) > + > + > +class _FlowMap(dict): > + pass > + > + > +class _FlowSeq(list): > + pass > + > + > +_Dumper.add_representer( > + _FlowMap, > + lambda dumper, value: dumper.represent_mapping( > + 'tag:yaml.org,2002:map', value, flow_style=True)) > +_Dumper.add_representer( > + _FlowSeq, > + lambda dumper, value: dumper.represent_sequence( > + 'tag:yaml.org,2002:seq', value, flow_style=True)) > + > + > +def build(family): > + """Return one family's ABI in the comparison form.""" > + return { > + 'family': family.name, > + 'family-id': family.family_id, > + 'parent': family.parent, > + 'pre-init': family.pre_init, > + 'post-init': family.post_init, > + 'pre-fini': family.pre_fini, > + 'definitions': _definitions(family.local_definitions), > + 'groups': _groups(family.local_groups), > + 'sets': _sets(family.local_sets), > + 'fills': _fills(family.fills), > + 'ops': _ops(family.local_ops), > + } > + > + > +def render(family): > + """Return one family's compact snapshot as YAML text.""" > + return render_data(build(family)) > + > + > +def render_data(data): > + """Render comparison-form data as a compact snapshot.""" > + body = yaml.dump(_compact(data), Dumper=_Dumper, sort_keys=False, > + default_flow_style=False, width=1000000) > + body = '\n'.join(line for source in body.splitlines() > + for line in _wrap_flow(source)) + '\n' > + return HEADER.format(name=data['family']) + body > + > + > +def _wrap_flow(line, width=80): > + """Wrap one flow collection with continuations aligned after its opener.""" > + if len(line) <= width: > + return (line,) > + starts = [index for index in (line.find('{'), line.find('[')) > + if index >= 0] > + if not starts: > + return (line,) > + start = min(starts) > + end, items = _flow_items(line, start) > + if end is None: > + return (line,) > + > + prefix = line[:start + 1] > + suffix = line[end:] > + indent = ' ' * (start + 1) > + wrapped = [] > + current = prefix > + for index, item in enumerate(items): > + last = index + 1 == len(items) > + token = item + ('' if last else ',') > + separator = '' if current.endswith(('{', '[')) else ' ' > + closing = len(suffix) if last else 0 > + if len(current) + len(separator) + len(token) + closing > width: > + wrapped.append(current) > + current = indent + token > + else: > + current += separator + token > + current += suffix > + wrapped.append(current) > + return tuple(wrapped) > + > + > +def _flow_items(line, start): Undocumented and non-trivial: It is a hand-rolled scanner that tracks nesting depth, quotes and backslash escapes at once, and returns a two-part result whose failure form is easy to misread. Suggested docstring: """Split one flow collection in a YAML line into its top-level items. Scans LINE from START, which must index the opening "{" or "[" of a flow mapping or sequence, and finds the matching closer. Brackets of either kind nest and must pair correctly. Text inside single or double quotes is skipped, and a backslash escapes the next character while inside a quote, so a quoted "," or "}" does not split or close anything. Items are the comma-separated pieces at nesting depth one, stripped of surrounding whitespace; nested collections stay whole inside their item, and empty pieces (for example from an empty "{}" or a trailing comma) are dropped. This only supports _wrap_flow, which re-wraps over-long lines emitted by the YAML dumper. It never raises: any line that cannot be parsed this way is reported through the failure return so the caller can leave it unwrapped. Args: line: One line of dumped YAML text. start: Index in LINE of the opening "{" or "[". Returns: tuple[int | None, tuple[str, ...]]: (end, items) where END is the index of the matching closing bracket and ITEMS are the top-level item strings. If the brackets are mismatched or never closed, returns (None, ()). """ > + pairs = {'{': '}', '[': ']'} > + stack = [line[start]] > + quote = None > + escape = False > + item_start = start + 1 > + items = [] > + for index in range(start + 1, len(line)): > + char = line[index] > + if quote is not None: > + if escape: > + escape = False > + elif char == '\\': > + escape = True > + elif char == quote: > + quote = None > + continue > + if char in ('"', "'"): > + quote = char > + elif char in pairs: > + stack.append(char) > + elif char in pairs.values(): > + if pairs[stack[-1]] != char: > + return None, () > + stack.pop() > + if not stack: > + items.append(line[item_start:index].strip()) > + return index, tuple(item for item in items if item) > + elif char == ',' and len(stack) == 1: > + items.append(line[item_start:index].strip()) > + item_start = index + 1 > + return None, () > + > + > +def check(family, path): > + """Compare a family against its snapshot, or raise with every problem.""" > + snapshot = _expand(yamlload.load(path)) > + problems = _compare(snapshot, build(family)) > + if problems: > + raise CtlvError(f'{family.name} against {path}', > + 'the schema is not compatible with its snapshot\n' + > + '\n'.join(f' {problem}' for problem in problems)) > + > + > +def _compact(data): Undocumented and non-trivial: It maps the comparison-form dict onto the compact snapshot layout, and the layout's omissions (empty sections, false flags, absent parent) are only implied by the conditionals. Suggested docstring: """Convert comparison-form family data into the compact snapshot layout. This is the writing half of the compact ABI snapshot format; _expand is its inverse. DATA is the dict produced by build(). The result is a plain dict of sections in a fixed order, ready for yaml.dump in render_data: family Flow map with name and id. "inherits" appears only when the family has a parent, and each of pre-init, post-init and pre-fini appears (as true) only when set; false is omitted. enums/flags One section each, mapping a definition name to a flow map of value name to number. The comparison form keeps both kinds in one "definitions" dict with a "kind" field; here the kind is the section name instead. attr-groups See _compact_groups. attr-sets Name to a flow map holding the set ID, plus "stub: true" only for stub sets. fills Name to the compact tree of an inherited stub fill. ops Name to the compact form of the op, see _compact_op. Every section except "family" is omitted entirely when empty, so a family with no flags has no "flags" key at all. The input is only read, never modified. Args: data: Comparison-form family dict as returned by build() (keys family, family-id, parent, pre-init, post-init, pre-fini, definitions, groups, sets, fills, ops). Returns: dict: The compact snapshot structure. Values that should print on one line are _FlowMap or _FlowSeq instances. """ > + family = _FlowMap({ > + 'name': data['family'], > + 'id': data['family-id'], > + }) > + if data['parent'] is not None: > + family['inherits'] = data['parent'] > + for name in ('pre-init', 'post-init', 'pre-fini'): > + if data[name]: > + family[name] = True > + > + compact = {'family': family} > + for kind, section in (('enum', 'enums'), ('flags', 'flags')): > + definitions = { > + name: _FlowMap(entry['values']) > + for name, entry in data['definitions'].items() > + if entry['kind'] == kind > + } > + if definitions: > + compact[section] = definitions > + if data['groups']: > + compact['attr-groups'] = _compact_groups(data['groups']) > + if data['sets']: > + compact['attr-sets'] = { > + name: _FlowMap({ > + 'id': entry['set-id'], > + **({'stub': True} if entry.get('stub') else {}), > + }) > + for name, entry in data['sets'].items() > + } > + if data['fills']: > + compact['fills'] = { > + name: _compact_tree(nodes) > + for name, nodes in data['fills'].items() > + } > + if data['ops']: > + compact['ops'] = { > + name: _compact_op(entry) > + for name, entry in data['ops'].items() > + } > + return compact > + > + > +def _compact_groups(groups): > + compact = {} > + for name, group in groups.items(): > + compact[name] = { > + 'id': group['group-id'], > + 'attrs': { > + attr_name: _compact_attr(attr) > + for attr_name, attr in group['attrs'].items() > + }, > + } > + return compact > + > + > +def _compact_attr(attr): Undocumented and non-trivial: each attribute kind omits a different set of default values, and the omission rules must mirror the defaults that _expand_attr restores. Suggested docstring: """Convert one comparison-form attribute into its compact flow map. Writes the attribute as a single-line flow map starting with "id" (the local ID). The kind is abbreviated by key: enum and flags attributes are written as "enum: NAME" or "flags: NAME" naming their definition, blob attributes as "blob: DIRECTION", and every other kind as "type: KIND". Values equal to the defaults are left out, and _expand_attr restores them: uint/sint "min" and "max" are written only when they differ from the full range of the type (0..U64_MAX, S64_MIN..S64_MAX). "unit" is written when present, and "scale" only when unit is present and scale is nonzero. string "min-length" is written only when nonzero; "max-length" is always written. binary As string, except that equal min and max lengths collapse to a single "length". blob "backings" (as a flow sequence), "max-length", and "alignment" only when it is not 1. all kinds "max-occurrences" is written only when it is not 1, with the spec.UNLIMITED sentinel printed as the word "unlimited". The input is only read, never modified. Args: attr: Comparison-form attribute dict as built by _attr(), with keys local-id, kind, max-occurrences, and the kind-specific keys definition, min, max, min-length, max-length, unit, scale or blob. Returns: _FlowMap: The compact attribute, printed in flow style. Throws: KeyError: If a required key is missing for the kind, for example max-length on a string or binary attribute. build() always supplies it. """ > + kind = attr['kind'] > + compact = _FlowMap({'id': attr['local-id']}) > + if kind in ('enum', 'flags'): > + compact[kind] = attr['definition'] > + elif kind == 'blob': > + blob = attr['blob'] > + compact['blob'] = blob['direction'] > + compact['backings'] = _FlowSeq(blob['backings']) > + compact['max-length'] = blob['max-length'] > + if blob['alignment'] != 1: > + compact['alignment'] = blob['alignment'] > + else: > + compact['type'] = kind > + > + if kind in ('uint', 'sint'): > + low, high = ((0, spec.U64_MAX) if kind == 'uint' > + else (spec.S64_MIN, spec.S64_MAX)) > + minimum = attr.get('min', low) > + maximum = attr.get('max', high) > + if minimum != low: > + compact['min'] = minimum > + if maximum != high: > + compact['max'] = maximum > + if 'unit' in attr: > + compact['unit'] = attr['unit'] > + if attr['scale']: > + compact['scale'] = attr['scale'] > + elif kind in ('string', 'binary'): > + minimum = attr.get('min-length', 0) > + maximum = attr['max-length'] > + if kind == 'binary' and minimum == maximum: > + compact['length'] = maximum > + else: > + if minimum: > + compact['min-length'] = minimum > + compact['max-length'] = maximum > + > + maximum = attr['max-occurrences'] > + if maximum != 1: > + compact['max-occurrences'] = ( > + 'unlimited' if maximum == spec.UNLIMITED else maximum) > + return compact > + > + > +def _compact_op(op): Undocumented and non-trivial: only the ABI-relevant op properties are written, and the defaults dropped from the "abi" map (fd-access, availability, optional, pre-post) are easy to confuse with required fields. Suggested docstring: """Convert one comparison-form op into its compact snapshot entry. The entry is a dict with an "abi" flow map and optional trees. The flow map always holds "id" (the op ID) and "type". The other properties are written only when they differ from their defaults, and _expand_ops restores them: fd-access Omitted for events (which have none) and when equal to spec.DEFAULT_FD_ACCESS. availability Omitted when equal to spec.DEFAULT_AVAILABILITY. optional Written as true only when set. pre-post Written as true only when set. "request" is written only when the request tree is non-empty. "reply" is written whenever the reply is not None, so an op that has an empty reply schema still records "reply: {}", which keeps it distinct from an op with no reply schema (such as an ACK-only action). Both trees use the compact tree form from _compact_tree. The input is only read, never modified. Args: op: Comparison-form op dict as built by _ops(), with keys op-id, type, availability, optional, pre-post, request, reply and, for non-events, fd-access. Request and reply are node lists, or None. Returns: dict: Mapping with the key "abi" (a _FlowMap) and, when present, "request" and "reply" compact trees. """ > + abi = _FlowMap({ > + 'id': op['op-id'], > + 'type': op['type'], > + }) > + if (op['type'] != 'event' and > + op.get('fd-access') != spec.DEFAULT_FD_ACCESS): > + abi['fd-access'] = op.get('fd-access') > + availability = op.get('availability', spec.DEFAULT_AVAILABILITY) > + if availability != spec.DEFAULT_AVAILABILITY: > + abi['availability'] = availability > + if op.get('optional', False): > + abi['optional'] = True > + if op.get('pre-post', False): > + abi['pre-post'] = True > + compact = {'abi': abi} > + if op.get('request'): > + compact['request'] = _compact_tree(op['request']) > + if op.get('reply') is not None: > + compact['reply'] = _compact_tree(op['reply']) > + return compact > + > + > +def _compact_tree(nodes): > + required = [] > + optional = [] > + for node in nodes: > + destination = required if node['required'] else optional > + destination.append(_compact_node(node)) > + compact = {} > + if required: > + compact['attrs'] = _compact_nodes(required) > + if optional: > + compact['optional-attrs'] = _compact_nodes(optional) > + if compact and _simple_tree_length(compact) <= 54: 54 and 55 (here and in `_simple_tree_length()`, which also hardcodes "+ 2", "+ 4") are unexplained magic numbers tuned to the 80-column `_wrap_flow()` output. A comment on how they derive from the width, or computing from width, is needed; any change to the snapshot layout silently changes every checked-in snapshot. > + return _FlowMap(compact) > + return compact > + > + > +def _simple_tree_length(tree): > + length = 2 > + for index, (name, selectors) in enumerate(tree.items()): > + if not isinstance(selectors, _FlowSeq): > + return 55 > + if index: > + length += 2 > + length += len(name) + 4 > + length += sum(len(selector) + 2 for selector in selectors) > + return length > + > + > +def _compact_nodes(nodes): > + if all(isinstance(node, str) for node in nodes): > + return _FlowSeq(nodes) > + return nodes > + > + > +def _compact_node(node): > + name = node.get('attr', node.get('stub')) > + children = _compact_tree(node.get('children', [])) > + if 'reenters' in node: > + children['reenters'] = node['reenters'] > + if not children: > + return name > + return {name: children} > + > + > +def _expand(compact): Undocumented and non-trivial: It is the inverse of _compact and has to restore every omitted default so the result compares equal to build() output, which is not visible from the code alone. Suggested docstring: """Convert a loaded compact snapshot back into comparison form. This is the reading half of the compact ABI snapshot format; _compact is its inverse. COMPACT is the parsed YAML of a snapshot file. The result has the same shape as build() returns, so check() can pass it to _compare as the "old" side against the freshly built schema. Conventions undone here: the separate "enums" and "flags" sections are merged into one "definitions" dict with an explicit "kind"; "inherits" becomes "parent" (None when absent); the pre-init, post-init and pre-fini flags default to False when absent; and missing "attr-groups", "attr-sets", "fills" and "ops" sections become empty dicts. Groups, sets, fills and ops are expanded by _expand_groups, _expand_sets, _expand_tree and _expand_ops. The input is not modified, though the definition value maps are shared with it rather than copied. Args: compact: Parsed compact snapshot mapping (dict). It must contain "family" with "name" and "id". Returns: dict: Comparison-form family data with keys family, family-id, parent, pre-init, post-init, pre-fini, definitions, groups, sets, fills and ops. Throws: CtlvError: Propagated from the helpers when an attribute or tree node is malformed (see _expand_attr and _expand_node). KeyError: If "family", "name" or "id" is missing, or a group lacks "id" or "attrs". """ > + family = compact['family'] > + family_id = family['id'] > + definitions = {} > + for kind, section in (('enum', 'enums'), ('flags', 'flags')): > + for name, values in compact.get(section, {}).items(): > + definitions[name] = { > + 'kind': kind, > + 'values': values, > + } > + return { > + 'family': family['name'], > + 'family-id': family_id, > + 'parent': family.get('inherits'), > + 'pre-init': family.get('pre-init', False), > + 'post-init': family.get('post-init', False), > + 'pre-fini': family.get('pre-fini', False), > + 'definitions': definitions, > + 'groups': _expand_groups(compact.get('attr-groups', {})), > + 'sets': _expand_sets(compact.get('attr-sets', {})), > + 'fills': { > + name: _expand_tree(tree) > + for name, tree in compact.get('fills', {}).items() > + }, > + 'ops': _expand_ops(compact.get('ops', {})), > + } > + > + > +def _expand_groups(groups): > + expanded = {} > + for name, group in groups.items(): > + expanded[name] = { > + 'group-id': group['id'], > + 'attrs': { > + attr_name: _expand_attr(attr) > + for attr_name, attr in group['attrs'].items() > + }, > + } > + return expanded > + > + > +def _expand_attr(compact): Undocumented and non-trivial: It must infer the attribute kind from which of four mutually exclusive keys is present and then reinstate per-kind defaults that _compact_attr dropped. Suggested docstring: """Convert one compact attribute flow map back into comparison form. Reverses _compact_attr. Exactly one of the keys "type", "enum", "flags" or "blob" must be present, and it determines the kind: for "type" the kind is the key's value (for example uint or string), for the others the kind is the key name itself. Omitted defaults are restored: max-occurrences 1 when absent; the word "unlimited" becomes the spec.UNLIMITED sentinel. enum/flags "definition" is the named definition. uint/sint "min" and "max" default to the full range of the type; "unit" and "scale" are set only when "unit" is present, with scale defaulting to 0. string/binary A "length" key sets both min-length and max-length; otherwise min-length defaults to 0 and max-length is required. blob A "blob" dict is built with direction, backings (copied to a list), max-length and alignment (default 1). The input is not modified. Kinds not listed (for example other scalar types) get only local-id, kind and max-occurrences. Args: compact: Parsed compact attribute mapping (dict) with an "id" key. Returns: dict: Comparison-form attribute in the shape built by _attr(). Throws: CtlvError: If the number of present keys among type, enum, flags and blob is not exactly one. KeyError: If a key required for the kind is missing, such as "id", "max-length" for string or binary, or "backings" for blob. """ > + primary = [name for name in ('type', 'enum', 'flags', 'blob') > + if name in compact] > + if len(primary) != 1: > + raise CtlvError('ABI snapshot attr', > + 'expected one of type, enum, flags or blob') > + kind = compact[primary[0]] if primary[0] == 'type' else primary[0] > + attr = { > + 'local-id': compact['id'], > + 'kind': kind, > + 'max-occurrences': compact.get('max-occurrences', 1), > + } > + if attr['max-occurrences'] == 'unlimited': > + attr['max-occurrences'] = spec.UNLIMITED > + if kind in ('enum', 'flags'): > + attr['definition'] = compact[kind] > + elif kind in ('uint', 'sint'): > + low, high = ((0, spec.U64_MAX) if kind == 'uint' > + else (spec.S64_MIN, spec.S64_MAX)) > + attr['min'] = compact.get('min', low) > + attr['max'] = compact.get('max', high) > + if 'unit' in compact: > + attr['unit'] = compact['unit'] > + attr['scale'] = compact.get('scale', 0) > + elif kind in ('string', 'binary'): > + if 'length' in compact: > + attr['min-length'] = compact['length'] > + attr['max-length'] = compact['length'] > + else: > + attr['min-length'] = compact.get('min-length', 0) > + attr['max-length'] = compact['max-length'] > + elif kind == 'blob': > + attr['blob'] = { > + 'direction': compact['blob'], > + 'backings': list(compact['backings']), > + 'max-length': compact['max-length'], > + 'alignment': compact.get('alignment', 1), > + } > + return attr > + > + > +def _expand_sets(attr_sets): > + return { > + name: { > + 'set-id': entry['id'], > + **({'stub': True} if entry.get('stub') else {}), > + } > + for name, entry in attr_sets.items() > + } > + > + > +def _expand_ops(ops): Undocumented and non-trivial: It restores several omitted defaults and distinguishes an absent reply or request (None) from an empty one, with the rules differing for events. Suggested docstring: """Convert the compact "ops" section back into comparison-form ops. Reverses _compact_op for every op. Each entry's "abi" map supplies op-id and type, and the omitted properties are restored: availability defaults to spec.DEFAULT_AVAILABILITY, and optional and pre-post to False. Requests and replies need care because None and an empty tree mean different things to the checker: request None for an event; otherwise the expanded tree, which is an empty list if the snapshot has no "request" key. reply None when the snapshot has no "reply" key; otherwise the expanded tree, even if it is empty. "fd-access" is set only for non-event ops, defaulting to spec.DEFAULT_FD_ACCESS, matching what _ops() produces. The input is not modified. Args: ops: The compact "ops" mapping (dict) of op name to a dict with an "abi" flow map and optional "request" and "reply" trees. Returns: dict[str, dict]: Op name to comparison-form op dict with keys op-id, type, availability, optional, pre-post, request, reply and (non-events) fd-access. Throws: CtlvError: Propagated from _expand_tree if a tree node is malformed. KeyError: If an op lacks "abi", or its "abi" map lacks "id" or "type". """ > + expanded = {} > + for name, op in ops.items(): > + abi = op['abi'] > + op_type = abi['type'] > + entry = { > + 'op-id': abi['id'], > + 'type': op_type, > + 'availability': abi.get( > + 'availability', spec.DEFAULT_AVAILABILITY), > + 'optional': abi.get('optional', False), > + 'pre-post': abi.get('pre-post', False), > + 'request': (None if op_type == 'event' > + else _expand_tree(op.get('request', {}))), > + 'reply': (None if 'reply' not in op > + else _expand_tree(op['reply'])), > + } > + if op_type != 'event': > + entry['fd-access'] = abi.get( > + 'fd-access', spec.DEFAULT_FD_ACCESS) > + expanded[name] = entry > + return expanded > + > + > +def _expand_tree(tree): > + nodes = [] > + for required, key in ((True, 'attrs'), (False, 'optional-attrs')): > + for selector in tree.get(key, []): > + nodes.append(_expand_node(selector, required)) > + return nodes > + > + > +def _expand_node(selector, required): Undocumented and non-trivial: a single YAML item can be a bare name, a one-key mapping, a stub or an attribute, and the stub/attribute distinction rests on whether the name contains a dot. Suggested docstring: """Convert one compact tree selector into a comparison-form tree node. A selector is either a bare string (a leaf, no children) or a mapping with exactly one key whose value holds that node's nested "attrs", "optional-attrs" and "reenters". This reverses _compact_node. The kind of node is decided by the name alone. Attribute names are fully qualified and always contain a "."; a name without one is a stub reference. A stub becomes {'stub': name, 'required': REQUIRED} and any nested content is ignored, since stubs have no children in the comparison form. An attribute becomes {'attr': name, 'required': REQUIRED}, gaining "children" (expanded recursively through _expand_tree) only when there are any, and "reenters" when the nested mapping carries it. REQUIRED comes from which list the selector was found in, not from the selector itself. Args: selector: A str, or a dict with exactly one item mapping the name to its nested tree mapping. required: True when the selector came from an "attrs" list and False when it came from "optional-attrs". Returns: dict: Node with key "stub" or "attr", "required", and optionally "children" (list of nodes) and "reenters". Throws: CtlvError: If the selector is neither a string nor a dict with exactly one key. """ > + if isinstance(selector, str): > + name = selector > + nested = {} > + else: > + if not isinstance(selector, dict) or len(selector) != 1: > + raise CtlvError('ABI snapshot tree', > + 'expected an attr name or one nested attr') > + name, nested = next(iter(selector.items())) > + if '.' not in name: > + return {'stub': name, 'required': required} > + node = {'attr': name, 'required': required} > + children = _expand_tree(nested) > + if children: > + node['children'] = children > + if 'reenters' in nested: > + node['reenters'] = nested['reenters'] > + return node > + > + > +def registry_build(registry): > + """Return the registry in the same normalized form as its source.""" > + families = [] > + for entry in registry.entries: > + if entry.deprecated: > + families.append({'name': entry.name, 'deprecated': True}) > + else: > + families.append(entry.name) > + return {'families': families} > + > + > +def registry_render(registry): > + body = yaml.dump(registry_build(registry), Dumper=_Dumper, > + sort_keys=False, default_flow_style=False, width=76) > + return REGISTRY_HEADER + body > + > + > +def registry_problems(old, current, exact): Undocumented and non-trivial: the positional ID-by-index comparison, the elif that hides the deprecation check on a rename, and the exact-only "needs updating" fallback are not obvious from the code. Suggested docstring: """List ABI breaks between the baseline registry and the current one. The registry is append-only and a family's ID is its index in entries, so the entries are compared position by position, walking only the baseline's IDs. Entries that exist only in current (new families appended at the end) are never a break. For each baseline ID the check reports, in order of ID: the ID no longer exists (current is shorter); the name at that ID changed; or the name is unchanged but the baseline entry was deprecated and the current one is not (the ID was reused after deprecation). A rename is reported instead of, not in addition to, the deprecation check. Becoming deprecated is allowed. If exact is true and no break was found, the two entry tuples are also compared for equality; any remaining difference (an appended family or a newly deprecated one) is reported as a stale baseline, which is not an ABI break but still fails the check. Nothing is mutated. Called by registry_check, which passes exact=False from the tree-level check that only needs compatibility (lib/tree.py) and the default True otherwise. Args: old: spec.Registry loaded from the checked-in baseline (paths.registry_abi). current: spec.Registry loaded from the live registry source. exact: If true, also report any difference from the baseline once no incompatibility has been found. Returns: list[str]: One message per problem, empty if compatible. Formats: "family ID N (NAME) was removed", "family ID N was OLD, now NEW", "family ID N (NAME) was reused after deprecation", and "the registry ABI baseline needs updating" (at most once, only when no other problem exists). """ > + problems = [] > + for family_id, was in enumerate(old.entries): > + if family_id >= len(current.entries): > + problems.append(f'family ID {family_id} ({was.name}) was removed') > + continue > + now = current.entries[family_id] > + if now.name != was.name: > + problems.append(f'family ID {family_id} was {was.name}, now ' > + f'{now.name}') > + elif was.deprecated and not now.deprecated: > + problems.append(f'family ID {family_id} ({was.name}) was reused ' > + 'after deprecation') > + if exact and not problems and old.entries != current.entries: > + problems.append('the registry ABI baseline needs updating') > + return problems > + > + > +def registry_check(paths, registry, exact=True): > + baseline = spec.load_registry(paths, paths.registry_abi) > + problems = registry_problems(baseline, registry, exact) > + if problems: > + raise CtlvError(f'{paths.registry} against {paths.registry_abi}', > + 'the family registry is not compatible with its ' > + 'baseline\n' + > + '\n'.join(f' {problem}' for problem in problems)) > + > + > +def _definitions(items): > + definitions = {} > + for definition in items: > + definitions[definition.name] = { > + 'kind': definition.kind, > + 'values': dict(definition.values), > + } > + return definitions > + > + > +def _groups(items): > + groups = {} > + for group in items: > + groups[group.name] = { > + 'group-id': group.group_id, > + 'attrs': {attr.name: _attr(attr) for attr in group.attrs}, > + } > + return groups > + > + > +def _attr(attr): Undocumented and non-trivial: It flattens an attribute declaration into the snapshot's comparison dict, where absent optional fields are omitted rather than None and the dict keys must line up with what _expand_attr and _compare_attr read. Suggested docstring: """Convert one attribute into its comparison-form dict. The result has the same shape that _expand_attr produces from a checked-in snapshot, so build() output and an expanded snapshot can be compared directly. Keys: "local-id" and "kind" always; "definition" (the enum or flags name from decl.ref) when the attribute refers to one; each of "min", "max", "min-length", "max-length", "unit" and "scale" only when its declaration value is not None (an unset bound is omitted, not stored as None); "blob" (direction, backings copied to a list, max-length, alignment) for blob attributes; and "max-occurrences" always, where spec.UNLIMITED is stored as its numeric value. The attribute is not modified. Args: attr: A spec attribute inside a local attr group, providing local_id and a decl with kind, ref, min, max, min_length, max_length, unit, scale, blob and max_occurrences. Returns: dict: The comparison-form entry described above. """ > + decl = attr.decl > + entry = {'local-id': attr.local_id, 'kind': decl.kind} > + if decl.ref is not None: > + entry['definition'] = decl.ref > + for name, value in (('min', decl.min), ('max', decl.max), > + ('min-length', decl.min_length), > + ('max-length', decl.max_length), > + ('unit', decl.unit), ('scale', decl.scale)): > + if value is not None: > + entry[name] = value > + if decl.blob is not None: > + entry['blob'] = { > + 'direction': decl.blob.direction, > + 'backings': list(decl.blob.backings), > + 'max-length': decl.blob.max_length, > + 'alignment': decl.blob.alignment, > + } > + entry['max-occurrences'] = decl.max_occurrences > + return entry > + > + > +def _sets(attr_sets): > + sets = {} > + for attr_set in attr_sets: > + entry = { > + 'set-id': attr_set.set_id, > + } > + if attr_set.stub: > + entry['stub'] = True > + sets[attr_set.name] = entry > + return sets > + > + > +def _fills(fills): > + return {fill.name: _tree(fill.root) for fill in fills} > + > + > +def _ops(items): Undocumented and non-trivial: It records each op's ABI-visible properties and request/reply trees, with several None conventions (absent reply, event ops without fd-access) that _compare_ops depends on. Suggested docstring: """Convert a family's own ops into comparison form, keyed by op name. Each entry carries "op-id" (the op's local_id), "type", "availability", "optional", "pre-post", and the "request" and "reply" trees as produced by _tree. A tree is None when the op has no such schema (_tree(None)), which is distinct from an empty list, which means a schema with no attributes; _compare_ops relies on that distinction. "fd-access" is recorded only when the op type is not "event", so event entries have no such key. Only the ops passed in are described, which build() supplies as family.local_ops (ops declared by this family, not inherited ones). Args: items: Iterable of spec ops, each with name, local_id, type, availability, optional, pre_post, fd_access, request and reply (a tree root or None). Returns: dict[str, dict]: Op name mapped to its comparison-form entry, in declaration order. """ > + ops = {} > + for op in items: > + entry = { > + 'op-id': op.local_id, > + 'type': op.type, > + 'availability': op.availability, > + 'optional': op.optional, > + 'pre-post': op.pre_post, > + 'request': _tree(op.request), > + 'reply': _tree(op.reply), > + } > + if op.type != 'event': > + entry['fd-access'] = op.fd_access > + ops[op.name] = entry > + return ops > + > + > +def _tree(root): > + if root is None: > + return None > + children = [_node(child) for child in root.children()] > + children += [_stub(stub) for stub in root.stubs] > + return children > + > + > +def _node(node): > + entry = {'attr': node.attr.full_name, 'required': node.required} > + if node.reenters is not None: > + # The members are recorded above, so record where they come from. > + entry['reenters'] = node.reenters > + return entry > + children = _tree(node) > + if children: > + entry['children'] = children > + return entry > + > + > +def _stub(stub): > + return {'stub': stub.name, 'required': not stub.optional} > + > + > +def _compare(old, new): Undocumented and non-trivial: It is the top-level driver whose contract (what counts as a family-level break, the order of problems, and which sub-checks run) is spread over six helpers. Suggested docstring: """Report every way the new family ABI breaks the old one. old is the snapshot (expanded from the checked-in file by _expand) and new is the current specification (from build()); both are in comparison form. Anything that exists in old must still exist in new with the same identity; additions are allowed, subject to the rules in the helpers (a new attribute placement must be optional). Family-level checks done here, each reporting a problem on any change: the family name, the family-id, and the parent (reparenting changes every symbol name, so it is a break), and each of the pre-init, post-init and pre-fini flags, where a key missing from the snapshot counts as False. It then delegates to _compare_definitions, _compare_groups, _compare_sets, _compare_fills and _compare_ops, in that order, which append to the same problem list. Nothing is raised here; check() turns a non-empty result into a CtlvError. The inputs are not modified. Args: old: dict, the snapshot-side comparison form. new: dict, the current-specification comparison form. Returns: list[str]: Human-readable problems, empty when compatible. Family-level formats are "family was renamed from A to B", "family-id was A, now B", "the inherited family was A, now B" and "family: KEY was A, now B". """ > + problems = [] > + if old['family'] != new['family']: > + problems.append(f'family was renamed from {old["family"]} to ' > + f'{new["family"]}') > + if old['family-id'] != new['family-id']: > + problems.append(f'family-id was {old["family-id"]}, now ' > + f'{new["family-id"]}') > + # A name carries its ancestry, so reparenting renames every symbol. > + if old['parent'] != new['parent']: > + problems.append(f'the inherited family was {old["parent"]}, now ' > + f'{new["parent"]}') > + for key in ('pre-init', 'post-init', 'pre-fini'): > + if old.get(key, False) != new[key]: > + problems.append(f'family: {key} was {old.get(key, False)}, now ' > + f'{new[key]}') > + _compare_definitions(problems, old['definitions'], new['definitions']) > + _compare_groups(problems, old['groups'], new['groups']) > + _compare_sets(problems, old['sets'], new['sets']) > + _compare_fills(problems, old['fills'], new['fills']) > + _compare_ops(problems, old['ops'], new['ops']) > + return problems > + > + > +def _compare_definitions(problems, old, new): Undocumented and non-trivial: It defines which enum and flags changes count as breaks (removal, kind switch, value removal or renumbering) while silently allowing new values. Suggested docstring: """Append problems for enum and flags definitions that were broken. Walks the definitions in old. A definition missing from new is reported as removed. For one that remains, a change of kind (enum versus flags) is reported, then each value in old is checked: removed values and values whose number changed are reported. Values and whole definitions that exist only in new are compatible and ignored. After a kind change the value checks still run, so one definition can yield several problems. problems is mutated in place. Args: problems: list[str] collecting messages; appended to. old: dict mapping definition name to {"kind": "enum" or "flags", "values": {value name: number}} from the snapshot. new: dict of the same shape from the current specification. Returns: None: Results are appended to problems. Messages look like "enum NAME was removed", "enum NAME is now a flags definition", "enum NAME: value V was removed" and "enum NAME: value V was A, now B" ("flags" replaces "enum" for flags definitions, based on the old kind). """ > + for name, was in old.items(): > + now = new.get(name) > + if now is None: > + problems.append(f'{was["kind"]} {name} was removed') > + continue > + where = f'{was["kind"]} {name}' > + if now['kind'] != was['kind']: > + problems.append(f'{where} is now a {now["kind"]} definition') > + for value, number in was['values'].items(): > + if value not in now['values']: > + problems.append(f'{where}: value {value} was removed') > + elif now['values'][value] != number: > + problems.append(f'{where}: value {value} was {number}, now ' > + f'{now["values"][value]}') > + > + > +def _compare_groups(problems, old, new): Undocumented and non-trivial: It must check both the group identity and each attribute inside it, and the attribute-level messages use a different prefix ("attr G.A") from the group-level ones. Suggested docstring: """Append problems for attr groups and their attributes that were broken. Walks the groups in old. A group missing from new is reported as removed (its attributes are not reported separately). For a remaining group, a changed group-id is reported. Then every attribute of the old group is looked up by name in the new group: a missing one is reported as removed, and a present one is compared in detail by _compare_attr. Groups and attributes that exist only in new are compatible and ignored. problems is mutated in place. Args: problems: list[str] collecting messages; appended to. old: dict mapping group name to {"group-id": int, "attrs": {attr name: comparison-form attr}} from the snapshot. new: dict of the same shape from the current specification. Returns: None: Results are appended to problems. Messages look like "attr group G was removed", "attr group G: group-id was A, now B" and "attr G.A was removed"; detailed ones come from _compare_attr with the location "attr G.A". """ > + for name, was in old.items(): > + now = new.get(name) > + if now is None: > + problems.append(f'attr group {name} was removed') > + continue > + where = f'attr group {name}' > + if was['group-id'] != now['group-id']: > + problems.append(f'{where}: group-id was {was["group-id"]}, now ' > + f'{now["group-id"]}') > + for attr, attr_was in was['attrs'].items(): > + attr_now = now['attrs'].get(attr) > + if attr_now is None: > + problems.append(f'attr {name}.{attr} was removed') > + continue > + _compare_attr(problems, f'attr {name}.{attr}', attr_was, attr_now) > + > + > +def _compare_attr(problems, where, was, now): Undocumented and non-trivial: identity keys must match exactly while bounds are only a break when narrowed, and the min/max comparisons use different directions and a default of 0 for a missing key. Suggested docstring: """Append problems for one attribute whose declaration changed. Applies three kinds of rule to the old (was) and new (now) comparison-form entries produced by _attr or _expand_attr: Exact identity: "local-id", "kind", "definition", "unit" and "scale" must be equal; a missing key is None, so adding or dropping a unit or definition is also reported. Bounds may widen but not narrow, checked only for keys present in was. A lower bound ("min", "min-length") is narrowed if the new value is greater than the old; an upper bound ("max", "max-length") is narrowed if the new value is smaller than the old. A key absent from now is read as 0, which reports an upper bound that disappeared and misreads a negative signed "min" that disappeared. Because a kind change is reported separately, bound messages can accompany it. Then max-occurrences is checked by _compare_occurrences, and if was has a "blob" entry, _compare_blob compares it with now's blob (an empty dict if now has none). Blob details are only checked when the old attribute was a blob. problems is mutated in place. Args: problems: list[str] collecting messages; appended to. where: str location prefix, such as "attr G.A", used in every message. was: dict, the old comparison-form attribute. now: dict, the new comparison-form attribute. Returns: None: Results are appended to problems. Messages look like "WHERE: KEY was A, now B" for identity keys and "WHERE: KEY was narrowed from A to B" for bounds. """ > + for key in ('local-id', 'kind', 'definition', 'unit', 'scale'): > + if was.get(key) != now.get(key): > + problems.append(f'{where}: {key} was {was.get(key)}, now ' > + f'{now.get(key)}') > + for key in ('min', 'min-length'): > + if key in was and now.get(key, 0) > was[key]: > + problems.append(f'{where}: {key} was narrowed from {was[key]} to ' > + f'{now[key]}') > + for key in ('max', 'max-length'): > + if key in was and now.get(key, 0) < was[key]: > + problems.append(f'{where}: {key} was narrowed from {was[key]} to ' > + f'{now[key]}') Bug (crash instead of diagnostic): if an attr changes kind (e.g. snapshot has a uint with min/max present, new schema has a string), then now.get('max', 0) < was['max'] is true because the new entry has no 'max', and the message then evaluates now[key] -> KeyError, which is not a CtlvError and so escapes `main()` as a traceback. The 'kind' mismatch has already been appended to problems but is never reported. Same for min-length/max-length (and 'min' for the other direction). Compare only when both sides have the key and the kinds match. > + _compare_occurrences(problems, where, was, now) > + if 'blob' in was: > + _compare_blob(problems, where, was['blob'], now.get('blob', {})) > + > + > +def _compare_occurrences(problems, where, was, now): > + old_max = was['max-occurrences'] > + new_max = now['max-occurrences'] > + if old_max == new_max: > + return > + if old_max == spec.UNLIMITED or new_max < old_max: > + problems.append(f'{where}: max-occurrences was narrowed from ' > + f'{_occurrences(old_max)} to {_occurrences(new_max)}') Bug: spec.UNLIMITED is 0, so `new_max < old_max` is true when a finite max-occurrences (say 4) becomes unlimited (0 < 4). Widening to unlimited is reported as "narrowed from 4 to unlimited". The sentinel needs to be mapped to infinity before comparing. > + > + > +def _compare_blob(problems, where, was, now): > + for key in ('direction', 'alignment'): > + if was.get(key) != now.get(key): > + problems.append(f'{where}: blob {key} was {was.get(key)}, now ' > + f'{now.get(key)}') > + for backing in was.get('backings', []): > + if backing not in now.get('backings', []): > + problems.append(f'{where}: blob backing {backing} was removed') > + if now.get('max-length', 0) < was.get('max-length', 0): > + problems.append(f'{where}: blob max-length was narrowed from ' > + f'{was["max-length"]} to {now["max-length"]}') > + > + > +def _compare_sets(problems, old, new): > + for name, was in old.items(): > + now = new.get(name) > + if now is None: > + problems.append(f'attr set {name} was removed') > + continue > + for key in ('set-id', 'stub'): > + was_value = was.get(key, False) > + now_value = now.get(key, False) > + if was_value != now_value: > + problems.append(f'attr set {name}: {key} was {was_value}, ' > + f'now {now_value}') > + > + > +def _compare_fills(problems, old, new): > + for name, was in old.items(): > + now = new.get(name) > + if now is None: > + problems.append(f'stub fill {name} was removed') > + continue > + _compare_nodes(problems, f'stub fill {name}', was, now) > + > + > +def _compare_ops(problems, old, new): Undocumented and non-trivial: besides the per-op ABI fields it has special- case rules for ACK-only actions and for removed versus newly added request/reply schemas that are easy to misread. Suggested docstring: """Append problems for ops that were removed or changed. Walks the ops in old. A missing op is reported as removed. Otherwise the fields "op-id", "type", "fd-access", "availability", "optional" and "pre-post" must be exactly equal (any change either way is a break, since userspace compiles against whether an op may be disabled or withheld, not against whether it currently is). A missing key such as fd-access on an event reads as None on both sides. Then the request and reply schemas are examined. For a reply on an op that was an "action" with reply None (ACK-only), a new non-None reply is a break. For anything else, a schema that was present and is now None is reported as removed; if both are present they are compared with _compare_nodes. A schema that was None and became present is otherwise allowed. Ops that exist only in new are ignored. problems is mutated in place. Args: problems: list[str] collecting messages; appended to. old: dict mapping op name to its comparison-form entry (see _ops) from the snapshot. new: dict of the same shape from the current specification. Returns: None: Results are appended to problems. Messages look like "op NAME was removed", "op NAME: KEY was A, now B", "op NAME: an ACK-only action gained a reply schema" and "op NAME: the request schema was removed" ("reply" likewise); tree problems come from _compare_nodes with location "op NAME request" or "op NAME reply". """ > + for name, was in old.items(): > + now = new.get(name) > + if now is None: > + problems.append(f'op {name} was removed') > + continue > + where = f'op {name}' > + # Whether an op may be disabled or withheld is what userspace compiles > + # against; whether it is either right now is per device. > + for key in ('op-id', 'type', 'fd-access', > + 'availability', 'optional', 'pre-post'): > + if was.get(key) != now.get(key): > + problems.append(f'{where}: {key} was {was.get(key)}, now ' > + f'{now.get(key)}') See the pre-post comment on the meta-schema: this makes a kernel-only knob an ABI property. Also 'optional'/'availability' compare fine, but the comparison ignores suggested-reply-length (kernel-only, fine) and the fill membership of stubs (ok), so "pre-post" is the odd one out. > + for what in ('request', 'reply'): > + if (what == 'reply' and was['type'] == 'action' and > + was[what] is None and now[what] is not None): > + problems.append(f'{where}: an ACK-only action gained a ' > + 'reply schema') > + elif was[what] is not None and now[what] is None: > + problems.append(f'{where}: the {what} schema was removed') > + elif was[what] is not None: > + _compare_nodes(problems, f'{where} {what}', was[what], > + now[what]) > + > + > +def _compare_nodes(problems, where, old, new): Undocumented and non-trivial: It recursively compares attribute placement trees keyed by (kind, name), with rules for removal, optional-to-required, re- entry and new nodes that are easy to get wrong. Suggested docstring: """Append problems for a placement tree that lost or tightened nodes. old and new are lists of tree nodes in comparison form, as built by _tree or _expand_tree for an op request, an op reply, a nested node's children, or a stub fill. A node is either an attribute placement ({"attr": full name, "required": bool, optional "children", optional "reenters"}) or a stub ({"stub": name, "required": bool}). Nodes are matched by (kind, name), so a stub and an attribute with the same name are different nodes, and order is ignored. For each old node: if there is no match in new it is reported as removed and its children are not examined. Otherwise it is a break if it became required (optional to required is a break; required to optional is fine), if its "reenters" value differs, and its children are compared recursively with a location extended by the node's name. After that, any node that exists only in new and is required is a break; new optional nodes are compatible. problems is mutated in place; new itself is not changed (a local dict is consumed). Args: problems: list[str] collecting messages; appended to. where: str location prefix, such as "op NAME request" or "stub fill NAME", extended with the node name when recursing. old: list[dict], the snapshot nodes. new: list[dict], the current nodes. Returns: None: Results are appended to problems. Messages look like "WHERE: A was removed", "WHERE: A became required", "WHERE: A re-entered X, now Y" and "WHERE: new A must be optional", where A is the full attr name or "stub NAME". """ > + by_name = {_node_key(node): node for node in new} > + for was in old: > + key = _node_key(was) > + name = _node_name(was) > + now = by_name.pop(key, None) > + if now is None: > + problems.append(f'{where}: {name} was removed') > + continue > + if now['required'] and not was['required']: > + problems.append(f'{where}: {name} became required') > + if was.get('reenters') != now.get('reenters'): > + problems.append(f'{where}: {name} re-entered ' > + f'{was.get("reenters")}, now ' > + f'{now.get("reenters")}') > + _compare_nodes(problems, f'{where} {name}', > + was.get('children', []), now.get('children', [])) > + for node in by_name.values(): > + if node['required']: > + problems.append( > + f'{where}: new {_node_name(node)} must be optional') > + > + > +def _node_key(node): > + if 'attr' in node: > + return 'attr', node['attr'] > + return 'stub', node['stub'] > + > + > +def _node_name(node): > + if 'attr' in node: > + return node['attr'] > + return f'stub {node["stub"]}' > + > + > +def _occurrences(max_occurrences): > + if max_occurrences == spec.UNLIMITED: > + return 'unlimited' > + return str(max_occurrences) > diff --git a/tools/ctlv/ctlvgen/lib/errors.py b/tools/ctlv/ctlvgen/lib/errors.py > new file mode 100644 > index 000000000000..a165218a381e > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/errors.py > @@ -0,0 +1,14 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""The single error type reported by the CTLV schema tool.""" > + > + > +class CtlvError(Exception): > + """A specification, registry or ABI snapshot is not acceptable. > + > + The test suite matches diagnostics, so a message's wording is contract. > + """ > + > + def __init__(self, where, what): > + self.where = where > + self.what = what > + super().__init__(f'{where}: {what}' if where else what) > diff --git a/tools/ctlv/ctlvgen/lib/gen_family.py b/tools/ctlv/ctlvgen/lib/gen_family.py > new file mode 100644 > index 000000000000..92309334f979 > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/gen_family.py > @@ -0,0 +1,347 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""The descriptor of one family: who answers the wire. > + > +One struct ctlv_family and one struct ctlv_op per action or query > +this family declares, plus the helpers a device uses to decide them. A family > +answers what it declared: an inherited op is answered through the descriptor > +of the family that declared it, by that family implementation. > +""" > + > +from __future__ import annotations > + > +import dataclasses > + > +from . import cgen > +from . import spec > + > +# The family's priv, the request/reply, and a context for core helpers. > +CALLBACK_PARAMS = ('void *priv', 'struct ctlv_op_ctx *ctx', > + 'const struct ctlv_cursor *request', > + 'struct ctlv_cursor *reply') > + > +# What a hook takes, and what each half returns: a pre may refuse the op. > +HOOK_PARAMS = ('void *priv', 'struct ctlv_op_ctx *ctx') > +HOOK_RETURNS = (('pre', 'int'), ('post', 'void')) > + > +# How every family prepares, initializes and finalizes its priv. > +LIFECYCLE_HOOKS = ( > + ('priv_size', 'size_t', ('void',)), > + ('pre_init', 'const void', ('const void *init_info',)), > + ('init', 'int', ('struct ctlv_device *ctlvdev', > + 'const void *init_info', 'void *priv')), > + ('post_init', 'int', ('struct ctlv_device *ctlvdev', 'void *priv')), > + ('pre_fini', 'void', ('struct ctlv_device *ctlvdev', 'void *priv')), > + ('fini', 'void', ('struct ctlv_device *ctlvdev', 'void *priv')), > +) > +OPTIONAL_LIFECYCLE_HOOKS = ('pre_init', 'post_init', 'pre_fini') > + > +# The reply-side callback for an extension point. > +STUB_HOOKS = ( > + ('fill', 'int', ('void *priv', 'struct ctlv_op_ctx *ctx', > + 'const struct ctlv_cursor *request', > + 'struct ctlv_cursor *nest')), > +) > + > +# The state helpers of a dynamically available op. > +STATES = ('enable', 'disable') > + > + > +@dataclasses.dataclass(frozen=True) > +class ReaderScope: > + name: tuple > + node: object > + tree: str > + node_index: int > + > + > +INPUT_MACROS = { > + 'uint': 'U64', > + 'sint': 'S64', > + 'bool': 'BOOL', > + 'string': 'STRING', > + 'binary': 'BINARY', > + 'enum': 'U64', > + 'flags': 'U64', > + 'nested': 'NEST', > + 'blob': 'BLOB', > +} `INPUT_MACROS['blob']` is dead: the only lookup (line 2776) is guarded by `cgen.attr_getter_value()`, which is False for blobs (they take the `CTLV_IN_[EACH_]BLOB` branch on line 2772). `INPUT_MACROS` also repeats the same mapping as `PUT_MACROS` for seven kinds; one table plus 'nested' would do. > + > +PUT_MACROS = { > + 'uint': 'U64', > + 'sint': 'S64', > + 'bool': 'BOOL', > + 'string': 'STRING', > + 'binary': 'BINARY', > + 'enum': 'U64', > + 'flags': 'U64', > +} > + > + > +def bindings(family): > + """The ops family binds, in the order it binds them. > + > + Its own actions and queries, in declaration order. An op it inherited is > + answered by the family that declared it, and an event by nobody. > + """ > + return tuple(op for op in family.local_ops if op.type != 'event') > + > + > +def _implementer(family): > + """Who writes the callbacks of a family.""" > + if family.name == spec.CORE_FAMILY_NAME: > + return 'the CTLV core' > + return 'this family' > + > + > +def _macro(writer, name, args): > + writer.parenthesized(f'{name}(', args, ')') > + > + > +def emit_header(writer, names, family, reader_scopes): > + """Append the family API to a generated kernel header.""" > + bound = bindings(family) > + if bound: > + writer.comment(f'Implemented by {_implementer(family)}, one per ' > + 'action and query it declares.') > + for op in bound: > + writer.signature('int', names.callback(op), CALLBACK_PARAMS) > + if bound: > + writer.line() > + _hooks(writer, names, family) > + _lifecycle_hooks(writer, names, family) > + _stub_hooks(writer, names, family) > + writer.line() > + writer.declaration('extern const struct ctlv_family', > + f'{names.descriptor()};') > + writer.line() > + writer.line('#ifndef _CTLV_GENERATED_METADATA_ONLY') > + _helpers(writer, names, family) > + _readers(writer, names, reader_scopes) > + _writers(writer, names, family) > + _creators(writer, names, family) > + writer.line('#endif /* _CTLV_GENERATED_METADATA_ONLY */') > + writer.line() > + > + > +def _hooks(writer, names, family): > + """The pair around every op, and the pair around one op.""" > + if family.name == spec.CORE_FAMILY_NAME: > + writer.comment('Root-family hooks around every core action or query.') > + else: > + writer.comment('Family-wide hooks around every effective non-core op.') > + for which, returns in HOOK_RETURNS: > + writer.signature(returns, names.family_hook(which), HOOK_PARAMS) > + > + marked = [op for op in bindings(family) if op.pre_post] > + if not marked: > + return > + writer.line() > + writer.comment('Per-op hooks nested inside the family-wide pair.') > + for first, op in enumerate(marked): > + if first: > + writer.line() > + for which, returns in HOOK_RETURNS: > + writer.signature(returns, names.op_hook(op, which), HOOK_PARAMS) > + > + > +def _declared_lifecycle_hooks(family): > + return tuple(hook for hook in LIFECYCLE_HOOKS > + if hook[0] not in OPTIONAL_LIFECYCLE_HOOKS or > + getattr(family, hook[0])) > + > + > +def _lifecycle_hooks(writer, names, family): > + """The generated lifecycle of this family's priv.""" > + writer.line() > + writer.comment('Core-allocated family priv lifecycle.') > + for which, returns, params in _declared_lifecycle_hooks(family): > + name = names.lifecycle_hook(which) > + > + writer.signature(returns, f'*{name}' if which == 'pre_init' else name, > + params) > + > + > +def _stub_hooks(writer, names, family): > + """What this family puts into each stub it fills.""" > + if not family.fills: > + return > + for fill in family.fills: > + owner = family.owners[fill.family_id] Redundant: `if not family.fills: return` - the for loop is a no-op for an empty tuple and no per-family output precedes it. > + writer.line() > + writer.comment(f'{fill.name} of {owner}: fill callback.') > + for which, returns, params in STUB_HOOKS: > + writer.signature(returns, names.stub_callback(fill, which), > + params) > + `STUB_HOOKS` has a single entry and is only looped over here. The only other user of `names.stub_callback()` is the reserved-name list in `cgen.symbols()`, which reserves a 'read' callback that nothing emits. A loop over a one-entry table, plus a reserved name for a callback that does not exist, is speculative generality. > + > +def _helpers(writer, names, family): > + """What one device may decide about each op this family declares.""" > + ops = [op for op in family.local_ops if op.type != 'event' and > + (op.optional or op.availability == 'dynamic')] > + if not ops: > + return > + writer.line() > + writer.comment('Optional publication and dynamic availability helpers.') > + for op in ops: > + if op.optional: > + _macro(writer, 'CTLV_OP_PROVIDE', > + (names.op_ident(op), names.op_id(op))) > + if op.availability == 'dynamic': > + for state in STATES: > + _macro(writer, f'CTLV_OP_{state.upper()}', > + (names.op_ident(op), names.op_id(op))) > + > + > +def _readers(writer, names, scopes): > + """Placement-specific indexed access to callback requests.""" > + if not any(scope.node.children() for scope in scopes): > + return > + writer.line() > + writer.comment('Reading a validated message. A blob has a resolver ' > + 'instead of a getter.') > + for scope in scopes: > + for slot, child in enumerate(scope.node.children()): > + attr = child.attr > + repeated = attr.decl.max_occurrences != 1 > + name = _input_name(names, scope, attr) > + _macro(writer, 'CTLV_IN_REF', > + _input_args(names, scope, slot, attr)) > + if repeated: > + _iterator(writer, names, scope, attr) > + else: > + _macro(writer, 'CTLV_IN_HAS', (name,)) > + if attr.decl.kind == 'blob': > + macro = 'CTLV_IN_' + ('EACH_' if repeated else '') + 'BLOB' > + _macro(writer, macro, (name,)) > + elif cgen.attr_getter_value(attr): > + macro = 'CTLV_IN_' + ('EACH_' if repeated else '') + \ > + INPUT_MACROS[attr.decl.kind] > + _macro(writer, macro, (name,)) > + > + > +def _writers(writer, names, family): > + """What a callback appends to the reply it was handed. > + > + One helper per attr, returning whether the attr made it into the message. > + """ > + attrs = [attr for group in family.local_groups for attr in group.attrs > + if cgen.attr_writers(names, attr)] > + if not attrs: > + return > + writer.line() > + writer.comment('Writing a reply. An append is atomic; a nest ends with ' > + 'ctlv_nest_end().') > + for attr in attrs: > + if attr.decl.kind == 'nested': > + macro = 'CTLV_NEST_START' > + else: > + macro = f'CTLV_PUT_{PUT_MACROS[attr.decl.kind]}' > + what = 'start' if attr.decl.kind == 'nested' else 'put' > + name = names.attr_helper(attr, what).removesuffix(f'_{what}') > + _macro(writer, macro, > + (name, > + names.attr(attr), cgen.KINDS[attr.decl.kind])) Roundabout: `attr_helper(attr, what)`.removesuffix(`_` + what) re-derives a stem that cgen.Names already has (compare `Names.input_name()`). Add a `Names.attr_stem()` and use it here and in `attr_helper`. > + > + > +def _creators(writer, names, family): > + """What begins one event of each kind this family declares. > + > + A creator hands back a context and the cursor of its root. > + """ > + events = [op for op in family.local_ops if op.type == 'event'] > + if not events: > + return > + writer.line() > + writer.comment('Building one event, passed to ctlv_event_emit() or ' > + 'ctlv_event_abort() once.') > + for op in events: > + _macro(writer, 'CTLV_EVENT_CREATE', > + (names.op_ident(op), names.op_id(op))) > + > + > +def _input_name(names, scope, attr): > + return names.input_ref(scope, attr).removesuffix('_ref') `_input_name()` duplicates `Names.input_name()` (which `_iterator()` about ten lines below calls directly): `input_ref()` is `input_name()` + `_ref`, so this strips and rebuilds the same string. Use `names.input_name(scope, attr)`. > + > + > +def _input_args(names, scope, slot, attr): > + return (_input_name(names, scope, attr), scope.tree, names.attr(attr), > + str(scope.node_index), str(slot), cgen.KINDS[attr.decl.kind]) > + > + > +def _iterator(writer, names, scope, attr): > + """Every occurrence of a repeated attr, in wire order.""" > + writer.define_call( > + f'{names.input_iterator(scope, attr)}(_attr, _nest)', > + 'CTLV_IN_FOR_EACH', names.input_name(scope, attr), '_attr', '_nest') > + > + > +def emit_source(writer, names, family): > + """Append the family bindings to a generated kernel source.""" > + _bindings(writer, names, family) > + _stub_ops(writer, names, family) > + _descriptor(writer, names, family) > + > + > +def _bindings(writer, names, family): > + bound = bindings(family) > + if not bound: > + return > + writer.comment('Local action and query bindings in declaration order.') > + writer.declaration('static const struct ctlv_op', > + f'{names.ops()}[] = {{') > + for op in bound: > + suffix = '_PRE_POST' if op.pre_post else '' > + writer.line(f'\tCTLV_OP_{op.type.upper()}{suffix}(' > + f'{names.op_ident(op)}),') > + writer.line('};') > + writer.line() > + > + > +def _stub_ops(writer, names, family): > + """One entry per extension point this family fills, in the same order.""" > + if not family.fills: > + return > + writer.comment('Extension-point callbacks.') > + writer.declaration('static const struct ctlv_stub_ops', > + f'{names.stub_ops()}[] = {{') > + for fill in family.fills: > + attr_set = family.sets[fill.name] > + args = (cgen.family_const(family.owners[fill.family_id]), > + names.attr_set(attr_set), names.stub_ident(fill)) > + writer.parenthesized('\tCTLV_STUB_OPS(', args, '),') > + writer.line('};') > + writer.line() > + > + > +def _descriptor(writer, names, family): Undocumented and non-trivial: It conditionally assembles a C designated- initializer for struct ctlv_family from several independent family properties (parent, bindings, hooks, fills), and the emitted names come from Names helpers that are not visible here. Suggested docstring: """Emit the struct ctlv_family descriptor of a family, and export it. Appends to writer, in order: a one-line comment (root-family wording for the core family, "used by ctlv_create()" for any other), the definition of the descriptor, and an EXPORT_SYMBOL_NS_GPL() line for it. No trailing blank line is written. The emitted text looks like this, with the optional members present only when the family has them: /* Family descriptor used by ctlv_create(). */ const struct ctlv_family tdx_family = { .schema = &tdx_schema, .parent = &ctlv_family, .ops = tdx_ops, .ops_count = ARRAY_SIZE(tdx_ops), .pre = tdx_op_pre, .post = tdx_op_post, .priv_size = tdx_priv_size, .init = tdx_init, .fini = tdx_fini, .stub_ops = tdx_stub_ops, .stub_ops_count = ARRAY_SIZE(tdx_stub_ops), }; EXPORT_SYMBOL_NS_GPL(tdx_family, "CTLV"); .schema always points at the schema node emitted by gen_kernel. .parent is the direct parent's descriptor (see _parent_descriptor) and is omitted only for the root family. .ops and .ops_count are emitted only if bindings(family) is non-empty, so they name the array written by _bindings (local actions and queries, never events or inherited ops). The per-family .pre and .post hooks (HOOK_RETURNS) are always emitted. The lifecycle hooks come from _declared_lifecycle_hooks: the required ones (priv_size, init, fini) always, the optional ones (pre_init, post_init, pre_fini) only when the family declares them. .stub_ops and .stub_ops_count are emitted only when the family fills inherited extension points, and refer to the array written by _stub_ops. The export is the one symbol an inheriting family's module needs from this one. This must run after _bindings and _stub_ops, because it refers to the arrays they declare. It is called from emit_source, and the actual hook functions it names are defined elsewhere in the generated source. Args: writer: cgen.Writer to append the generated lines to; it is mutated. names: cgen.Names for family, which spells every emitted identifier. family: the resolved family (semantics model) being generated. Returns: None: output is only appended to writer. """ > + if family.name == spec.CORE_FAMILY_NAME: > + writer.comment('Root-family descriptor used by the CTLV core.') > + else: > + writer.comment('Family descriptor used by ctlv_create().') > + writer.declaration('const struct ctlv_family', > + f'{names.descriptor()} = {{') > + writer.line(f'\t.schema = &{names.schema()},') > + parent = _parent_descriptor(family) > + if parent: > + writer.line(f'\t.parent = &{parent},') > + if bindings(family): > + writer.line(f'\t.ops = {names.ops()},') > + writer.line(f'\t.ops_count = ARRAY_SIZE({names.ops()}),') > + for which, _ in HOOK_RETURNS: > + writer.line(f'\t.{which} = {names.family_hook(which)},') > + for which, _, _ in _declared_lifecycle_hooks(family): > + writer.line(f'\t.{which} = {names.lifecycle_hook(which)},') > + if family.fills: > + stub_ops = names.stub_ops() > + writer.line(f'\t.stub_ops = {stub_ops},') > + writer.line(f'\t.stub_ops_count = ARRAY_SIZE({stub_ops}),') > + writer.line('};') > + # The one symbol an inheriting family's module needs of this one. > + writer.line(f'EXPORT_SYMBOL_NS_GPL({names.descriptor()}, "CTLV");') > + > + > +def _parent_descriptor(family): > + """The direct parent descriptor, none only for the root family.""" > + if family.parent is None: > + return None > + return cgen.Names(family.chain[-2]).descriptor() > diff --git a/tools/ctlv/ctlvgen/lib/gen_kernel.py b/tools/ctlv/ctlvgen/lib/gen_kernel.py > new file mode 100644 > index 000000000000..1854bd004f36 > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/gen_kernel.py > @@ -0,0 +1,517 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""The kernel metadata of one family. > + > +Each family describes only what it declares. Each op request and reply becomes > +one flat, breadth-first array of struct ctlv_node_schema the core walks by > +index. > +""" > + > +from __future__ import annotations > + > +import dataclasses > + > +from . import cgen > +from . import gen_family > +from . import semantics > +from . import spec > + > + > +def root_macro(*names): > + """Return a constant of the root family's generated UAPI header.""" > + return cgen.macro(spec.CORE_FAMILY_NAME, *names) > + > + > +FD_ACCESSES = {access: root_macro('op-fd-access', access) > + for access in spec.FD_ACCESSES} > + > +AVAILABILITIES = {availability: root_macro('op-availability', availability) > + for availability in spec.AVAILABILITIES} > + > +# Lengths that have to fit the inline message limit. An event record is checked > +# against the event queue instead. > +INLINE_FIELDS = ('request_max_len', 'action_reply_max_len', > + 'query_reply_min_len', 'query_reply_suggested_len') > + > +# A bound written as its symbolic extreme stays readable. > +U64_MAX = (1 << 64) - 1 > +S64_MIN = -(1 << 63) > +S64_MAX = (1 << 63) - 1 > +EXTREMES = { > + ('uint', spec.U32_MAX): 'U32_MAX', > + ('uint', U64_MAX): 'U64_MAX', > + ('sint', S64_MIN): 'S64_MIN', > + ('sint', S64_MAX): 'S64_MAX', `U64_MAX`, `S64_MIN` and `S64_MAX` are redefined here although `spec.py` already defines the same three constants (and `gen_kernel` uses `spec.U32_MAX` and `spec.S64_MAX` from there in the same function). Use the spec ones. > +} > + > + > +@dataclasses.dataclass(frozen=True) > +class FlatNode: > + """One node of a flattened tree, with its children as an index range.""" > + > + node: object > + first_child: int > + children_count: int > + required_children_count: int > + > + > +def flatten(root): > + """Return the nodes of one tree breadth first, the root first. > + > + A re-entering node points back at nodes already laid out, which is what > + keeps the array finite. > + """ > + flat = [] > + queue = [(root, {})] > + next_index = 1 > + while queue: > + node, levels = queue.pop(0) queue.pop(0) on a list is O(n) per pop, so `flatten()` is O(n^2) in the number of nodes; use collections.deque. `flatten()` is also re-run per tree from `stub_schemas()`, `_kinds()`, `_node_schemas()`, `_reader_scopes()` and `_limits()` with no caching. > + if node.reenters is not None: > + first_child, children_count, required_count = levels[node.reenters] > + flat.append(FlatNode(node=node, first_child=first_child, > + children_count=children_count, > + required_children_count=required_count)) > + continue > + children = node.children() > + level = (next_index if children else 0, len(children), > + len(node.required_children)) > + flat.append(FlatNode(node=node, first_child=level[0], > + children_count=level[1], > + required_children_count=level[2])) > + next_index += len(children) > + if node.expands: > + levels = levels | {node.expands: level} > + queue += [(child, levels) for child in children] > + return tuple(flat) > + > + > +def stub_schemas(root): > + """Return each stub schema with the node index it joins.""" > + schemas = [] > + for index, flat in enumerate(flatten(root)): > + for stub in flat.node.stubs: > + schemas.append((index, stub)) > + return tuple(schemas) > + > + > +def render_header(family, *, family_api): > + """Return one family's generated kernel header.""" > + names = cgen.Names(family) > + cgen.symbols(family) > + guard = names.kernel_guard() > + local = family.local_ops > + > + writer = cgen.preamble(family, 'kernel', 'header') > + what = 'Kernel metadata and family API' if family_api else \ > + 'Schema metadata' > + writer.comment(f'{what} of the {family.name} CTLV family.') > + writer.line() > + writer.open_guard(guard) > + writer.includes(*_header_includes(family, names, family_api)) > + > + if local: > + writer.comment('Ops this family declares, in declaration order.') > + writer.declaration('extern const struct ctlv_op_schema', > + f'{names.op_schemas()}[];') > + writer.line() > + writer.declaration('extern const struct ctlv_family_schema', > + f'{names.schema()};') > + writer.line() > + _message_decls(writer, names, family) > + if family_api: > + gen_family.emit_header(writer, names, family, > + _reader_scopes(family, names)) > + writer.close_guard(guard) > + return writer.text() > + > + > +def _reader_scopes(family, names): Undocumented and non-trivial: It cross-references two different views of the same trees, the semantic input scopes and the breadth-first flattened node arrays, to compute a C expression and an array index for each readable nested scope. Suggested docstring: """Describe every request scope family code may read, with its schema node. For each scope yielded by semantics.input_scopes(family) (the request tree of each local op, each of its nested attrs, and the same for every extension-point fill), build a gen_family.ReaderScope holding: name the unique scope name tuple from the semantic scope, node the semantic tree node of that scope, tree a C expression for the ctlv_schema_tree that contains it, node_index the node's position in that tree's flat node array. The tree expression is "&[i].request" for an op scope and "&.stub_fill_schemas[i].members" for a fill scope, with i the scope's index among the family's local ops or fills. node_index is the index flatten() assigns, so it is the index of the matching entry in the emitted ctlv_node_schema array; flatten() is run once per (kind, index) tree and cached, and nodes are matched by identity (id()), not equality. Nothing is written to a Writer here. render_header passes the result to gen_family.emit_header, which turns it into the family's typed readers. Args: family: the resolved family being generated. names: cgen.Names for family, used to spell the C symbols in the tree expressions. Returns: tuple[gen_family.ReaderScope, ...]: one entry per input scope, in the order semantics.input_scopes yields them. Throws: KeyError: if a scope's node is not part of the flattened tree of its root, which would mean the semantic model is inconsistent. """ > + scopes = [] > + indexes = {} > + for scope in semantics.input_scopes(family): > + if scope.kind == 'op': > + root = family.local_ops[scope.index].request > + tree = f'&{names.op_schemas()}[{scope.index}].request' > + else: > + root = family.fills[scope.index].root > + tree = f'&{names.schema()}.stub_fill_schemas[' \ > + f'{scope.index}].members' > + key = (scope.kind, scope.index) > + if key not in indexes: > + indexes[key] = {id(flat.node): index > + for index, flat in enumerate(flatten(root))} > + scopes.append(gen_family.ReaderScope( > + name=scope.name, node=scope.node, tree=tree, > + node_index=indexes[key][id(scope.node)])) > + return tuple(scopes) > + > + > +def _message_decls(writer, names, family): > + """The messages the core builds itself, and the room each one needs.""" > + for message in family.messages: > + writer.comment(f'The {message.name} message, built by the core.') > + writer.define(names.message_max_len(message), str(message.max_len)) > + writer.line() > + writer.declaration('extern const struct ctlv_schema_tree', > + f'{names.message_schema(message)};') > + writer.line() > + > + > +def _header_includes(family, names, family_api): > + """The generated types, constants and direct parent API.""" > + headers = [''] > + if not family_api: > + return tuple(headers) > + headers += ['', '', '', > + names.uapi_header(in_kernel=True)] `` and `` are included by every generated header on behalf of the `CTLV_IN_*`/`CTLV_PUT_*` macros. A header providing macros should include what its macros need (`ctlv_family.h` at HEAD does not include either), rather than each generated file compensating; as it stands a generated header without a prior include of those would break only for users of specific macros. > + if family.parent not in (None, spec.CORE_FAMILY_NAME): > + headers += ['', cgen.Names(family.chain[-2]).kernel_header()] > + return tuple(headers) > + > + > +def render_source(family, *, family_api): > + """Return one family's generated kernel source.""" > + names = cgen.Names(family) > + cgen.symbols(family) > + trees = _trees(family) > + > + writer = cgen.preamble(family, 'kernel', 'source') > + if family_api: > + writer.line('#define _CTLV_GENERATED_METADATA_ONLY') > + writer.includes(*_includes(names, family, trees, family_api)) > + for op, which, root in trees: > + _node_schemas(writer, family, names, > + names.node_schemas(op, which), root) > + _stub_schemas(writer, family, names, op, which, root) > + _stub_fill_schemas(writer, family, names) > + _messages(writer, family, names) > + _table(writer, names, family) > + _descriptor(writer, names, family) > + _uevent(writer, family) > + _limits(writer, family, trees) > + if family_api: > + if family.local_ops: > + writer.line() > + gen_family.emit_source(writer, names, family) > + return writer.text() > + > + > +def _trees(family): > + """Every tree of every local op, in the order they are emitted.""" > + trees = [] > + for op in family.local_ops: > + if op.request is not None: > + trees.append((op, 'request', op.request)) > + if op.reply is not None: > + trees.append((op, 'reply', op.reply)) > + return trees > + > + > +def _roots(family, trees): > + """Every emitted tree root: an op's, a fill's and a core message's.""" > + return [root for _, _, root in trees] + \ > + [fill.root for fill in family.fills] + \ > + [message.root for message in family.messages] > + > + > +def _kinds(roots): > + return {flat.node.attr.decl.kind > + for root in roots for flat in flatten(root) > + if not flat.node.is_root} > + > + > +def _includes(names, family, trees, family_api): Undocumented and non-trivial: It builds a grouped include list whose members depend on the family, on whether the family API is generated, and on which attribute kinds occur anywhere in the family's trees. Suggested docstring: """Return the #include lines of the generated kernel source. The result is a flat list of header strings in which an empty string stands for the blank line between groups; writer.includes() emits it verbatim, one "#include " per non-empty entry. Nothing is written to a Writer here. The groups, in order, are: 1. Core headers, sorted: , , , plus when family_api is true (the descriptor is exported), when some node is a blob (BIT() builds the backing mask) and when some node is a uint or sint (U32_MAX and friends in the bounds). 2. UAPI headers: always the root family's, plus the family's own unless it is the root. The family's own is spelled with the "uapi/" prefix only when family_api is true. 3. The family's own generated kernel header, via names.kernel_header(). Which kinds occur is found by walking every emitted tree root (ops, fills and core messages) with flatten(), ignoring the root node itself. Args: names: cgen.Names for family. family: the resolved family being generated. trees: list of (op, which, root) tuples from _trees(family). family_api: bool, true when the family API (callbacks, descriptor) is generated into this source as well as the metadata. Returns: list[str]: include operands such as "" or '"x_gen.h"', with "" separating groups. """ > + headers = ['', '', > + ''] > + if family_api: > + headers.append('') > + # The root family's enums name op types, availabilities and blob backings. > + uapi = [cgen.uapi_header(spec.CORE_FAMILY_NAME, in_kernel=True)] > + if family.name != spec.CORE_FAMILY_NAME: > + uapi.append(names.uapi_header(in_kernel=family_api)) > + kinds = _kinds(_roots(family, trees)) > + if 'blob' in kinds: > + # BIT(), for the mask of permitted blob backings. > + headers.append('') > + if kinds & {'uint', 'sint'}: > + # The symbolic extremes of an unbounded integer bound. > + headers.append('') > + groups = [sorted(headers), uapi, [names.kernel_header()]] > + lines = [] > + for group in groups: > + if not group: > + continue > + if lines: > + lines.append('') > + lines += group Dead branch: none of the three groups can be empty ('headers' always has three entries, 'uapi' at least one, [`names.kernel_header()`] exactly one), so `if not group: continue` never runs. > + return lines > + > + > +def _has_event(family): > + return any(op.type == 'event' for op in family.local_ops) > + > + > +def _node_schemas(writer, family, names, symbol, root): > + writer.declaration('static const struct ctlv_node_schema', > + f'{symbol}[] = {{') > + for flat in flatten(root): > + _node(writer, family, names, flat) > + writer.line('};') > + writer.line() > + > + > +def _node(writer, family, names, flat): > + macro, args = _node_initializer(family, names, flat) > + if macro == 'CTLV_NODE_SCHEMA_BLOB': > + _blob_node(writer, macro, args) > + else: > + writer.parenthesized(f'\t{macro}(', args, '),') > + > + > +def _blob_node(writer, macro, args): > + opening = f'\t{macro}(' > + > + for line in cgen.wrapped(opening, args[:4], ','): > + writer.line(line) > + column = cgen.width(opening) > + indent = '\t' * (column // cgen.TAB_WIDTH) + \ > + ' ' * (column % cgen.TAB_WIDTH) > + backings = args[4].split(' | ') > + for backing in backings[:-1]: > + writer.line(f'{indent}{backing} |') > + writer.line(f'{indent}{backings[-1]},') > + writer.line(f'{indent}{args[5]}),') > + > + > +def _node_initializer(family, names, flat): Undocumented and non-trivial: the macro name and argument list of each node's initializer depend on the root/nested/leaf split, on re-entry, on units, and on the constraint arguments of each attribute kind, all encoded in string suffixes. Suggested docstring: """Choose the CTLV_NODE_SCHEMA_* macro and arguments for one flat node. Returns the pieces of one initializer of the static struct ctlv_node_schema array; the caller (_node) writes it as a line "MACRO(arg, arg, ...)," into the writer, and _blob_node does so in a special layout for blobs. Nothing is written here. The macro depends on the node: root CTLV_NODE_SCHEMA_ROOT(first_child, children_count, required_children_count), with no attribute. nested CTLV_NODE_SCHEMA_NEST(attr, occurrences, first_child, children_count, required_children_count). A node that re-enters an ancestor gets the macro with a _REENTERS suffix and the child range of the node it re-enters, from flatten(). leaf CTLV_NODE_SCHEMA_(attr, occurrences, ), where KIND is the decl kind from cgen.KINDS (UINT, SINT, BOOL, STRING, BINARY, ENUM, FLAGS, BLOB). A uint or sint with a unit gets a _UNIT suffix and two extra arguments (unit constant and scale). The other constraints come from _constraint_args. occurrences is node.max_occurrences, or CTLV_NODE_SCHEMA_UNLIMITED when that is zero or None. An example of the emitted text: CTLV_NODE_SCHEMA_UINT_UNIT(TDX_ATTR_REPORT_SIZE, 1, 0, U32_MAX, CTLV_UNIT_BYTES, 1), Args: family: the resolved family being generated. names: cgen.Names for family, used to spell attribute constants. flat: gen_kernel.FlatNode, a node together with its child index range in the flattened array. Returns: tuple[str, tuple[str, ...]]: the macro name and its arguments, all already formatted as C text. Throws: KeyError: if the decl kind is not in cgen.KINDS, or an enum/flags reference is not among family.definitions (from the constraint helpers). """ > + node = flat.node > + > + if node.is_root: > + return 'CTLV_NODE_SCHEMA_ROOT', ( > + str(flat.first_child), > + str(flat.children_count), > + str(flat.required_children_count), > + ) > + > + decl = node.attr.decl > + kind = cgen.KINDS[decl.kind].removeprefix('CTLV_SCHEMA_') > + macro = f'CTLV_NODE_SCHEMA_{kind}' > + occurrences = str(node.max_occurrences) if node.max_occurrences else \ > + 'CTLV_NODE_SCHEMA_UNLIMITED' The UNLIMITED == 0 sentinel is relied on implicitly here ("if `node.max_occurrences` else `CTLV_NODE_SCHEMA_UNLIMITED`") rather than comparing with spec.UNLIMITED. A truthiness test on a magic zero is fragile (and is exactly what bit `compat._compare_occurrences`). > + args = [names.attr(node.attr), occurrences] > + if decl.kind == 'nested': > + if node.reenters is not None: > + macro += '_REENTERS' > + args += [str(flat.first_child), str(flat.children_count), > + str(flat.required_children_count)] > + else: > + if decl.kind in ('uint', 'sint') and decl.unit is not None: > + macro += '_UNIT' > + args += _constraint_args(family, names, decl) > + return macro, tuple(args) > + > + > +def _stub_schemas(writer, family, names, op, which, root): > + """The extension points of one tree, by the node they widen.""" > + schemas = stub_schemas(root) > + if not schemas: > + return > + writer.comment('Extension points selected by this tree.') > + writer.declaration('static const struct ctlv_stub_schema', > + f'{names.stub_schemas(op, which)}[] = {{') > + for index, stub in schemas: > + macro = 'CTLV_STUB_SCHEMA_OPTIONAL' if stub.optional else \ > + 'CTLV_STUB_SCHEMA' > + attr_set = family.sets[stub.name] > + args = (cgen.family_const(family.owners[stub.family_id]), > + names.attr_set(attr_set), str(index)) > + writer.parenthesized(f'\t{macro}(', args, '),') > + writer.line('};') > + writer.line() > + > + > +def _stub_fill_schemas(writer, family, names): > + """The members this family puts into the stubs it inherited.""" > + if not family.fills: > + return > + for fill in family.fills: > + _node_schemas(writer, family, names, > + names.fill_node_schemas(fill), fill.root) > + writer.comment('The extension points this family fills.') > + writer.declaration('static const struct ctlv_stub_fill_schema', > + f'{names.stub_fill_schemas()}[] = {{') > + for fill in family.fills: > + attr_set = family.sets[fill.name] > + args = (cgen.family_const(family.owners[fill.family_id]), > + names.attr_set(attr_set), cgen.ident(fill.name)) > + writer.parenthesized('\tCTLV_STUB_FILL_SCHEMA(', args, '),') > + writer.line('};') > + writer.line() > + > + > +def _messages(writer, family, names): > + """The messages the core builds where no op declared one.""" > + for message in family.messages: > + _node_schemas(writer, family, names, > + names.message_node_schemas(message), message.root) > + writer.declaration('const struct ctlv_schema_tree', > + f'{names.message_schema(message)} = {{') > + node_schemas = names.message_node_schemas(message) > + writer.line(f'\t.node_schemas = {node_schemas},') > + writer.line('\t.node_schemas_count =') > + writer.line(f'\t\tARRAY_SIZE({node_schemas}),') > + writer.line('};') > + writer.line() > + writer.assertion(names.message_max_len(message), '<=', message.limit) > + writer.line() > + > + > +def _constraint_args(family, names, decl): > + """The kind-specific arguments of one node-schema macro.""" > + if decl.kind in ('uint', 'sint'): > + low = _bound(decl.kind, decl.min) > + high = _bound(decl.kind, decl.max) > + args = [low, high] > + if decl.unit is not None: > + args += [f'CTLV_UNIT_{cgen.macro(decl.unit)}', str(decl.scale)] > + return args > + if decl.kind == 'enum': > + highest = max(family.definitions[decl.ref].numbers()) > + return [str(highest)] > + if decl.kind == 'flags': > + return [names.value_mask(family.definitions[decl.ref])] > + if decl.kind in ('string', 'binary'): > + return [str(decl.min_length), str(decl.max_length)] > + if decl.kind == 'blob': > + return _blob_args(decl.blob) > + return [] > + > + > +def _bound(kind, value): > + extreme = EXTREMES.get((kind, value)) > + > + if extreme: > + return extreme > + if kind == 'uint' and value > spec.S64_MAX: > + return f'{value}ULL' > + return str(value) `_bound()` can return `U32_MAX`/`U64_MAX` (also for blob max-length, via `_blob_args()`), but `` is added only when a uint/sint attr is present (see `_includes()`). A family whose only large bound is a blob max-length of `U32_MAX`/`U64_MAX` depends on a transitive include for `U32_MAX`/`U64_MAX`. > + > + > +def _blob_args(blob): > + direction = root_macro('blob-direction', blob.direction) > + return [_bound('uint', blob.max_length), str(blob.alignment), > + _backings(blob.backings), direction] > + > + > +def _backings(backings): > + terms = [f'BIT({root_macro("blob-type", backing)})' > + for backing in backings] > + return ' | '.join(terms) > + > + > +def _table(writer, names, family): > + local = family.local_ops > + if not local: > + return > + writer.comment('The ops this family declares, in declaration order.') > + writer.declaration('const struct ctlv_op_schema', > + f'{names.op_schemas()}[] = {{') > + for op in local: > + macro, args = _op_schema_initializer(names, op) > + writer.parenthesized(f'\t{macro}(', args, '),') > + writer.line('};') > + writer.line() > + > + > +def _op_schema_initializer(names, op): Undocumented and non-trivial: the three op types take different macros with different argument orders and size fields, and two optional designated initializers are appended on top. Suggested docstring: """Choose the CTLV_OP_SCHEMA_* macro and arguments for one local op. Returns the pieces of one element of the const struct ctlv_op_schema array; _table writes it as "MACRO(arg, ...)," into the writer, and this function itself writes nothing. The macro and positional arguments depend on op.type, with sizes taken from op.sizes: action CTLV_OP_SCHEMA_ACTION(op_id, fd_access, availability, request_max_len, action_reply_max_len, request_tree, reply_tree) query CTLV_OP_SCHEMA_QUERY(op_id, fd_access, availability, request_max_len, query_reply_min_len, query_reply_suggested_len, request_tree, reply_tree) event CTLV_OP_SCHEMA_EVENT(op_id, event_record_max_len, reply_tree) (an event has no request, fd access or availability) fd_access and availability are mapped to the root family's constants via FD_ACCESSES and AVAILABILITIES. The tree arguments come from _op_tree (CTLV_SCHEMA_TREE, CTLV_SCHEMA_TREE_STUBS or CTLV_SCHEMA_TREE_NONE()). After the positional arguments, ".optional = true" and ".pre_post = true" are appended, in that order, only when the op sets them. Example: CTLV_OP_SCHEMA_ACTION(TDX_OP_GET_REPORT, CTLV_OP_FD_ACCESS_READ, CTLV_OP_AVAILABILITY_STATIC, 64, 1024, CTLV_SCHEMA_TREE(get_report_request), CTLV_SCHEMA_TREE(get_report_reply)), Args: names: cgen.Names for the family, used for the op ID constant. op: the local op to describe; its sizes must already be computed. Returns: tuple[str, tuple[str, ...]]: the macro name and its C-text arguments. Throws: KeyError: if an action's or query's fd_access or availability is not one of spec.FD_ACCESSES / spec.AVAILABILITIES. """ > + sizes = op.sizes > + if op.type == 'action': > + macro = 'CTLV_OP_SCHEMA_ACTION' > + args = [names.op_id(op), FD_ACCESSES[op.fd_access], > + AVAILABILITIES[op.availability], > + str(sizes.request_max_len), > + str(sizes.action_reply_max_len), > + _op_tree(op, 'request'), _op_tree(op, 'reply')] > + elif op.type == 'query': > + macro = 'CTLV_OP_SCHEMA_QUERY' > + args = [names.op_id(op), FD_ACCESSES[op.fd_access], > + AVAILABILITIES[op.availability], > + str(sizes.request_max_len), > + str(sizes.query_reply_min_len), > + str(sizes.query_reply_suggested_len), > + _op_tree(op, 'request'), _op_tree(op, 'reply')] > + else: > + macro = 'CTLV_OP_SCHEMA_EVENT' > + args = [names.op_id(op), str(sizes.event_record_max_len), > + _op_tree(op, 'reply')] > + if op.optional: > + args.append('.optional = true') > + if op.pre_post: > + args.append('.pre_post = true') > + return macro, tuple(args) > + > + > +def _op_tree(op, which): > + root = getattr(op, which) > + if root is None: > + return 'CTLV_SCHEMA_TREE_NONE()' > + macro = 'CTLV_SCHEMA_TREE_STUBS' if stub_schemas(root) else \ > + 'CTLV_SCHEMA_TREE' > + return f'{macro}({cgen.ident(op.name, which)})' > + > + > +def _descriptor(writer, names, family): > + """This family's node of the schema graph.""" > + writer.comment('The family ID is its position in the central registry.') > + writer.declaration('const struct ctlv_family_schema', > + f'{names.schema()} = {{') > + writer.line(f'\t.name = "{family.name}",') > + writer.line(f'\t.family_id = {family.family_id},') > + if family.local_ops: > + writer.line(f'\t.ops = {names.op_schemas()},') > + writer.line(f'\t.ops_count = ARRAY_SIZE({names.op_schemas()}),') > + if family.fills: > + stub_fill_schemas = names.stub_fill_schemas() > + writer.line(f'\t.stub_fill_schemas = {stub_fill_schemas},') > + writer.line(f'\t.stub_fill_schemas_count = ' > + f'ARRAY_SIZE({stub_fill_schemas}),') > + writer.line('};') > + writer.line() > + > + > +def _uevent(writer, family): > + """What this chain's names cost a device's uevent environment.""" > + names = [item.name for item in family.chain] > + variables = [f'CTLV_FAMILY={names[-1]}'] > + variables += [f'CTLV_COMPATIBLE_{index}={name}' > + for index, name in enumerate(names)] > + variables.append(f'CTLV_COMPATIBLE_N={len(names)}') > + # "KEY=value" plus the NUL, as add_uevent_var() charges it. > + writer.comment('Maximum uevent cost for this family chain.') > + writer.line(f'static_assert({sum(len(item) + 1 for item in variables)} <= ' > + 'CTLV_UEVENT_MAX_LEN);') > + writer.line(f'static_assert({len(variables)} <= CTLV_UEVENT_MAX_VARS);') The uevent variable names and counting here (`CTLV_FAMILY=`, `CTLV_COMPATIBLE_%u=`, `CTLV_COMPATIBLE_N=`) duplicate `ctlv_device_uevent_families()` in `drivers/ctlv/core.c`. If `core.c` changes the format, this `static_assert` stops reflecting reality at build time; only the runtime check in `ctlv_device_uevent_check()` (which fails with -E2BIG and a `pr_err`) would catch it. Export the format from one place (a header macro). > + writer.line() > + > + > +def _limits(writer, family, trees): Undocumented and non-trivial: It is a sequence of conditional compile-time guards, each derived from a different aggregate (message size, event queue, nest depth) over different sets of trees, with an early return that suppresses all output. Suggested docstring: """Emit static_assert guards for the generated size and depth maxima. Appends to writer the comment "Guards for generated maxima." followed by up to three static_assert() lines, so the build fails if a schema outgrows a core limit: static_assert(1024 <= CTLV_MAX_INLINE_MESSAGE_LEN); static_assert(256 <= CTLV_MAX_EVENT_QUEUE_LEN); static_assert(3 <= CTLV_MAX_DECLARED_NEST_DEPTH); The first is the largest, over the family's local ops, of the sizes in INLINE_FIELDS (request max, action reply max, query reply min and suggested lengths). The second is emitted only if some local op is an event, and bounds the largest event_record_max_len (event records are checked against the event queue, not the inline limit). The third bounds the deepest nest over every emitted root (ops, fills and core messages, via _roots) and is omitted when the depth is zero. If the family has no local ops, nothing at all is written, even if it has fills or messages with nests. No trailing blank line is written. It is called from render_source after the uevent guards. Args: writer: cgen.Writer to append the generated lines to; it is mutated. family: the resolved family being generated. trees: list of (op, which, root) tuples from _trees(family). Returns: None: output is only appended to writer. """ > + local = family.local_ops > + if not local: > + return > + writer.comment('Guards for generated maxima.') > + inline = max(getattr(op.sizes, field) Early return for families without local ops skips the nest-depth guard (line 3418) even though `_roots()` also covers stub fills and core messages, which are nests as well. Conversely the depth `static_assert` is redundant for ops: `semantics._node()` already rejects depth > `MAX_DECLARED_NEST_DEPTH`. If the `static_assert` is meant as a guard against `sizes.py` constants drifting from the C header, it should be emitted for every root, and `ATTR_HDR_LEN`, `EVENT_HDR_LEN` and ALIGNTO (`sizes.py`) should be asserted too: no generated check covers them. > + for op in local for field in INLINE_FIELDS) > + writer.line(f'static_assert({inline} <= CTLV_MAX_INLINE_MESSAGE_LEN);') > + if _has_event(family): > + record = max(op.sizes.event_record_max_len for op in local) > + writer.line(f'static_assert({record} <= CTLV_MAX_EVENT_QUEUE_LEN);') > + depth = max(_nest_depth(root) for root in _roots(family, trees)) > + if depth: > + writer.line(f'static_assert({depth} <= ' > + 'CTLV_MAX_DECLARED_NEST_DEPTH);') > + > + > +def _nest_depth(node): > + """Deepest nest below node, which is itself at depth zero.""" > + depths = [1 + _nest_depth(child) for child in node.children() > + if child.attr.decl.kind == 'nested'] > + return max(depths, default=0) > diff --git a/tools/ctlv/ctlvgen/lib/gen_uapi.py b/tools/ctlv/ctlvgen/lib/gen_uapi.py > new file mode 100644 > index 000000000000..14330248f144 > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/gen_uapi.py > @@ -0,0 +1,223 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""The UAPI header of one family. > + > +Names for numbers: complete attr and op IDs, group and attr-set IDs, enum > +values, flag bits and units. The wire format is the transport > +header's. An inherited declaration is not repeated, already having a name in > +the header of the family that owns it. > +""" > + > +from __future__ import annotations > + > +import textwrap > + > +from . import cgen > + > +# " * " plus the text has to stay inside the line limit. > +COMMENT_WIDTH = cgen.LINE_LIMIT - 3 > + > + > +def render(family): > + """Return the UAPI header of family as text.""" > + names = cgen.Names(family) > + cgen.symbols(family) > + guard = names.uapi_guard() > + > + writer = cgen.preamble(family, 'uapi', 'header') > + _file_comment(writer, family) > + writer.line() > + writer.open_guard(guard) > + writer.includes(*_includes(family)) > + > + _definitions(writer, family, names) > + _groups(writer, family, names) > + _sets(writer, family, names) > + _ops(writer, family, names) > + > + writer.close_guard(guard) > + return writer.text() > + > + > +def render_families(registry): > + """Return the family-number header of one registry.""" > + guard = '_UAPI_LINUX_CTLV_FAMILIES_GEN_H' > + values = [(cgen.family_const(entry.name), family_id) > + for family_id, entry in enumerate(registry.entries)] > + writer = cgen.preamble(registry, 'uapi', 'header', registry.path) > + > + writer.comment('CTLV family numbers.') > + writer.line() > + writer.open_guard(guard) > + _sequential_enum(writer, '_ctlv_family', values) > + writer.line() > + writer.close_guard(guard) > + return writer.text() > + > + > +def _includes(family): > + headers = ['', cgen.families_uapi_header()] > + if family.parent: > + # Inherited constants live in the parent's header, so one include is > + # enough for the whole effective schema. > + headers.append(cgen.uapi_header(family.parent)) > + if any(definition.kind == 'flags' > + for definition in _own(family, family.definitions.values())): > + # _BITULL(), which is the form of BIT_ULL() that userspace has. > + headers.append('') > + return sorted(headers) > + > + > +def _file_comment(writer, family): Undocumented and non-trivial: It builds a multi-paragraph comment by wrapping text to a width that leaves room for the comment prefix, and the blank separators are easy to misread. Suggested docstring: """Emit the explanatory comment block at the top of a UAPI header. Appends one C comment to writer, directly after the generated-file preamble, saying that the header holds the family's constants and that the wire format is in . If the family has a parent, a second paragraph says the parent's header is included so one include covers the whole effective schema. The result looks like: /* * Constants of the tdx CTLV family. Attr and op IDs are complete; the * wire format carrying them is in . * * The inherited ctlv header is included here, so one include covers * the whole effective schema. */ Each paragraph is wrapped with textwrap to COMMENT_WIDTH, which is the line limit minus the three columns of the " * " prefix, and paragraphs are separated by an empty line (which Writer.comment renders as a bare " *"). If the text wraps to a single line, Writer.comment emits it as a one-line "/* ... */" comment instead. Args: writer: cgen.Writer to append the comment to; it is mutated. family: the resolved family; its name and parent are quoted. Returns: None: output is only appended to writer. """ > + paragraphs = [ > + f'Constants of the {family.name} CTLV family. Attr and op IDs are ' > + 'complete; the wire format carrying them is in .', > + ] > + if family.parent: > + paragraphs.append( > + f'The inherited {family.parent} header is included here, so one ' > + 'include covers the whole effective schema.') > + lines = [] > + for paragraph in paragraphs: > + if lines: > + lines.append('') > + lines += textwrap.wrap(paragraph, COMMENT_WIDTH) > + writer.comment(*lines) > + > + > +def _own(family, declarations): > + """Only what this family declares itself, in declaration order.""" > + return [item for item in declarations > + if item.family_id == family.family_id] > + > + > +def _definitions(writer, family, names): > + for definition in _own(family, family.definitions.values()): > + if definition.kind == 'enum': > + _enum(writer, names, definition) > + else: > + _flags(writer, names, definition) > + writer.line() > + > + > +def _enum(writer, names, definition): > + writer.comment(f'enum {definition.name}') > + values = [(names.value(definition, value_name), number) > + for value_name, number in definition.values] > + _sequential_enum(writer, names.definition(definition), values) > + > + > +def _flags(writer, names, definition): > + writer.comment(f'flags {definition.name}') > + bits = [(names.value_bit(definition, value_name), bit) > + for value_name, bit in definition.values] > + _sequential_enum(writer, names.bit_enum(definition), bits) > + writer.line() > + values = [] > + for value_name, _bit in definition.values: > + value = names.value(definition, value_name) > + writer.define(value, > + f'_BITULL({names.value_bit(definition, value_name)})') > + values.append(value) > + _flags_mask(writer, names.value_mask(definition), values) > + > + > +def _groups(writer, family, names): Undocumented and non-trivial: It emits several interlocking constructs (group enum, group-to-attr macro, per-group attr macro and enum, attr macros, metadata) whose names are all derived through Names and must stay consistent with each other. Suggested docstring: """Emit the attr-group IDs, attr IDs and attr macros of a family. Only groups the family itself declares are handled (see _own); an inherited group is already named in its owner's header. If there are none, nothing is written. Otherwise this appends to writer, in order: 1. A "attr groups" comment and a sequential enum of group IDs, e.g. enum _tdx_group { TDX_GROUP_REPORT, TDX_GROUP_QUOTE, }; The first value gets an explicit "= N" only when it is nonzero. 2. A family-level macro that builds a complete attr ID from a group and a local attr: #define _TDX_ATTR(_group, _attr) \ CTLV_ATTR(CTLV_FAMILY_TDX, (_group), (_attr)) 3. For each group that has attrs: an "attrs of group" comment; a per-group macro that pastes the local name onto the group's prefix, #define _TDX_ATTR_REPORT(_attr) \ _TDX_ATTR(TDX_GROUP_REPORT, _TDX_ATTR_REPORT_##_attr) a sequential enum of the local attr numbers; one #define per attr (via _attr) giving its complete ID; and, through _attr_metadata, an enum of unit and scale constants for attrs that declare a unit. Groups without attrs only contribute their enum entry. The sequential enums assert that the numbers are consecutive, since only the first value is written out. Args: writer: cgen.Writer to append the generated lines to; it is mutated. family: the resolved family whose own groups are emitted. names: cgen.Names for family, which spells every emitted identifier. Returns: None: output is only appended to writer. Throws: AssertionError: from _sequential_enum, if group IDs or local attr IDs are not consecutive in declaration order. """ > + groups = _own(family, family.groups.values()) > + if not groups: > + return > + writer.comment('attr groups') > + _sequential_enum(writer, names.group_enum(groups[0]), > + [(names.group(group), group.group_id) > + for group in groups]) > + writer.line() > + writer.define_call( > + f'{names.family_attr(groups[0])}(_group, _attr)', 'CTLV_ATTR', > + names.family_const(), '(_group)', '(_attr)') > + writer.line() > + for group in groups: > + if not group.attrs: > + continue > + writer.comment(f'attrs of group {group.name}') > + writer.define_call(f'{names.group_attr(group)}(_attr)', > + names.family_attr(group), > + names.group(group), > + f'{names.local_attr_prefix(group)}##_attr') > + writer.line() > + _sequential_enum(writer, names.group_attr_enum(group), > + [(names.local_attr(attr), attr.local_id) > + for attr in group.attrs]) > + writer.line() > + for attr in group.attrs: > + _attr(writer, names, group, attr) > + _attr_metadata(writer, names, group) > + writer.line() > + > + > +def _attr(writer, names, group, attr): > + writer.define_call(names.attr(attr), names.group_attr(group), > + cgen.macro(attr.name)) > + > + > +def _attr_metadata(writer, names, group): > + attrs = [attr for attr in group.attrs if attr.decl.unit is not None] > + if not attrs: > + return > + writer.line() > + writer.comment(f'metadata of group {group.name}') > + writer.line(f'enum {names.group_metadata_enum(group)} {{') > + for attr in attrs: > + writer.line(f'\t{names.attr_unit(attr)} = ' > + f'CTLV_UNIT_{cgen.macro(attr.decl.unit)},') > + writer.line(f'\t{names.attr_scale(attr)} = {attr.decl.scale},') > + writer.line('};') > + > + > +def _sequential_enum(writer, name, values): > + first = values[0][1] > + > + assert all(number == first + index > + for index, (_value, number) in enumerate(values)) A bare "assert" is used for input validation in generator code: it is compiled out under "python -O" and otherwise produces a traceback instead of a CtlvError. If enum numbering must be sequential, raise CtlvError; `_sequential_enum()` also IndexErrors on an empty list. > + writer.line(f'enum {name} {{') > + for index, (value, number) in enumerate(values): > + assignment = f' = {number}' if not index and number else '' > + writer.line(f'\t{value}{assignment},') > + writer.line('};') > + > + > +def _flags_mask(writer, name, values): > + if len(values) == 1: > + writer.define(name, values[0]) > + return > + lines = [f'\t({values[0]} |'] > + lines += [f'\t {value} |' for value in values[1:-1]] > + lines.append(f'\t {values[-1]})') > + writer.define_block(name, *lines) > + > + > +def _sets(writer, family, names): > + own = _own(family, family.sets.values()) > + if not own: > + return > + writer.comment('attr sets') > + _sequential_enum(writer, names.set_enum(own[0]), > + [(names.attr_set(attr_set), attr_set.set_id) > + for attr_set in own]) > + writer.line() > + > + > +def _ops(writer, family, names): Undocumented and non-trivial: It emits an enum plus a two-level macro scheme for op IDs, where the helper macro name, the pasted prefix and the per-op macros are derived separately. Suggested docstring: """Emit the op IDs and their complete-ID macros for a family. Only ops the family declares itself are handled (see _own); inherited ops are named in their owner's header. If there are none, nothing is written. Otherwise this appends to writer, in order: an "ops" comment, a sequential enum of the local op numbers, a blank line, a helper macro that builds a complete op ID from a local op name by pasting it onto the family's op prefix, one #define per op, and a closing blank line. Events are included, as every op type has an ID. The text looks like: /* ops */ enum _tdx_op { _TDX_OP_GET_REPORT, _TDX_OP_GET_QUOTE, }; #define _TDX_OP(_op) CTLV_OP(CTLV_FAMILY_TDX, _TDX_OP_##_op) #define TDX_OP_GET_REPORT _TDX_OP(GET_REPORT) #define TDX_OP_GET_QUOTE _TDX_OP(GET_QUOTE) The enum holds the local numbers, the public TDX_OP_* macros hold the complete IDs that carry the family number. The enum numbers must be consecutive, which _sequential_enum asserts. Args: writer: cgen.Writer to append the generated lines to; it is mutated. family: the resolved family whose own ops are emitted. names: cgen.Names for family, which spells every emitted identifier. Returns: None: output is only appended to writer. Throws: AssertionError: from _sequential_enum, if the local op IDs are not consecutive in declaration order. """ > + own = _own(family, family.ops) > + if not own: > + return > + writer.comment('ops') > + _sequential_enum(writer, names.op_enum(own[0]), > + [(names.op(op), op.local_id) > + for op in own]) > + writer.line() > + writer.define_call(f'{names.operation(own[0])}(_op)', > + 'CTLV_OP', names.family_const(), > + f'{names.op_prefix(own[0])}##_op') > + for op in own: > + writer.define_call(names.op_id(op), names.operation(op), > + cgen.macro(op.name)) > + writer.line() > diff --git a/tools/ctlv/ctlvgen/lib/outputs.py b/tools/ctlv/ctlvgen/lib/outputs.py > new file mode 100644 > index 000000000000..4df7ae60c1c1 > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/outputs.py > @@ -0,0 +1,329 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""The complete set of generated files and its reconciliation.""" > + > +from __future__ import annotations > + > +import dataclasses > +import os > +import pathlib > +import subprocess > +import tempfile > + > +from . import gen_kernel > +from . import gen_uapi > +from . import semantics > +from . import spec > +from . import yamlload > +from .errors import CtlvError > + > + > +FLAVOURS = { > + 'uapi-header': gen_uapi.render, > + 'kernel-header': gen_kernel.render_header, > + 'kernel-source': gen_kernel.render_source, > +} > + > +IDENTITIES = { > + 'uapi-header': ('uapi', 'header', '.h'), > + 'kernel-header': ('kernel', 'header', '.h'), > + 'kernel-source': ('kernel', 'source', '.c'), > +} > + > +ROOT_FLAVOURS = frozenset(FLAVOURS) > +FAMILY_FLAVOURS = frozenset(FLAVOURS) > +FIXTURE_FLAVOURS = frozenset({ > + 'uapi-header', 'kernel-header', 'kernel-source', > +}) `ROOT_FLAVOURS`, `FAMILY_FLAVOURS` and `FIXTURE_FLAVOURS` are all frozenset(FLAVOURS) with the same three members, so the selection at lines 3796-3798 ("required = `FIXTURE_FLAVOURS` if fixture else `FAMILY_FLAVOURS`; if ... CORE: required = `ROOT_FLAVOURS`") always yields the same set. Dead configurability; collapse to a single constant. IDENTITIES and FLAVOURS are also two parallel tables over the same keys. > + > + > +@dataclasses.dataclass(frozen=True) > +class Output: > + family: str > + source: pathlib.PurePosixPath > + registry: pathlib.PurePosixPath > + flavour: str > + path: pathlib.PurePosixPath > + > + > +def _spec_paths(paths, source, registry): > + return spec.Paths(paths.kernel_dir / source, schema=paths.schema, > + registry=paths.kernel_dir / registry) > + > + > +def _relative(where, value): > + path = pathlib.PurePosixPath(value) > + if path.is_absolute() or '..' in path.parts or \ > + any(part.startswith('.') for part in path.parts) or \ > + str(path) != value: > + raise CtlvError(where, f'"{value}" is not a normalized relative path') > + return path > + > + > +def _repository_path(paths, path): > + return pathlib.PurePosixPath( > + pathlib.Path(path).relative_to(paths.kernel_dir).as_posix()) > + > + > +def _manifest_contexts(paths): > + fixtures_dir = paths.kernel_dir / 'tools/ctlv/tests/specs' `tools/ctlv/tests/specs` and `tools/ctlv/tests/generated-files.yaml` do not exist anywhere in this series (git ls-files `tools/ctlv` at HEAD lists no tests/). The 'fixture' context (and the fixture/`expected_names` plumbing in `load()`, `render()`'s `family_api`=False path and `gen_kernel`'s schema-only mode, cli `--schema-only`) exists solely for tests that are not submitted. That is dead code as posted. > + contexts = [ > + (paths.generated, _repository_path(paths, paths.registry), > + _repository_path(paths, paths.spec_dir), None, False), > + ] > + manifest = fixtures_dir.parent / 'generated-files.yaml' > + if manifest.exists(): > + fixtures = sorted(path.stem for path in fixtures_dir.glob('*.yaml')) > + contexts.append(( > + manifest, > + _repository_path(paths, fixtures_dir.parent / 'families.yaml'), > + _repository_path(paths, fixtures_dir), fixtures, True)) > + return tuple(contexts) > + > + > +def load(paths): > + """Load and validate every generated-file manifest.""" > + outputs = [] > + families = set() > + destinations = set() > + for manifest, expected_registry, source_dir, expected_names, fixture in \ > + _manifest_contexts(paths): > + data = yamlload.load(manifest) > + spec._validate_syntax(manifest, data, paths.generated_schema) `spec._validate_syntax()` is private (leading underscore) yet called from here. Make it public or move the manifest validation into `spec.py`. > + registry_record = data['registry'] > + where = f'{manifest}: registry' > + registry_source = _relative(where, registry_record['source']) > + if registry_source != expected_registry: > + raise CtlvError(where, f'source is {registry_source}, expected ' > + f'{expected_registry}') > + registry = spec.load_registry( > + paths, paths.kernel_dir / registry_source) > + if expected_names is None: > + expected_names = [entry.name for entry in registry.entries] > + records = data['families'] > + if [record['name'] for record in records] != expected_names: > + raise CtlvError(str(manifest), > + 'manifest families do not match their sources') > + > + for flavour, value in registry_record['outputs'].items(): > + destination = _relative(f'{where} {flavour}', value) > + suffix = IDENTITIES[flavour][2] > + if destination.suffix != suffix: > + raise CtlvError(f'{where} {flavour}', > + f'output must end in {suffix}') > + if destination in destinations: > + raise CtlvError(f'{where} {flavour}', > + f'duplicate output path {destination}') > + destinations.add(destination) > + outputs.append(Output(None, registry_source, registry_source, > + flavour, destination)) > + > + for record in records: > + name = record['name'] > + where = f'{manifest}: family {name}' > + if name in families: > + raise CtlvError(where, 'duplicate family') > + families.add(name) > + > + expected_source = source_dir / f'{name}.yaml' > + source = _relative(where, record['source']) > + if source != expected_source: > + raise CtlvError(where, f'source is {source}, expected ' > + f'{expected_source}') > + > + required = FIXTURE_FLAVOURS if fixture else FAMILY_FLAVOURS > + if not fixture and name == spec.CORE_FAMILY_NAME: > + required = ROOT_FLAVOURS > + flavours = frozenset(record['outputs']) > + if flavours != required: > + missing = sorted(required - flavours) > + extra = sorted(flavours - required) > + detail = [] > + if missing: > + detail.append(f'missing {", ".join(missing)}') > + if extra: > + detail.append(f'unexpected {", ".join(extra)}') > + raise CtlvError(where, '; '.join(detail)) > + > + family = semantics.load( > + _spec_paths(paths, source, registry_source)) > + if family.name != name: > + raise CtlvError(where, f'source declares family ' > + f'{family.name}') > + for flavour, value in record['outputs'].items(): > + destination = _relative(f'{where} {flavour}', value) > + suffix = IDENTITIES[flavour][2] > + if destination.suffix != suffix: > + raise CtlvError(f'{where} {flavour}', > + f'output must end in {suffix}') > + if destination in destinations: > + raise CtlvError(f'{where} {flavour}', > + f'duplicate output path {destination}') > + destinations.add(destination) > + outputs.append(Output(name, source, registry_source, flavour, > + destination)) The loop at 3770-3781 (registry outputs) and the loop at 3815-3826 (family outputs) are near-verbatim copies of the suffix/duplicate-path checks (they differ only in indentation and in the `Output(...)` arguments). Factor out a helper. > + return tuple(outputs) > + > + > +def render(paths, plan): > + """Render every planned output before any destination is changed.""" > + families = {} > + rendered = {} > + registered_registry = _repository_path(paths, paths.registry) > + for output in plan: > + source = _output_source(output) > + if output.family is None: > + registry = spec.load_registry( > + paths, paths.kernel_dir / output.registry) > + text = gen_uapi.render_families(registry) > + else: > + key = (output.source, output.registry) > + family = families.get(key) > + if family is None: > + family = semantics.load( > + _spec_paths(paths, output.source, output.registry)) > + families[key] = family > + renderer = FLAVOURS[output.flavour] > + if output.flavour in ('kernel-header', 'kernel-source'): > + text = renderer( > + family, family_api=output.registry == registered_registry) > + else: > + text = renderer(family) > + mode, kind, _ = IDENTITIES[output.flavour] > + if _identity(text) != (source, mode, kind): > + raise CtlvError(str(output.path), > + 'generator emitted the wrong output marker') > + rendered[output.path] = text > + return rendered > + > + > +def _output_source(output): > + return output.source.as_posix() > + > + > +def _marked_files(root): Undocumented and non-trivial: It finds generated files by a content marker through two different search strategies (git or a directory walk) and tolerates git's "no match" exit status, so what it returns and what it skips is not obvious. Suggested docstring: """Find every file under root that claims to be a generated CTLV file. reconcile() uses this to detect stale or orphaned generated files, that is, files carrying a CTLV-GEN marker that no manifest entry accounts for. A file counts as marked only when _identity() accepts its leading header lines, so a file that merely mentions CTLV-GEN does not. When root is a git checkout (it has a .git entry), candidates are the *.c, *.h and *.rst files that "git grep --untracked --files-with-matches CTLV-GEN" lists. That covers tracked and untracked files but not ignored ones. Otherwise the tree is walked with rglob() for the same three suffixes. Symlinks and non-regular files are skipped in both cases. Files are read as UTF-8 with undecodable bytes replaced, so binary junk cannot make the scan fail. Args: root: pathlib.Path of the tree to scan, normally the kernel source root or the --output-root. Returns: set[str]: paths of the marked files relative to root, in POSIX form. Throws: CtlvError: when git grep exits with a status other than 0 (matches) or 1 (no matches). The message is git's standard error output. OSError: when git cannot be run or a candidate file cannot be read. """ > + if (root / '.git').exists(): > + result = subprocess.run( > + ['git', 'grep', '--untracked', '--files-with-matches', > + 'CTLV-GEN', '--', '*.c', '*.h', '*.rst'], git grep without -z quotes paths with special characters (core.quotePath), so such a file would be mis-resolved by root / line. The '*.rst' pathspec, and the '.. SPDX' branch of `_identity()` below, handle a file kind no generator in this series emits: dead. > + cwd=root, capture_output=True, text=True, check=False) > + if result.returncode not in (0, 1): > + raise CtlvError(str(root), result.stderr.strip()) > + candidates = (root / line for line in result.stdout.splitlines()) > + else: > + candidates = (path for suffix in ('*.c', '*.h', '*.rst') > + for path in root.rglob(suffix)) > + > + marked = set() > + for path in candidates: > + if path.is_symlink() or not path.is_file(): > + continue > + text = path.read_text(encoding='utf-8', errors='replace') > + if _identity(text) is not None: > + marked.add(path.relative_to(root).as_posix()) > + return marked > + > + > +def _destination(root, relative): > + destination = root.joinpath(*relative.parts) > + if destination.is_symlink(): > + raise CtlvError(str(destination), > + 'generated output cannot be a symlink') > + parent = destination.parent > + while parent != root and not parent.exists(): > + parent = parent.parent > + if parent.is_symlink() or root.resolve() not in ( > + parent.resolve(), *parent.resolve().parents): > + raise CtlvError(str(destination), > + 'generated output escapes its output root') > + return destination > + > + > +def _replace(destination, text): > + destination.parent.mkdir(parents=True, exist_ok=True) > + fd, temporary = tempfile.mkstemp(dir=destination.parent, > + prefix=f'.{destination.name}.') > + try: > + with os.fdopen(fd, 'w', encoding='utf-8') as stream: > + stream.write(text) > + os.chmod(temporary, 0o644) > + os.replace(temporary, destination) > + finally: > + if os.path.exists(temporary): > + os.unlink(temporary) > + > + > +def _identity(text): Undocumented and non-trivial: It is a hand-written parser for the four-line generated-file header in two comment syntaxes, and it signals "not a generated file" by returning None rather than raising. Suggested docstring: """Parse the CTLV-GEN header of a generated file, if it has one. A generated file starts with four lines: an SPDX line, a "Do not edit, regenerate with tools/ctlv/ctlv-regen.sh, from:" line, a line holding a tab and the repository-relative path of the specification, and a line "CTLV-GEN ". C files (.c and .h) use comment forms ("//" or "/* ... */" for the SPDX line, "/* ... */" for the others). reStructuredText files use ".." comments. Any deviation, such as a different wording, a missing comment terminator, fewer than four lines or a CTLV-GEN line that does not have exactly three fields, makes the text not a generated file. The result is what render() checks against the expected IDENTITIES entry of an output, and what reconcile() checks before replacing an existing file, so that an update refuses to overwrite a hand-written file that happens to sit at a planned destination. Only the header is inspected, never the body. Args: text: str, the full contents of a candidate file. Returns: tuple[str, str, str] | None: (source, mode, kind), where source is the specification path from the third line, mode is "uapi" or "kernel", and kind is "header" or "source". None if text has no valid header. The mode and kind are not checked against the known values here. """ > + lines = text.splitlines() > + if len(lines) < 4: > + return None > + c_spdx = (lines[0].startswith('// SPDX-License-Identifier: ') or > + (lines[0].startswith('/* SPDX-License-Identifier: ') and > + lines[0].endswith(' */'))) > + if c_spdx and \ > + lines[1] == ('/* Do not edit, regenerate with ' > + 'tools/ctlv/ctlv-regen.sh, from: */') and \ > + lines[2].startswith('/*\t') and lines[2].endswith(' */') and \ > + lines[3].startswith('/* CTLV-GEN ') and lines[3].endswith(' */'): > + source = lines[2][3:-3] > + fields = lines[3][3:-3].split() > + elif lines[0].startswith('.. SPDX-License-Identifier: ') and \ > + lines[1] == ('.. Do not edit, regenerate with ' > + 'tools/ctlv/ctlv-regen.sh, from:') and \ > + lines[2].startswith('..\t') and \ > + lines[3].startswith('.. CTLV-GEN '): > + source = lines[2][3:] > + fields = lines[3][3:].split() > + else: > + return None > + if len(fields) == 3 and fields[0] == 'CTLV-GEN': > + return source, fields[1], fields[2] > + return None See above: nothing generates `.rst` (`IDENTITIES` allows only `.h` and `.c`), so the reStructuredText branch (lines 3932-3938) is speculative. > + > + > +def reconcile(paths, output_root=None, check=False): > + """Check or atomically update every generated output.""" The docstring promises "atomically update every generated output", but each file is replaced separately in the loop at 3990; if one `os.replace()` fails midway the tree is left partially updated. The same applies to the docstring of `render()`, which is fine, but this one overstates. Also `destination.read_text(encoding='utf-8')` at 3974 can raise UnicodeDecodeError (uncaught, not a CtlvError) on a corrupted/binary destination. > + plan = load(paths) > + rendered = render(paths, plan) > + root = pathlib.Path(paths.kernel_dir if output_root is None > + else output_root) > + if not root.is_dir(): > + raise CtlvError(str(root), 'output root is not a directory') > + expected = {output.path.as_posix() for output in plan} > + unexpected = sorted(_marked_files(root) - expected) > + if unexpected: > + raise CtlvError('generated outputs', > + 'unexpected files\n' + > + '\n'.join(f' {path}' for path in unexpected)) > + > + states = {} > + problems = [] > + unsafe = [] > + by_path = {output.path: output for output in plan} > + for relative, text in rendered.items(): > + destination = _destination(root, relative) > + if not destination.exists(): > + problems.append(f'missing {relative}') > + states[relative] = (destination, text) > + continue > + if not destination.is_file(): > + raise CtlvError(str(destination), > + 'generated output is not a regular file') > + current = destination.read_text(encoding='utf-8') > + if current == text: > + continue > + problems.append(f'stale {relative}') > + output = by_path[relative] > + mode, kind, _ = IDENTITIES[output.flavour] > + if not check and _identity(current) != ( > + _output_source(output), mode, kind): > + unsafe.append(f'{relative} is not the planned generated file') > + states[relative] = (destination, text) > + if unsafe: > + raise CtlvError('generated outputs', > + 'refusing to overwrite files\n' + > + '\n'.join(f' {problem}' for problem in unsafe)) > + if check and problems: > + raise CtlvError('generated outputs', '\n'.join(problems)) > + for destination, text in states.values(): > + _replace(destination, text) > + return tuple(problems) > diff --git a/tools/ctlv/ctlvgen/lib/semantics.py b/tools/ctlv/ctlvgen/lib/semantics.py > new file mode 100644 > index 000000000000..83290bfb2f46 > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/semantics.py > @@ -0,0 +1,893 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""The effective form of a CTLV family. > + > +The specifications of one inheritance chain resolved into one schema: numeric > +IDs from declaration positions, selectors expanded into placements, and one > +tree per op request and reply. A rule enforced here is not re-enforced > +downstream. > +""" > + > +from __future__ import annotations > + > +import dataclasses > + > +from . import sizes > +from . import spec > +from .errors import CtlvError > + > +MAX_FLAG_BITS = 64 > + > +# These root enums are stored more narrowly than a generic enum range. > +ROOT_ENUM_VALUES_MAX = { > + 'op-type': spec.U8_MAX, > + 'op-availability': spec.U8_MAX, > + 'op-fd-access': spec.U8_MAX, > + 'blob-direction': spec.U8_MAX, > + # BIT(value) is stored in struct ctlv_node_schema.blob.backings. > + 'blob-type': 15, > +} `ROOT_ENUM_VALUES_MAX` bakes in the C field widths of the root family's structs (`u8` op-type etc., "15" for blob-type) with no cross-check against the headers; this is the only place those widths are recorded. > + > + > +def _check_count(where, what, count, maximum): > + if count > maximum: > + raise CtlvError(where, f'{count} {what} exceed the {maximum} maximum') > + > + > +@dataclasses.dataclass(frozen=True) > +class Definition: > + """An effective enum or flags definition.""" > + > + name: str > + kind: str > + family_id: int > + values: tuple > + > + def numbers(self): > + return tuple(number for _, number in self.values) > + > + > +@dataclasses.dataclass(frozen=True) > +class Attr: > + """An effective attr, with its complete wire ID.""" > + > + group: str > + name: str > + family_id: int > + group_id: int > + local_id: int > + attr_id: int > + decl: spec.AttrDecl > + > + @property > + def full_name(self): > + return f'{self.group}.{self.name}' > + > + > +@dataclasses.dataclass(frozen=True) > +class Group: > + name: str > + family_id: int > + group_id: int > + attrs: tuple > + > + > +@dataclasses.dataclass(frozen=True) > +class AttrSet: > + """An effective attr set. A stub that nobody filled stays a stub.""" > + > + name: str > + family_id: int > + set_id: int > + stub: bool > + attrs: tuple > + optional_attrs: tuple > + > + > +@dataclasses.dataclass(frozen=True) > +class Stub: > + """One place where a descendant's fill of a stub splices in. > + > + The declaring family's metadata carries this in place of those members. > + """ > + > + name: str > + family_id: int > + set_id: int > + optional: bool > + > + > +@dataclasses.dataclass(frozen=True) > +class Fill: > + """The members one family puts into a stub it inherited. > + > + root stands for the stub itself, so its children are what splices in. > + """ > + > + name: str > + owner_family_id: int > + family_id: int > + set_id: int > + root: Node > + > + > +@dataclasses.dataclass(frozen=True) > +class Node: > + """One node of an expanded op tree. The root has no attr. > + > + stubs are the extension points selected here and still unfilled. A > + recursive nest becomes a node naming the set in expands and childless > + nodes below it naming it in reenters, so walking children terminates. > + """ > + > + attr: Attr > + required: bool > + max_occurrences: int > + required_children: tuple > + optional_children: tuple > + stubs: tuple = () > + expands: str = None > + reenters: str = None > + > + @property > + def is_root(self): > + return self.attr is None > + > + def children(self): > + return self.required_children + self.optional_children > + > + > +@dataclasses.dataclass(frozen=True) > +class Message: > + """One message root that no op declares, built by the core unasked. > + > + Only the root family declares one. > + """ > + > + name: str > + family_id: int > + set_id: int > + root: Node > + max_len: int > + limit: str > + > + > +@dataclasses.dataclass(frozen=True) > +class Op: > + """An effective op. > + > + availability is the permission to enable and disable the op per device, > + not a state. optional means a device has the op only once its family opts > + in. pre_post asks for a pair of this op's own, on top of the family-wide > + one. > + """ > + > + name: str > + type: str > + fd_access: str > + availability: str > + optional: bool > + pre_post: bool > + family_id: int > + local_id: int > + op_id: int > + request: Node > + reply: Node > + sizes: sizes.OpSizes > + > + > +@dataclasses.dataclass(frozen=True) > +class Family: > + """One effective family: itself plus everything it inherits. > + > + owners maps a family ID to its name for every family in the chain. fills > + is what a generator emits, chain_fills what a device answers with. > + """ > + > + name: str > + family_id: int > + path: str > + chain: tuple > + owners: dict > + definitions: dict > + groups: dict > + attrs: dict > + sets: dict > + ops: tuple > + fills: tuple > + chain_fills: tuple Family.attrs (the flat `full_name` -> Attr dict) is never read outside the resolver (grep for family.attrs finds nothing); only family.groups[...].attrs is used. Unused field. `Message.set_id` is likewise never read. > + messages: tuple > + > + @property > + def parent(self): > + return self.chain[-1].inherits > + > + @property > + def license(self): > + return self.chain[-1].license > + > + @property > + def pre_init(self): > + return self.chain[-1].pre_init > + > + @property > + def post_init(self): > + return self.chain[-1].post_init > + > + @property > + def pre_fini(self): > + return self.chain[-1].pre_fini > + > + def _local(self, items): > + return tuple(item for item in items > + if item.family_id == self.family_id) > + > + @property > + def local_definitions(self): > + return self._local(self.definitions.values()) > + > + @property > + def local_groups(self): > + return self._local(self.groups.values()) > + > + @property > + def local_sets(self): > + """Sets this family declares, a stub it inherited excluded.""" > + return self._local(self.sets.values()) > + > + @property > + def local_ops(self): > + return self._local(self.ops) > + > + > +@dataclasses.dataclass(frozen=True) > +class InputScope: > + kind: str > + index: int > + name: tuple > + node: Node > + > + > +def input_scopes(family, owner_family_id=None): > + """Every schema scope family code may read, with a unique name.""" > + def walk(kind, index, name, node): > + yield InputScope(kind, index, name, node) > + for child in node.children(): > + if child.attr.decl.kind != 'nested': > + continue > + child_name = name + (child.attr.name,) > + yield from walk(kind, index, child_name, child) > + > + if owner_family_id is None: > + ops = family.local_ops > + fills = family.fills > + else: > + ops = tuple(op for op in family.ops > + if op.family_id == owner_family_id) > + fills = tuple(fill for fill in family.chain_fills > + if fill.owner_family_id == owner_family_id) > + > + for index, op in enumerate(ops): > + if op.request is not None: > + yield from walk('op', index, (op.name,), op.request) > + for index, fill in enumerate(fills): > + yield from walk('fill', index, (fill.name,), fill.root) > + > + > +def load(paths): > + """Load the family at paths.spec, resolved against its whole chain.""" > + registry = spec.load_registry(paths) > + return _Resolver(paths, registry).resolve() > + > + > +class _Resolver: > + def __init__(self, paths, registry): > + self.paths = paths > + self.registry = registry > + self.definitions = {} > + self.groups = {} > + self.attrs = {} > + self.sets = {} > + self.ops = [] > + # Which family filled each stub, to name it if a second one tries. > + self.stub_fillers = {} > + # Where each set was declared, for diagnostics. > + self.set_paths = {} > + > + def resolve(self): Undocumented and non-trivial: It is the orchestrator whose correctness depends on the order of eight helper phases that all communicate through mutable _Resolver fields. Suggested docstring: """Resolve the selected specification into one effective Family. Drives the whole resolution in a fixed order that matters. The chain is loaded and its names and registry order checked; then every family is declared root first (enums and flags, groups and attrs, attr sets); then every set is expanded once with _check_sets, so a set that no op selects is still validated and every stub fill is already known; only then are the ops of each family added, because expanding an op tree needs the complete set, group and stub-filler tables. Stub fills are collected last, once for the target family (fills) and once for the whole chain (chain_fills), and the message roots the target declares are built. Reads self.paths and self.registry. Populates self.definitions, self.groups, self.attrs, self.sets, self.ops, self.stub_fillers and self.set_paths through the helpers. The returned Family holds the resolver's own definitions, groups, attrs and sets dicts (not copies), so the resolver must not be reused afterwards. The finished Family is passed through _check_composed, which enforces the limits that only exist once descendant fills are spliced into the stubs. Returns: Family: The effective family for self.paths.spec, with chain ordered from the root family to the target. Throws: CtlvError: If any specification, naming, registration, declaration, op or composed-size rule is violated; the first violation aborts resolution. """ > + chain = self._load_chain() > + self._check_names(chain) > + self._check_order(chain) > + target = chain[-1] > + for family in chain: > + self._add_family(family) > + self._check_sets() > + for family in chain: > + self._add_ops(family) > + fills = self._fills(target) > + chain_fills = self._fills(None) > + _check_count(f'{target.path}: family {target.name}', 'stub fills', > + len(fills), spec.U32_MAX) > + resolved = Family( > + name=target.name, > + family_id=self._family_id(target), > + path=target.path, > + chain=chain, > + owners={self._family_id(item): item.name for item in chain}, > + definitions=self.definitions, > + groups=self.groups, > + attrs=self.attrs, > + sets=self.sets, > + ops=tuple(self.ops), > + fills=fills, > + chain_fills=chain_fills, > + messages=self._messages(target), > + ) Most of the `_check_count(..., spec.U32_MAX)` calls on in-memory Python tuples (stub fills, attr groups, ops, enum values, new attr sets, composed nodes ...) duplicate the meta-schema `maxItems` (enum values, attrs per group and ops at 2^32-1, groups at 65535, flags at 64) and guard counts that cannot realistically reach 2^32 from a YAML file. Only the derived counts (effective ops, composed nodes, fills) need a Python check. If they mirror C `u32`/`u16` fields, say so once, at one place. > + _check_composed(resolved) > + return resolved > + > + def _messages(self, family): > + """The message roots this family declares that no op selects. > + > + The root family alone has them, and has to have them. > + """ > + messages = [] > + for name in sizes.CORE_MESSAGES: > + attr_set = self.sets.get(name) > + if attr_set is None: > + if family.name != spec.CORE_FAMILY_NAME: > + continue > + raise CtlvError(f'{family.path}: family {family.name}', > + f'the root family has to declare the "{name}" ' > + 'attr set, which the core builds a message of') > + if attr_set.family_id != self._family_id(family): > + continue > + where = f'{self.set_paths[name]}: attr set {name}' > + nodes, stubs = self._expand_placements( > + spec.Placements(attr_set.attrs, attr_set.optional_attrs), > + where, (), 0, (name,)) > + if stubs: > + raise CtlvError(where, 'a message the core builds cannot ' > + 'select an extension point') > + root = _root(nodes) > + _check_tree_size(where, root) > + messages.append(Message( > + name=name, > + family_id=attr_set.family_id, > + set_id=attr_set.set_id, > + root=root, > + max_len=sizes.message_max_len(where, name, root), > + limit=sizes.CORE_MESSAGES[name][0], > + )) > + return tuple(messages) > + > + def _fills(self, family): > + """The stubs one family fills, with the members it puts in each. > + > + Passing None asks for the fills of the whole chain. > + """ > + fills = [] > + for name, filler in self.stub_fillers.items(): > + if family is not None and filler != family.name: > + continue > + attr_set = self.sets[name] > + nodes, stubs = self._expand_placements( > + spec.Placements(attr_set.attrs, attr_set.optional_attrs), > + f'{self.set_paths[name]}: attr set {name}', (), 0, (name,)) > + root = _root(nodes, stubs) > + where = f'{self.set_paths[name]}: attr set {name}' > + _check_tree_size(where, root) > + if _reenters_anywhere(root): > + raise CtlvError( > + where, 'a stub fill cannot contain a recursive attr') > + fills.append(Fill( > + name=name, > + owner_family_id=self.registry.family_id(filler), > + family_id=attr_set.family_id, > + set_id=attr_set.set_id, > + root=root, > + )) > + return tuple(fills) > + > + # Loading and identity > + > + def _load_chain(self): Undocumented and non-trivial: It walks the inheritance links upward while prepending, with cycle, naming and root checks interleaved, so the ordering of the result and the meaning of the seen list are easy to get backwards. Suggested docstring: """Load the selected family and every ancestor it inherits. Starts from the specification at self.paths.spec, which must be named after its family (.yaml), and follows each family's inherits name to a sibling specification file until the root family is reached. Every family is checked by _check_root: the root family inherits nothing and every other family has to name a parent. Each parent file must declare the family name it was looked up under. chain is built by inserting each parent at the front, so it ends up ordered root first with the selected family last; chain[0] is always the family whose parent is loaded next. seen runs the other way (selected family up to its ancestors) and is only used to report an inheritance cycle in child-to-ancestor order. This method reads self.paths and does not modify any resolver field; resolve stores the result. Returns: tuple: The loaded family specifications (spec family objects), root family first and the selected family last. Throws: CtlvError: If the file name does not match the family name, the parent file declares a different name, the root family inherits something, a non-root family inherits nothing, or the inheritance chain is cyclic. Also propagated from spec.load when a specification fails to load or validate. """ > + family = spec.load(self.paths.spec, self.paths.schema) > + if self.paths.spec.stem != family.name: > + raise CtlvError(f'{family.path}: family {family.name}', > + 'a specification must be named after its family, ' > + f'so this file must be {family.name}.yaml') > + chain = [family] > + seen = [family.name] > + self._check_root(family) > + while chain[0].inherits: > + parent_name = chain[0].inherits > + child = chain[0] > + if parent_name in seen: > + # seen runs from the selected family to its ancestors. > + raise CtlvError(f'{child.path}: family {child.name}', > + 'inheritance cycle: ' + > + ' -> '.join(seen + [parent_name])) > + parent_path = self.paths.sibling_spec(parent_name) > + parent = spec.load(parent_path, self.paths.schema) > + if parent.name != parent_name: > + raise CtlvError(f'{parent_path}: family {parent.name}', > + f'expected family {parent_name}') > + self._check_root(parent) > + seen.append(parent_name) > + chain.insert(0, parent) > + return tuple(chain) > + > + def _check_root(self, family): > + """Require every family but the root one to name what it inherits.""" > + if family.name == spec.CORE_FAMILY_NAME: > + if family.inherits: > + raise CtlvError(f'{family.path}: family {family.name}', > + 'the root family heads every chain, so it ' > + 'inherits nothing') > + elif not family.inherits: > + raise CtlvError(f'{family.path}: family {family.name}', > + 'must name the family it inherits, which is the ' > + f'root {spec.CORE_FAMILY_NAME} for a family ' > + 'directly below it') > + > + def _check_names(self, chain): > + """Require every child family name to carry its whole ancestry. > + > + A dash is the only separator; a dot selects group.attr. > + """ > + for parent, child in zip(chain, chain[1:]): > + suffix = child.name.removeprefix(f'{parent.name}-') > + if not suffix or suffix == child.name: > + raise CtlvError(f'{child.path}: family {child.name}', > + f'a family that inherits {parent.name} must ' > + f'be named {parent.name}-') > + > + def _check_order(self, chain): > + """Require a parent to be registered before what inherits it. > + > + Ascending positions let the core walk a chain once and index it. > + """ > + for parent, child in zip(chain, chain[1:]): > + if self._family_id(parent) >= self._family_id(child): > + raise CtlvError(f'{child.path}: family {child.name}', > + f'is registered before {parent.name}, the ' > + 'family it inherits') > + > + def _family_id(self, family): > + family_id = self.registry.family_id(family.name) > + if family_id is None: > + raise CtlvError(f'{family.path}: family {family.name}', > + f'not registered in {self.registry.path}') > + return family_id > + > + # Declarations > + > + def _add_family(self, family): > + family_id = self._family_id(family) > + self._add_definitions(family, family_id, family.enums, 'enum') > + self._add_definitions(family, family_id, family.flags, 'flags') > + self._add_groups(family, family_id) > + self._add_sets(family, family_id) > + > + def _add_definitions(self, family, family_id, definitions, kind): Undocumented and non-trivial: one routine handles two kinds with different numbering rules and different limit checks, including an extra limit that applies only to certain root-family enums. Suggested docstring: """Add one family's enum or flags definitions to the effective table. Called by _add_family once with the family's enums (kind 'enum') and once with its flags (kind 'flags'). Enum and flags names share one namespace, so a name already present from this or any ancestor family is rejected. Value numbering is assigned from declaration position: enum values start at 1 because 0 is reserved, flag bits start at 0. Limits: a flags definition may have at most MAX_FLAG_BITS bits; an enum at most spec.U32_MAX values, and an enum of the root family (spec.CORE_FAMILY_ID) named in ROOT_ENUM_VALUES_MAX gets that tighter limit because it is stored in a narrower field. Populates self.definitions, keyed by definition name, with immutable Definition objects whose values are (name, number) tuples and whose family_id is the declaring family. Reads self.definitions for the redeclaration check; nothing else in the resolver is touched. Args: family: The spec family object being declared; supplies path for diagnostics. family_id: int registry ID of that family, recorded in each Definition. definitions: Iterable of the spec enum or flags declarations to add (each has name and values). kind: str, either 'enum' or 'flags'. Throws: CtlvError: If a name is already declared as an enum or flags, a flags definition has more than MAX_FLAG_BITS bits, or an enum has more values than its limit allows. """ > + for definition in definitions: > + where = f'{family.path}: {kind} {definition.name}' > + self._reject_redeclaration(where, self.definitions, > + definition.name, 'enum or flags') > + if kind == 'flags' and len(definition.values) > MAX_FLAG_BITS: > + raise CtlvError(where, f'more than {MAX_FLAG_BITS} flag bits') > + if kind == 'enum': > + _check_count(where, 'enum values', len(definition.values), > + spec.U32_MAX) > + if family_id == spec.CORE_FAMILY_ID: > + maximum = ROOT_ENUM_VALUES_MAX.get(definition.name) > + if maximum is not None: > + _check_count(where, 'enum values', > + len(definition.values), maximum) > + # Enum values from one, zero being reserved; flag bits from zero. > + first = 1 if kind == 'enum' else 0 > + values = tuple((name, first + index) > + for index, name in enumerate(definition.values)) > + self.definitions[definition.name] = Definition( > + definition.name, kind, family_id, values) > + > + def _add_groups(self, family, family_id): Undocumented and non-trivial: It derives the packed 64-bit attr IDs from declaration positions and keeps three tables (groups, attrs, and the group/set namespace) consistent. Suggested docstring: """Add one family's attr groups and their attrs to the effective tables. IDs come from declaration positions, so reordering a specification changes the wire format. A group's ID is its 1-based position within this family's groups; an attr's local ID is its 1-based position within its group. The complete attr ID packs them as (family_id << 48) | (group_id << 32) | local_id. Attr groups and attr sets share one selector namespace, so a group name that matches any previously declared group or set (from this or an ancestor family) is rejected. Attr names must be unique within a group. Each attr that is an enum or flags attr must reference an existing definition of the right kind (_check_reference), so _add_definitions has to have run for this family first; the check also sees definitions inherited from ancestors. Populates self.groups (keyed by group name) and self.attrs (keyed by 'group.name'). Reads self.groups, self.sets and self.definitions. Args: family: The spec family object being declared. family_id: int registry ID of that family, stored in every Group and Attr. Throws: CtlvError: If the family has more than spec.U16_MAX groups or a group more than spec.U32_MAX attrs, a group name is already a group or a set, an attr name repeats within its group, or an enum or flags attr names an unknown definition or one of the other kind. """ > + _check_count(f'{family.path}: family {family.name}', 'attr groups', > + len(family.groups), spec.U16_MAX) > + for group_id, group in enumerate(family.groups, start=1): > + where = f'{family.path}: attr group {group.name}' > + self._reject_redeclaration(where, self.groups, group.name, > + 'attr group') > + self._reject_redeclaration(where, self.sets, group.name, > + 'attr set') > + attrs = [] > + names = set() > + _check_count(where, 'attrs', len(group.attrs), spec.U32_MAX) > + for local_id, decl in enumerate(group.attrs, start=1): > + attr_where = f'{family.path}: attr {group.name}.{decl.name}' > + if decl.name in names: > + raise CtlvError(attr_where, 'duplicate attr') > + names.add(decl.name) > + self._check_reference(attr_where, decl) > + attrs.append(Attr( > + group=group.name, > + name=decl.name, > + family_id=family_id, > + group_id=group_id, > + local_id=local_id, > + attr_id=(family_id << 48) | (group_id << 32) | local_id, > + decl=decl, > + )) > + effective = Group(group.name, family_id, group_id, tuple(attrs)) > + self.groups[group.name] = effective > + for attr in attrs: > + self.attrs[attr.full_name] = attr > + > + def _check_reference(self, where, decl): > + if decl.kind not in ('enum', 'flags'): > + return > + definition = self.definitions.get(decl.ref) > + if definition is None: > + raise CtlvError(where, f'unknown {decl.kind} "{decl.ref}"') > + if definition.kind != decl.kind: > + raise CtlvError(where, f'"{decl.ref}" is a {definition.kind} ' > + f'definition, not {decl.kind}') > + > + def _add_sets(self, family, family_id): Undocumented and non-trivial: the same loop either declares a new set or turns a redeclaration into a stub fill, and set IDs are counted only for the new ones. Suggested docstring: """Add one family's attr sets, filling inherited stubs where redeclared. A set whose name is not yet in self.sets is a new declaration: its name must not be an attr group, and it gets the next set ID, counted from 1 per family over new sets only. A set whose name is already present is not a new set at all; it must be a fill of an inherited stub, which _fill_stub validates and applies, and it consumes no set ID here. Stubs are attr sets declared with no members as extension points. A stub stays a stub, with its declaring family's IDs, until a descendant fills it. Populates self.sets (name to AttrSet) and self.set_paths (name to the path of the file that last defined the set, used in diagnostics). Via _fill_stub it also updates self.stub_fillers. Reads self.groups for the namespace check. The members of the sets are stored unexpanded; expansion happens later in _check_sets and the op trees. Args: family: The spec family object being declared. family_id: int registry ID of that family, used as the owner of new sets. Throws: CtlvError: If a new set's name is already an attr group, there are more than spec.U32_MAX new sets, or a redeclaration is not a valid stub fill (raised from _fill_stub). """ > + set_id = 0 > + for attr_set in family.sets: > + where = f'{family.path}: attr set {attr_set.name}' > + existing = self.sets.get(attr_set.name) > + if existing is not None: > + self._fill_stub(where, family, existing, attr_set) > + continue > + self._reject_redeclaration(where, self.groups, attr_set.name, > + 'attr group') > + _check_count(where, 'new attr sets', set_id + 1, > + spec.U32_MAX) > + set_id += 1 > + self.set_paths[attr_set.name] = family.path > + self.sets[attr_set.name] = AttrSet( > + name=attr_set.name, > + family_id=family_id, > + set_id=set_id, > + stub=attr_set.stub, > + attrs=attr_set.attrs, > + optional_attrs=attr_set.optional_attrs, > + ) > + > + def _fill_stub(self, where, family, existing, filler): > + """Replace an inherited empty stub with a descendant's members.""" > + filled_by = self.stub_fillers.get(existing.name) > + if filled_by is not None: > + raise CtlvError(where, f'stub was already filled by family ' > + f'{filled_by}') > + if not existing.stub: > + raise CtlvError(where, 'attr set is already declared by an ' > + 'inherited family') > + if filler.stub: > + raise CtlvError(where, 'a stub cannot be redeclared; a further ' > + 'extension point needs a new name') > + self.stub_fillers[existing.name] = family.name > + self.set_paths[existing.name] = family.path > + # A fill completes an inherited set, so the IDs are the declarer's. > + self.sets[existing.name] = dataclasses.replace( > + existing, stub=False, attrs=filler.attrs, > + optional_attrs=filler.optional_attrs) > + > + def _check_sets(self): > + """Expand every set, so one that no op selects is still checked.""" > + for name, attr_set in self.sets.items(): > + where = f'{self.set_paths[name]}: attr set {name}' > + # Seeding the stack makes a cycle read from the set expanded. > + _nodes, _stubs = self._expand_placements( > + spec.Placements(attr_set.attrs, attr_set.optional_attrs), > + where, (), 0, set_stack=(attr_set.name,), > + opened=(attr_set.name,)) > + > + def _reject_redeclaration(self, where, namespace, name, what): > + if name in namespace: > + raise CtlvError(where, f'{what} "{name}" is already declared') > + > + # Ops > + > + def _add_ops(self, family): Undocumented and non-trivial: It builds the op IDs, rejects duplicates across the whole chain, and ties together tree expansion, context checks and size computation in one step. Suggested docstring: """Add one family's ops to the effective op list. Called for each family in chain order, after all families' definitions, groups and sets are declared. Each op's request and reply placements are expanded into Node trees by _tree, validated for where their attrs may appear by _check_context, and measured by sizes.compute. An op's local ID is its 1-based position in its family and its op ID is (family_id << 48) | local_id. Op names are unique across the whole chain, so an op in a descendant may not reuse an inherited op name. The effective count (inherited plus this family's) is limited too. Appends Op objects to self.ops, which therefore accumulates inherited ops before the target's. Reads self.ops (duplicate and count checks), and, through _tree and the helpers it uses, self.attrs, self.groups, self.sets and self.stub_fillers. An op without a request or reply keeps None for it. Args: family: The spec family object whose ops are added. Throws: CtlvError: If the family is not registered, an op count exceeds spec.U32_MAX, an op name is already declared, a tree is invalid (unknown selector, cycle, too deep, too large), or an attr appears in a context it may not (_check_context); also whatever sizes.compute raises. """ > + family_id = self._family_id(family) > + _check_count(f'{family.path}: family {family.name}', 'ops', > + len(family.ops), spec.U32_MAX) > + _check_count(f'{family.path}: family {family.name}', 'effective ops', > + len(self.ops) + len(family.ops), spec.U32_MAX) > + for local_id, op in enumerate(family.ops, start=1): > + where = f'{family.path}: op {op.name}' > + if any(existing.name == op.name for existing in self.ops): > + raise CtlvError(where, f'op "{op.name}" is already declared') > + request = None > + if op.request is not None: > + request = self._tree(op.request, where + ' request') > + reply = None > + if op.reply is not None: > + reply = self._tree(op.reply, where + ' reply') > + self._check_context(where, op, request, reply) > + self.ops.append(Op( > + name=op.name, > + type=op.type, > + fd_access=op.fd_access, > + availability=op.availability, > + optional=op.optional, > + pre_post=op.pre_post, > + family_id=family_id, > + local_id=local_id, > + op_id=(family_id << 48) | local_id, > + request=request, > + reply=reply, > + sizes=sizes.compute(where, op, request, reply), > + )) > + > + def _tree(self, placements, where): > + children, stubs = self._expand_placements(placements, where, (), 0) > + root = _root(children, stubs) > + _check_tree_size(where, root) > + return root > + > + def _expand_placements(self, placements, where, stack, depth, > + set_stack=(), opened=()): > + """Return the child nodes of one level and the stubs selected there.""" > + merged = {} > + expanded, stubs = self._expand(placements.attrs, False, where, > + set_stack) > + optional, more = self._expand(placements.optional_attrs, True, where, > + set_stack) > + expanded += optional > + stubs += more > + for attr, required, fill in expanded: > + entry = merged.get(attr.attr_id) > + if entry is None: > + merged[attr.attr_id] = [attr, required, fill] > + elif entry[2] != fill and (entry[2] is not None or > + fill is not None): Redundant condition: "entry[2] != fill and (entry[2] is not None or fill is not None)". If the two values differ at least one of them is not None, so the second conjunct is always true. > + raise CtlvError( > + where, f'attr {attr.full_name} is selected more than ' > + 'once in one composed nest') > + elif required: > + # A required selection at this level beats an optional one. > + entry[1] = True > + nodes = tuple(self._node(attr, required, where, stack, depth, opened) > + for attr, required, _fill in merged.values()) > + return nodes, _merge_stubs(stubs) > + > + def _expand(self, selectors, force_optional, where, stack): > + expanded = [] > + stubs = () > + for selector in selectors: > + attrs, found = self._expand_one(selector, force_optional, where, > + stack) > + expanded += attrs > + stubs += found > + return expanded, stubs > + > + def _expand_one(self, selector, force_optional, where, stack): Undocumented and non-trivial: It returns a pair whose first element carries a per-attr fill tag and whose optionality is rewritten on the way down, and the three selector forms resolve in a subtle priority order. Suggested docstring: """Expand a single selector into attrs and unfilled stubs. A selector is one of, in this lookup order: 'group.attr', naming one attr; a group name, naming all its attrs; or an attr set name. A set that is still a stub expands to nothing and yields a Stub reference instead, so a descendant can splice its fill in later. A filled or ordinary set expands recursively into the selectors of its attrs and optional_attrs; the stub check comes before the cycle check, so a stub is never reported as a cycle. Optionality: each returned attr carries a required flag, True unless force_optional. Under an optional selection of a set every member becomes optional, and the set's own optional_attrs are always optional. The third element of each triple is a fill tag. It is None except for attrs reached through a set that some family filled (a name in self.stub_fillers), when it is that set's name; the outer set overrides any tag from inside. The caller uses the tag to tell "the same set reached twice", which is allowed, from a genuine duplicate selection. Reads self.attrs, self.groups, self.sets and self.stub_fillers; modifies no state. Args: selector: str, an attr 'group.name', a group name or a set name. force_optional: bool, True when an enclosing selection is optional. where: str, location used in diagnostics. stack: tuple of the set names being expanded, outermost first, used to detect a set that contains itself. Returns: tuple: (list of (Attr, bool required, str-or-None fill tag), tuple of Stub), the attrs selected and the extension points found. Throws: CtlvError: If the selector names nothing known ("unknown attr" for a dotted name, "unknown selector" otherwise), or sets select each other in a cycle. """ > + if '.' in selector: > + attr = self.attrs.get(selector) > + if attr is None: > + raise CtlvError(where, f'unknown attr "{selector}"') > + return [(attr, not force_optional, None)], () > + group = self.groups.get(selector) > + if group is not None: > + return [(attr, not force_optional, None) > + for attr in group.attrs], () > + attr_set = self.sets.get(selector) > + if attr_set is None: > + raise CtlvError(where, f'unknown selector "{selector}"') > + if attr_set.stub: > + # Nothing to expand until a descendant fills it. > + return [], (Stub(attr_set.name, attr_set.family_id, > + attr_set.set_id, force_optional),) > + if selector in stack: > + raise CtlvError(where, 'attr set cycle: ' + > + ' -> '.join(stack + (selector,))) > + stack += (selector,) > + # Under an optional selection every member becomes optional. > + expanded, stubs = self._expand(attr_set.attrs, force_optional, where, > + stack) > + optional, more = self._expand(attr_set.optional_attrs, True, where, > + stack) > + expanded += optional > + if selector in self.stub_fillers: > + expanded = [(attr, required, selector) > + for attr, required, _fill in expanded] > + return expanded, stubs + more > + > + def _node(self, attr, required, where, stack, depth, opened=()): Undocumented and non-trivial: It handles plain attrs, nests, recursive nests and re-entry in one body, with stack, depth and opened tracking three different kinds of "already inside". Suggested docstring: """Build the Node for one selected attr, expanding it if it is a nest. A non-nested attr becomes a leaf. A nested attr is expanded through _expand_placements into required and optional child nodes plus unfilled stubs. Three pieces of context are threaded through the recursion: stack is the attr IDs of the enclosing nests (a repeat means an illegal nesting cycle), depth is how many nests deep this one is (limited to sizes.MAX_DECLARED_NEST_DEPTH), and opened is the names of attr sets currently open above (what a recursive attr may re-enter). holds, from _nested_set, is the one attr set the nest contains, if it contains only that. A recursive attr whose set is already in opened is not expanded again: it becomes a childless Node with reenters set to that set name, so walking children terminates. Otherwise the node is expanded with expands=holds, and a recursive attr must then re-enter its set somewhere below, checked by _reenters. A non-recursive nest that returns to an enclosing attr is a nesting cycle, not a re-entry. Reads self.attrs (through _attr_by_id for diagnostics) and self.sets and self.groups (through _nested_set and expansion); modifies no state. Args: attr: Attr, the effective attr being placed. required: bool, whether the placement is required (False for optional). where: str, location used in diagnostics. stack: tuple of int attr IDs of the enclosing nested attrs. depth: int, the number of enclosing nests. opened: tuple of str set names open above this attr. Returns: Node: The leaf, re-entry node or fully expanded nest node. Throws: CtlvError: If the nest forms a cycle, exceeds the depth limit, is recursive but not over a single attr set, or never re-selects itself; also from expanding its children. """ > + decl = attr.decl > + if decl.kind != 'nested': > + return Node(attr, required, decl.max_occurrences, (), ()) > + holds = self._nested_set(where, attr) > + if decl.recursive and holds in opened: > + # Open above, so one level further down rather than a new one. > + return Node(attr, required, decl.max_occurrences, (), (), > + reenters=holds) > + if attr.attr_id in stack: > + names = [self._attr_by_id(i).full_name for i in stack] > + raise CtlvError(where, 'nesting cycle: ' + > + ' -> '.join(names + [attr.full_name])) > + if depth + 1 > sizes.MAX_DECLARED_NEST_DEPTH: > + raise CtlvError(where, f'{attr.full_name} nests deeper than the ' > + f'{sizes.MAX_DECLARED_NEST_DEPTH} level ' > + 'limit') > + children, stubs = self._expand_placements( > + spec.Placements(decl.attrs, decl.optional_attrs), where, > + stack + (attr.attr_id,), depth + 1, > + opened=opened + ((holds,) if holds else ())) > + node = Node(attr, required, decl.max_occurrences, > + tuple(child for child in children if child.required), > + tuple(child for child in children if not child.required), > + stubs, expands=holds) > + if decl.recursive and not _reenters(node, holds): > + raise CtlvError(where, f'{attr.full_name} is recursive, but attr ' > + f'set "{holds}" never selects it again') > + return node > + > + def _nested_set(self, where, attr): > + """The one attr set a nest holds, when that is all it holds. > + > + Only such a nest can be re-entered, a re-entry meaning the members > + open above. > + """ > + decl = attr.decl > + if not decl.recursive: > + if decl.optional_attrs or len(decl.attrs) != 1: > + return None > + return decl.attrs[0] if decl.attrs[0] in self.sets else None > + target = decl.attrs[0] > + if target in self.groups: > + raise CtlvError(where, f'{attr.full_name} is recursive, so it ' > + f'nests an attr set, and "{target}" is an ' > + 'attr group') > + if target not in self.sets: > + raise CtlvError(where, f'unknown selector "{target}"') > + return target > + > + def _attr_by_id(self, attr_id): > + for attr in self.attrs.values(): > + if attr.attr_id == attr_id: > + return attr > + raise AssertionError(f'no attr with ID {attr_id:#x}') Raising AssertionError from a lookup helper and, in `sizes.py`, `_children_max_len()` (unlimited / re-entering child -> AssertionError) are only unreachable if `_check_subtree()` ran first. See below: it does not cover core messages (`_messages()` calls `sizes.message_max_len()` without `_check_context`), so "max-occurrences: unlimited" or a recursive attr in the root family's "error-info" set produces a traceback, not a CtlvError. > + > + def _check_context(self, where, op, request, reply): > + if request is not None: > + self._check_subtree(where, request, 'request') > + if reply is None: > + return > + if op.type == 'query': > + self._check_subtree(where, reply, 'query reply') > + self._check_reentry(where, reply) > + elif op.type == 'event': > + self._check_subtree(where, reply, 'event') > + else: > + self._check_subtree(where, reply, 'action reply') > + > + def _check_reentry(self, where, root): > + """A repeating level cannot also be widened by a descendant.""" > + if not _reenters_anywhere(root) or not _stubs_anywhere(root): > + return > + raise CtlvError(f'{where} query reply', > + 're-enters a level and selects an extension point, ' > + 'and a descendant cannot widen a level that repeats') > + > + def _check_subtree(self, where, node, context): Undocumented and non-trivial: its context argument is an unchecked string tag that selects which of three restrictions apply, and the rules are only visible by reading every branch. Suggested docstring: """Reject attrs that may not appear in this kind of message. Walks the tree below node depth first and applies per-attr rules chosen by context, the role of the tree: 'request', 'query reply', 'event' or 'action reply' (as passed by _check_context). Rules: an attr with unlimited occurrences and an attr that re-enters a level (so has no maximum length) are allowed only in a query reply; a blob attr is allowed only in a request. The root node has no attr and is skipped, but its children are checked. Stubs are not looked into, since their fills are not part of the tree yet (_check_composed covers the composed form). Reads only the node tree; modifies no resolver state. Args: where: str, op location used as the prefix of diagnostics. node: Node, the root or any subtree to check. context: str, one of 'request', 'query reply', 'event' or 'action reply'. Throws: CtlvError: If an attr violates a rule for this context; the message names the attr and the rule. """ > + if node.attr is not None: > + name = node.attr.full_name > + if (node.max_occurrences == spec.UNLIMITED and > + context != 'query reply'): > + raise CtlvError(f'{where} {context}', > + f'{name} has unlimited occurrences, which ' > + 'only a query reply may contain') > + if node.reenters is not None and context != 'query reply': > + raise CtlvError(f'{where} {context}', > + f'{name} re-enters "{node.reenters}", so it ' > + 'has no maximum length, which only a query ' > + 'reply may contain') > + if node.attr.decl.kind == 'blob' and context != 'request': > + raise CtlvError(f'{where} {context}', > + f'{name} is a blob attr, which only a ' > + 'request may contain') > + for child in node.children(): > + self._check_subtree(where, child, context) > + > + > +def _reenters(node, name): > + """Whether anything below node re-enters name.""" > + return any(child.reenters == name or _reenters(child, name) > + for child in node.children()) > + > + > +def _reenters_anywhere(node): > + return any(child.reenters is not None or _reenters_anywhere(child) > + for child in node.children()) > + > + > +def _stubs_anywhere(node): > + return bool(node.stubs) or any(_stubs_anywhere(child) > + for child in node.children()) `_reenters_anywhere()`/`_stubs_anywhere()` here are duplicated in `sizes.py` (`_reenters`) and `cgen.py` (`_stubs_anywhere`). > + > + > +def _composed_members(fills, node, force_optional=False): > + members = [(child, force_optional) > + for child in node.required_children] > + members += [(child, True) for child in node.optional_children] > + for stub in node.stubs: > + fill = fills.get((stub.family_id, stub.set_id)) > + if fill is not None: > + members += _composed_members( > + fills, fill.root, force_optional or stub.optional) > + return members > + > + > +def _check_composed(family): Undocumented and non-trivial: It validates the family as a descendant will see it, splicing stub fills into trees through a closure whose counts and duplicate checks apply to the composed members, not the declared ones. Suggested docstring: """Check the limits and uniqueness of the fully composed op trees. The earlier checks run on each tree as declared, with stubs left as references. A device's schema, however, has the fills of every family in the chain spliced in at the stubs. This check redoes the member accounting on that composed form for every op's request and reply: at each node, _composed_members gathers the children plus the members of any fill chosen by the node's stubs (an optional stub makes the fill's members optional), and walk counts required and optional members and the nodes below. A fill is found by (family_id, set_id) in chain_fills. Stubs nobody filled add nothing. Checks: composed required and optional member counts per node, the node count per request or reply, and the combined request plus reply count per op, each against spec.U32_MAX. An attr must also not appear twice among a node's composed members, which can only happen after a fill is spliced in (for example a fill repeating an attr the declaring set already selects). Re-entry nodes are childless, so they are not unrolled. Reads family.ops, family.chain_fills and family.path; modifies nothing. Args: family: Family, the fully resolved family to validate. Throws: CtlvError: If a composed count exceeds its limit, or an attr is selected more than once in one composed nest. """ > + fills = {(fill.family_id, fill.set_id): fill > + for fill in family.chain_fills} > + > + def walk(where, node): > + members = _composed_members(fills, node) > + required = sum(not optional for _child, optional in members) > + optional = len(members) - required > + seen = set() > + > + _check_count(where, 'composed required attrs', required, > + spec.U32_MAX) > + _check_count(where, 'composed optional attrs', optional, > + spec.U32_MAX) > + count = 1 > + for child, _optional in members: > + if child.attr.attr_id in seen: > + raise CtlvError( > + where, f'attr {child.attr.full_name} is selected more ' > + 'than once in one composed nest') > + seen.add(child.attr.attr_id) > + count += walk(where, child) > + return count > + > + for op in family.ops: > + total = 0 > + for which, root in (('request', op.request), ('reply', op.reply)): > + if root is None: > + continue > + where = f'{family.path}: op {op.name} {which}' > + count = walk(where, root) > + _check_count(where, 'composed schema nodes', count, > + spec.U32_MAX) > + total += count > + _check_count(f'{family.path}: op {op.name}', > + 'composed request and reply nodes', total, > + spec.U32_MAX) > + > + > +def _node_count(node): > + return 1 + sum(_node_count(child) for child in node.children()) > + > + > +def _stub_count(node): > + return len(node.stubs) + sum(_stub_count(child) > + for child in node.children()) > + > + > +def _check_tree_size(where, root): > + _check_count(where, 'schema nodes', _node_count(root), spec.U32_MAX) > + _check_count(where, 'schema stubs', _stub_count(root), spec.U32_MAX) > + > + > +def _root(children, stubs=()): > + return Node(None, True, 1, > + tuple(node for node in children if node.required), > + tuple(node for node in children if not node.required), > + stubs) > + > + > +def _merge_stubs(stubs): > + """One reference per stub selected at a level, required winning.""" > + merged = {} > + for stub in stubs: > + previous = merged.get(stub.name) > + if previous is None or not stub.optional: > + merged[stub.name] = stub > + return tuple(merged.values()) > diff --git a/tools/ctlv/ctlvgen/lib/sizes.py b/tools/ctlv/ctlvgen/lib/sizes.py > new file mode 100644 > index 000000000000..864e3b6873e7 > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/sizes.py > @@ -0,0 +1,238 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""Length arithmetic over an expanded op tree. > + > +These numbers decide whether an op can run at all, so they are kept apart from > +the rest of resolution and tested against independently calculated lengths. > +The constants mirror include/uapi/linux/ctlv.h, and a test checks that. "a test checks that" - there is no test. `ATTR_HDR_LEN` (16) and `EVENT_HDR_LEN` (24) are also not asserted against `sizeof(struct ctlv_attr)`/the event record header anywhere, so a layout change in `include/uapi/linux/ctlv.h` would make every computed maximum wrong while the generated `static_asserts` (which use these numbers) still pass. > +""" > + > +from __future__ import annotations > + > +import dataclasses > + > +from . import spec > +from .errors import CtlvError > + > +ALIGNTO = 8 > +ATTR_HDR_LEN = 16 > +EVENT_HDR_LEN = 24 > + > +MAX_INLINE_MESSAGE_LEN = 0x100000 > +MAX_ERROR_LEN = 0x4000 > +MAX_NEST_DEPTH = 32 > +DEFAULT_QUERY_REPLY_LEN = 0x10000 > +MAX_EVENT_QUEUE_LEN = 0x1000000 > + > +# One level short of what the transport carries: op-schema-get describes a > +# message, and a description nests one level deeper than what it describes. > +MAX_DECLARED_NEST_DEPTH = MAX_NEST_DEPTH - 1 > + > +# Every integer, enum and flags payload is 64-bit. > +SCALAR_PAYLOAD_LEN = 8 > + > +# A bool payload is one byte, padded to eight only as a placement. > +BOOL_PAYLOAD_LEN = 1 > + > +# How many core scalar descriptor attrs each blob backing carries. A test > +# checks these against the blob group of the core family. > +BLOB_DESCRIPTOR_ATTRS = { > + 'va': 3, > +} > + > +# Attr sets of the root family that are message roots in their own right, each > +# with the transport header's limit and the name the header gives it. > +CORE_MESSAGES = { > + 'error-info': ('CTLV_MAX_ERROR_LEN', MAX_ERROR_LEN), > +} > + > + > +@dataclasses.dataclass(frozen=True) > +class OpSizes: > + """Generated lengths of one op. Zero means "does not apply".""" > + > + request_max_len: int > + action_reply_max_len: int > + query_reply_min_len: int > + query_reply_suggested_len: int > + event_record_max_len: int > + > + > +def align(length): > + return (length + ALIGNTO - 1) & ~(ALIGNTO - 1) > + > + > +def attr_max_len(node): > + """Maximum length of one occurrence of node, excluding its own padding.""" > + return ATTR_HDR_LEN + _payload_max_len(node) > + > + > +def root_max_len(root): > + """Maximum length of a complete message root.""" > + return ATTR_HDR_LEN + _payload_max_len(root) `root_max_len()` and `attr_max_len()` have identical bodies (`ATTR_HDR_LEN` + `_payload_max_len(x)`); keep one. > + > + > +def min_reply_len(root): > + """Buffer length needed for the largest indivisible leaf and its path. > + > + A leaf below a re-entered level sits as deep as the transport allows. > + """ > + if _reenters(root): > + return (MAX_NEST_DEPTH + 1) * ATTR_HDR_LEN + _largest_leaf(root) > + return _min_walk(root, 0) > + > + > +def compute(where, op, request, reply): > + """Return the generated lengths of one op.""" > + request_max_len = 0 > + if request is not None: > + request_max_len = root_max_len(request) > + _check_inline(where, 'request', request_max_len) > + > + action_reply_max_len = 0 > + query_reply_min_len = 0 > + query_reply_suggested_len = 0 > + event_record_max_len = 0 > + > + if op.type == 'action' and reply is not None: > + action_reply_max_len = root_max_len(reply) > + _check_inline(where, 'action reply', action_reply_max_len) > + elif op.type == 'query': > + query_reply_min_len = min_reply_len(reply) > + _check_inline(where, 'largest query reply leaf and its path', > + query_reply_min_len) > + query_reply_suggested_len = _suggested_len( > + where, op.suggested_reply_length, query_reply_min_len) > + elif op.type == 'event': > + payload_max_len = root_max_len(reply) > + _check_inline(where, 'event payload', payload_max_len) > + event_record_max_len = EVENT_HDR_LEN + align(payload_max_len) > + > + return OpSizes(request_max_len=request_max_len, > + action_reply_max_len=action_reply_max_len, > + query_reply_min_len=query_reply_min_len, > + query_reply_suggested_len=query_reply_suggested_len, > + event_record_max_len=event_record_max_len) > + > + > +def composed(where, op, request, reply): > + """Return the lengths one op needs for one family. > + > + compute()'s numbers over the composed tree, fills spliced in. A suggestion > + is raised here rather than refused. > + """ > + sizes = OpSizes(request_max_len=root_max_len(request) if request else 0, > + action_reply_max_len=0, query_reply_min_len=0, > + query_reply_suggested_len=0, event_record_max_len=0) > + _check_inline(where, 'request', sizes.request_max_len) > + > + if op.type == 'action' and reply is not None: > + sizes = dataclasses.replace(sizes, > + action_reply_max_len=root_max_len(reply)) > + _check_inline(where, 'action reply', sizes.action_reply_max_len) > + elif op.type == 'query': > + minimum = min_reply_len(reply) > + _check_inline(where, 'largest query reply leaf and its path', minimum) > + sizes = dataclasses.replace( > + sizes, query_reply_min_len=minimum, > + query_reply_suggested_len=max(op.sizes.query_reply_suggested_len, > + minimum)) > + elif op.type == 'event': > + payload_max_len = root_max_len(reply) > + _check_inline(where, 'event payload', payload_max_len) > + record = EVENT_HDR_LEN + align(payload_max_len) > + if record > MAX_EVENT_QUEUE_LEN: > + raise CtlvError(where, f'event record needs {record} bytes, more ' > + f'than the {MAX_EVENT_QUEUE_LEN} byte ' > + 'event queue limit') > + sizes = dataclasses.replace(sizes, event_record_max_len=record) > + return sizes Dead and, as written, wrong-for-its-purpose: `sizes.composed()` has no caller (grep over `tools/ctlv`: only the definition). It also takes an Op (it reads op.sizes) while `compute()` takes an OpSpec. The effect is that the lengths of an op are computed with unfilled stubs contributing nothing (`_children_max_len()` ignores node.stubs), so a descendant's fill that grows a request/reply beyond `CTLV_MAX_INLINE_MESSAGE_LEN`, or a query's minimum reply length, is never checked at generation time. Either wire `composed()` into `_check_composed()` (what its docstring says it is for) or delete it. The event-queue limit it checks is not checked in `compute()` either; only the generated C `static_assert` (`gen_kernel.py`) catches it, at C compile time. > + > + > +def message_max_len(where, name, root): > + """Return the maximum length of one core message, against its limit.""" > + limit_name, limit = CORE_MESSAGES[name] > + length = root_max_len(root) > + if length > limit: > + raise CtlvError(where, f'needs {length} bytes, more than the {limit} ' > + f'of {limit_name}') > + return length > + > + > +def _payload_max_len(node): Undocumented and non-trivial: It computes the worst-case payload length of a node by wire kind, with sizing rules (NUL terminator, bool as one byte, blob as descriptor attrs) that depend on conventions elsewhere, and it is mutually recursive with attr_max_len through _children_max_len. Suggested docstring: """Return the maximum payload length of node, without its attr header. The payload is what follows the ATTR_HDR_LEN header of one occurrence of an attr, or of the message root. The rule depends on the declared kind: - uint, sint, enum, flags: SCALAR_PAYLOAD_LEN (8); all are 64-bit on the wire. - bool: BOOL_PAYLOAD_LEN (1); the padding to eight bytes is a placement matter and is added by the caller, not here. - string: max-length plus one for the terminating NUL, which max-length excludes. - binary: max-length. - blob: the widest backing among the declared ones, counted as that many core scalar descriptor attrs (BLOB_DESCRIPTOR_ATTRS), each aligned to ALIGNTO. The blob's own content is not part of the message. - anything else (nested), and the root: the sum over the children, see _children_max_len(), which multiplies each child by its max-occurrences and pads it. The result is a worst case used by attr_max_len(), root_max_len() and the inline message limit checks in compute() and composed(). It is only defined for trees that are bounded; see Throws. Args: node: semantics.Node of the expanded op tree. A root has no attr, other nodes have attr.decl (a spec.AttrDecl) giving the kind and limits. Returns: int: the payload length in bytes, not aligned. Throws: AssertionError: when a nested child has unlimited occurrences or re-enters an ancestor (a recursive attr set), since such a tree has no maximum. Semantic checks are supposed to reject these before sizes is used, so this signals a bug in the caller rather than a user error. KeyError: when a blob backing is missing from BLOB_DESCRIPTOR_ATTRS. """ > + if node.is_root: > + return _children_max_len(node) > + > + decl = node.attr.decl > + if decl.kind in ('uint', 'sint', 'enum', 'flags'): > + return SCALAR_PAYLOAD_LEN > + if decl.kind == 'bool': > + return BOOL_PAYLOAD_LEN > + if decl.kind == 'string': > + # One terminating NUL, which max-length excludes. > + return decl.max_length + 1 > + if decl.kind == 'binary': > + return decl.max_length > + if decl.kind == 'blob': > + widest = max(BLOB_DESCRIPTOR_ATTRS[backing] > + for backing in decl.blob.backings) > + return widest * align(ATTR_HDR_LEN + SCALAR_PAYLOAD_LEN) > + return _children_max_len(node) > + > + > +def _children_max_len(node): > + total = 0 > + for child in node.children(): > + if child.max_occurrences == spec.UNLIMITED: > + raise AssertionError( > + f'{child.attr.full_name} has unlimited occurrences, so it has ' > + 'no maximum length') > + if child.reenters is not None: > + raise AssertionError( > + f'{child.attr.full_name} re-enters {child.reenters}, so it ' > + 'has no maximum length') > + total += child.max_occurrences * align(attr_max_len(child)) > + return total > + > + > +def _min_walk(node, ancestors_len): > + children = node.children() > + if children: > + return max(_min_walk(child, ancestors_len + ATTR_HDR_LEN) > + for child in children) > + if node.is_root: > + return ancestors_len + ATTR_HDR_LEN > + return ancestors_len + align(attr_max_len(node)) > + > + > +def _reenters(node): > + return any(child.reenters is not None or _reenters(child) > + for child in node.children()) > + > + > +def _largest_leaf(node): > + """The longest placement of the tree that a fragment cannot split.""" > + lengths = [align(attr_max_len(child)) > + for child in node.children() > + if child.attr.decl.kind != 'nested'] > + lengths += [_largest_leaf(child) for child in node.children()] > + return max(lengths, default=0) > + > + > +def _check_inline(where, what, length): > + if length > MAX_INLINE_MESSAGE_LEN: > + raise CtlvError(where, f'{what} needs {length} bytes, more than the ' > + f'{MAX_INLINE_MESSAGE_LEN} byte inline ' > + 'message limit') > + > + > +def _suggested_len(where, declared, minimum): > + if declared is None: > + return max(DEFAULT_QUERY_REPLY_LEN, minimum) > + if declared < minimum: > + raise CtlvError(where, f'suggested-reply-length {declared} is below ' > + f'the computed minimum of {minimum}') > + if declared > MAX_INLINE_MESSAGE_LEN: > + raise CtlvError(where, f'suggested-reply-length {declared} exceeds ' > + f'the {MAX_INLINE_MESSAGE_LEN} byte inline ' > + 'message limit') > + return declared > diff --git a/tools/ctlv/ctlvgen/lib/spec.py b/tools/ctlv/ctlvgen/lib/spec.py > new file mode 100644 > index 000000000000..8942664d88a7 > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/spec.py > @@ -0,0 +1,583 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""The as-written form of a CTLV family specification. > + > +The strict YAML subset, the meta-schema, and every rule that one declaration > +decides on its own. Comparing declarations and walking a chain are > +semantics.py's. Omitted values are normalized here, once. > +""" > + > +from __future__ import annotations > + > +import dataclasses > +import pathlib > +import re > + > +import jsonschema > + > +from . import yamlload > +from .errors import CtlvError > + > +# A generated file carries the license of its specification's first line. > +LICENSE = re.compile(r'#\s*SPDX-License-Identifier:\s*(\S.*?)\s*$') The generated kernel header and source reuse this license verbatim, and the checked-in specs use `((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause)`. `include/linux/ctlv_gen.h` and the generated .c files at HEAD therefore carry Linux-syscall-note, an exception intended for UAPI headers only; it is wrong for kernel-internal files and checkpatch/license checks will not like it. The license of kernel-header/kernel-source outputs needs to be derived (e.g. strip the note) rather than copied. > + > +# The registry is indexed by family ID, so the root family is its first entry. > +CORE_FAMILY_NAME = 'ctlv' > +CORE_FAMILY_ID = 0 > + > +MAX_FAMILY_NAME_LEN = 63 > +U8_MAX = (1 << 8) - 1 > +U16_MAX = (1 << 16) - 1 > +U32_MAX = (1 << 32) - 1 > +MAX_FAMILY_ID = U16_MAX > + > +# Zero means unlimited, as the introspection wire form has it. > +UNLIMITED = 0 > + > +U64_MAX = (1 << 64) - 1 > +S64_MIN = -(1 << 63) > +S64_MAX = (1 << 63) - 1 > +S8_MIN = -(1 << 7) > +S8_MAX = (1 << 7) - 1 > + > +PRIMARY_KEYS = ('type', 'enum', 'flags', 'attrs', 'blob') > + > +INTEGER_KINDS = ('uint', 'sint') > +SCALAR_KINDS = ('uint', 'sint', 'bool', 'string', 'binary', 'enum', 'flags') `SCALAR_KINDS` is unused. > + > +FD_ACCESSES = ('read', 'read-write') > +DEFAULT_FD_ACCESS = 'read-write' > + > +# Every op has one of these, and only an action or a query may be dynamic. > +AVAILABILITIES = ('fixed', 'dynamic') > +DEFAULT_AVAILABILITY = 'fixed' > + > +# Keys each primary form may add, beyond its own, name and max-occurrences. > +_ACCESSORY_KEYS = { > + 'uint': {'min', 'max', 'unit', 'scale'}, > + 'sint': {'min', 'max', 'unit', 'scale'}, > + 'bool': set(), > + 'string': {'min-length', 'max-length'}, > + 'binary': {'length', 'min-length', 'max-length'}, > + 'enum': set(), > + 'flags': set(), > + 'nested': {'optional-attrs', 'recursive'}, > + 'blob': set(), > +} > +_COMMON_KEYS = {'name', 'max-occurrences'} > + > + > +@dataclasses.dataclass(frozen=True) > +class ValueList: > + """An enum or a flags definition.""" > + > + name: str > + values: tuple > + > + > +@dataclasses.dataclass(frozen=True) > +class BlobDecl: > + """The blob properties of one blob attr declaration.""" > + > + direction: str > + backings: tuple > + max_length: int > + alignment: int > + > + > +@dataclasses.dataclass(frozen=True) > +class AttrDecl: > + """One attr declaration inside an attr group. > + > + kind is the resolved primary form, and a field that does not apply stays > + None. recursive marks a nest whose attr set may select this attr again. > + """ > + > + name: str > + kind: str > + ref: str = None > + attrs: tuple = () > + optional_attrs: tuple = () > + recursive: bool = False > + blob: BlobDecl = None > + min: int = None > + max: int = None > + min_length: int = None > + max_length: int = None > + unit: str = None > + scale: int = None > + max_occurrences: int = 1 > + > + > +@dataclasses.dataclass(frozen=True) > +class AttrGroupSpec: > + name: str > + attrs: tuple > + > + > +@dataclasses.dataclass(frozen=True) > +class AttrSetSpec: > + name: str > + stub: bool > + attrs: tuple > + optional_attrs: tuple > + > + > +@dataclasses.dataclass(frozen=True) > +class Placements: > + """The required and optional selectors declared at one level.""" > + > + attrs: tuple = () > + optional_attrs: tuple = () > + > + > +@dataclasses.dataclass(frozen=True) > +class OpSpec: > + name: str > + type: str > + request: Placements > + reply: Placements = None > + fd_access: str = DEFAULT_FD_ACCESS > + availability: str = DEFAULT_AVAILABILITY > + optional: bool = False > + pre_post: bool = False > + suggested_reply_length: int = None > + > + > +@dataclasses.dataclass(frozen=True) > +class FamilySpec: > + """One specification file, syntax-checked and normalized.""" > + > + path: str > + name: str > + license: str > + inherits: str > + enums: tuple > + flags: tuple > + groups: tuple > + sets: tuple > + ops: tuple > + pre_init: bool = False > + post_init: bool = False > + pre_fini: bool = False > + > + > +@dataclasses.dataclass(frozen=True) > +class RegistryEntry: > + name: str > + deprecated: bool > + > + > +@dataclasses.dataclass(frozen=True) > +class Registry: > + """The append-only family registry, indexed by family ID.""" > + > + path: str > + entries: tuple > + > + @property > + def license(self): > + return _license(self.path) > + > + def family_id(self, name): > + """Return the registered family ID of name, or None if unregistered.""" > + for family_id, entry in enumerate(self.entries): > + if entry.name == name: > + return family_id > + return None > + > + > +class Paths: > + """Where the specs, the registry and the meta-schemas are.""" > + > + def __init__(self, spec, schema=None, registry=None, abi=None): Undocumented and non-trivial: the constructor derives a dozen related paths from three optional overrides by directory-layout conventions, and which path is derived from which (for example kernel_dir from the schema, not the spec) is not obvious. Suggested docstring: """Compute every input and output location from one specification path. Nothing is read or checked on disk; the attributes are only pathlib.Path objects, so a path may name a file or directory that does not exist yet (the ABI directory, for one). The layout convention is that a specification lives in a "specs" directory inside the documentation directory (doc_dir), next to the meta-schema, the family registry, the ABI snapshot directory and the manifest of generated files: spec, spec_dir: the specification file and its directory. doc_dir: the parent of spec_dir. schema: schema if given, else doc_dir/ctlv-schema.yaml. registry: registry if given, else doc_dir/families.yaml. registry_schema: ctlv-registry-schema.yaml next to schema, so overriding the schema also moves the registry meta-schema. abi_dir: abi if given, else doc_dir/abi. registry_abi is the registry baseline, families.registry.yaml, inside it. generated, generated_schema: generated-files.yaml and generated-files-schema.yaml in doc_dir. These do not follow overrides. kernel_dir: the source tree root, three directory levels above the schema file (schema.parent.parent.parent). It follows schema, not spec, and is the base of the repository-relative paths in the output manifests. An empty string for schema, registry or abi counts as not given. Args: spec: str or path-like, the specification file (the --spec argument). schema: str or path-like, optional path of the meta-schema. registry: str or path-like, optional path of the family registry. abi: str or path-like, optional directory of the ABI snapshots. """ > + self.spec = pathlib.Path(spec) > + self.spec_dir = self.spec.parent > + self.doc_dir = self.spec_dir.parent > + self.schema = pathlib.Path(schema or self.doc_dir / 'ctlv-schema.yaml') > + self.registry = pathlib.Path(registry or > + self.doc_dir / 'families.yaml') > + # The two meta-schemas ship together, so overriding one locates both. > + self.registry_schema = (self.schema.parent / > + 'ctlv-registry-schema.yaml') > + self.abi_dir = pathlib.Path(abi or self.doc_dir / 'abi') > + self.registry_abi = self.abi_dir / 'families.registry.yaml' > + self.generated = self.doc_dir / 'generated-files.yaml' > + self.generated_schema = self.doc_dir / 'generated-files-schema.yaml' > + self.kernel_dir = self.schema.parent.parent.parent > + > + def sibling_spec(self, name): > + """Return the path of another family's specification. > + > + The one root family lives where the meta-schema does, so a corpus of > + fixtures inherits it rather than a copy. > + """ > + if name == CORE_FAMILY_NAME: > + return self.schema.parent / 'specs' / f'{name}.yaml' > + return self.spec_dir / f'{name}.yaml' > + > + def abi(self, name): > + """Return the path of one family's checked-in ABI snapshot.""" > + return self.abi_dir / f'{name}.yaml' > + > + > +def load_registry(paths, path=None): > + """Load and check the family registry.""" > + path = pathlib.Path(path or paths.registry) > + data = yamlload.load(path) > + _validate_syntax(path, data, paths.registry_schema) > + > + entries = [] > + seen = set() > + for item in data['families']: > + if isinstance(item, dict): > + entry = RegistryEntry(item['name'], item['deprecated']) > + else: > + entry = RegistryEntry(item, False) > + _family_name(f'{path}: family {entry.name}', entry.name) > + if entry.name in seen: > + raise CtlvError(f'{path}: {entry.name}', 'duplicate family') > + seen.add(entry.name) > + entries.append(entry) > + > + core = entries[CORE_FAMILY_ID] > + if core.name != CORE_FAMILY_NAME: > + raise CtlvError(f'{path}: {core.name}', > + f'family ID {CORE_FAMILY_ID} belongs to ' > + f'{CORE_FAMILY_NAME}, so it must be the first entry') > + if core.deprecated: > + raise CtlvError(f'{path}: {core.name}', > + 'the core family cannot be deprecated') > + if len(entries) - 1 > MAX_FAMILY_ID: > + raise CtlvError(str(path), > + f'{len(entries)} entries need a family ID past the ' > + f'{MAX_FAMILY_ID} maximum') > + return Registry(str(path), tuple(entries)) > + > + > +def load(path, schema=None): > + """Load, syntax-check and normalize one family specification.""" > + path = pathlib.Path(path) > + data = yamlload.load(path) > + _validate_syntax(path, data, schema) > + > + family = data['family'] > + name = _family_name(f'{path}: family {family["name"]}', family['name']) > + inherits = family.get('inherits') > + if inherits is not None: > + inherits = _family_name(f'{path}: family {name} parent', inherits) > + return FamilySpec( > + path=str(path), > + name=name, > + license=_license(path), > + inherits=inherits, > + enums=_value_lists(path, data.get('enums', []), 'enum'), > + flags=_value_lists(path, data.get('flags', []), 'flags'), > + groups=_groups(path, data.get('attr-groups', [])), > + sets=_sets(path, data.get('attr-sets', [])), > + ops=_ops(path, data.get('ops', [])), > + pre_init=family.get('pre-init', False), > + post_init=family.get('post-init', False), > + pre_fini=family.get('pre-fini', False), > + ) > + > + > +def _license(path): > + """Return the SPDX identifier the specification's first line carries.""" > + with open(path, 'r', encoding='utf-8') as stream: > + found = LICENSE.match(stream.readline()) > + if found is None: > + raise CtlvError(str(path), > + 'the first line must be an SPDX-License-Identifier ' > + 'comment, which every generated file inherits') > + return found.group(1) > + > + > +def _validate_syntax(path, data, schema_path): > + schema = yamlload.load(schema_path) > + validator = jsonschema.Draft7Validator(schema) > + error = jsonschema.exceptions.best_match(validator.iter_errors(data)) > + if error is None: > + return > + where = f'{path}: {error.json_path}' if error.absolute_path else str(path) > + raise CtlvError(where, error.message) `_validate_syntax()` reloads and re-parses the meta-schema YAML and builds a new Draft7Validator on every call (each spec in a chain, the registry, each manifest, each output); there is no `check_schema()` on the schema, and `best_match()` reports a single error. With the chain walking in `semantics._load_chain()` and `outputs.render()` reloading every family per output, a full `--outputs-check` loads each file many times. Cache the validators. > + > + > +def _family_name(where, name): > + if len(name) > MAX_FAMILY_NAME_LEN: > + raise CtlvError(where, f'name is {len(name)} characters, more than ' > + f'the {MAX_FAMILY_NAME_LEN} maximum') > + return name Redundant with the meta-schema (family-name/name maxLength: 63): this check cannot fire after `_validate_syntax()`. Same for the `len(entries) - 1 > MAX_FAMILY_ID` test at 5391 (registry maxItems 65536). > + > + > +def _value_lists(path, raw_lists, what): Undocumented and non-trivial: the same function serves both enums and flags through a label argument, and it is easy to miss that it only rejects duplicate value names and leaves numbering and name uniqueness to others. Suggested docstring: """Normalize the enum or flags definitions of a specification. Converts the schema-validated top-level "enums" or "flags" list into ValueList objects. Each raw entry has a "name" and a "values" list of kebab-case value names in declaration order. The order is preserved exactly, because it is the numbering: an enum value gets its 1-based position (0 is reserved) and a flag its bit index from bit 0. This function assigns no numbers itself. The only rule it enforces is that a value name occurs once within its own list. It does not check that list names are unique, that the length limits (4294967295 for enums, 64 for flags) hold, or that attrs refer to defined lists; the meta-schema and semantics.py do those. Nothing passed in is modified. Called twice by load(), once for enums and once for flags. Args: path: The specification file path (str or pathlib.Path), used only to build the error location. raw_lists: The list of raw YAML dicts from the "enums" or "flags" key, or an empty list when the key is absent. what: The kind label, 'enum' or 'flags', used only in error locations. Returns: tuple: ValueList objects (name and a tuple of value names), in the order declared. Throws: CtlvError: A value name appears more than once in one list; the error is located at ": ". """ > + lists = [] > + for raw in raw_lists: > + name = raw['name'] > + where = f'{path}: {what} {name}' > + values = [] > + seen = set() > + for raw_value in raw['values']: > + value = raw_value `value = raw_value` is a pointless alias. > + if value in seen: > + raise CtlvError(where, f'duplicate value "{value}"') > + seen.add(value) > + values.append(value) > + lists.append(ValueList(name, tuple(values))) > + return tuple(lists) > + > + > +def _groups(path, raw_groups): > + groups = [] > + for raw in raw_groups: > + name = raw['name'] > + attrs = tuple(_attr(f'{path}: attr {name}.{a["name"]}', a) > + for a in raw['attrs']) > + groups.append(AttrGroupSpec(name, attrs)) > + return tuple(groups) > + > + > +def _attr(where, raw): Undocumented and non-trivial: It dispatches on a "kind" derived from five mutually exclusive primary keys and then pulls different normalized fields from different helpers, so the flow from raw YAML keys to AttrDecl fields is hard to see. Suggested docstring: """Normalize one attr declaration into an AttrDecl. Takes one raw entry of an attr group's "attrs" list. Exactly one primary key must be present: "type" (uint, sint, bool, string or binary), "enum", "flags", "attrs" (a nested payload) or "blob". The resolved kind is the "type" value, 'enum', 'flags', 'blob', or 'nested' when the primary key was "attrs". Keys that do not belong to that kind (for example "min" on a string) are rejected. The remaining fields depend on the kind. Integers (uint, sint) get min, max, unit and scale. A string gets min_length and max_length. A binary gets min_length and max_length from either "length" or the min-length and max-length pair. An enum or flags attr keeps only the referenced definition name in ref; whether it exists is checked later by semantics.py. A nested attr gets attrs and optional_attrs as tuples of selector strings, plus recursive (from "recursive", default False); a recursive nest is validated by _check_recursive. A blob gets a BlobDecl. A bool has nothing further. max_occurrences always comes from "max-occurrences" (default 1; 'unlimited' becomes UNLIMITED, which is 0). Fields that do not apply to the kind stay None or their dataclass default. Args: where: Error location prefix, such as ": attr .". raw: One schema-validated attr dict from the YAML specification. Returns: AttrDecl: The frozen, normalized declaration. The argument is not modified. Throws: CtlvError: There is no primary key or more than one; a key is not allowed for the kind; max-occurrences, a bound, a length, scale or blob property is out of range or inconsistent (raised by the helpers); or a recursive nest declares optional-attrs, does not name exactly one selector, or names a "group.attr" selector instead of an attr set. """ > + primary = _primary_key(where, raw) > + kind = _kind(primary, raw) > + _reject_unusable_keys(where, raw, primary, kind) > + > + attr = { > + 'name': raw['name'], > + 'kind': kind, > + 'max_occurrences': _max_occurrences(where, raw), > + } > + if kind in INTEGER_KINDS: > + attr.update(_integer_bounds(where, raw, kind)) > + elif kind == 'string': > + attr.update(_string_lengths(where, raw)) > + elif kind == 'binary': > + attr.update(_binary_lengths(where, raw)) > + elif kind in ('enum', 'flags'): > + attr['ref'] = raw[kind] > + elif kind == 'nested': > + attr['attrs'] = _selectors(raw['attrs']) > + attr['optional_attrs'] = _selectors(raw.get('optional-attrs')) > + attr['recursive'] = raw.get('recursive', False) > + if attr['recursive']: > + _check_recursive(where, attr) > + elif kind == 'blob': > + attr['blob'] = _blob(where, raw['blob']) > + # A bool has nothing to normalize. > + return AttrDecl(**attr) > + > + > +def _check_recursive(where, attr): > + """A recursive nest is one named set, which is what it re-enters.""" > + if attr['optional_attrs']: > + raise CtlvError(where, 'a recursive attr cannot declare ' > + 'optional-attrs; the attr set it names says ' > + 'which of its own members are optional') > + if len(attr['attrs']) != 1: > + raise CtlvError(where, 'a recursive attr nests exactly one attr set') > + if '.' in attr['attrs'][0]: > + raise CtlvError(where, f'"{attr["attrs"][0]}" is one attr, and a ' > + 'recursive attr nests an attr set') > + > + > +def _primary_key(where, raw): > + present = [key for key in PRIMARY_KEYS if key in raw] > + if not present: > + raise CtlvError(where, 'no primary declaration; expected one of ' > + 'type, enum, flags, attrs or blob') > + if len(present) > 1: > + raise CtlvError(where, 'conflicting primary declarations: ' + > + ', '.join(present)) > + return present[0] > + > + > +def _kind(primary, raw): > + if primary == 'type': > + return raw['type'] > + if primary == 'attrs': > + return 'nested' > + return primary > + > + > +def _reject_unusable_keys(where, raw, primary, kind): > + allowed = _COMMON_KEYS | {primary} | _ACCESSORY_KEYS[kind] > + unusable = sorted(set(raw) - allowed) > + if unusable: > + raise CtlvError(where, f'a {kind} attr cannot declare ' + > + ', '.join(unusable)) > + > + > +def _max_occurrences(where, raw): > + value = raw.get('max-occurrences', 1) > + if value == 'unlimited': > + return UNLIMITED > + if value < 1 or value > U32_MAX: > + raise CtlvError(where, f'max-occurrences {value} is outside the ' > + 'u32 range') > + return value > + > + > +def _integer_bounds(where, raw, kind): Undocumented and non-trivial: the default and legal ranges depend on signedness (uint vs sint), and the unit and scale interplay, where scale defaults to 0 only when a unit is present, is not obvious. Suggested docstring: """Validate and normalize the min, max, unit and scale of an integer attr. Reads the optional YAML keys "min", "max", "unit" and "scale" from an attr of kind uint or sint. Integers are always 64-bit, so an omitted bound defaults to the full range of the kind: 0..U64_MAX for uint and S64_MIN..S64_MAX for sint. The meta-schema accepts any integer from S64_MIN to U64_MAX for either key, so this function enforces the narrower per-kind range (a uint cannot have a negative min, a sint max cannot exceed S64_MAX) and min <= max. Both bounds are inclusive. "unit" is one of the CTLV-owned units (seconds, hertz, bytes, celsius) and "scale" is a decimal exponent in the s8 range -128..127. A scale without a unit is an error. When a unit is given and scale is omitted, scale is 0. Without a unit, both unit and scale are None. Args: where: Error location prefix for the attr. raw: The schema-validated attr dict. It is read, not modified. kind: 'uint' or 'sint'; selects the legal range. Returns: dict: The keys 'min' (int), 'max' (int), 'unit' (str or None) and 'scale' (int or None), ready to be merged into the AttrDecl constructor arguments. Throws: CtlvError: min or max is outside the range of kind, min is greater than max, scale is given without unit, or scale is outside the s8 range. """ > + low, high = (0, U64_MAX) if kind == 'uint' else (S64_MIN, S64_MAX) > + minimum = raw.get('min', low) > + maximum = raw.get('max', high) > + if minimum < low or minimum > high: > + raise CtlvError(where, f'min {minimum} is outside the {kind} range') > + if maximum < low or maximum > high: > + raise CtlvError(where, f'max {maximum} is outside the {kind} range') > + if minimum > maximum: > + raise CtlvError(where, f'min {minimum} is greater than max {maximum}') > + if 'scale' in raw and 'unit' not in raw: > + raise CtlvError(where, 'scale requires unit') > + scale = raw.get('scale', 0) if 'unit' in raw else None > + if scale is not None and (scale < S8_MIN or scale > S8_MAX): > + raise CtlvError(where, f'scale {scale} is outside the s8 range') > + return { > + 'min': minimum, > + 'max': maximum, > + 'unit': raw.get('unit'), > + 'scale': scale, > + } > + > + > +def _u32_value(where, what, value): > + if value < 0 or value > U32_MAX: > + raise CtlvError(where, f'{what} {value} is outside the u32 range') > + return value > + > + > +def _string_lengths(where, raw): > + if 'max-length' not in raw: > + raise CtlvError(where, 'a string attr requires max-length') > + minimum = _u32_value(where, 'min-length', raw.get('min-length', 0)) > + maximum = _u32_value(where, 'max-length', raw['max-length']) > + if minimum > maximum: > + raise CtlvError(where, f'min-length {minimum} is greater than ' > + f'max-length {maximum}') > + return {'min_length': minimum, 'max_length': maximum} > + > + > +def _binary_lengths(where, raw): Undocumented and non-trivial: a binary attr can state its size in two mutually exclusive ways (exact "length" or a "min-length"/"max-length" range), and both are folded into one min/max pair. Suggested docstring: """Validate and normalize the payload length limits of a binary attr. A binary attr is sized either by "length", an exact byte count, or by "max-length" with an optional "min-length" (default 0). The two forms cannot be mixed. An exact length is represented as min_length == max_length, so later stages see one shape. Every value must be in the u32 range, and min-length must not exceed max-length. Unlike a string, nothing is excluded for a terminator, since a binary has none. Unlike _string_lengths, which always requires max-length, this function accepts "length" as an alternative. Nothing in raw is modified. Args: where: Error location prefix for the attr. raw: The schema-validated attr dict of a binary attr. Returns: dict: The keys 'min_length' (int) and 'max_length' (int), ready to be merged into the AttrDecl constructor arguments. Throws: CtlvError: "length" is combined with "min-length" or "max-length"; neither "length" nor "max-length" is present; a value is outside the u32 range; or min-length is greater than max-length. """ > + if 'length' in raw: > + if 'min-length' in raw or 'max-length' in raw: > + raise CtlvError(where, 'length cannot be combined with ' > + 'min-length or max-length') > + length = _u32_value(where, 'length', raw['length']) > + return {'min_length': length, 'max_length': length} > + if 'max-length' not in raw: > + raise CtlvError(where, 'a binary attr requires length or max-length') > + minimum = _u32_value(where, 'min-length', raw.get('min-length', 0)) > + maximum = _u32_value(where, 'max-length', raw['max-length']) > + if minimum > maximum: > + raise CtlvError(where, f'min-length {minimum} is greater than ' > + f'max-length {maximum}') > + return {'min_length': minimum, 'max_length': maximum} > + > + > +def _blob(where, raw): Undocumented and non-trivial: blob placements carry several interlocking numeric constraints (u64 length, u32 power-of-two alignment, length a multiple of alignment) that are not visible from the schema alone. Suggested docstring: """Validate and normalize the "blob" mapping of a blob attr. A blob attr describes external memory rather than inline payload. Its "blob" mapping has "direction" (input, output or inout: how the family may use the memory), "allowed-backings" (a non-empty, unique list; the schema currently permits only 'va') and "max-length", the authoritative maximum external payload length, independent of the inline message limit. "alignment" is optional and defaults to 1. The meta-schema bounds the types and basic ranges, but this function enforces the relationships: max-length is 1..U64_MAX, alignment is within the u32 range and a power of two, and max-length is an exact multiple of alignment. direction and allowed-backings are copied through unchanged (backings become a tuple). Args: where: Error location prefix for the attr. raw: The schema-validated dict under the attr's "blob" key (not the whole attr). It is not modified. Returns: BlobDecl: The frozen declaration with direction, backings (tuple of str), max_length and alignment. Throws: CtlvError: max-length is outside 1..U64_MAX; alignment is outside the u32 range, is not a power of two, or does not divide max-length. """ > + alignment = raw.get('alignment', 1) > + maximum = raw['max-length'] > + if maximum < 1 or maximum > U64_MAX: > + raise CtlvError(where, f'blob max-length {maximum} is outside the ' > + 'u64 range') > + if alignment < 1 or alignment > U32_MAX: > + raise CtlvError(where, f'blob alignment {alignment} is outside the ' > + 'u32 range') > + if alignment & (alignment - 1): > + raise CtlvError(where, f'blob alignment {alignment} is not a power ' > + 'of two') > + if maximum % alignment: > + raise CtlvError(where, f'blob max-length {maximum} is not ' > + f'a multiple of alignment {alignment}') > + return BlobDecl(direction=raw['direction'], > + backings=tuple(raw['allowed-backings']), > + max_length=maximum, > + alignment=alignment) > + > + > +def _sets(path, raw_sets): Undocumented and non-trivial: an attr set is either a stub (an empty extension point) or a non-empty member list, and the two cases are separated only by checks that are not in the schema. Suggested docstring: """Normalize the "attr-sets" list into AttrSetSpec objects. An attr set is a compositional unit that attrs and ops select by name. It is either a stub, declared with "stub: true", which has no members and may be filled once by one descendant family, or a set of members given by "attrs" (required) and "optional-attrs" (optional placements). Each member list may be a single selector string or a list; selectors can name an attr set, a whole attr group, or one attr as "group.attr". They are only normalized to tuples of strings here (an omitted list becomes an empty tuple); whether they resolve is checked by semantics.py. This function enforces the exclusivity the schema cannot express: a stub must have no members, and a non-stub must have at least one member in attrs or optional-attrs. Duplicate set names and the single-fill rule across families are not checked here. Nothing in raw_sets is modified. Args: path: The specification file path, used to build the error location. raw_sets: The list of raw dicts from "attr-sets", or an empty list when the key is absent. Returns: tuple: AttrSetSpec objects (name, stub, attrs, optional_attrs) in the order declared. Throws: CtlvError: A stub declares attrs or optional-attrs, or a non-stub declares neither. """ > + sets = [] > + for raw in raw_sets: > + name = raw['name'] > + where = f'{path}: attr set {name}' > + stub = raw.get('stub', False) > + attrs = _selectors(raw.get('attrs')) > + optional = _selectors(raw.get('optional-attrs')) > + if stub and (attrs or optional): > + raise CtlvError(where, 'a stub attr set cannot declare members') > + if not stub and not attrs and not optional: > + raise CtlvError(where, 'attr set has no members; declare ' > + 'stub: true for an extension point') > + sets.append(AttrSetSpec(name, stub, attrs, optional)) > + return tuple(sets) > + > + > +def _ops(path, raw_ops): Undocumented and non-trivial: op type (action, query, event) decides which keys are legal and what an omitted request or reply means, and these per-type rules are spread over many inline checks. Suggested docstring: """Normalize the "ops" list into OpSpec objects. Each op has a "name" and a "type": an action (invoked once, never fragments), a query (may fragment its reply) or an event (an unsolicited, reply-only schema). The type decides which other keys may appear and how omissions are normalized. Key rules, enforced here because the schema cannot tie them to the type: an event cannot declare "request", "fd-access", "availability", "optional" or "pre-post"; a query must declare "reply"; "suggested-reply-length" is valid only on a query and must be in the u32 range. Whether it lies between the computed minimum and CTLV_MAX_INLINE_MESSAGE_LEN needs the resolved schema and is not checked here. Normalization: for an action or query, an omitted "request" becomes an empty Placements (a request root with no children); an event's request is None. An omitted action "reply" stays None, meaning ACK-only, while an omitted event "reply" becomes an empty Placements. Omitted "fd-access" becomes 'read-write', "availability" becomes 'fixed', and "optional" and "pre-post" become False. Placement selectors are only normalized into tuples of strings; resolving them is done by semantics.py. Nothing in raw_ops is modified. Args: path: The specification file path, used to build the error location. raw_ops: The list of raw dicts from "ops", or an empty list when the key is absent. Returns: tuple: OpSpec objects in declaration order. Throws: CtlvError: An event declares request, fd-access, availability, optional or pre-post; a query has no reply; suggested-reply-length is on a non-query or outside the u32 range. """ > + ops = [] > + for raw in raw_ops: > + name = raw['name'] > + where = f'{path}: op {name}' > + op_type = raw['type'] > + suggested = raw.get('suggested-reply-length') > + if suggested is not None: > + suggested = _u32_value(where, 'suggested-reply-length', > + suggested) > + if op_type == 'event' and 'request' in raw: > + raise CtlvError(where, 'an event op cannot declare a request') > + if op_type == 'query' and 'reply' not in raw: > + raise CtlvError(where, 'a query op must declare a reply') > + if 'suggested-reply-length' in raw and op_type != 'query': > + raise CtlvError(where, 'suggested-reply-length is only valid on ' > + 'a query op') > + if 'fd-access' in raw and op_type == 'event': > + raise CtlvError(where, 'fd-access is only valid on an action or ' > + 'a query op') > + # An event is not executed, so there is nothing to enable or disable. > + if 'availability' in raw and op_type == 'event': > + raise CtlvError(where, 'availability is only valid on an action ' > + 'or a query op') > + # An event is produced rather than answered, so there is nothing for a > + # device to opt in to. > + if 'optional' in raw and op_type == 'event': > + raise CtlvError(where, 'optional is only valid on an action or a ' > + 'query op') > + # A hook pair wraps the execution of an op, and an event has none. > + if 'pre-post' in raw and op_type == 'event': > + raise CtlvError(where, 'pre-post is only valid on an action or a ' > + 'query op') Five consecutive "X is only valid on an action or a query op" blocks that only differ by key name: table-drive them (for key in ('fd-access', 'availability', 'optional', 'pre-post')). > + # An omitted action or query request means a root with no children. > + request = None > + if op_type != 'event': > + request = _placements(raw.get('request')) > + # Omitted: an action reply is ACK-only, an event reply payload-free. > + reply = None > + if 'reply' in raw: > + reply = _placements(raw['reply']) > + elif op_type == 'event': > + reply = Placements() > + ops.append(OpSpec( > + name=name, > + type=op_type, > + request=request, > + reply=reply, > + fd_access=raw.get('fd-access', DEFAULT_FD_ACCESS), > + availability=raw.get('availability', DEFAULT_AVAILABILITY), > + optional=raw.get('optional', False), > + pre_post=raw.get('pre-post', False), > + suggested_reply_length=suggested, > + )) > + return tuple(ops) > + > + > +def _placements(raw): > + if raw is None: > + return Placements() > + return Placements(_selectors(raw.get('attrs')), > + _selectors(raw.get('optional-attrs'))) > + > + > +def _selectors(raw): > + if raw is None: > + return () > + if isinstance(raw, str): > + return (raw,) > + return tuple(raw) > diff --git a/tools/ctlv/ctlvgen/lib/tree.py b/tools/ctlv/ctlvgen/lib/tree.py > new file mode 100644 > index 000000000000..62f4fb363469 > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/tree.py > @@ -0,0 +1,71 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""Whole-tree registry, specification and ABI coverage checks.""" > + > +from __future__ import annotations > + > +from . import compat > +from . import outputs > +from . import spec > +from .errors import CtlvError > + > + > +def _stems(directory, exclude=()): > + return {path.stem for path in directory.glob('*.yaml') > + if path.name not in exclude} > + `tree.coverage_problems()`: the 'snapshots' parameter is never passed as False by any caller; unused knob. `_stems(exclude=)` is only for the registry baseline file. > + > +def coverage_problems(paths, registry, snapshots=True, > + allowed_missing_snapshots=()): Undocumented and non-trivial: It returns a list of problem strings instead of raising, and the optional snapshot checks and the allowance for not-yet- written snapshots are driven by flags that only make sense in the update flow. Suggested docstring: """List the ways the spec and ABI snapshot files differ from the registry. The registry is the authority: its entry names are the expected set. The function compares that set with the *.yaml file stems in the specification directory (paths.spec_dir) and, unless snapshots is false, in the ABI snapshot directory (paths.abi_dir). The registry baseline file, paths.registry_abi, is excluded from the snapshot stems because it is not a family snapshot. Nothing is raised or modified here; check(), check_for_update() and finish_update() turn a non-empty result into a CtlvError. Every problem is one line of the form "missing specification X.yaml", "unexpected specification X.yaml", "missing ABI snapshot X.yaml" or "unexpected ABI snapshot X.yaml". Within each group the names are sorted, so the output is stable. Deprecated registry entries still count as expected, as they keep their specification and snapshot. Args: paths: spec.Paths locating the specification and ABI snapshot directories. registry: spec.Registry whose entries define the expected family names. snapshots: bool, whether to check the ABI snapshots at all. A missing spec_dir or abi_dir simply has no files, so every family is then missing. allowed_missing_snapshots: iterable of family names whose snapshot may be absent without being reported. check_for_update() passes the families appended to the registry since the baseline, since their first snapshots are written by the update itself. An unexpected snapshot is still reported for these. Returns: list[str]: problem descriptions, empty when everything is covered. """ > + expected = {entry.name for entry in registry.entries} > + problems = [] > + > + actual_specs = _stems(paths.spec_dir) > + for name in sorted(expected - actual_specs): > + problems.append(f'missing specification {name}.yaml') > + for name in sorted(actual_specs - expected): > + problems.append(f'unexpected specification {name}.yaml') > + > + if snapshots: > + actual_abi = _stems(paths.abi_dir, > + exclude=(paths.registry_abi.name,)) > + missing = expected - actual_abi - set(allowed_missing_snapshots) > + for name in sorted(missing): > + problems.append(f'missing ABI snapshot {name}.yaml') > + for name in sorted(actual_abi - expected): > + problems.append(f'unexpected ABI snapshot {name}.yaml') > + return problems > + > + > +def check(paths): > + """Check all inputs whose expected set comes from the registry.""" > + registry = spec.load_registry(paths) > + compat.registry_check(paths, registry) > + problems = coverage_problems(paths, registry) > + if problems: > + raise CtlvError('CTLV tree', '\n'.join(problems)) > + outputs.load(paths) > + return registry > + > + > +def check_for_update(paths): > + """Validate a registry update without requiring new snapshots yet.""" > + registry = spec.load_registry(paths) > + compat.registry_check(paths, registry, exact=False) > + baseline = spec.load_registry(paths, paths.registry_abi) > + appended = {entry.name > + for entry in registry.entries[len(baseline.entries):]} > + problems = coverage_problems(paths, registry, > + allowed_missing_snapshots=appended) > + if problems: > + raise CtlvError('CTLV tree', '\n'.join(problems)) > + outputs.load(paths) > + return registry > + > + > +def finish_update(paths): > + """Require complete snapshots before advancing the registry baseline.""" > + registry = check_for_update(paths) > + problems = coverage_problems(paths, registry) > + if problems: > + raise CtlvError('CTLV tree', '\n'.join(problems)) > + return registry > diff --git a/tools/ctlv/ctlvgen/lib/yamlload.py b/tools/ctlv/ctlvgen/lib/yamlload.py > new file mode 100644 > index 000000000000..f043d87c9818 > --- /dev/null > +++ b/tools/ctlv/ctlvgen/lib/yamlload.py > @@ -0,0 +1,83 @@ > +# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause > +"""Strict YAML subset loader. > + > +Anchors, aliases, merge keys, explicit tags, duplicate mapping keys and > +multiple documents per file are all rejected: they let two readers of a file > +that defines an ABI disagree about what it says. > +""" > + > +import yaml Correctness: PyYAML's SafeLoader implements YAML 1.1 scalar resolution even though every file here starts with "%YAML 1.2" (PyYAML only checks the major version). Unquoted on/off/yes/no/y/n resolve to booleans and 010, 1:20 to ints. Since names are kebab-case strings, an enum value or attr named "on", "off", "yes" or "no" is loaded as a bool and rejected by the schema with an unhelpful "False is not of type 'string'". "Strict YAML subset" should reject these (override the bool/int resolvers) or the docs should say that such names are unusable. > + > +from .errors import CtlvError > + > + > +def _mark(path, mark): > + if mark is None: > + return str(path) > + return f'{path}:{mark.line + 1}' > + > + > +class _StrictLoader(yaml.SafeLoader): > + """SafeLoader without the features that make a document ambiguous.""" > + > + def __init__(self, stream, path): > + super().__init__(stream) > + self.ctlv_path = path > + > + def _reject(self, mark, what): > + raise CtlvError(_mark(self.ctlv_path, mark), what) > + > + def compose_node(self, parent, index): > + event = self.peek_event() > + if isinstance(event, yaml.events.AliasEvent): > + self._reject(event.start_mark, 'YAML aliases are not allowed') > + if getattr(event, 'anchor', None) is not None: > + self._reject(event.start_mark, 'YAML anchors are not allowed') > + if _has_explicit_tag(event): > + self._reject(event.start_mark, > + f'explicit YAML tag "{event.tag}" is not allowed') > + return super().compose_node(parent, index) > + > + def construct_mapping(self, node, deep=False): > + seen = set() > + for key_node, _ in node.value: > + if key_node.tag == 'tag:yaml.org,2002:merge': > + self._reject(key_node.start_mark, > + 'YAML merge keys are not allowed') > + key = self.construct_object(key_node, deep=deep) > + if key in seen: > + self._reject(key_node.start_mark, f'duplicate key "{key}"') > + seen.add(key) > + return super().construct_mapping(node, deep=deep) > + > + > +def _has_explicit_tag(event): > + if event.tag is None: > + return False > + implicit = getattr(event, 'implicit', False) > + if isinstance(implicit, tuple): > + return not (implicit[0] or implicit[1]) > + return not implicit > + > + > +def load(path): > + """Return the single YAML document in path as plain Python objects.""" > + try: > + with open(path, 'r', encoding='utf-8') as stream: > + loader = _StrictLoader(stream, path) > + try: > + data = loader.get_single_data() > + finally: > + loader.dispose() > + except OSError as exc: > + raise CtlvError(str(path), f'cannot read: {exc.strerror}') from exc > + except yaml.YAMLError as exc: > + mark = getattr(exc, 'problem_mark', None) > + described = [getattr(exc, 'context', None), > + getattr(exc, 'problem', None)] > + problem = ', '.join(part for part in described if part) or str(exc) > + raise CtlvError(_mark(path, mark), problem) from exc > + > + if not isinstance(data, dict): > + raise CtlvError(str(path), 'expected a YAML mapping at the top level') > + return data > diff --git a/tools/ctlv/pyproject.toml b/tools/ctlv/pyproject.toml > new file mode 100644 > index 000000000000..7ecae2de3065 > --- /dev/null > +++ b/tools/ctlv/pyproject.toml > @@ -0,0 +1,22 @@ > +[build-system] > +requires = ["setuptools>=61.0"] > +build-backend = "setuptools.build_meta" > + > +[project] > +name = "ctlvgen" > +authors = [ > + {name = "Jiri Pirko", email = "jiri@resnulli.us"}, Email differs from the Signed-off-by again. pyproject.toml also duplicates `requirements.txt`; and the console-script entry point "ctlvgen.cli:main" works only because `cli.py` appends its own directory to sys.path and imports the sub-package as a second top-level module named "lib". sys.path.append (not insert) means any other module called "lib" earlier on sys.path wins. Nothing in the kernel build uses this packaging; consider dropping pyproject.toml. > +] > +description = "CTLV schema tool" > +version = "0.0.1" > +requires-python = ">=3.9" > +dependencies = [ > + "pyyaml==6.*", > + "jsonschema==4.*" > +] > + > +[tool.setuptools.packages.find] > +include = ["ctlvgen", "ctlvgen.lib"] > + > +[project.scripts] > +ctlvgen = "ctlvgen.cli:main" > diff --git a/tools/ctlv/requirements.txt b/tools/ctlv/requirements.txt > new file mode 100644 > index 000000000000..2a73e762f9b9 > --- /dev/null > +++ b/tools/ctlv/requirements.txt > @@ -0,0 +1,2 @@ > +pyyaml==6.* > +jsonschema==4.* > -- > 2.55.0 > -- Summary of the most serious issues: - Documentation: by a rough count 226 of the 338 functions in `tools/ctlv/ctlvgen` have no docstring. The 50 that are non-trivial (15+ lines or 8+ branches) are called out inline with a suggested docstring; the shorter helpers they depend on need the same treatment. - Series/bisectability: the Makefile and `ctlv-regen.sh` operate on specs, `families.yaml`, abi/ and `generated-files.yaml` that do not exist until a later patch; the tool is untestable here. Docstrings and `outputs.py` refer to tests/fixtures (`tools/ctlv/tests`) that are never added. - Generator bugs: `compat._compare_occurrences` flags finite -> unlimited as "narrowed" (UNLIMITED == 0); `_compare_attr` raises KeyError on a kind change; `--mode uapi --source` KeyErrors; assertions used for validation; core messages can hit AssertionError. - `sizes.composed()`, meant to bound fill-grown lengths and the event-queue limit, is dead; only the generated C `static_assert`s catch those. - Licensing: kernel-header/kernel-source outputs inherit "WITH Linux-syscall-note" from the UAPI spec license. - Substantial dead/speculative code: `user_symbols` and the `Names.user_*` family, sizes.composed, tests/fixture plumbing in `outputs.py`, .rst handling, ROOT/FAMILY/`FIXTURE_FLAVOURS`, GENERATORS, tree/registry CLI modes with no Makefile wrapper, duplicated helpers and constants. - Makefile: "echo -e" under dash; abi-update does not update the registry baseline and blesses breaking changes; no wrapper for several modes. - yamlload is YAML 1.1 despite the "%YAML 1.2" headers (on/off/yes/no). - Author/SoB (nvidia) vs MAINTAINERS/pyproject (resnulli) address mismatch.