Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
569410e
fix: include loggers, tags, groups and other referenced resources tra…
cdayne Nov 4, 2026
f49214a
fix: ignore policy references in XML comments and Liquid templates
cdayne Nov 4, 2026
0dc6d5d
feat: allow wildcard API sub-filters and filtering API tag descriptions
cdayne Nov 4, 2026
669d0fc
refactor: share dependency, link and ARM ID helpers in the filtering …
cdayne Nov 4, 2026
d4eb96e
fix: apply workspace sub-filters keyed by a pattern
cdayne Nov 4, 2026
9dc92f6
feat: remove pulled-in dependencies nothing references with --remove-…
cdayne Nov 4, 2026
c0296f5
fix: pull in the backend a policy names through a named value
cdayne Nov 4, 2026
fd70bba
fix: keep a product's link to administrators when the filter narrows …
cdayne Nov 4, 2026
e23d7d1
fix: publish a named value after the backend its value names
cdayne Nov 5, 2026
07f69bb
fix: follow named values APIM substitutes inside Liquid bodies
cdayne Nov 5, 2026
5b75c26
fix: read ARM IDs as collection/name pairs
cdayne Nov 5, 2026
e3691fd
fix: report a failed workspace tag listing as an error
cdayne Nov 5, 2026
d4673f9
fix: keep stale dependencies when a named value can't be read
cdayne Nov 5, 2026
7189485
fix: follow a re-extracted dependency's old references in --remove-stale
cdayne Nov 5, 2026
0d8fc78
fix: check a named value exclusion against the name a reference resol…
cdayne Nov 5, 2026
3c7e7a6
fix: follow what an artifact references after overrides on publish
cdayne Nov 5, 2026
00861b4
fix: publish a credential user after the named value and what it names
cdayne Nov 5, 2026
87f5e8c
docs: describe when a backend using a named value in its credentials …
cdayne Nov 5, 2026
21e0857
fix: resolve a reference to an extracted named value's name
cdayne Nov 5, 2026
2ecc3db
fix: report an error when Liquid named values can't be told apart
cdayne Nov 5, 2026
5c1cc39
fix: match logger credential references by display name first
cdayne Nov 5, 2026
b6e4b17
fix: follow the overridden XML of policies and fragments on publish
cdayne Nov 5, 2026
7869b61
fix: order named values and credential users by what each needs
cdayne Nov 5, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,10 @@ apiops publish \
```

The publish filter uses the same YAML file and matching rules as `apiops extract --filter`.
Referenced named values, backends (including backend pool members), policy fragments, and version
sets are included automatically. Add `--no-transitive` to publish only the exact filter matches.
The resources the selected ones depend on, such as referenced named values, backends (including
backend pool members), policy fragments, version sets, tags, groups and loggers, are included
automatically; see [Transitive Dependency Behavior](docs/guides/filtering-resources.md#transitive-dependency-behavior).
Add `--no-transitive` to publish only the exact filter matches.

### `apiops init`

Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ flowchart TD
### Key Extract Behaviors

- **Topological ordering** — Resources are extracted in dependency order across 4 tiers, ensuring parent resources are written before children.
- **Transitive dependency resolution** — Policy XML is scanned for references to named values, backends, and policy fragments. Referenced resources are included even if not explicitly listed in the filter.
- **Transitive dependency resolution** — Policy XML is scanned for references to named values, backends, policy fragments, loggers and schemas; APIs, diagnostics, loggers, backends and products are scanned for their version set, logger, credential named values, tags and groups. Referenced resources are included even if not explicitly listed in the filter.
- **Dual output format** — Resource metadata is written as JSON (`*.info.json`), while policies are written as raw XML (`policy.xml`).
- **LIST then GET** — Some APIM list endpoints omit fields. When `listOmitsFields` is set for a resource type, individual GET calls fetch the full representation.

Expand Down
18 changes: 14 additions & 4 deletions docs/commands/extract.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ apiops extract \
| `--output <dir>` | string | `./apim-artifacts` | No | Output directory path |
| `--filter <path>` | string | — | No | Filter configuration YAML file |
| `--no-transitive` | boolean | false (transitive ON) | No | Disable transitive dependency inclusion |
| `--remove-stale` | boolean | false | No | Remove stale managed artifacts in the selected filter and workspace scope after a fully successful extraction |
| `--remove-stale` | boolean | false | No | Remove stale managed artifacts in the selected filter and workspace scope, and pulled-in dependencies nothing references any more, after a fully successful extraction |

### Global flags

Expand Down Expand Up @@ -159,7 +159,7 @@ For every key, the value semantics are:
### API sub-filters

To restrict which child resources of an API are extracted (operations,
diagnostics, schemas, releases), use the **nested object entry** form under
diagnostics, schemas, releases, tag descriptions), use the **nested object entry** form under
`apis:`. Each entry becomes a `name: { subKey: [...] }` map:

```yaml
Expand All @@ -175,9 +175,13 @@ apis:
releases: []
```

Supported API sub-filter keys: `operations`, `diagnostics`, `schemas`, `releases`.
Supported API sub-filter keys: `operations`, `diagnostics`, `schemas`, `releases`, `tagDescriptions`.
Same value semantics as top-level keys (omitted = include all, `[]` = include none, list = allowlist).

The key of a nested entry can be a wildcard pattern (`'*'`, `'v2-*'`), so one
sub-filter applies to every matching API. An entry keyed by an API's name takes
precedence over a pattern. See [One sub-filter for many APIs](../guides/filtering-resources.md#one-sub-filter-for-many-apis).

### Workspace sub-filters

Workspaces accept the same nested object entry form. Each workspace can have
Expand All @@ -197,12 +201,18 @@ Supported workspace sub-filter keys: `apis`, `backends`, `diagnostics`,
`groups`, `loggers`, `namedValues`, `policyFragments`, `products`, `schemas`,
`subscriptions`, `tags`, `versionSets`.

As for APIs, the key can be a wildcard pattern (`'team-*'`), and an entry keyed
by the workspace's name takes precedence over a pattern. See
[Workspace sub-resource filters](../guides/filtering-resources.md#workspace-sub-resource-filters).

### Transitive dependencies

When extracting with a filter, transitive dependencies are **included automatically**. For example, extracting `echo-api` also extracts any backends, named values, and policy fragments that `echo-api` references in its policies.
When extracting with a filter, transitive dependencies are **included automatically**. For example, extracting `echo-api` also extracts the backends, named values and policy fragments its policies reference, its version set, its tags, and the loggers its diagnostics use. See [Transitive Dependency Behavior](../guides/filtering-resources.md#transitive-dependency-behavior) for the full list.

Use `--no-transitive` to disable this behavior and extract only the explicitly listed resources.

With `--remove-stale`, a pulled-in dependency the filter otherwise leaves out (for example a tag when `tags: []`) is removed once the selected resources no longer reference it and no other artifact in the output does. A dependency another filtered extract into the same output still uses is kept.

## Output format

Extracted artifacts are organized as JSON info files and XML policy files in a directory tree:
Expand Down
10 changes: 6 additions & 4 deletions docs/commands/publish.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,12 @@ apiops publish \
--filter ./configuration.extractor.yaml
```

The filter uses the same YAML format as `apiops extract --filter`. Referenced named values,
backends (including backend pool members), policy fragments, and version sets are included by
default; use `--no-transitive` to publish only exact matches. Product, gateway, and subscription
links do not pull their API or Product targets into the publish set.
The filter uses the same YAML format as `apiops extract --filter`. The resources the selected ones
depend on — such as referenced named values, backends (including backend pool members), policy
fragments, version sets, tags, groups and loggers — are included by default; see
[Transitive Dependency Behavior](../guides/filtering-resources.md#transitive-dependency-behavior).
Use `--no-transitive` to publish only exact matches. Product, gateway, and subscription links do
not pull their API or Product targets into the publish set.

### Delete resources not in source

Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ apiops extract \
--filter filter.yaml
```

Referenced backends, named values, and policy fragments are included automatically (transitive dependencies). Use `--no-transitive` to disable this.
Referenced backends, named values, policy fragments and other dependencies are included automatically ([transitive dependencies](guides/filtering-resources.md#transitive-dependency-behavior)). Use `--no-transitive` to disable this.

## 4. Publish to a Target Environment

Expand Down
70 changes: 51 additions & 19 deletions docs/guides/filtering-resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ documentations: []
workspaces: []
```

With transitive resolution enabled (the default), any version set, backend, named value, policy fragment, or tag directly referenced by `my-own-api` or its policies is still pulled in automatically — even though those keys are set to `[]`. Use `--no-transitive` to disable that behavior.
With transitive resolution enabled (the default), the resources `my-own-api` depends on — such as the backends, named values and policy fragments its policies reference, its version set, its tags, and its diagnostics' loggers — are still pulled in automatically, even though those keys are set to `[]`. See [Transitive Dependency Behavior](#transitive-dependency-behavior) for the full list. Use `--no-transitive` to disable that behavior.

The three states for every key are:

Expand Down Expand Up @@ -228,7 +228,7 @@ apis:

### API sub-resource filters

For APIs, you can control which sub-resources (operations, diagnostics, schemas, releases) are extracted. Use an object entry instead of a plain string:
For APIs, you can control which sub-resources (operations, diagnostics, schemas, releases, tag descriptions) are extracted. Use an object entry instead of a plain string:

```yaml
apis:
Expand All @@ -242,13 +242,28 @@ apis:
schemas: [] # Empty = exclude ALL schemas
releases:
- v1-release
tagDescriptions: [] # By name, as under apis/<api>/tagDescriptions/
```

**Sub-filter rules:**
- If a sub-resource key is **omitted**, all sub-resources of that type are included
- If a sub-resource key is an **empty array** (`[]`), all sub-resources of that type are excluded
- If a sub-resource key lists **names**, only those sub-resources are included

#### One sub-filter for many APIs

The key of a nested entry can be a wildcard pattern (`*`, `?`) instead of an API name. Like a plain pattern, it selects the matching APIs, and its sub-filter applies to each of them. For example, to extract every API without its operations, schemas or tag descriptions:

```yaml
apis:
- '*':
operations: []
schemas: []
tagDescriptions: []
```

An entry keyed by an API's name takes precedence over a pattern; otherwise the first matching pattern applies. Quote patterns that start with `*` so YAML reads them as strings.

During extraction, the `operations` filter also applies to the API's OpenAPI specification
(`specification.yaml` or `specification.json`, including Swagger 2.0). Excluded operations
are removed from the specification, and paths with no remaining operations are removed.
Expand All @@ -275,6 +290,10 @@ workspaces:

Supported workspace sub-filter keys: `apis`, `backends`, `diagnostics`, `groups`, `loggers`, `namedValues`, `policyFragments`, `products`, `schemas`, `subscriptions`, `tags`, `versionSets`.

As for APIs, the key can be a wildcard pattern: `- 'team-*': { apis: [orders] }` selects every matching workspace and applies the sub-filter to each. An entry keyed by the workspace's name takes precedence over a pattern; otherwise the first matching pattern applies.

A sub-filter can't be attached to an exclusion (`- '!team-c': {...}`), for APIs or workspaces: the exclusion still applies, but its sub-filter is ignored with a warning.

---

## All 17 Filterable Resource Types
Expand Down Expand Up @@ -303,20 +322,28 @@ Supported workspace sub-filter keys: `apis`, `backends`, `diagnostics`, `groups`

## Transitive Dependency Behavior

When you filter by API name, `apiops extract` automatically includes resources referenced by those APIs. This is **transitive dependency resolution** — it ensures the extracted artifacts are self-contained and publishable.
When you filter resources, `apiops extract` and `apiops publish` automatically include the resources the selected ones depend on. This is **transitive dependency resolution** — it keeps the extracted artifacts self-contained and publishable. Dependencies are followed until nothing new is found, so a pulled-in resource's own dependencies are included too.

### What Gets Auto-Included

```mermaid
flowchart TD
A[Filtered API] --> B[Backends referenced in policies]
A --> C[Named Values referenced in policies]
A --> D[Policy Fragments included in policies]
A --> E[Tags attached to the API]
A --> F[Products containing the API]
A --> G[Diagnostics configured on the API]
A --> H[Version Set the API belongs to]
```
| Selected resource | Includes |
|-------------------|----------|
| Any policy (service, API, operation, GraphQL resolver, product) and policy fragment | Named values referenced as `{{displayName}}`; backends named by `<set-backend-service backend-id>`, `<llm-content-safety backend-id>` and `embeddings-backend-id` on the semantic cache lookup policies; fragments named by `<include-fragment fragment-id>`; loggers named by `<log-to-eventhub logger-id>`; schemas named by validate-content's `<content schema-id>` |
| API | Its version set, and the tags it is linked to |
| Diagnostic (service-level or on an API) | The logger it sends data to (`loggerId`) |
| Logger, backend | Named values referenced in their credentials |
| Backend pool | Its member backends |
| Product | The groups and tags it is linked to |

A reference written as a policy expression (`backend-id="@(...)"`) is not followed, because its value is only known at run time. One set to a named value (`backend-id="{{backend-name}}"`) pulls in the named value and the resource its value names, in the same scope (except `<include-fragment fragment-id>`, where APIM doesn't substitute named values); publish reads the value with your `--overrides` applied, so each environment gets the resource its own value names. A named value that is published also brings the resource its value names whenever any policy or fragment in the source uses it that way, even one the filter leaves out: APIM checks every policy on the target that uses a named value when its value changes. A resource the filter excludes by name, or that isn't in the source, isn't published, so it has to exist on the target already. When a policy uses a named value to name a backend, logger or schema that is published in the same run, publish creates that resource before it updates the named value, and publishes policy fragments, and any backend or logger using the named value in its credentials, after it, since APIM rejects a named value change that would leave a policy using it pointing at a resource that doesn't exist. A secret or Key Vault named value's value isn't known, so the resource it names isn't pulled in; add that resource to the filter.

A `{{...}}` reference, in a policy or in credentials, names a named value by its display name, as APIM does; when no named value has that display name, it is read as a resource name. The `namedValues` list and its exclusions (`'!name'`) use the resource name, which can differ from the display name. References inside XML comments are ignored. Inside a `<set-body template="liquid">` body, a `{{name}}` is followed only when a named value has that name: APIM substitutes those there and leaves every other `{{ ... }}` (`{{body.id}}`, `{{ name }}`, `{{x | upcase}}`) to Liquid, so those are skipped without an error.

A resource the filter excludes by name (`'!name'`) is never pulled in, and links to it are not extracted: an API's or product's link to an excluded tag, or a product's link to an excluded group. Built-in groups (`administrators`, `developers`, `guests`) exist at service level in every APIM instance, so they are not pulled in either; a product's links to them are published only when the filter includes them, as for any group. The exception is the link to `administrators`, which APIM adds to every product itself: it is published whatever the `groups` filter says, unless the filter excludes `administrators` by name.

**Not included:** the products, gateways and subscriptions an API is linked to, and the APIs a product contains. These are independent resources, not dependencies; add them to the filter to include them. On publish, links to APIs or products outside the publish set are skipped.

> **Logger keys:** APIM stores an Application Insights logger's key as an auto-generated secret named value, named with a 24-character hex ID (its display name is `Logger-Credentials--<id>`). It is pulled in with the logger and extracted with its value redacted. Publish skips it unless you provide an override for it: an instance that already has the logger already has its key, but on a new instance the logger can't be published without one. See [Logger credentials with auto-generated named values](environment-overrides.md#logger-credentials-with-auto-generated-named-values).

### Example

Expand All @@ -325,18 +352,23 @@ Given this filter:
```yaml
apis:
- petstore-api
backends: []
namedValues: []
policyFragments: []
loggers: []
tags: []
products: []
```

If `petstore-api` has a policy that references:
- Backend `petstore-backend` → **auto-included**
- Named value `petstore-api-key` → **auto-included**
- Policy fragment `rate-limit-fragment` → **auto-included**

And `petstore-api` is assigned to:
- Product `starter` → **auto-included**
- Tag `production` → **auto-included**

The extract output includes all of these, even though only `apiNames` was specified in the filter.
And `petstore-api`:
- has a diagnostic that sends data to logger `appinsights` → **auto-included**
- is tagged `production` → **auto-included**
- is in product `starter` → **not included**; add `starter` to `products` to include it

### Why Transitive Matters

Expand Down Expand Up @@ -379,7 +411,7 @@ apis:
- orders-admin-api
```

Transitive dependencies (backends, named values, policy fragments) are auto-included.
Transitive dependencies (backends, named values, policy fragments, version sets, tags and loggers) are auto-included.

### Product-Focused Extract

Expand Down
11 changes: 7 additions & 4 deletions docs/reference/dependency-graph.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ apiops-cli solves this by organizing all 34 resource types into **4 tiers**. Eac

| Tier | Resources | Rule |
|------|-----------|------|
| **1** | NamedValue, Tag, Gateway, VersionSet, Backend, Logger, Group, PolicyFragment, GlobalSchema, PolicyRestriction, Documentation | No dependencies — safe to process in any order |
| **1** | NamedValue, Tag, Gateway, VersionSet, Backend, Logger, Group, PolicyFragment, GlobalSchema, PolicyRestriction, Documentation | No dependencies on other tiers. Within the tier, named values go first, since backends and loggers can use them; pool backends follow the backends they contain; and a named value a policy uses to name a backend, logger or schema published in the same run goes after that resource, with policy fragments after it in that case. A backend or logger in the run that uses such a named value in its credentials goes after the named value, and these are ordered so each comes after what it needs; only where that is impossible (a cycle, as when a backend uses in its credentials the named value that names it, or a pool containing it), the named values in it go first |
| **2** | Diagnostic, ServicePolicy, Product, Api | Depend only on Tier 1 resources |
| **3** | ProductPolicy, ProductGroup, ProductTag, ProductApi, ProductWiki, ApiPolicy, ApiTag, ApiDiagnostic, ApiOperation, ApiSchema, ApiRelease, ApiTagDescription, ApiWiki, McpServer, GraphQLResolver, GatewayApi, Subscription | Depend on Tier 1 and/or Tier 2 resources |
| **4** | ApiOperationPolicy, GraphQLResolverPolicy | Depend on Tier 3 resources |
Expand Down Expand Up @@ -163,9 +163,12 @@ If a Tier 2 resource fails to publish, its dependents in Tier 3 and 4 will also

When using `filter.yaml` to include specific resources, be aware of dependencies.

**During publish**, apiops-cli does **not** auto-include dependencies at the
artifact level — your filter (and hence your extracted artifact tree) must
already contain every required parent. For example, publishing an **ApiPolicy**
**During publish** with a filter, apiops-cli includes the resources the
selected ones depend on (named values, backends, policy fragments, version
sets, tags, groups, loggers and so on) unless `--no-transitive` is passed; see
[Transitive Dependency Behavior](../guides/filtering-resources.md#transitive-dependency-behavior).
It can only include what is in the artifact tree, so the tree must already
contain every required resource. For example, publishing an **ApiPolicy**
without its parent **Api** present in the artifact tree will fail.

**During extract**, apiops-cli **does** auto-follow a limited set of transitive
Expand Down
Loading