Repository navigation
JS: convert data extensions to JSON - #23074
Merged
Merged
Conversation
In preparation for converting data extensions to JSON, these model.yml files need to be renamed to avoid clashes with existing model.json files in the same directories.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Approval is blocked by an unresolved model-file glob regression, a comment typo, and missing DCA validation for the production loading change.
3 open issues
What changed in this PR
Migrates JavaScript CodeQL library and test data extensions from YAML to JSON, enabling faster loading with CodeQL CLI 2.27.3 while preserving existing model tuples.
Changes:
- Converts framework, security-domain, external-library, query-test, and library-test extensions to JSON.
- Renames conflicting framework model files where necessary during conversion.
- Updates the JavaScript library pack manifest to discover JSON data extensions.
| File | Description |
|---|---|
| javascript/ql/test/query-tests/Security/CWE-918/RequestForgery.ext.yml | Removes the YAML request-forgery test extension replaced by JSON. |
| javascript/ql/test/query-tests/Security/CWE-918/RequestForgery.ext.json | Adds the JSON request-forgery test extension. |
| javascript/ql/test/query-tests/Security/CWE-918/ClientSideRequestForgery.ext.yml | Removes the YAML client-side request-forgery test extension replaced by JSON. |
| javascript/ql/test/query-tests/Security/CWE-918/ClientSideRequestForgery.ext.json | Adds the JSON client-side request-forgery test extension. |
| javascript/ql/test/query-tests/Security/CWE-730/Threat-models-enabled/RegExpInjection.ext.yml | Removes the YAML regular-expression-injection test extension replaced by JSON. |
| javascript/ql/test/query-tests/Security/CWE-730/Threat-models-enabled/RegExpInjection.ext.json | Adds the JSON regular-expression-injection test extension. |
| javascript/ql/test/query-tests/Security/CWE-089/local-threat-source/SqlInjection.ext.yml | Removes the YAML SQL-injection test extension replaced by JSON. |
| javascript/ql/test/query-tests/Security/CWE-089/local-threat-source/SqlInjection.ext.json | Adds the JSON SQL-injection test extension. |
| javascript/ql/test/query-tests/Security/CWE-079/DomBasedXssWithResponseThreat/Xss.ext.yml | Removes the YAML XSS test extension replaced by JSON. |
| javascript/ql/test/query-tests/Security/CWE-079/DomBasedXssWithResponseThreat/Xss.ext.json | Adds the JSON XSS test extension. |
| javascript/ql/test/library-tests/threat-models/ai-enabled/ActiveAiThreatModel.ext.yml | Removes the YAML active AI threat-model test extension replaced by JSON. |
| javascript/ql/test/library-tests/threat-models/ai-enabled/ActiveAiThreatModel.ext.json | Adds the JSON active AI threat-model test extension. |
| javascript/ql/test/library-tests/ModelGeneration/ModelGeneration.ext.yml | Removes the YAML model-generation test extension replaced by JSON. |
| javascript/ql/test/library-tests/ModelGeneration/ModelGeneration.ext.json | Adds the JSON model-generation test extension. |
| javascript/ql/test/library-tests/frameworks/data/warnings.ext.yml | Removes the YAML framework warnings test extension replaced by JSON. |
| javascript/ql/test/library-tests/frameworks/data/warnings.ext.json | Adds the JSON framework warnings test extension. |
| javascript/ql/test/library-tests/frameworks/data/test.ext.yml | Removes the YAML framework data test extension replaced by JSON. |
| javascript/ql/test/library-tests/frameworks/data/test.ext.json | Adds the JSON framework data test extension. |
| javascript/ql/test/library-tests/DataExtensions/message.ext.yml | Removes the YAML message data extension replaced by JSON. |
| javascript/ql/test/library-tests/DataExtensions/message.ext.json | Adds the JSON message data extension. |
| javascript/ql/test/library-tests/DataExtensions/execa.ext.yml | Removes the YAML execa data extension replaced by JSON. |
| javascript/ql/test/library-tests/DataExtensions/execa.ext.json | Adds the JSON execa data extension. |
| javascript/ql/test/library-tests/DataExtensions/connection.ext.yml | Removes the YAML connection data extension replaced by JSON. |
| javascript/ql/test/library-tests/DataExtensions/connection.ext.json | Adds the JSON connection data extension. |
| javascript/ql/lib/semmle/javascript/security/domains/untrusted/untrusted_domains.model.yml | Removes the YAML untrusted-domains model. |
| javascript/ql/lib/semmle/javascript/security/domains/untrusted/untrusted_domains.model.json | Adds the JSON untrusted-domains model. |
| javascript/ql/lib/semmle/javascript/security/domains/IntegrityCheckingRequired/integrity_checking_required.model.yml | Removes the YAML integrity-checking-required model. |
| javascript/ql/lib/semmle/javascript/security/domains/IntegrityCheckingRequired/integrity_checking_required.model.json | Adds the JSON integrity-checking-required model. |
| javascript/ql/lib/semmle/javascript/security/domains/compromised/compromised_domains.model.yml | Removes the YAML compromised-domains model. |
| javascript/ql/lib/semmle/javascript/security/domains/compromised/compromised_domains.model.json | Adds the JSON compromised-domains model. |
| javascript/ql/lib/semmle/javascript/frameworks/sqlite3/sqlite3.model.json | Adds the converted JSON SQLite3 framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/sqlite3/model.yml | Removes the generic YAML SQLite3 model filename after conversion. |
| javascript/ql/lib/semmle/javascript/frameworks/SQL.model.yml | Removes the YAML SQL framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/SQL.model.json | Adds the JSON SQL framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/spanner/spanner.model.json | Adds the converted JSON Spanner framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/spanner/model.yml | Removes the generic YAML Spanner model filename after conversion. |
| javascript/ql/lib/semmle/javascript/frameworks/sequelize/sequelize.model.json | Adds the converted JSON Sequelize framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/sequelize/model.yml | Removes the generic YAML Sequelize model filename after conversion. |
| javascript/ql/lib/semmle/javascript/frameworks/pg/pg.model.json | Adds the converted JSON PostgreSQL framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/pg/model.yml | Removes the generic YAML PostgreSQL model filename after conversion. |
| javascript/ql/lib/semmle/javascript/frameworks/NoSQL.model.yml | Removes the YAML NoSQL framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/NoSQL.model.json | Adds the JSON NoSQL framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/NodeJSLib.model.yml | Removes the YAML Node.js library model. |
| javascript/ql/lib/semmle/javascript/frameworks/NodeJSLib.model.json | Adds the JSON Node.js library model. |
| javascript/ql/lib/semmle/javascript/frameworks/mysql/mysql.model.json | Adds the converted JSON MySQL framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/mysql/model.yml | Removes the generic YAML MySQL model filename after conversion. |
| javascript/ql/lib/semmle/javascript/frameworks/mssql/mssql.model.json | Adds the converted JSON MSSQL framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/mssql/model.yml | Removes the generic YAML MSSQL model filename after conversion. |
| javascript/ql/lib/semmle/javascript/frameworks/minimongo/model.yml | Removes the generic YAML Minimongo model filename after conversion. |
| javascript/ql/lib/semmle/javascript/frameworks/minimongo/minimongo.model.json | Adds the converted JSON Minimongo framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/helmet/Helmet.Required.Setting.model.yml | Removes the YAML Helmet framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/helmet/Helmet.Required.Setting.model.json | Adds the JSON Helmet framework model. |
| javascript/ql/lib/semmle/javascript/frameworks/data/internal/empty.model.yml | Removes the YAML empty framework model definitions. |
| javascript/ql/lib/semmle/javascript/frameworks/data/internal/empty.model.json | Adds the JSON empty framework model definitions. |
| javascript/ql/lib/qlpack.yml | Updates data-extension discovery patterns for JSON and YAML model files. |
| javascript/ql/lib/ext/zod.model.yml | Removes the YAML Zod model extension. |
| javascript/ql/lib/ext/zod.model.json | Adds the JSON Zod model extension. |
| javascript/ql/lib/ext/vue.model.yml | Removes the YAML Vue model extension. |
| javascript/ql/lib/ext/vue.model.json | Adds the JSON Vue model extension. |
| javascript/ql/lib/ext/underscore.string.model.yml | Removes the YAML underscore.string model extension. |
| javascript/ql/lib/ext/underscore.string.model.json | Adds the JSON underscore.string model extension. |
| javascript/ql/lib/ext/tanstack.model.yml | Removes the YAML TanStack model extension. |
| javascript/ql/lib/ext/tanstack.model.json | Adds the JSON TanStack model extension. |
| javascript/ql/lib/ext/shelljs.model.yml | Removes the YAML ShellJS model extension. |
| javascript/ql/lib/ext/shelljs.model.json | Adds the JSON ShellJS model extension. |
| javascript/ql/lib/ext/rimraf.model.yml | Removes the YAML rimraf model extension. |
| javascript/ql/lib/ext/rimraf.model.json | Adds the JSON rimraf model extension. |
| javascript/ql/lib/ext/react.model.yml | Removes the YAML React model extension. |
| javascript/ql/lib/ext/react.model.json | Adds the JSON React model extension. |
| javascript/ql/lib/ext/react-relay-threat.model.yml | Removes the YAML React Relay threat model extension. |
| javascript/ql/lib/ext/react-relay-threat.model.json | Adds the JSON React Relay threat model extension. |
| javascript/ql/lib/ext/openrouter.model.yml | Removes the YAML OpenRouter model extension. |
| javascript/ql/lib/ext/openrouter.model.json | Adds the JSON OpenRouter model extension. |
| javascript/ql/lib/ext/openai.model.yml | Removes the YAML OpenAI model extension. |
| javascript/ql/lib/ext/openai.model.json | Adds the JSON OpenAI model extension. |
| javascript/ql/lib/ext/open.model.yml | Removes the YAML open model extension. |
| javascript/ql/lib/ext/open.model.json | Adds the JSON open model extension. |
| javascript/ql/lib/ext/mkdirp.model.yml | Removes the YAML mkdirp model extension. |
| javascript/ql/lib/ext/mkdirp.model.json | Adds the JSON mkdirp model extension. |
| javascript/ql/lib/ext/mcp.model.yml | Removes the YAML MCP model extension. |
| javascript/ql/lib/ext/mcp.model.json | Adds the JSON MCP model extension. |
| javascript/ql/lib/ext/markdown-table.model.yml | Removes the YAML markdown-table model extension. |
| javascript/ql/lib/ext/markdown-table.model.json | Adds the JSON markdown-table model extension. |
| javascript/ql/lib/ext/make-dir.model.yml | Removes the YAML make-dir model extension. |
| javascript/ql/lib/ext/make-dir.model.json | Adds the JSON make-dir model extension. |
| javascript/ql/lib/ext/langchain.model.yml | Removes the YAML LangChain model extension. |
| javascript/ql/lib/ext/langchain.model.json | Adds the JSON LangChain model extension. |
| javascript/ql/lib/ext/hana-db-client.model.yml | Removes the YAML SAP HANA model extension. |
| javascript/ql/lib/ext/hana-db-client.model.json | Adds the JSON SAP HANA model extension. |
| javascript/ql/lib/ext/graph-ql.model.yml | Removes the YAML GraphQL model extension. |
| javascript/ql/lib/ext/graph-ql.model.json | Adds the JSON GraphQL model extension. |
| javascript/ql/lib/ext/google-genai.model.yml | Removes the YAML Google GenAI model extension. |
| javascript/ql/lib/ext/google-genai.model.json | Adds the JSON Google GenAI model extension. |
| javascript/ql/lib/ext/fastmcp.model.yml | Removes the YAML FastMCP model extension. |
| javascript/ql/lib/ext/fastmcp.model.json | Adds the JSON FastMCP model extension. |
| javascript/ql/lib/ext/default-threat-models-fixup.model.yml | Removes the YAML default threat-model fixup extension. |
| javascript/ql/lib/ext/default-threat-models-fixup.model.json | Adds the JSON default threat-model fixup extension. |
| javascript/ql/lib/ext/cors.model.yml | Removes the YAML CORS model extension. |
| javascript/ql/lib/ext/cors.model.json | Adds the JSON CORS model extension. |
| javascript/ql/lib/ext/call-me-maybe.model.yml | Removes the YAML call-me-maybe model extension. |
| javascript/ql/lib/ext/call-me-maybe.model.json | Adds the JSON call-me-maybe model extension. |
| javascript/ql/lib/ext/axios.model.yml | Removes the YAML Axios model extension. |
| javascript/ql/lib/ext/axios.model.json | Adds the JSON Axios model extension. |
| javascript/ql/lib/ext/aws-sdk.model.yml | Removes the YAML AWS SDK model extension. |
| javascript/ql/lib/ext/aws-sdk.model.json | Adds the JSON AWS SDK model extension. |
| javascript/ql/lib/ext/apollo-server.model.yml | Removes the YAML Apollo Server model extension. |
| javascript/ql/lib/ext/apollo-server.model.json | Adds the JSON Apollo Server model extension. |
| javascript/ql/lib/ext/anthropic.model.yml | Removes the YAML Anthropic model extension. |
| javascript/ql/lib/ext/anthropic.model.json | Adds the JSON Anthropic model extension. |
Review effort: Lite (auto)
| codeql/yaml: ${workspace} | ||
| dataExtensions: | ||
| - semmle/javascript/frameworks/**/model.yml | ||
| - semmle/javascript/frameworks/**/*.model.yml |
Comment on lines
+24
to
+26
| - semmle/javascript/frameworks/**/*.model.json | ||
| - semmle/javascript/security/domains/**/*.model.json | ||
| - ext/*.model.json |
| // they were separated. To handle everything with a single model, we treat them as the same here. | ||
| ["mongodb.Db", "mongodb.MongoClient", ""], | ||
| // 'marsdb' has no typings and is archived. | ||
| // We just model is as a variant of 'mongoose'. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CodeQL CLI 2.27.3 supports JSON data extensions (aka MaD models), and can load them significantly faster than YAML. This PR converts those in the JS library pack (first two commits), along with
$TEST.ext.ymlfiles (third commit). The conversions were automated by an LLM-generated Python script (which I will add in a separate PR).Note that the first commit renames some
model.ymldata extension files that havemodel.jsonfiles in the same directory, and which would have caused name clashes upon conversion.Although JSON is preferred going forward,
javascript/ql/lib/qlpack.ymlnow has patterns that will pick up both.model.ymland.model.jsonfiles, so that any YAML files that do get checked in won't get silently dropped. (I also took care to ensure the patterns don't pick up the existingmodel.jsonfiles, which aren't data extensions).To verify the first two commits, I cherry-picked them onto #23073, which has a temporary test that enumerates all tuples from data extensions. The test still passes after conversion.
For the third commit, I sampled several of the converted files and manually inspected the diffs.