Compare commits

...

35 Commits

Author SHA1 Message Date
github-actions[bot] bdf39710a2 Rebuild 2026-08-05 17:59:47 +00:00
dependabot[bot] 74cfae9be6 Bump actions/setup-java
Bumps the actions-minor group with 1 update in the /.github/workflows directory: [actions/setup-java](https://github.com/actions/setup-java).


Updates `actions/setup-java` from 5.6.0 to 5.7.0
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/03ad4de0992f5dab5e18fcb136590ce7c4a0ac95...b6effb05e454b25005698d916606bdc6ffcbf961)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-version: 5.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-05 17:56:46 +00:00
Michael B. Gale 6a90bf1f54 Merge pull request #4075 from github/dependabot/npm_and_yarn/brace-expansion-1.1.18
Bump brace-expansion from 1.1.16 to 1.1.18
2026-08-04 14:52:59 +00:00
github-actions[bot] c5995f544d Rebuild 2026-08-04 14:19:52 +00:00
dependabot[bot] 76c44396d3 Bump brace-expansion from 1.1.16 to 1.1.18
Bumps [brace-expansion](https://github.com/juliangruber/brace-expansion) from 1.1.16 to 1.1.18.
- [Release notes](https://github.com/juliangruber/brace-expansion/releases)
- [Commits](https://github.com/juliangruber/brace-expansion/compare/v1.1.16...v1.1.18)

---
updated-dependencies:
- dependency-name: brace-expansion
  dependency-version: 1.1.18
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-04 14:18:04 +00:00
Michael B. Gale fad141fa6c Merge pull request #4073 from github/mergeback/v4.37.6-to-main-5595ccaf
Mergeback v4.37.6 refs/heads/releases/v4 into main
2026-08-04 14:03:40 +00:00
github-actions[bot] 7d82f1132f Rebuild 2026-08-04 13:34:54 +00:00
github-actions[bot] 37bdbde050 Update changelog and version after v4.37.6 2026-08-04 13:34:41 +00:00
Michael B. Gale 5595ccaf91 Merge pull request #4071 from github/update-v4.37.6-6a9359a1b
Merge main into releases/v4
2026-08-04 14:33:02 +01:00
Michael B. Gale ec9c75796a Add change note for PR 4070 2026-08-04 14:19:36 +01:00
github-actions[bot] 45c8742e17 Update changelog for v4.37.6 2026-08-04 13:15:26 +00:00
Michael B. Gale 6a9359a1bd Merge pull request #4070 from github/mbg/remote-address/change-file-default
Change `DEFAULT_CONFIG_FILE_NAME`
2026-08-03 14:18:48 +00:00
Michael B. Gale 065cdc0394 Change DEFAULT_CONFIG_FILE_NAME 2026-08-03 15:02:48 +01:00
Michael B. Gale f99dd5aeee Merge pull request #4066 from github/dependabot/npm_and_yarn/js-yaml-5.2.2
Bump js-yaml from 5.2.1 to 5.2.2
2026-08-03 13:26:42 +00:00
Henry Mercer 1804b211a3 Merge pull request #4068 from github/mergeback/v4.37.5-to-main-d1ba80a1
Mergeback v4.37.5 refs/heads/releases/v4 into main
2026-08-03 11:43:09 +00:00
github-actions[bot] 3020a2f462 Rebuild 2026-08-03 11:03:06 +00:00
github-actions[bot] 93c3a5a40b Update changelog and version after v4.37.5 2026-08-03 11:02:52 +00:00
Henry Mercer d1ba80a13d Merge pull request #4067 from github/update-v4.37.5-1cd4d01d5
Merge main into releases/v4
2026-08-03 12:01:20 +01:00
github-actions[bot] e74600b0d9 Update changelog for v4.37.5 2026-08-03 09:18:21 +00:00
github-actions[bot] 266c7bdbd2 Rebuild 2026-08-01 14:26:29 +00:00
dependabot[bot] daa7fe6fba Bump js-yaml from 5.2.1 to 5.2.2
Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 5.2.1 to 5.2.2.
- [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/compare/5.2.1...5.2.2)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 5.2.2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-08-01 14:24:54 +00:00
Michael B. Gale 1cd4d01d58 Merge pull request #4061 from github/henrymercer/turbo-system
Handle network errors when streaming the CodeQL bundle download
2026-07-31 16:04:48 +00:00
Michael B. Gale d2bfc30bc3 Merge pull request #4050 from github/mbg/status/registries
Include `registry_types` in more status reports
2026-07-31 15:32:50 +00:00
Sam Robson 68028fcb16 Merge pull request #4062 from github/sam-robson/migrate-enterprise-release-pat
ci: update Enterprise release checkout credential
2026-07-31 14:05:01 +00:00
Sam Robson c29563eeaa ci: use federated enterprise release PAT 2026-07-31 10:10:39 +01:00
sim 155e522997 Link the PR from the changelog entry
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-07-30 18:48:10 +01:00
sim 2d3b351ea6 Handle network errors when streaming the CodeQL bundle download
A network error such as `ECONNRESET` while streaming the download and
extraction of the CodeQL bundle terminated the `init` Action rather than
falling back to downloading the bundle before extracting it, since no
`error` listener was attached to the request returned by `https.get`.

Also pipe the response into `tar` using `stream.pipeline` so that errors
on the response itself are surfaced and `tar`'s standard input is closed,
and abort the request if it stalls.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-07-30 18:47:27 +01:00
Michael B. Gale 780c00da3f Merge remote-tracking branch 'origin/main' into mbg/status/registries 2026-07-29 16:52:25 +01:00
Michael B. Gale 13d4882649 Validate JSON more 2026-07-29 11:35:46 +01:00
Michael B. Gale e55a57b808 Add RegistryBase schema and type 2026-07-29 11:21:19 +01:00
Michael B. Gale 51d51e8121 Add boolean Validator to json module 2026-07-29 11:19:57 +01:00
Michael B. Gale e893985e8b Fix makeValidator returning required: boolean 2026-07-29 11:19:21 +01:00
Michael B. Gale eb692f8b49 Add check to createStatusReportBase test 2026-07-29 10:57:52 +01:00
Michael B. Gale aac07d2a41 Include registry_types whenever CODEQL_PROXY_URLS is set 2026-07-29 10:55:46 +01:00
Michael B. Gale de57c4a441 Move registry_types to StatusReportBase 2026-07-29 10:38:08 +01:00
18 changed files with 1096 additions and 590 deletions
@@ -63,7 +63,7 @@ jobs:
- name: Check out repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Install Java
uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0
uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0
with:
java-version: ${{ inputs.java-version || '17' }}
distribution: temurin
+1 -1
View File
@@ -63,7 +63,7 @@ jobs:
- name: Check out repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Install Java
uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5.6.0
uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0
with:
java-version: ${{ inputs.java-version || '17' }}
distribution: temurin
@@ -38,7 +38,7 @@ jobs:
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: github/enterprise-releases
token: ${{ secrets.ENTERPRISE_RELEASE_TOKEN }}
token: ${{ secrets.CODEQL_CI_ENTERPRISE_RELEASE_PAT }}
path: ${{ github.workspace }}/enterprise-releases/
sparse-checkout: releases.json
+8
View File
@@ -6,6 +6,14 @@ See the [releases page](https://github.com/github/codeql-action/releases) for th
No user facing changes.
## 4.37.6 - 04 Aug 2026
- Changed the default filepath for the new remote file address format that was introduced in CodeQL Action 4.37.0 / 3.37.0 to `.github/codeql-config.yml` to align it with the suggested path that is used elsewhere. [#4070](https://github.com/github/codeql-action/pull/4070)
## 4.37.5 - 03 Aug 2026
- Fixed a bug where a network error while streaming the download of the CodeQL bundle could terminate the `init` Action instead of falling back to downloading the bundle before extracting it. [#4061](https://github.com/github/codeql-action/pull/4061)
## 4.37.4 - 29 Jul 2026
- This version of the CodeQL Action adds support for the `tools` input for the `codeql-action/init` step to be specified using a `github-codeql-tools` [repository property](https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization). This feature will gradually be rolled out following the release of this version. Once rolled out, this allows for the CodeQL CLI version that is used in GitHub-managed workflows, such as Default Setup, to be set to a custom value. For example, customers who run into issues with rate limits when a new CodeQL CLI version is released can set the value to `toolcache` to always use the CodeQL CLI version that is available in the runner toolcache. For Advanced Setup workflows, the value provided for `tools` in the workflow definition always takes precedence unless the value of the repository property starts with `!`. [#4037](https://github.com/github/codeql-action/pull/4037)
+793 -528
View File
File diff suppressed because it is too large Load Diff
+27 -27
View File
@@ -1,12 +1,12 @@
{
"name": "codeql",
"version": "4.37.5",
"version": "4.37.7",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "codeql",
"version": "4.37.5",
"version": "4.37.7",
"license": "MIT",
"workspaces": [
"pr-checks"
@@ -31,7 +31,7 @@
"follow-redirects": "^1.16.0",
"get-folder-size": "^5.0.0",
"https-proxy-agent": "^7.0.6",
"js-yaml": "^5.2.1",
"js-yaml": "^5.2.2",
"jsonschema": "1.5.0",
"long": "^5.3.2",
"node-forge": "^1.4.0",
@@ -374,9 +374,9 @@
"license": "Apache-2.0"
},
"node_modules/@actions/artifact/node_modules/brace-expansion": {
"version": "2.1.2",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.2.tgz",
"integrity": "sha512-w5JZcKgdhDOgOwm8H+KgbosopHMuGcl6qbulwjtz3SM7I7P3yW1eAjzMPLrIE+NQ9vjgANKHWeMHnrT0OXW1oA==",
"version": "2.1.4",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.1.4.tgz",
"integrity": "sha512-hGfVzPxthbf3+2yjg/RBs60cB0FhqBS/zvdV/4wn4/BmN0bNMMHPc4V/BbFieqf1TKAGGAHnY4eSjajCl0f2Xg==",
"license": "MIT",
"dependencies": {
"balanced-match": "^1.0.0"
@@ -2843,9 +2843,9 @@
}
},
"node_modules/@typescript-eslint/typescript-estree/node_modules/brace-expansion": {
"version": "5.0.8",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.8.tgz",
"integrity": "sha512-JZyDyq3D4AUifKTPOB7DELf6XsB3WdPuNxCtob1vFXPsSXhdAiHBWJ/tJ8HAc9aH84BK+5JFZLNkJKx3G9kzQg==",
"version": "5.0.9",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.9.tgz",
"integrity": "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -3864,9 +3864,9 @@
"license": "MIT"
},
"node_modules/brace-expansion": {
"version": "1.1.16",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.16.tgz",
"integrity": "sha512-IDw48K2/2kRkg9LdJxurvq3lV3aBgq0REY89duEqFRthjlPdXHKMj7EnQOXVckxzgisinf3nHfrcE2FufFLXMw==",
"version": "1.1.18",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.18.tgz",
"integrity": "sha512-Edep/X9fGqVNmzKBVsDYIOtD+z1tuezV70LBjdCst9Tqu76lsnvRiZ6oTic1n+/BIwX6QDGAO94PN4N2SADvtw==",
"license": "MIT",
"dependencies": {
"balanced-match": "^1.0.0",
@@ -5115,16 +5115,16 @@
}
},
"node_modules/eslint-plugin-import-x/node_modules/brace-expansion": {
"version": "5.0.7",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.7.tgz",
"integrity": "sha512-7oFy703dxfY3/NLxC1fh2SUCQ0H9rmAY+5EpDVfXjUTTs+HEwR2nYaqLv+GWcTsumwxPfiz6CzCNkwXwBUwqCA==",
"version": "5.0.9",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.9.tgz",
"integrity": "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==",
"dev": true,
"license": "MIT",
"dependencies": {
"balanced-match": "^4.0.2"
},
"engines": {
"node": "18 || 20 || >=22"
"node": "20 || >=22"
}
},
"node_modules/eslint-plugin-import-x/node_modules/minimatch": {
@@ -6111,15 +6111,15 @@
}
},
"node_modules/glob/node_modules/brace-expansion": {
"version": "5.0.7",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.7.tgz",
"integrity": "sha512-7oFy703dxfY3/NLxC1fh2SUCQ0H9rmAY+5EpDVfXjUTTs+HEwR2nYaqLv+GWcTsumwxPfiz6CzCNkwXwBUwqCA==",
"version": "5.0.9",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.9.tgz",
"integrity": "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==",
"license": "MIT",
"dependencies": {
"balanced-match": "^4.0.2"
},
"engines": {
"node": "18 || 20 || >=22"
"node": "20 || >=22"
}
},
"node_modules/glob/node_modules/minimatch": {
@@ -6981,9 +6981,9 @@
}
},
"node_modules/js-yaml": {
"version": "5.2.1",
"resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-5.2.1.tgz",
"integrity": "sha512-zfLtNfQqxVqq3uaTqSkh4x4hZw3KHobGUA0fJUj4wawW8bsQLTVqpHdXSIzidh7o+4lEW36tANuAGdaFx6Zgnw==",
"version": "5.2.2",
"resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-5.2.2.tgz",
"integrity": "sha512-dayzUzKkJ1MkuUtZglSebU43utNXH0OWQByK9rKOOuYIO8M5TV1y+n8ALMdG0rdzBnfNkOmZEqrURepb0ejqBw==",
"funding": [
{
"type": "github",
@@ -8090,15 +8090,15 @@
}
},
"node_modules/readdir-glob/node_modules/brace-expansion": {
"version": "5.0.7",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.7.tgz",
"integrity": "sha512-7oFy703dxfY3/NLxC1fh2SUCQ0H9rmAY+5EpDVfXjUTTs+HEwR2nYaqLv+GWcTsumwxPfiz6CzCNkwXwBUwqCA==",
"version": "5.0.9",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.9.tgz",
"integrity": "sha512-ScQ4IuvIEF1TMlP7Zt+vjJ//9zlPb2SDcxWxM3bk8s6t6GGdJ7KO1dCcTidOPJKePW30LE/2cT7wCyPho9/Wxg==",
"license": "MIT",
"dependencies": {
"balanced-match": "^4.0.2"
},
"engines": {
"node": "18 || 20 || >=22"
"node": "20 || >=22"
}
},
"node_modules/readdir-glob/node_modules/minimatch": {
+2 -2
View File
@@ -1,6 +1,6 @@
{
"name": "codeql",
"version": "4.37.5",
"version": "4.37.7",
"private": true,
"description": "CodeQL action",
"scripts": {
@@ -39,7 +39,7 @@
"follow-redirects": "^1.16.0",
"get-folder-size": "^5.0.0",
"https-proxy-agent": "^7.0.6",
"js-yaml": "^5.2.1",
"js-yaml": "^5.2.2",
"jsonschema": "1.5.0",
"long": "^5.3.2",
"node-forge": "^1.4.0",
+2 -2
View File
@@ -253,8 +253,8 @@ const languageSetups: LanguageSetups = {
name: "Install Java",
uses: pinnedUses(
"actions/setup-java",
"03ad4de0992f5dab5e18fcb136590ce7c4a0ac95",
"v5.6.0",
"b6effb05e454b25005698d916606bdc6ffcbf961",
"v5.7.0",
),
with: {
"java-version": `\${{ inputs.java-version || '${defaultLanguageVersions.java}' }}`,
+1 -1
View File
@@ -16,7 +16,7 @@ export interface RemoteFileAddress {
}
/** The default file path to use in configuration file shorthands. */
export const DEFAULT_CONFIG_FILE_NAME = ".github/codeql-action.yaml";
export const DEFAULT_CONFIG_FILE_NAME = ".github/codeql-config.yml";
/** The default ref to use in configuration file shorthands. */
export const DEFAULT_CONFIG_FILE_REF = "main";
+27 -5
View File
@@ -35,6 +35,11 @@ export function isNumber(value: unknown): value is number {
return typeof value === "number";
}
/** Asserts that `value` is a boolean. */
export function isBoolean(value: unknown): value is boolean {
return typeof value === "boolean";
}
/** Asserts that `value` is either a string or undefined. */
export function isStringOrUndefined(
value: unknown,
@@ -62,14 +67,11 @@ function defaultCheck(
return (arg) => ({ unknownKeys: [], invalidKeys: [], valid: validate(arg) });
}
function makeValidator<T>(
validate: (arg: unknown) => arg is T,
required: boolean = true,
) {
function makeValidator<T>(validate: (arg: unknown) => arg is T) {
return {
validate,
check: defaultCheck(validate),
required,
required: true,
} as const satisfies Validator<T>;
}
@@ -82,6 +84,9 @@ export const string = makeValidator(isString);
/** A validator for number fields in schemas. */
export const number = makeValidator(isNumber);
/** A validator for boolean fields in schemas. */
export const boolean = makeValidator(isBoolean);
/** A validator for arrays. */
export function array<T>(validator: Validator<T>) {
const validate = (val: unknown) => {
@@ -221,6 +226,23 @@ export function validateSchema<
return result.valid;
}
/**
* Validates that `arr` is an array whose elements satisfy at least `elementSchema`.
* Additional keys are accepted in each element.
*
* @param elementSchema The schema to validate the elements against.
* @param arr The array to validate.
* @returns Asserts that `arr` has elements of `schema`'s type if validation is successful.
*/
export function validateArray<
S extends Schema,
T extends UnvalidatedArray = Array<FromSchema<S>>,
>(elementSchema: S, arr: UnvalidatedArray): arr is T {
const elementValidator = object(elementSchema);
return array(elementValidator).validate(arr);
}
export interface CheckSchemaOptions {
/** Whether to stop validation after the first error. */
failFast?: boolean;
+1 -7
View File
@@ -83,12 +83,6 @@ export class StartProxyError extends Error {
}
}
interface StartProxyStatus extends StatusReportBase {
// A comma-separated list of registry types which are configured for CodeQL.
// This only includes registry types we support, not all that are configured.
registry_types: string;
}
/**
* Sends a status report for the `start-proxy` action indicating a successful outcome.
*
@@ -112,7 +106,7 @@ export async function sendSuccessStatusReport(
logger,
);
if (statusReportBase !== undefined) {
const statusReport: StartProxyStatus = {
const statusReport: StatusReportBase = {
...statusReportBase,
registry_types: registry_types.join(","),
};
+11 -5
View File
@@ -254,13 +254,19 @@ export function credentialToStr(credential: Credential): string {
return result;
}
/** A package registry is identified by its type and address. */
export type Registry = {
/** The schema for `RegistryBase` objects. */
export const registryBaseSchema = {
/** The type of the package registry. */
type: string;
type: json.string,
/** Whether the registry replaces the base registry for the ecosystem. */
"replaces-base"?: boolean;
} & Address;
"replaces-base": json.optional(json.boolean),
} as const satisfies json.Schema;
/** Information about a registry, other than its address. */
export type RegistryBase = json.FromSchema<typeof registryBaseSchema>;
/** A package registry is identified by its type and address. */
export type Registry = RegistryBase & Address;
// If a registry has an `url`, then that takes precedence over the `host` which may or may
// not be defined.
+64 -1
View File
@@ -4,15 +4,17 @@ import * as uuid from "uuid";
import * as actionsUtil from "./actions-util";
import { Config } from "./config-utils";
import { EnvVar } from "./environment";
import { EnvVar, RegistryProxyVars } from "./environment";
import { BuiltInLanguage } from "./languages";
import { getRunnerLogger } from "./logging";
import { ToolsSource } from "./setup-codeql";
import type { Registry } from "./start-proxy";
import {
ActionName,
createInitWithConfigStatusReport,
createStatusReportBase,
getActionsStatus,
getRegistryTypesFromEnv,
getJobUUID,
InitStatusReport,
InitWithConfigStatusReport,
@@ -22,12 +24,69 @@ import {
setupActionsVars,
createTestConfig,
makeMacro,
getTestEnv,
RecordingLogger,
callee,
} from "./testing-utils";
import { BuildMode, ConfigurationError, withTmpDir, wrapError } from "./util";
setupTests(test);
test("getRegistryTypesFromEnv - gets unique registry types from environment", async (t) => {
const logger = new RecordingLogger(true);
const env = getTestEnv({
[RegistryProxyVars.PROXY_URLS]: JSON.stringify([
{ type: "git_source", url: "https://example.com" },
{ type: "git_source", url: "https://github.com" },
{ type: "docker_registry", url: "https://registry.example.com" },
] satisfies Array<Partial<Registry>>),
});
const result = getRegistryTypesFromEnv(logger, env);
t.deepEqual(result, ["git_source", "docker_registry"].sort().join(","));
});
test("getRegistryTypesFromEnv - returns undefined if the env var is not set", async (t) => {
const logger = new RecordingLogger(true);
const env = getTestEnv({});
const result = getRegistryTypesFromEnv(logger, env);
t.is(result, undefined);
});
test("getRegistryTypesFromEnv - returns undefined if the env var is not valid JSON", async (t) => {
const logger = new RecordingLogger(true);
const env = getTestEnv({ [RegistryProxyVars.PROXY_URLS]: "[" });
const result = getRegistryTypesFromEnv(logger, env);
t.is(result, undefined);
});
test("getRegistryTypesFromEnv - returns undefined if the env var is unexpected JSON", async (t) => {
const logger = new RecordingLogger(true);
t.is(
getRegistryTypesFromEnv(
logger,
getTestEnv({
// Top-level object rather than an array of objects.
[RegistryProxyVars.PROXY_URLS]: JSON.stringify({ type: "git_source" }),
}),
),
undefined,
);
t.is(
getRegistryTypesFromEnv(
logger,
getTestEnv({
// Object has no "type" key.
[RegistryProxyVars.PROXY_URLS]: JSON.stringify([{}]),
}),
),
undefined,
);
});
test("getJobUUID - generates valid UUIDs", async (t) => {
await callee(getJobUUID)
.withArgs()
@@ -74,6 +133,9 @@ function setupEnvironmentAndStub(tmpDir: string) {
process.env[EnvVar.ANALYSIS_KEY] = "analysis-key";
process.env["ImageVersion"] = "2023.05.19.1";
process.env[RegistryProxyVars.PROXY_URLS] = JSON.stringify([
{ type: "maven_repository" },
] satisfies Array<Partial<Registry>>);
const getRequiredInput = sinon.stub(actionsUtil, "getRequiredInput");
getRequiredInput.withArgs("matrix").resolves("input/matrix");
@@ -117,6 +179,7 @@ test.serial("createStatusReportBase", async (t) => {
t.is(typeof statusReport.job_run_uuid, "string");
t.is(statusReport.languages, "java,swift");
t.is(statusReport.ref, process.env["GITHUB_REF"]!);
t.is(statusReport.registry_types, "maven_repository");
t.is(statusReport.runner_available_disk_space_bytes, 100);
t.is(statusReport.runner_image_version, process.env["ImageVersion"]);
t.is(statusReport.runner_os, process.env["RUNNER_OS"]!);
+54 -1
View File
@@ -19,12 +19,14 @@ import type { ComputedInput, InputName } from "./config/inputs";
import { parseRegistriesWithoutCredentials } from "./config/pack-registries";
import type { DependencyCacheRestoreStatusReport } from "./dependency-caching";
import { DocUrl } from "./doc-url";
import { EnvVar } from "./environment";
import { EnvVar, getEnv, ReadOnlyEnv, RegistryProxyVars } from "./environment";
import { getRef } from "./git-utils";
import * as json from "./json";
import type { Logger } from "./logging";
import type { OverlayBaseDatabaseDownloadStats } from "./overlay/caching";
import { getRepositoryNwo } from "./repository";
import type { ToolsSource } from "./setup-codeql";
import { registryBaseSchema } from "./start-proxy/types";
import {
ConfigurationError,
getRequiredEnvParam,
@@ -185,6 +187,12 @@ export interface StatusReportBase {
ml_powered_javascript_queries?: string;
/** Ref that the workflow was triggered on. */
ref: string;
/**
* A comma-separated list of private registry types which are configured for CodeQL.
* This only includes registry types we support (as determined by the `start-proxy` action),
* not all that are configured.
*/
registry_types?: string;
/** Action runner hardware architecture (context runner.arch). */
runner_arch?: string;
/** Available disk space on the runner, in bytes. */
@@ -288,6 +296,50 @@ export interface EventReport {
started_at: string;
}
/**
* Attempts to retrieve a list of private registry types from the `CODEQL_PROXY_URLS` environment
* variable and returns it as a comma-separated string if successful. Returns `undefined` otherwise.
*/
export function getRegistryTypesFromEnv(
logger: Logger,
env: ReadOnlyEnv = getEnv(),
): string | undefined {
// Try to get the value of the environment variable.
const value = env.getOptional(RegistryProxyVars.PROXY_URLS);
if (value === undefined) {
return undefined;
}
// Try to parse the JSON we expect to find in it and return the comma-separated list of
// (unique) registry types.
try {
const data = JSON.parse(value) as unknown;
// Check that the parsed JSON meets our expectations.
if (!json.isArray(data)) {
logger.debug(
`Expected '${RegistryProxyVars.PROXY_URLS}' to contain a JSON array, but got '${typeof data}'.`,
);
return undefined;
}
if (!json.validateArray(registryBaseSchema, data)) {
logger.debug(
`Expected '${RegistryProxyVars.PROXY_URLS}' to contain a JSON array of registry objects, but got something else.`,
);
return undefined;
}
const types = new Set(data.map((r) => r.type));
return Array.from(types).sort().join(",");
} catch (err) {
logger.debug(
`Failed to parse '${RegistryProxyVars.PROXY_URLS}': ${getErrorMessage(err)}.`,
);
return undefined;
}
}
/**
* Compose a StatusReport.
*
@@ -350,6 +402,7 @@ export async function createStatusReportBase(
job_name: jobName,
job_run_uuid: jobRunUUID,
ref,
registry_types: getRegistryTypesFromEnv(logger),
runner_os: runnerOs,
started_at: workflowStartedAt,
status,
+33
View File
@@ -0,0 +1,33 @@
import * as path from "path";
import * as stream from "stream";
import test from "ava";
import { getRunnerLogger } from "./logging";
import { extractTarZst } from "./tar";
import { setupTests } from "./testing-utils";
import { withTmpDir } from "./util";
setupTests(test);
test("extractTarZst rejects if the input stream errors", async (t) => {
await withTmpDir(async (tmpDir) => {
const archive = new stream.PassThrough();
const promise = extractTarZst(
archive,
path.join(tmpDir, "dest"),
{ type: "gnu", version: "1.34" },
getRunnerLogger(true),
);
archive.destroy(
Object.assign(new Error("socket hang up"), {
code: "ECONNRESET",
}),
);
await t.throwsAsync(promise, {
message: /Error while downloading and extracting tar/,
});
});
});
+9 -4
View File
@@ -194,10 +194,15 @@ export async function extractTarZst(
});
if (tar instanceof stream.Readable) {
tar.pipe(tarProcess.stdin).on("error", (err) => {
reject(
new Error(`Error while downloading and extracting tar: ${err}`),
);
// Use `pipeline` rather than `pipe` so that an error on either stream is reported here
// rather than being emitted as an unhandled `error` event, and so that `tar`'s standard
// input is closed if the download fails partway through.
stream.pipeline(tar, tarProcess.stdin, (err) => {
if (err) {
reject(
new Error(`Error while downloading and extracting tar: ${err}`),
);
}
});
}
+37
View File
@@ -38,6 +38,43 @@ test.serial(
},
);
test.serial(
"downloadAndExtract falls back to downloading before extracting if streaming fails",
async (t) => {
await withTmpDir(async (tmpDir) => {
sinon.stub(process, "platform").value("linux");
const archivePath = path.join(tmpDir, "codeql-bundle.tar.zst");
const destination = path.join(tmpDir, "codeql");
const downloadTool = sinon
.stub(toolcache, "downloadTool")
.resolves(archivePath);
const extract = sinon.stub(tar, "extract").resolves(destination);
const extractTarZst = sinon.stub(tar, "extractTarZst").resolves();
const request = nock("https://example.com")
.get("/codeql-bundle.tar.zst")
.replyWithError(
Object.assign(new Error("socket hang up"), { code: "ECONNRESET" }),
);
const statusReport = await downloadAndExtract(
"https://example.com/codeql-bundle.tar.zst",
"zstd",
destination,
undefined,
{},
{ type: "gnu", version: "1.34" },
getRunnerLogger(true),
);
t.assert(Number.isInteger(statusReport.downloadDurationMs));
t.true(request.isDone());
t.false(extractTarZst.called);
t.true(downloadTool.calledOnce);
t.true(extract.calledOnce);
});
},
);
test.serial(
"downloadAndExtract omits the download duration when streaming extraction",
async (t) => {
+24 -4
View File
@@ -19,6 +19,12 @@ import { cleanUpPath, getErrorMessage, getRequiredEnvParam } from "./util";
*/
const STREAMING_HIGH_WATERMARK_BYTES = 4 * 1024 * 1024; // 4 MiB
/**
* How long the streaming download of the CodeQL tools may stall for before we abort it. This
* applies both to establishing the connection and to gaps between chunks of the response body.
*/
const STREAMING_STALL_TIMEOUT_MS = 5 * 60 * 1000; // 5 minutes
/**
* The name of the tool cache directory for the CodeQL tools.
*/
@@ -137,8 +143,8 @@ async function downloadAndExtractZstdWithStreaming(
authorization ? { authorization } : {},
headers,
);
const response = await new Promise<IncomingMessage>((resolve) =>
https.get(
const response = await new Promise<IncomingMessage>((resolve, reject) => {
const request = https.get(
codeqlURL,
{
headers,
@@ -148,10 +154,24 @@ async function downloadAndExtractZstdWithStreaming(
agent,
} as unknown as RequestOptions,
(r) => resolve(r),
),
);
);
// Without this listener, connection failures such as `ECONNRESET` are emitted as unhandled
// `error` events, which terminate the process instead of letting us fall back to downloading
// the bundle before extracting it. This listener stays attached after the response arrives, so
// it also handles errors that occur while the response is being streamed.
request.on("error", reject);
request.setTimeout(STREAMING_STALL_TIMEOUT_MS, () => {
request.destroy(
new Error(
`No data received for ${formatDuration(STREAMING_STALL_TIMEOUT_MS)}.`,
),
);
});
});
if (response.statusCode !== 200) {
// Discard the response body so that the connection can be released.
response.resume();
throw new Error(
`Failed to download CodeQL bundle from ${codeqlURL}. HTTP status code: ${response.statusCode}.`,
);