gecko_taskgraph.transforms.job package

Submodules

gecko_taskgraph.transforms.job.common module

Common support for various job types. These functions are all named after the worker implementation they operate on, and take the same three parameters, for consistency.

gecko_taskgraph.transforms.job.common.add_artifacts(config, job, taskdesc, path)

gecko_taskgraph.transforms.job.common.add_tooltool(config, job, taskdesc, internal=False)

Give the task access to tooltool.

Enables the tooltool cache. Adds releng proxy. Configures scopes.

By default, only public tooltool access will be granted. Access to internal tooltool can be enabled via internal=True.

This can only be used with run-task tasks, as the cache name is reserved for use with run-task.

gecko_taskgraph.transforms.job.common.docker_worker_add_artifacts(config, job, taskdesc): Adds an artifact directory to the task

gecko_taskgraph.transforms.job.common.generic_worker_add_artifacts(config, job, taskdesc): Adds an artifact directory to the task

gecko_taskgraph.transforms.job.common.get_cache_name(config, job)

gecko_taskgraph.transforms.job.common.get_expiration(config, policy='default')

gecko_taskgraph.transforms.job.common.setup_secrets(config, job, taskdesc): Set up access to secrets via taskcluster-proxy. The value of run[‘secrets’] should be a boolean or a list of secret names that can be accessed.

gecko_taskgraph.transforms.job.common.support_vcs_checkout(config, job, taskdesc, repo_configs)

Update a job/task with parameters to enable a VCS checkout.

This can only be used with run-task tasks, as the cache name is reserved for run-task tasks.

gecko_taskgraph.transforms.job.distro_package module

Support for running spidermonkey jobs via dedicated scripts

Bases: Schema

arch: str | None

dsc: SourceDefinition | None

name: str | None

packages: list[str] | None

patch: str | None

pre_build_command: str | None

resolver: Literal['apt-get', 'aptitude'] | None

tarball: SourceDefinition | None

workdir: str

class gecko_taskgraph.transforms.job.distro_package.DebianSchema(*, dsc: SourceDefinition | None = None, tarball: SourceDefinition | None = None, name: str | None = None, patch: str | None = None, pre_build_command: str | None = None, arch: str | None = None, packages: list[str] | None = None, resolver: Literal['apt-get', 'aptitude'] | None = None, workdir: str, using: Literal['debian-package'], dist: str)

Bases: CommonSchema

dist: str

using: Literal['debian-package']

class gecko_taskgraph.transforms.job.distro_package.SourceDefinition(*, url: str, sha256: str)

Bases: Schema

sha256: str

url: str

class gecko_taskgraph.transforms.job.distro_package.UbuntuSchema(*, dsc: SourceDefinition | None = None, tarball: SourceDefinition | None = None, name: str | None = None, patch: str | None = None, pre_build_command: str | None = None, arch: str | None = None, packages: list[str] | None = None, resolver: Literal['apt-get', 'aptitude'] | None = None, workdir: str, using: Literal['ubuntu-package'], dist: str)

Bases: CommonSchema

dist: str

using: Literal['ubuntu-package']

gecko_taskgraph.transforms.job.distro_package.common_package(config, job, taskdesc, distro, version)

gecko_taskgraph.transforms.job.distro_package.docker_worker_debian_package(config, job, taskdesc)

gecko_taskgraph.transforms.job.distro_package.docker_worker_ubuntu_package(config, job, taskdesc)

gecko_taskgraph.transforms.job.hazard module

Support for running hazard jobs via dedicated scripts

class gecko_taskgraph.transforms.job.hazard.HazRunSchema(*, using: Literal['hazard'], command: str, mozconfig: str | None = None, secrets: bool | list[str] | None = None, workdir: str | None = None)

Bases: Schema

command: str

mozconfig: str | None

secrets: bool | list[str] | None

using: Literal['hazard']

workdir: str | None

gecko_taskgraph.transforms.job.hazard.docker_worker_hazard(config, job, taskdesc)

gecko_taskgraph.transforms.job.mach module

Support for running mach tasks (via run-task)

Bases: Schema

comm_checkout: bool

mach: str | TaskRefTypeSchema

prepend_env: dict[str, str] | None

sparse_profile: str | None

use_caches: bool | list[str] | None

using: Literal['mach']

workdir: str | None

gecko_taskgraph.transforms.job.mach.configure_mach(config, job, taskdesc)

gecko_taskgraph.transforms.job.mozharness module

Support for running jobs via mozharness. Ideally, most stuff gets run this way, and certainly anything using mozharness should use this approach.

class gecko_taskgraph.transforms.job.mozharness.MozharnessRunSchema(*, using: Literal['mozharness'], script: str, config_paths: list[str] | None = None, config: list[str], actions: list[str] | None = None, options: list[str] | None = None, custom_build_variant_cfg: str | None = None, extra_config: dict | None = None, tooltool_downloads: bool | Literal['public', 'internal'], secrets: bool | list[str], taskcluster_proxy: bool, keep_artifacts: bool, job_script: str | None = None, requires_signed_builds: bool, use_caches: bool | list[str] | None = None, use_simple_package: bool, use_magic_mh_args: bool, comm_checkout: bool, workdir: str | None = None, run_as_root: bool | None = None)

Bases: Schema

actions: list[str] | None

comm_checkout: bool

config: list[str]

config_paths: list[str] | None

custom_build_variant_cfg: str | None

extra_config: dict | None

job_script: str | None

keep_artifacts: bool

options: list[str] | None

requires_signed_builds: bool

run_as_root: bool | None

script: str

secrets: bool | list[str]

taskcluster_proxy: bool

tooltool_downloads: bool | Literal['public', 'internal']

use_caches: bool | list[str] | None

use_magic_mh_args: bool

use_simple_package: bool

using: Literal['mozharness']

workdir: str | None

gecko_taskgraph.transforms.job.mozharness.mozharness_on_docker_worker_setup(config, job, taskdesc)

gecko_taskgraph.transforms.job.mozharness.mozharness_on_generic_worker(config, job, taskdesc)

gecko_taskgraph.transforms.job.mozharness_test module

gecko_taskgraph.transforms.job.mozharness_test.get_variant(test_platform)

gecko_taskgraph.transforms.job.mozharness_test.installer_url(taskdesc)

gecko_taskgraph.transforms.job.mozharness_test.mozharness_test_on_docker(config, job, taskdesc)

gecko_taskgraph.transforms.job.mozharness_test.mozharness_test_on_generic_worker(config, job, taskdesc)

gecko_taskgraph.transforms.job.mozharness_test.test_packages_url(taskdesc): Account for different platforms that name their test packages differently

gecko_taskgraph.transforms.job.python_test module

Support for running mach python-test tasks (via run-task)

class gecko_taskgraph.transforms.job.python_test.PythonTestSchema(*, using: Literal['python-test'], subsuite: str, workdir: str | None = None, use_caches: bool | list[str] | None = None, prepend_env: dict[str, str] | None = None)

Bases: Schema

prepend_env: dict[str, str] | None

subsuite: str

use_caches: bool | list[str] | None

using: Literal['python-test']

workdir: str | None

gecko_taskgraph.transforms.job.python_test.configure_python_test(config, job, taskdesc)

gecko_taskgraph.transforms.job.run_task module

Support for running jobs that are invoked via the run-task script.

Bases: Schema

checkout: bool

comm_checkout: bool

command: list[str | TaskRefTypeSchema] | str | TaskRefTypeSchema

cwd: str | None

run_as_root: bool | None

shallow_clone: bool | None

sparse_profile: str | None

sparse_profile_prefix: str | None

tooltool_downloads: bool | Literal['public', 'internal']

use_caches: bool | list[str] | None

using: Literal['run-task']

workdir: str | None

gecko_taskgraph.transforms.job.run_task.common_setup(config, job, taskdesc, command)

gecko_taskgraph.transforms.job.run_task.docker_worker_run_task(config, job, taskdesc)

gecko_taskgraph.transforms.job.run_task.generic_worker_run_task(config, job, taskdesc)

gecko_taskgraph.transforms.job.run_task.script_url(config, script)

gecko_taskgraph.transforms.job.spidermonkey module

Support for running spidermonkey jobs via dedicated scripts

class gecko_taskgraph.transforms.job.spidermonkey.SmRunSchema(*, using: Literal['spidermonkey', 'spidermonkey-package'], spidermonkey_variant: str, spidermonkey_platform: str | None = None, workdir: str | None = None, tooltool_downloads: bool | Literal['public', 'internal'])

Bases: Schema

spidermonkey_platform: str | None

spidermonkey_variant: str

tooltool_downloads: bool | Literal['public', 'internal']

using: Literal['spidermonkey', 'spidermonkey-package']

workdir: str | None

gecko_taskgraph.transforms.job.spidermonkey.docker_worker_spidermonkey(config, job, taskdesc)

gecko_taskgraph.transforms.job.spidermonkey.generic_worker_spidermonkey(config, job, taskdesc)

gecko_taskgraph.transforms.job.toolchain module

Support for running toolchain-building jobs via dedicated scripts

class gecko_taskgraph.transforms.job.toolchain.ToolchainRunSchema(*, using: ~typing.Literal['toolchain-script'], script: str, arguments: list[str] | None = None, tooltool_downloads: bool | ~typing.Literal['public', 'internal'], sparse_profile: str | None, sparse_profile_prefix: str | None = None, resources: list[str] | None = None, toolchain_artifact: str, toolchain_alias: ~typing.Annotated[~typing.Any, <taskgraph.util.schema.OptionallyKeyedBy object at 0x7fe21c19ff10>] | None = None, toolchain_env: dict[str, object] | None = None, toolchain_extract: bool | None = None, workdir: str | None = None)

Bases: Schema

arguments: list[str] | None

resources: list[str] | None

script: str

sparse_profile: str | None

sparse_profile_prefix: str | None

toolchain_alias: OptionallyKeyedBy object at 0x7fe21c19ff10>] | None

toolchain_artifact: str

toolchain_env: dict[str, object] | None

toolchain_extract: bool | None

tooltool_downloads: bool | Literal['public', 'internal']

using: Literal['toolchain-script']

workdir: str | None

gecko_taskgraph.transforms.job.toolchain.common_toolchain(config, job, taskdesc, is_docker)

gecko_taskgraph.transforms.job.toolchain.docker_worker_toolchain(config, job, taskdesc)

gecko_taskgraph.transforms.job.toolchain.generic_worker_toolchain(config, job, taskdesc)

gecko_taskgraph.transforms.job.toolchain.get_digest_data(config, run, taskdesc)

Module contents

Convert a job description into a task description.

Jobs descriptions are similar to task descriptions, but they specify how to run the job at a higher level, using a “run” field that can be interpreted by run-using handlers in taskcluster/gecko_taskgraph/transforms/job.

class gecko_taskgraph.transforms.job.FetchArtifactSchema(*, artifact: str, dest: str | None = None, extract: bool | None = None, verify_hash: bool | None = None)

Bases: Schema

artifact: str

dest: str | None

extract: bool | None

verify_hash: bool | None

class gecko_taskgraph.transforms.job.JobDescriptionSchema(*, name: str | None = None, label: str | None = None, description: str, attributes: dict[str, object] | None = None, task_from: str | None = None, dependencies: dict[str, object] | None = None, if_dependencies: list[str] | None = None, soft_dependencies: list[str] | None = None, requires: ~typing.Literal['all-completed', 'all-resolved'] | None = None, expires_after: str | None = None, expiration_policy: str | None = None, routes: list[str] | None = None, scopes: list[str] | None = None, tags: dict[str, str] | None = None, extra: dict[str, object] | None = None, treeherder: ~gecko_taskgraph.transforms.task.TreeherderSchema | None = None, index: ~gecko_taskgraph.transforms.task.IndexSchema | None = None, run_on_repo_type: list[~typing.Literal['git', 'hg']] | None = None, run_on_projects: ~typing.Annotated[~typing.Any, <taskgraph.util.schema.OptionallyKeyedBy object at 0x7fe231eac910>] | None = None, run_on_git_branches: list[str] | None = None, shipping_phase: ~typing.Literal['build', 'promote', 'push', 'ship'] | None = None, shipping_product: str | None = None, always_target: bool | None = None, optimization: ~gecko_taskgraph.optimize.schema.OptimizationSchema | None = None, use_sccache: bool | None = None, use_python: ~typing.Literal['system', 'default'] | str | None = None, use_uv: bool | None = None, priority: str | None = None, when: ~gecko_taskgraph.transforms.job.WhenSchema | None = None, fetches: dict[str, list[str | ~gecko_taskgraph.transforms.job.FetchArtifactSchema]] | None = None, run: ~gecko_taskgraph.transforms.job.JobRunSchema, worker_type: str | None, worker: dict | None = None)

Bases: Schema

always_target: bool | None

attributes: dict[str, object] | None

dependencies: dict[str, object] | None

description: str

expiration_policy: str | None

expires_after: str | None

extra: dict[str, object] | None

fetches: dict[str, list[str | FetchArtifactSchema]] | None

if_dependencies: list[str] | None

index: IndexSchema | None

label: str | None

name: str | None

optimization: OptimizationSchema | None

priority: str | None

requires: Literal['all-completed', 'all-resolved'] | None

routes: list[str] | None

run: JobRunSchema

run_on_git_branches: list[str] | None

run_on_projects: OptionallyKeyedBy object at 0x7fe231eac910>] | None

run_on_repo_type: list[Literal['git', 'hg']] | None

scopes: list[str] | None

shipping_phase: Literal['build', 'promote', 'push', 'ship'] | None

shipping_product: str | None

soft_dependencies: list[str] | None

tags: dict[str, str] | None

task_from: str | None

treeherder: TreeherderSchema | None

use_python: Literal['system', 'default'] | str | None

use_sccache: bool | None

use_uv: bool | None

when: WhenSchema | None

worker: dict | None

worker_type: str | None

class gecko_taskgraph.transforms.job.JobRunSchema(*, using: str, workdir: str | None = None)

Bases: Schema

using: str

workdir: str | None

class gecko_taskgraph.transforms.job.WhenSchema(*, files_changed: list[str] | None = None)

Bases: Schema

files_changed: list[str] | None

gecko_taskgraph.transforms.job.add_perfherder_fetch_content_artifact(config, jobs)

gecko_taskgraph.transforms.job.always_optimized(config, job, taskdesc)

gecko_taskgraph.transforms.job.configure_taskdesc_for_run(config, job, taskdesc, worker_implementation)

Run the appropriate function for this job against the given task description.

This will raise an appropriate error if no function exists, or if the job’s run is not valid according to the schema.

gecko_taskgraph.transforms.job.get_attribute(dict, key, attributes, attribute_name): Get attribute_name from the given attributes dict, and if there is a corresponding value, set key in dict to that value.

gecko_taskgraph.transforms.job.get_platform(job)

gecko_taskgraph.transforms.job.make_task_description(config, jobs): Given a build description, create a task description

gecko_taskgraph.transforms.job.run_job_using(worker_implementation, run_using, schema=None, defaults={})

Register the decorated function as able to set up a task description for jobs with the given worker implementation and run.using property. If schema is given, the job’s run field will be verified to match it.

The decorated function should have the signature using_foo(config, job, taskdesc) and should modify the task description in-place. The skeleton of the task description is already set up, but without a payload.

gecko_taskgraph.transforms.job.set_implementation(config, jobs)

gecko_taskgraph.transforms.job.set_label(config, jobs)

gecko_taskgraph.transforms.job.use_fetches(config, jobs)

gecko_taskgraph.transforms.job.use_system_python(config, jobs)

gecko_taskgraph.transforms.job.use_uv(config, jobs)