telemetry-tests-client integration tests

The telemetry-tests-client suite verifies that Firefox collects telemetry, aggregates it, and submits pings to an HTTP server. The suite uses Marionette Python tests to mimic user behavior while making as few assumptions as possible about Firefox’s internal implementation. wptserve provides the HTTP ping server.

The integration test suite for Firefox Client Telemetry runs on CI with Treeherder symbol tt(c) at each platform’s default tier; see the task definition. The client tests are listed in the client test manifest and stored in the client test directory.

The client manifest is the authoritative list of client tests. The suite covers areas such as:

  • Legacy Telemetry ping contents and behavior across browser sessions and subsessions.

  • Firefox on Glean pings, and opt-out and deletion-request behavior for both Legacy Telemetry and FOG.

  • Ping submission and payload correctness.

Running the tests locally

Run the commands in this section from the source root using mach.

./mach telemetry-tests-client

The default command also runs the harness self-tests. To run only the client tests while preserving manifest filtering, use:

./mach telemetry-tests-client --tag client

The client manifest sets tags = "client" in its [DEFAULT] section, while the harness self-tests are tagged unit; use --tag unit to run them.

You can also pass a test manifest, test file, or directory to the command. Passing a test file or a directory bypasses the manifest. This means that skip-if conditions are ignored and tests that are not expected to pass on your platform might run. The --tag option also has no effect in this mode.

See Marionette testing for runner options such as --gecko-log, --headless, and --binary.

Adding a test

Name new test files test_*.py and subclass TelemetryTestCase or, for tests that need a Glean ping server, FOGTestCase. Add the test to the client test manifest so that the default command and CI run it. See Marionette Python tests for test structure and assertions.

A new directory of tests additionally needs its own manifest.toml. Set its tags value in the [DEFAULT] section and include it from the suite’s top-level manifest. It also needs an entry in default_tests in testing/mozharness/scripts/telemetry/telemetry_client.py, which CI reads instead of the top-level manifest.

Running the tests on try

You can run the tests across all platforms on the try server using mach:

./mach try fuzzy -q "'telemetry-tests-client"

Disabling an individual failing test

To disable a test only in affected configurations, add a skip-if condition with a bug number to its entry in the client test manifest. See manifest conditional expressions for the syntax.

To skip an individual test method in code, call self.skipTest(“reason”).

Who to contact for help

Ask in the #telemetry Matrix room. See the Telemetry module for the current owners and peers.

Bugzilla

Bugs can be filed under the Toolkit product for the Telemetry component.