Creates and runs a matrix of tests according to the given specifications. Unsupported environments will be returned in the state UNSUPPORTED. A test matrix is limited to use at most 2000 devices in parallel. May return any of the following canonical error codes: - PERMISSION_DENIED - if the user is not authorized to write to project - INVALID_ARGUMENT - if the request is malformed or if the matrix tries to use too many simultaneous devices.

Scopes

You will need authorization for the https://www.googleapis.com/auth/cloud-platform scope to make a valid call.

If unset, the scope for this method defaults to https://www.googleapis.com/auth/cloud-platform. You can set the scope for this method like this: testing1 --scope <scope> projects test-matrices-create ...

Required Scalar Argument

• <project-id> (string)
  • The GCE project under which this job will run.

Required Request Value

The request value is a data-structure with various fields. Each field may be a simple scalar or another data-structure. In the latter case it is advised to set the field-cursor to the data-structure's field to specify values more concisely.

For example, a structure like this:

TestMatrix:
  client-info:
    name: string
  environment-matrix:
    android-matrix:
      android-model-ids: [string]
      android-version-ids: [string]
      locales: [string]
      orientations: [string]
  fail-fast: boolean
  flaky-test-attempts: integer
  invalid-matrix-details: string
  outcome-summary: string
  project-id: string
  result-storage:
    google-cloud-storage:
      gcs-path: string
    results-url: string
    tool-results-execution:
      execution-id: string
      history-id: string
      project-id: string
    tool-results-history:
      history-id: string
      project-id: string
  state: string
  test-matrix-id: string
  test-specification:
    android-instrumentation-test:
      app-apk:
        gcs-path: string
      app-bundle:
        bundle-location:
          gcs-path: string
      app-package-id: string
      orchestrator-option: string
      sharding-option:
        uniform-sharding:
          num-shards: integer
      test-apk:
        gcs-path: string
      test-package-id: string
      test-runner-class: string
      test-targets: [string]
    android-robo-test:
      app-apk:
        gcs-path: string
      app-bundle:
        bundle-location:
          gcs-path: string
      app-initial-activity: string
      app-package-id: string
      max-depth: integer
      max-steps: integer
      robo-script:
        gcs-path: string
    android-test-loop:
      app-apk:
        gcs-path: string
      app-bundle:
        bundle-location:
          gcs-path: string
      app-package-id: string
      scenario-labels: [string]
      scenarios: [integer]
    disable-performance-metrics: boolean
    disable-video-recording: boolean
    ios-test-loop:
      app-bundle-id: string
      app-ipa:
        gcs-path: string
      scenarios: [integer]
    ios-test-setup:
      network-profile: string
    ios-xc-test:
      app-bundle-id: string
      test-special-entitlements: boolean
      tests-zip:
        gcs-path: string
      xcode-version: string
      xctestrun:
        gcs-path: string
    test-setup:
      directories-to-pull: [string]
      dont-autogrant-permissions: boolean
      network-profile: string
      systrace:
        duration-seconds: integer
    test-timeout: string
  timestamp: string

can be set completely with the following arguments which are assumed to be executed in the given order. Note how the cursor position is adjusted to the respective structures, allowing simple field names to be used most of the time.

• -r .client-info name=magna

  • Required. Client name, such as gcloud.

• ..environment-matrix.android-matrix android-model-ids=no

  • Required. The ids of the set of Android device to be used. Use the TestEnvironmentDiscoveryService to get supported options.
  • Each invocation of this argument appends the given value to the array.
• android-version-ids=ipsum
  • Required. The ids of the set of Android OS version to be used. Use the TestEnvironmentDiscoveryService to get supported options.
  • Each invocation of this argument appends the given value to the array.
• locales=voluptua.
  • Required. The set of locales the test device will enable for testing. Use the TestEnvironmentDiscoveryService to get supported options.
  • Each invocation of this argument appends the given value to the array.

• orientations=at

  • Required. The set of orientations to test with. Use the TestEnvironmentDiscoveryService to get supported options.
  • Each invocation of this argument appends the given value to the array.

• ... fail-fast=false

  • If true, only a single attempt at most will be made to run each execution/shard in the matrix. Flaky test attempts are not affected. Normally, 2 or more attempts are made if a potential infrastructure issue is detected. This feature is for latency sensitive workloads. The incidence of execution failures may be significantly greater for fail-fast matrices and support is more limited because of that expectation.
• flaky-test-attempts=99
  • The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
• invalid-matrix-details=takimata
  • Output only. Describes why the matrix is considered invalid. Only useful for matrices in the INVALID state.
• outcome-summary=amet.
  • Output Only. The overall outcome of the test. Only set when the test matrix state is FINISHED.
• project-id=duo
  • The cloud project that owns the test matrix.

• result-storage.google-cloud-storage gcs-path=ipsum

  • Required. The path to a directory in GCS that will eventually contain the results for this test. The requesting user must have write access on the bucket in the supplied path.

• .. results-url=gubergren

  • Output only. URL to the results in the Firebase Web Console.
• tool-results-execution execution-id=lorem
  • Output only. A tool results execution ID.
• history-id=gubergren
  • Output only. A tool results history ID.

• project-id=eos

  • Output only. The cloud project that owns the tool results execution.

• ..tool-results-history history-id=dolor

  • Required. A tool results history ID.

• project-id=ea

  • Required. The cloud project that owns the tool results history.

• ... state=ipsum

  • Output only. Indicates the current progress of the test matrix.
• test-matrix-id=invidunt
  • Output only. Unique id set by the service.

• test-specification.android-instrumentation-test.app-apk gcs-path=amet

  • A path to a file in Google Cloud Storage. Example: gs://build-app-1414623860166/app%40debug-unaligned.apk These paths are expected to be url encoded (percent encoding)

• ..app-bundle.bundle-location gcs-path=duo

  • A path to a file in Google Cloud Storage. Example: gs://build-app-1414623860166/app%40debug-unaligned.apk These paths are expected to be url encoded (percent encoding)

• ... app-package-id=ipsum

  • The java package for the application under test. The default value is determined by examining the application's manifest.
• orchestrator-option=sed
  • The option of whether running each test within its own invocation of instrumentation with Android Test Orchestrator or not. Orchestrator is only compatible with AndroidJUnitRunner version 1.0 or higher! Orchestrator offers the following benefits: - No shared state - Crashes are isolated - Logs are scoped per test See for more information about Android Test Orchestrator. If not set, the test will be run without the orchestrator.

• sharding-option.uniform-sharding num-shards=64

  • Required. Total number of shards. When any physical devices are selected, the number must be >= 1 and <= 50. When no physical devices are selected, the number must be >= 1 and <= 500.

• ...test-apk gcs-path=gubergren

  • A path to a file in Google Cloud Storage. Example: gs://build-app-1414623860166/app%40debug-unaligned.apk These paths are expected to be url encoded (percent encoding)

• .. test-package-id=rebum.

  • The java package for the test to be executed. The default value is determined by examining the application's manifest.
• test-runner-class=est
  • The InstrumentationTestRunner class. The default value is determined by examining the application's manifest.

• test-targets=ipsum

  • Each target must be fully qualified with the package name or class name, in one of these formats: - "package package_name" - "class package_name.class_name" - "class package_name.class_name#method_name" If empty, all targets in the module will be run.
  • Each invocation of this argument appends the given value to the array.

• ..android-robo-test.app-apk gcs-path=ipsum

  • A path to a file in Google Cloud Storage. Example: gs://build-app-1414623860166/app%40debug-unaligned.apk These paths are expected to be url encoded (percent encoding)

• ..app-bundle.bundle-location gcs-path=est

  • A path to a file in Google Cloud Storage. Example: gs://build-app-1414623860166/app%40debug-unaligned.apk These paths are expected to be url encoded (percent encoding)

• ... app-initial-activity=gubergren

  • The initial activity that should be used to start the app.
• app-package-id=ea
  • The java package for the application under test. The default value is determined by examining the application's manifest.
• max-depth=2
  • The max depth of the traversal stack Robo can explore. Needs to be at least 2 to make Robo explore the app beyond the first activity. Default is 50.
• max-steps=45
  • The max number of steps Robo can execute. Default is no limit.

• robo-script gcs-path=eos

  • A path to a file in Google Cloud Storage. Example: gs://build-app-1414623860166/app%40debug-unaligned.apk These paths are expected to be url encoded (percent encoding)

• ...android-test-loop.app-apk gcs-path=labore

  • A path to a file in Google Cloud Storage. Example: gs://build-app-1414623860166/app%40debug-unaligned.apk These paths are expected to be url encoded (percent encoding)

• ..app-bundle.bundle-location gcs-path=sed

  • A path to a file in Google Cloud Storage. Example: gs://build-app-1414623860166/app%40debug-unaligned.apk These paths are expected to be url encoded (percent encoding)

• ... app-package-id=duo

  • The java package for the application under test. The default is determined by examining the application's manifest.
• scenario-labels=sed
  • The list of scenario labels that should be run during the test. The scenario labels should map to labels defined in the application's manifest. For example, player_experience and com.google.test.loops.player_experience add all of the loops labeled in the manifest with the com.google.test.loops.player_experience name to the execution. Scenarios can also be specified in the scenarios field.
  • Each invocation of this argument appends the given value to the array.

• scenarios=40

  • The list of scenarios that should be run during the test. The default is all test loops, derived from the application's manifest.
  • Each invocation of this argument appends the given value to the array.

• .. disable-performance-metrics=true

  • Disables performance metrics recording. May reduce test latency.
• disable-video-recording=true
  • Disables video recording. May reduce test latency.
• ios-test-loop app-bundle-id=vero
  • Output only. The bundle id for the application under test.

• app-ipa gcs-path=erat

  • A path to a file in Google Cloud Storage. Example: gs://build-app-1414623860166/app%40debug-unaligned.apk These paths are expected to be url encoded (percent encoding)

• .. scenarios=8

  • The list of scenarios that should be run during the test. Defaults to the single scenario 0 if unspecified.
  • Each invocation of this argument appends the given value to the array.

• ..ios-test-setup network-profile=duo

  • The network traffic profile used for running the test. Available network profiles can be queried by using the NETWORK_CONFIGURATION environment type when calling TestEnvironmentDiscoveryService.GetTestEnvironmentCatalog.

• ..ios-xc-test app-bundle-id=dolore

  • Output only. The bundle id for the application under test.
• test-special-entitlements=false
  • The option to test special app entitlements. Setting this would re-sign the app having special entitlements with an explicit application-identifier. Currently supports testing aps-environment entitlement.

• tests-zip gcs-path=diam

  • A path to a file in Google Cloud Storage. Example: gs://build-app-1414623860166/app%40debug-unaligned.apk These paths are expected to be url encoded (percent encoding)

• .. xcode-version=dolor

  • The Xcode version that should be used for the test. Use the TestEnvironmentDiscoveryService to get supported options. Defaults to the latest Xcode version Firebase Test Lab supports.

• xctestrun gcs-path=et

  • A path to a file in Google Cloud Storage. Example: gs://build-app-1414623860166/app%40debug-unaligned.apk These paths are expected to be url encoded (percent encoding)

• ...test-setup directories-to-pull=et

  • List of directories on the device to upload to GCS at the end of the test; they must be absolute paths under /sdcard, /storage or /data/local/tmp. Path names are restricted to characters a-z A-Z 0-9 _ - . + and / Note: The paths /sdcard and /data will be made available and treated as implicit path substitutions. E.g. if /sdcard on a particular device does not map to external storage, the system will replace it with the external storage path prefix for that device.
  • Each invocation of this argument appends the given value to the array.
• dont-autogrant-permissions=false
  • Whether to prevent all runtime permissions to be granted at app install
• network-profile=stet
  • The network traffic profile used for running the test. Available network profiles can be queried by using the NETWORK_CONFIGURATION environment type when calling TestEnvironmentDiscoveryService.GetTestEnvironmentCatalog.

• systrace duration-seconds=2

  • Systrace duration in seconds. Should be between 1 and 30 seconds. 0 disables systrace.

• ... test-timeout=duo

  • Max time a test execution is allowed to run before it is automatically cancelled. The default value is 5 min.

• .. timestamp=vero

  • Output only. The time this test matrix was initially created.

About Cursors

The cursor position is key to comfortably set complex nested structures. The following rules apply:

• The cursor position is always set relative to the current one, unless the field name starts with the . character. Fields can be nested such as in -r f.s.o .
• The cursor position is set relative to the top-level structure if it starts with ., e.g. -r .s.s
• You can also set nested fields without setting the cursor explicitly. For example, to set a value relative to the current cursor position, you would specify -r struct.sub_struct=bar.
• You can move the cursor one level up by using ... Each additional . moves it up one additional level. E.g. ... would go three levels up.

Optional Output Flags

The method's return value a JSON encoded structure, which will be written to standard output by default.

• -o out
  • out specifies the destination to which to write the server's result to. It will be a JSON-encoded structure. The destination may be - to indicate standard output, or a filepath that is to contain the received bytes. If unset, it defaults to standard output.

Optional Method Properties

You may set the following properties to further configure the call. Please note that -p is followed by one or more key-value-pairs, and is called like this -p k1=v1 k2=v2 even though the listing below repeats the -p for completeness.

• -p request-id=string
  • A string id used to detect duplicated requests. Ids are automatically scoped to a project, so users should ensure the ID is unique per-project. A UUID is recommended. Optional, but strongly recommended.

Optional General Properties

The following properties can configure any call, and are not specific to this method.

• -p $-xgafv=string

  • V1 error format.

• -p access-token=string

  • OAuth access token.

• -p alt=string

  • Data format for response.

• -p callback=string

  • JSONP

• -p fields=string

  • Selector specifying which fields to include in a partial response.

• -p key=string

  • API key. Your API key identifies your project and provides you with API access, quota, and reports. Required unless you provide an OAuth 2.0 token.

• -p oauth-token=string

  • OAuth 2.0 token for the current user.

• -p pretty-print=boolean

  • Returns response with indentations and line breaks.

• -p quota-user=string

  • Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters.

• -p upload-type=string

  • Legacy upload protocol for media (e.g. "media", "multipart").

• -p upload-protocol=string

  • Upload protocol for media (e.g. "raw", "multipart").