qa-e2e-tests-reporting

Testomat.io Reporter Setup

Install and configure the Testomat.io reporter in an automation project, then push test results to the Testomat.io TMS.

Step 1: Detect Language & Framework

• Detect the language from the project manifest (package.json, pyproject.toml/requirements.txt, pom.xml/build.gradle, *.csproj, composer.json, Gemfile).
• Detect the framework from its config file: playwright.config.*, codecept.conf.js, wdio.conf.*, jest.config.*, .mocharc.*, pytest.ini/conftest.py, *.robot/robot.toml, testng.xml/JUnit deps in pom.xml.
• No native reporter for the stack (C#, PHP, Ruby, Go, etc.): use JUnit XML import (Step 4).

Step 2: Configure Credentials

• Check .env for a TESTOMATIO API key. If present, use it.
• If missing, add a placeholder line to .env:

TESTOMATIO=tstmt_xxxxx

• Ask the user to replace the placeholder with their "Project Reporting API key" themselves — never ask them to paste the key into chat.

Message to the user:

✅ Added `TESTOMATIO=tstmt_xxxxx` placeholder to .env

To activate reporting:
1. Go to Testomat.io → your project → Settings → Project
   (https://app.testomat.io/projects/<project-id>/settings/project)
2. Copy "Project Reporting API key"
3. Replace `tstmt_xxxxx` in .env with your actual key
4. Run tests: TESTOMATIO=tstmt_xxxxx npx <test-command>

Step 3: Install & Configure Reporter

Pick the section for the detected language and framework.

JavaScript / TypeScript

Use @testomatio/reporter:

npm install @testomatio/reporter --save-dev

Playwright — playwright.config.ts or playwright.config.js:

reporter: [
  ['@testomatio/reporter/playwright'],
  // other reporters...
],

CodeceptJS — codecept.conf.js or codecept.config.js:

plugins: {
  testomatio: {
    enabled: true,
    require: '@testomatio/reporter/codecept',
  }
}

WebdriverIO — wdio.conf.js:

import testomatio from '@testomatio/reporter/webdriver';

exports.config = {
  // ...
  reporters: [
    [testomatio, { apiKey: process.env.TESTOMATIO }],
  ],
};

Jest — jest.config.js or jest.config.ts:

module.exports = {
  reporters: ['default', ['@testomatio/reporter/jest', { apiKey: process.env.TESTOMATIO }]],
};

Mocha — run with CLI flags:

mocha --reporter @testomatio/reporter/mocha --reporter-options apiKey=tstmt_xxx

• Run commands and more config options: ./references/TESTOMATIO_REPORTERS_CONFIG.md
• Docs: Testomat.io NodeJS Frameworks

Python

Use pytestomatio for pytest, robot-framework-reporter for Robot Framework.

• Use pip3 if Python 2.x and 3.x coexist.

pytest:

pip install pytestomatio

# Run and report tests
pytest --testomatio report

Robot Framework:

pip install robot-framework-reporter

# Run and report tests
robot --listener Testomatio.Report path/to/tests

Docs: Testomat.io Python Reporting

Java (JUnit, TestNG, Cucumber, Karate)

Use the io.testomat Java reporters.

• Ask the user to set the API key before running tests, then verify it is set and starts with tstmt_:

export TESTOMATIO=<tstmt_api_key>

• Add the dependency for the framework in use:

<dependency>
  <groupId>io.testomat</groupId>
  <artifactId>java-reporter-junit</artifactId>
  <version>{LATEST_STABLE_VERSION}</version>
</dependency>

Artifact IDs: java-reporter-junit, java-reporter-testng, java-reporter-cucumber, java-reporter-karate.

• Run tests:

mvn clean test

• Scan the output for a run ID. If found, show the link:

View results: https://app.testomat.io/projects/{project-id}/runs/{run-id}

Docs: Testomat.io Java/JUnit Reporting

Step 4: Import Automated Tests to Testomat.io TMS

Import test source code into Testomat.io to see test structure and enable codebase-TMS synchronization.

Docs: Testomat.io Import Overview

JavaScript / TypeScript

Use the check-tests CLI:

npx check-tests@latest <framework> "<glob-pattern>" [options]

Framework	Example
Playwright	npx check-tests@latest Playwright "tests/**/*.spec.js"
CodeceptJS	npx check-tests@latest CodeceptJS "tests/**_test.js"
Cypress	npx check-tests@latest cypress "cypress/e2e/**/*.js"
Jest	npx check-tests@latest Jest "tests/**/*.test.js"
Mocha	npx check-tests@latest mocha "test/**/*_test.js"
WebdriverIO	npx check-tests@latest webdriverio "test/**/*.js"

Options:

• --typescript — for TypeScript projects: npx check-tests@latest Playwright "tests/**/*.spec.ts" --typescript
• --update-ids — sync and auto-assign test IDs in source code: npx check-tests@latest CodeceptJS "tests/**_test.js" --update-ids

Python

pytest — sync tests with Testomat.io:

pytest --testomatio sync

Robot Framework — import tests:

robot --listener Testomatio.Import path/to/tests

Java (JUnit, TestNG)

Use the testomatio.jar CLI. Re-run after every test execution.

• Run only from the project root:

cd <project-root>

export TESTOMATIO=tstmt_xxxxx && \
curl -L -O https://github.com/testomatio/java-check-tests/releases/latest/download/testomatio.jar && \
java -jar testomatio.jar sync

Commands:

• import — import test code to Testomat.io (dry-run without API key).
• clean-ids — import tests without system ids.
• sync — import tests and pull IDs into source code (alias: update-ids).

JUnit XML (C#, PHP, Ruby, Go, etc.)

For stacks without a native reporter, generate a JUnit XML report, then import it.

• NUnit (C#): dotnet test --logger:"trx;LogFileName=results.xml" — convert to JUnit XML if needed.
• PHPUnit (PHP): ./vendor/bin/phpunit --log-junit=results.xml
• RSpec (Ruby): rspec --format json --out results.json — then convert to JUnit XML.

Import to Testomat.io:

npx check-tests@latest junit "path/to/results.xml"

# Multiple XML files
npx check-tests@latest junit "results/*.xml"

Step 5: Verify Setup

Run the tests with the reporter enabled:

TESTOMATIO=tstmt_xxxxx npx <your-test-command>

Via Testomat.io MCP (preferred, if enabled)

• After the run, fetch the latest run via MCP to confirm data was received.
• Success indicators: a new run ID appears in the logs; run status shows passed/failed as expected; the run is retrievable by its ID.
• Setup is complete when the new run is fetchable via MCP.
• MCP tools: Testomat.io MCP Run Management

Debug Mode (alternative)

• Capture reporter output locally for only 1-2 tests to verify data is generated correctly before sending:

DEBUG=@testomatio/reporter:pipe:testomatio npx <your-test-command>

• Read the execution logs to confirm the reporter is enabled and configured.
• Docs: Testomat.io Debugging Logs

Step 6: Configure Artifacts (only on explicit request)

Store screenshots, videos, traces, and logs in S3-compatible storage alongside test results.

• Only perform this step if the user explicitly asks for artifacts configuration (or mentions saving screenshots, videos, traces, logs).
• Prerequisite: first test report executed successfully.
• Check .env for existing S3 credentials (S3_ACCESS_KEY_ID, S3_BUCKET, S3_REGION). If present, skip to verification.

Create S3 Bucket (if needed)

• Recommended: one bucket per project.
• Guide the user through their provider, based on official documentation:

Provider	Link
AWS S3	https://s3.console.aws.amazon.com/s3
DigitalOcean	https://cloud.digitalocean.com/spaces
Google Cloud Storage	https://console.cloud.google.com/storage
Cloudflare R2	https://dash.cloudflare.com/
Minio	Self-hosted or cloud instance

• The user obtains the Access Key ID and Secret Access Key from the provider.

Option A: Testomat.io UI (recommended)

• The agent cannot access Testomat.io project settings — the user configures this manually:
  1. Open https://app.testomat.io/projects/<project-id>/settings/artifacts
  2. Enable the "Share credentials" toggle.
  3. Enter Access Key ID, Secret Access Key, bucket name, region, and endpoint (endpoint only for non-AWS providers).

Option B: Environment Variables

Add S3 credentials to .env or the CI pipeline:

# Required for all providers
S3_ACCESS_KEY_ID=your_access_key
S3_SECRET_ACCESS_KEY=your_secret_key
S3_BUCKET=your_bucket_name
S3_REGION=us-west-1

# Optional: for non-AWS providers (DigitalOcean, Minio, Cloudflare R2, GCS)
S3_ENDPOINT=https://your-endpoint-url

# Optional: private access mode (recommended)
TESTOMATIO_PRIVATE_ARTIFACTS=1

Cloudflare R2 example:

S3_ACCESS_KEY_ID=xxx
S3_SECRET_ACCESS_KEY=xxx
S3_BUCKET=testomatio-artifacts
S3_REGION=auto
S3_ENDPOINT=https://<account-id>.r2.cloudflarestorage.com
S3_FORCE_PATH_STYLE=true

• Disable uploads: TESTOMATIO_DISABLE_ARTIFACTS=1
• Provider examples (AWS, DigitalOcean, Minio, GCS): ./references/TESTOMATIO_ARTIFACTS.md
• More options: Testomat.io Artifacts

Verify Artifacts

• Run tests that produce screenshots, videos, or traces: TESTOMATIO=tstmt_xxxxx npx <your-test-command>
• Ask the user to open https://app.testomat.io/projects/<project-id>/runs and click a test with artifacts (failed tests typically have screenshots).
• Success: artifacts appear in the test results ("Artifacts" section or attachment icon) and are clickable; Playwright traces open in the trace viewer.

References

Description	Link
Getting started	https://docs.testomat.io/getting-started/
Reporter configuration	https://docs.testomat.io/test-reporting/frameworks/
Framework config snippets	./references/TESTOMATIO_REPORTERS_CONFIG.md
Python test reporting	https://docs.testomat.io/test-reporting/python/
Python Robot Framework	https://github.com/testomatio/robot-framework-reporter/blob/master/README.md
Java/JUnit reporting	https://docs.testomat.io/test-reporting/junit/
HTML pipe	./references/TESTOMATIO_HTML_REPORT.md
Debug pipe	TESTOMATIO_DEBUG=1 <test-command>
Artifacts (S3)	./references/TESTOMATIO_ARTIFACTS.md