railflow-cypress-junit-reporter

Railflow Cypress JUnit reporter is a custom reporter for use with Mocha and Cypress testing frameworks.
The reporter helps you to integrate your Cypress/Mocha test with TestRail easily by producing enriched JUnit-style XML test report files which can be uploaded into TestRail by using powerful Railflow NPM CLI tool.
The reporter is built on the top of mocha-junit-reporter.

Installation

$ npm install railflow-cypress-junit-reporter --save-dev

or as a global module

$ npm install -g railflow-cypress-junit-reporter

Usage

Run cypress with railflow-cypress-junit-reporter:

Specify railflow cypress junit reporter in your configuration file (cypress.json by default) or via the command line.

Configuration file

{
"reporter": "railflow-cypress-junit-reporter"
}

Command line

cypress run --reporter railflow-cypress-junit-reporter

Adding TestRail-related data to the report

Railflow elements can be configured in test suite level and test level in cypress tests.

Test suite level

describe('My First Test',
    {
        env: {
            railflow: {
                title: 'Sample Test Suite',
                case_type: 'Automated',
                case_priority: 'High',
                case_fields: ['field1=value1', 'field2=value2'],
                result_fields: ['field1=value1', 'field2=value2'],
                jira_ids: ['jid1', 'jid2', 'jid3'],
                smart_failure_assignment: ['[e-mail]', '[e-mail]']
            }
        },
    }, () => {
        it('Does not do much!', () => {
            expect(true).to.equal(true);
        })
    })

Test level

describe('My First Test', () => {
    it('Does not do much!',
        {
            env: {
                railflow: {
                    title: 'Sample Test Case',
                    case_type: 'Automated',
                    case_priority: 'High',
                    case_fields: ['field1=value1', 'field2=value2'],
                    result_fields: ['field1=value1', 'field2=value2'],
                    jira_ids: ['jid1', 'jid2', 'jid3'],
                    smart_failure_assignment: ['[e-mail]', '[e-mail]'],
                    testrail_ids: [1, 2, 3]

                }
            }
        }, () => {
            expect(true).to.equal(true)
        })
})

Railflow configuration params description

Config Name	Description
title	Name of the test suite or test case
case_type	One of the test case types defined in TestRail, e.g.: Automated, Compatibility
case_priority	One of the case priority defined in TestRail, e.g.: Critical, High
case_fields	Custom case fields defined in TestRail. Should input as an array of strings, e.g.: ['field1=value1','field2=value2']
result_fields	Custom result fields defined in TestRail. Should input as an array of strings, e.g.: ['field1=value1','field2=value2']
jira_ids	Jira IDs.These values will be populated as a case field 'refs'. Should input as an array of strings, e.g.: ['jid1','jid2']
testrail_ids	IDs of test cases in TestRail. Should input as an array of integers, e.g.: [1,2,3]
smart_failure_assignment	Array of TestRail users to automatically assign failed test cases. Should input as a string array, e.g.: ['[e-mail]','[e-mail]']

Enable automatic screenshot capturing for failed tests

To add screenshots into report XML file for failed tests automatically, the following steps need to be done:

1. Add global hook into index.js file.
2. Enable attachments in report configuration

Customize index.js file

To add screenshots for failed tests, add the code fragment below into index.js file located in "<Cypress_project_root>/cypress/support"

Cypress.on('test:after:run', function (test, runnable) {
    if (test.state === "failed") {
        const screenshot = `${Cypress.config('screenshotsFolder')}/${Cypress.spec.name}/${runnable.parent.title} -- ${test.title} (failed).png`;
        console.log(screenshot)
        if (!test.attachments) {
            test.attachments = [];
        }
        test.attachments.push(screenshot);
    }
});

If test has any screenshots attached, they will be output into XML report in the following way:

<system-out>[[ATTACHMENT|path/to/file]]</system-out>

Disable attachments in configurations

By default, adding attachments is enabled in the test report. To disable adding attachments to the report, the attachments property should set to false in the configuration file or via command line.

Configuration file

{
"reporter": "railflow-cypress-junit-reporter",
 "reporterOptions": {
    "attachments": false
  }
}

Command line

cypress run --reporter railflow-cypress-junit-reporter \
  --reporter-options "attachments=false"

Additional configuration

Disable adding skipped/pending tests in the report

By default, skipped tests are included in the test report. To disable adding skipped tests into the report includePending property should set to false.

Configuration file

{
"reporter": "railflow-cypress-junit-reporter",
 "reporterOptions": {
    "includePending": false
  }
}

Command line

cypress run --reporter railflow-cypress-junit-reporter \
  --reporter-options "includePending=false"

Example

1. Install 'railflow-cypress-junit-reporter' to the cypress project

$ npm install railflow-cypress-junit-reporter --save-dev

2. Configure 'railflow-cypress-junit-reporter' as the reporter and do other configurations as necessary. Configurations can be done in the cypress.json or in the command line. Add this to cypress.json

{
"reporter": "railflow-cypress-junit-reporter",
 "reporterOptions": {
    "mochaFile": "cypress/results/my-test-output-[hash].xml"
  }
}

'mochaFile' is the path to the report file. When there are several test files run it will generate a report file for each test file, so in order to generate unique file names and not overwrite the existing ones, the [hash] is added to the name of the report file.

3. Customize index.js to add screenshots into the report.

Add the code below into index.js file located in <Cypress_project_root>/cypress/support

Cypress.on('test:after:run', function (test, runnable) {
    if (test.state === "failed") {
        const screenshot = `${Cypress.config('screenshotsFolder')}/${Cypress.spec.name}/${runnable.parent.title} -- ${test.title} (failed).png`;
        console.log(screenshot)
        if (!test.attachments) {
            test.attachments = [];
        }
        test.attachments.push(screenshot);
    }
});

4. Add cypress test with TestRail-related elements

describe('My First Test',
    {
        env: {
            railflow: {
                title: 'Sample Test Suite',
                case_type: 'Automated',
                case_priority: 'High',
                case_fields: ['field1=value1', 'field2=value2'],
                result_fields: ['field1=value1', 'field2=value2'],
                jira_ids: ['jid1', 'jid2', 'jid3'],
                smart_failure_assignment: ['[e-mail]', '[e-mail]']
            }
        },
    }, () => {
        it('Does not do much!',
            {
                env: {
                    railflow: {
                        title: 'Sample Test Case',
                        case_type: 'Automated',
                        case_priority: 'High',
                        case_fields: ['field1=value1', 'field2=value2'],
                        result_fields: ['field1=value1', 'field2=value2'],
                        jira_ids: ['jid1', 'jid2', 'jid3'],
                        smart_failure_assignment: ['[e-mail]', '[e-mail]'],
                        testrail_ids: [1, 2, 3]

                    }
                }
            }, () => {
                expect(true).to.equal(false)
            })
    })

5. Run tests and generate report

./node_modules/.bin/cypress run

6. View report file

Report file generated at '<Cypress_project_root>/cypress/results'.

my-test-output-828a1c4885dc687b1a19e11e24b9437e.xml

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0.3620" tests="1" failures="1">
  <testsuite name="Root Suite" timestamp="2021-11-03T11:49:27" tests="0" file="cypress/integration/MySampleCypressTests.spec.js" time="0.0000" failures="0">
  </testsuite>
  <testsuite name="My First Test" timestamp="2021-11-03T11:49:27" tests="1" time="0.3430" failures="0">
    <testcase name="My First Test Does not do much!" time="0.2730" classname="Does not do much!">
      <system-out>[[ATTACHMENT|/home/my_sample_cypress_project/raiflow-cypress-junit-reporter/cypress/screenshots/MySampleCypressTests.spec.js/My First Test -- Does not do much! (failed).png]]</system-out>
      <failure message="expected true to equal false" type="AssertionError"><![CDATA[AssertionError: expected true to equal false
    at Context.eval (http://localhost:37195/__cypress/tests?p=cypress/integration/MySampleCypressTests.spec.js:126:21)]]></failure>
      <railflow>
        <title>Sample Test Case</title>
        <case_type>Automated</case_type>
        <case_priority>High</case_priority>
        <testrail_id>1</testrail_id>
        <testrail_id>2</testrail_id>
        <testrail_id>3</testrail_id>
        <jira_id>jid1</jira_id>
        <jira_id>jid2</jira_id>
        <jira_id>jid3</jira_id>
        <smart_failure_assignment>
          <user>[e-mail]</user>
          <user>[e-mail]</user>
        </smart_failure_assignment>
        <case_fields>
          <field name="field1">value1</field>
          <field name="field2">value2</field>
        </case_fields>
        <result_fields>
          <field name="field1">value1</field>
          <field name="field2">value2</field>
        </result_fields>
      </railflow>
    </testcase>
    <railflow>
      <title>Sample Test Suite</title>
      <case_type>Automated</case_type>
      <case_priority>High</case_priority>
      <jira_id>jid1</jira_id>
      <jira_id>jid2</jira_id>
      <jira_id>jid3</jira_id>
      <smart_failure_assignment>
        <user>[e-mail]</user>
        <user>[e-mail]</user>
      </smart_failure_assignment>
      <case_fields>
        <field name="field1">value1</field>
        <field name="field2">value2</field>
      </case_fields>
      <result_fields>
        <field name="field1">value1</field>
        <field name="field2">value2</field>
      </result_fields>
    </railflow>
  </testsuite>
</testsuites>

7. Install Railflow CLI to the project

npm install railflow

8. Run Railflow CLI and upload test results into TestRail

npx railflow -k ABCDE-12345-FGHIJ-67890 -url https://testrail.your-server.com/ -u testrail-username -p testrail-password -pr "Railflow Demo" -path section1/section2 -f junit -r cypress/results/*.xml -sm path

Where:

k - Railflow license key url - the URL of the TestRail server u - TestRail user name p - TestRail user password pr - name of the project in TestRail path - path to the subsection in TestRail where test cases will be exported to f - test report format - junit r - path to the report XML files

Please see Railflow NPM documentation for the all the details about Railflow CLI options.

9. View results in TestRail

Test run

Test result details