Test Result

SUN* CI can collects the test metadata from XML file and uses it to display build details. You can see detailed errors and some test cases not running, as seen below.

Attribute	Description
Test Case	The total number of tests in the suite, required
Assertions	Number of assertions in the test case. optional. not supported by maven surefire
Failures	Total number of failed tests from all testsuites.
Warnings	The total number of warning tests from all testsuites.
Errors	The total number of tests in the suite that errored
Skipped	The total number of skipped tests.
Time	Time taken to execute the tests in the suite

SUN * CI uses percentages to calculate product quality based on the following formula:

For the formula used, SUN* CI have three criteria to apply, as seen below:

• Line Coverage: Total line and indicates that each line of execution has been executed.
• Method coverage: Total method and indicates that each method of execution has been executed.
• Branch coverage: Metric measures whether the boolean expression of each control structure evaluated to both true and false while running the test suite.

test_result will exist at the same time artifacts and to use test result, you have to provide a properly formatted Junit.

• Junit for PHP
• Junit for Ruby

Example test result configurations

PHP example

The following .sun-ci.yml example use PHPUnit for genarate Junit result file.

jobs:
- name: test:phpunit
  stage: test
  image: sunasteriskrnd/php-workspace:7.4
  workspace: shared
  script:
  - phpunit --coverage-clover ./coverage.xml --coverage-html=coverage --log-junit ./junit.xml
  coverage:
    type: clover
    path: coverage.xml
  artifacts:
    paths:
    - coverage
    expires_in: 3 days
  test_result:
    type: junit
    path: junit.xml

Ruby example

Install gem rspec_junit_formatter:

gem install rspec_junit_formatter

Config .sun-ci.yml:

jobs:
- name: test:rspec
  stage: test
  image: sunci/ruby:2.5.1
  before_script:
  - bundle _2.1.4_ install --path vendor/bundle
  - bundle exec rake db:structure:load db:migrate
  script:
  - bundle _2.1.4_ exec rspec --format progress --format  --out ./.sun-ci-reports/rspec.html
  - bundle _2.1.4_ exec rspec --format progress --format RspecJunitFormatter -o .sun-ci-reports/junit.xml
  coverage:
    type: cobertura
    path: ./.sun-ci-reports/coverage/coverage.xml
  artifacts:
    paths:
    - ./.sun-ci-reports/coverage/
    - ./.sun-ci-reports/rspec.html
    expires_in: 3 days
  test_result:
    type: junit
    path: ~/sun-ci-reports/junit.xml 

Kotlin example

config .sun-ci.yml:

jobs:
  - name: test result
    stage: lint
    image: androidsdk/android-29:latest
    before_script:
      - chmod +x ./gradlew
    script:
      - ./gradlew lint test
    artifacts:
      paths:
        - app/build/reports
        - app/build/test-results
      expires_in: 3 days
    test_result:
      type: junit
      path: app/build/test-results/test**/TEST**.xml

Python example

This example uses pytest the --junitxml=junit.xml to output into the Junit report XML.

jobs:
- name: test result
  stage: test
  image: python:3.8-slim-buster
  script:
  - pip3 install -U pytest
  - pytest --junitxml=junit.xml
  test_result:
    type: junit
    path: junit.xml