Usually after the new version of AOSP is released by Google, the SoC vendor needs to upgrade the HAL, and the OEM can upgrade the HAL and Framework before the user can receive the push of the OTA upgrade package on the device.

VTS itself means the compliance test suite of Android Vendor Interface (VINTF).

VINTF is a versioned, stable interface for Android vendor implementation. This concept is introduced from Android version 8.0 (Oreo) in order to improve the engineering productivity, launch velocity, security, and reliability of Android device ecosystem.

VTS consists of a set of testing frameworks and test cases, designed to help enhance the robustness, reliability, and compliance of:

  • the Android system (e.g., Core HALs and libraries) and
  • low-level system software (e.g., kernel, module, and firmware).

VTS has mainly two types of test suites:

  • automated (e.g., record-and-replay and fuzzing) and
  • structural (e.g., gTest and host-driven python) testing.

While the VTS framework and test cases are designed for userdebug or eng build devices, some test cases can be run on user build devices using its app namely, VTS Agent App.

This is achieved by the introduction of a new vendor interface between the Android OS framework and the vendor implementation. The new vendor interface is validated by a Vendor Test Suite (VTS), analogous to the CTS, to ensure forward compatibility of the vendor implementation.

This means that the Vendor Interface is introduced in Project Treble to separate the Android Framework from the HAL and test these Vendor Interfaces through VTS testing to ensure forward compatibility of the HAL.
It may not be clear to see this paragraph. We know that only the APP layer and the Framework layer are designed to be separate, but through CTS testing, a consistent call interface (API) between the APP and the Android Framework is ensured.This allows the same program written by the APP developer to run on different system versions (forward compatibility), different hardware platforms, and different devices made by different vendors.

Similar to CTS, VTS tests the Vendor Interface to ensure that the same version of the Android Framework can run on different HALs, or that different Android Frameworks can run on the same HAL.
Through such a Framework / HAL separation design and interface consistency guarantee, the Android system after the 8.0 version can be upgraded directly when the platform is upgraded without considering the HAL layer changes, thereby shortening the system for the user to get the system. Upgrade the time of OTA push.

CTS VTS struct

For CTS can go through below blog:

VTS test environment

  • Install Python development kit:

$ sudo apt-get install python-dev

  • Install Protocol Buffer tools (for Python):

$ sudo apt-get install python-protobuf

$ sudo apt-get install protobuf-compiler

  • Install Python virtual environment-related tools:

$ sudo apt-get install python-virtualenv

$ sudo apt-get install python-pip

VTS test method

1. MTK User load boot up to home screen switched to the main interface

2. Please enable OEM unlocking in settings open developer options open to OEM unlocking

3. Please enable USB debugging in settings to turn on USB debugging

4. Press Volume up key + power key into fastboot mode press volume up key and power on key to enter fastboot mode

5. Connect phone to PC, input “fastboot flashing unlock” / “fastboot oem unlock” connect to computer and enter cmd Enter the command fastboot flashing unlock or fastboot oem unlock

6. fastboot flash system system.img and then enter the command fastboot flash system + system.img path
eg: fastboot flash system d:\Users\ system\system_aosp_arm_a_20180205.img
brush the corresponding security patch corresponding to system.img

7. fastboot reboot
Finally, type the command fastboot reboot to restart the prototype

VTS test command

VTS tests two items:

run vts

run cts-on-gsi

VTS User Manual

Linux is officially supported for building and running VTS. Building on Windows is not supported, but it is possible to run VTS on Windows with Python, Java, and ADB installed.

The following instructions assume Linux environment.

1. Setup

1.1. Host setup

(Optional) During VTS test runs, required Python packages are downloaded from the Python Package Index. There is an option to instead install these packages from a local directory during test runtime by predownloading the packages. First run the lunch command, then set an environment variable VTS_PYPI_PATH as a new local directory to host the Python packages. Then run the download-pypi-packages.sh script:

$ cd ${branch}

$ . test/vts/script/download-pypi-packages.sh

1.2. Checkout master git repository

Download Android Source Code

$ export branch=master

$ mkdir ${branch}

$ cd ${branch}

$ repo init -b ${branch} -u persistent https://android.googlesource.com/platform/manifest

$ repo sync -j 8

1.3. Build an Android image

$ cd ${branch}

$ . build/make/envsetup.sh

$ lunch aosp_arm64-userdebug # or <your device>-userdebug

The below is an optional step:

$ make -j 8

If this fails, please do:

$ repo sync -j 8

$ make -j 8

Such can happen because tip of tree may not always be buildable.

1.4. Build a VTS package

$ cd ${branch}

$ make vts -j8

Or use the exact command:

$ make -j8 vts showcommands dist TARGET_PRODUCT=aosp_arm64 WITH_DEXPREOPT=false TARGET_BUILD_VARIANT=userdebug

1.5. Connect to an Android device

Let’s connect an Android device and a host computer using a USB cable.

· On an Android device, Setting -> About Phone -> Click repeatedly ‘Build number’ until developer mode is enabled.

· On an Android device, Setting -> Developer options -> Turn on ‘USB debugging’

· On a host, run adb devices from a command line shell.

· On a Android device, confirm that the host is trusted.

· On a host, type adb shell and if that works, we’re ready.

Run VTS Tests

2.1. Run a VTS test plan

For Linux users,

$ vts10-tradefed

> run vts10

For Windows users,

please build on Linux. Then copy the following zip file to Windows and extract it.

out/host/linux-x86/vts10/android-vts10.zip

Launch the batch file in the extracted folder.

$ android-vts10\tools\vts10-tradefed_win.bat

> run vts10

Example stdout:

08–16 09:36:03 I/ResultReporter: Saved logs for device_logcat in …/out/host/linux-x86/vts10/android-vts10/logs/2016.08.16_09.17.13/device_logcat_7912321856562095748.zip

08–16 09:36:03 I/ResultReporter: Saved logs for host_log in …/out/host/linux-x86/vts10/android-vts10/logs/2016.08.16_09.17.13/host_log_2775945280523850018.zip

08–16 09:36:04 I/ResultReporter: Invocation finished in 18m 50s. PASSED: 18, FAILED: 0, NOT EXECUTED: 2, MODULES: 8 of 10

08–16 09:36:04 I/ResultReporter: Test Result: …/out/host/linux-x86/vts10/android-vts10/results/2016.08.16_09.17.13/test_result.xml

08–16 09:36:04 I/ResultReporter: Full Result: …/out/host/linux-x86/vts10/android-vts10/results/2016.08.16_09.17.13.zip

2.2. Test report for APFE (Android Partner Front-End)

The uploadable report xml file can be found at

out/host/linux-x86/vts10/android-vts10/results/

After Android O release, you will be able to upload that xml file to AFPE and obtain a certificate.

2.3. Check the test logs

$ vi out/host/linux-x86/vts10/android-vts10/logs/

Then select a directory which captures the time stamp of your test run (e.g., 2016.08.16_09.17.13).

Then select

host_log_<timestamp>.zip

and host_log.txt in that zip file for host log.

Then select

device_logcat_<timestamp>.zip

and device_logcat.txt in that zip file for device log.

Run Options for Advanced Users

3.1. List of VTS10 Plans

A VTS test plan consists of a set of VTS tests. Typically the tests within a plan are for the same target component or testing the same or similar aspect (e.g., functionality, performance, robustness, or power). There are three kinds of plans in VTS:

3.1.1. Official Plans

Official plans contain only verified tests, and are for all Android developers and partners.

  • vts: For all default VTS tests.
  • vts-firmware: For all default VTS System Firmware tests.
  • vts-fuzz: For all default VTS fuzz tests.
  • vts-hal: For all default VTS HAL (hardware abstraction layer) module tests.
  • vts-hal-profiling: For all default VTS HAL performance profiling tests.
  • vts-hal-replay: For all default VTS HAL replay tests.
  • vts-kernel: For all default VTS kernel tests.
  • vts-library: For all default VTS library tests.
  • vts-performance: For all default VTS performance tests

3.1.2. Staging Plans

Serving plans contain experimental tests, and are for Android partners to use as part of their continuous integration infrastructure. The name of a serving plan always starts with ‘vts-staging-

3.1.3. Other Plans

The following plans are also available for development purposes.

  • vts-camera-its: For camera ITS (Image Test Suite) tests ported to VTS.
  • vts-codelab: For VTS codelab.
  • vts-codelab-multi-device: For VTS codelab of multi-device testing.
  • vts-gce: For VTS tests which can be run on Google Compute Engine (GCE)
  • vts-hal-auto: For VTS automotive vehicle HAL test.
  • vts-hal-tv: For VTS tv HAL test.
  • vts-host: For VTS host-driven tests.
  • vts-performance-systrace: For VTS performance tests with systrace enabled.
  • vts-presubmit: For VTS pre-submit time tests.
  • vts-security: For VTS security tests.
  • vts-system: For VTS system tests.
  • vts-vndk: for VTS vndk tests.

Debugging

Run VTS tests directly for debugging

First of all, if you have not done VTS setup, that is required here.

Download required Python packages to local host

Build Binaries

or

Copy Binaries

or

Run a test direclty

For example, for SampleShellTest, please run:

More examples are in .

Additional Step for LTP and Linux-Kselftest

Add to your config file (e.g., ).

Add a new test

In order to add a new test, the following two files needed to be extended.

Optionally, the command used to add a new test can be also added to:

Where to find VTS test cases

In principle, VTS test cases are co-located with their respective projects.

  • HIDL HAL Tests: test/vts-testcase/hal/<hal name>/<version> and hardware/interfaces/<hal name>/<version>/vts
  • Kernel Tests: test/vts-testcase/kernel
  • LTP: external/ltp
  • Linux Kselftest: external/linux-kselftest
  • VNDK (Vendor Native Development Kit) Tests: test/vts-testcase/vndk
  • Performance Tests: test/vts-testcase/performance
  • Fuzz Tests: test/vts-testcase/fuzz
  • Security Tests: test/vts-testcase/security

Possible causes of VTS Fail

At the moment, I have summarized the following reasons why VTS tests may fail:

  • Kernel interface problem
  • No corresponding SEpolicy configuration added
  • No binderized HAL is used
  • There is an implementation in the HAL that does not match the interface specification.

A developer is responsible for developing a framework and system applications for devices powered by the Android operating system.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store