Android VTS
Usually after the new version of AOSP is released by Google, the SoC vendor needs to upgrade the HAL, and the OEM can upgrade the HAL and Framework before the user can receive the push of the OTA upgrade package on the device.
VTS itself means the compliance test suite of Android Vendor Interface (VINTF).
VINTF is a versioned, stable interface for Android vendor implementation. This concept is introduced from Android version 8.0 (Oreo) in order to improve the engineering productivity, launch velocity, security, and reliability of Android device ecosystem.
VTS consists of a set of testing frameworks and test cases, designed to help enhance the robustness, reliability, and compliance of:
- the Android system (e.g., Core HALs and libraries) and
- low-level system software (e.g., kernel, module, and firmware).
VTS has mainly two types of test suites:
- automated (e.g., record-and-replay and fuzzing) and
- structural (e.g., gTest and host-driven python) testing.
While the VTS framework and test cases are designed for userdebug or eng build devices, some test cases can be run on user build devices using its app namely, VTS Agent App.
This is achieved by the introduction of a new vendor interface between the Android OS framework and the vendor implementation. The new vendor interface is validated by a Vendor Test Suite (VTS), analogous to the CTS, to ensure forward compatibility of the vendor implementation.
This means that the Vendor Interface is introduced in Project Treble to separate the Android Framework from the HAL and test these Vendor Interfaces through VTS testing to ensure forward compatibility of the HAL.
It may not be clear to see this paragraph. We know that only the APP layer and the Framework layer are designed to be separate, but through CTS testing, a consistent call interface (API) between the APP and the Android Framework is ensured.This allows the same program written by the APP developer to run on different system versions (forward compatibility), different hardware platforms, and different devices made by different vendors.
Similar to CTS, VTS tests the Vendor Interface to ensure that the same version of the Android Framework can run on different HALs, or that different Android Frameworks can run on the same HAL.
Through such a Framework / HAL separation design and interface consistency guarantee, the Android system after the 8.0 version can be upgraded directly when the platform is upgraded without considering the HAL layer changes, thereby shortening the system for the user to get the system. Upgrade the time of OTA push.
For CTS can go through below blog:
VTS test environment
- Install Python development kit:
$ sudo apt-get install python-dev
- Install Protocol Buffer tools (for Python):
$ sudo apt-get install python-protobuf
$ sudo apt-get install protobuf-compiler
- Install Python virtual environment-related tools:
$ sudo apt-get install python-virtualenv
$ sudo apt-get install python-pip
VTS test method
1. MTK User load boot up to home screen switched to the main interface
2. Please enable OEM unlocking in settings open developer options open to OEM unlocking
3. Please enable USB debugging in settings to turn on USB debugging
4. Press Volume up key + power key into fastboot mode press volume up key and power on key to enter fastboot mode
5. Connect phone to PC, input “fastboot flashing unlock” / “fastboot oem unlock” connect to computer and enter cmd Enter the command fastboot flashing unlock or fastboot oem unlock
6. fastboot flash system system.img and then enter the command fastboot flash system + system.img path
eg: fastboot flash system d:\Users\ system\system_aosp_arm_a_20180205.img
brush the corresponding security patch corresponding to system.img
7. fastboot reboot
Finally, type the command fastboot reboot to restart the prototype
VTS test command
VTS tests two items:
run vts
run cts-on-gsi
VTS User Manual
Linux is officially supported for building and running VTS. Building on Windows is not supported, but it is possible to run VTS on Windows with Python, Java, and ADB installed.
The following instructions assume Linux environment.
1. Setup
1.1. Host setup
(Optional) During VTS test runs, required Python packages are downloaded from the Python Package Index. There is an option to instead install these packages from a local directory during test runtime by predownloading the packages. First run the lunch command, then set an environment variable VTS_PYPI_PATH as a new local directory to host the Python packages. Then run the download-pypi-packages.sh script:
$ cd ${branch}
$ . test/vts/script/download-pypi-packages.sh
1.2. Checkout master git repository
Download Android Source Code
$ export branch=master
$ mkdir ${branch}
$ cd ${branch}
$ repo init -b ${branch} -u persistent https://android.googlesource.com/platform/manifest
$ repo sync -j 8
1.3. Build an Android image
$ cd ${branch}
$ . build/make/envsetup.sh
$ lunch aosp_arm64-userdebug # or <your device>-userdebug
The below is an optional step:
$ make -j 8
If this fails, please do:
$ repo sync -j 8
$ make -j 8
Such can happen because tip of tree may not always be buildable.
1.4. Build a VTS package
$ cd ${branch}
$ make vts -j8
Or use the exact command:
$ make -j8 vts showcommands dist TARGET_PRODUCT=aosp_arm64 WITH_DEXPREOPT=false TARGET_BUILD_VARIANT=userdebug
1.5. Connect to an Android device
Let’s connect an Android device and a host computer using a USB cable.
· On an Android device, Setting -> About Phone -> Click repeatedly ‘Build number’ until developer mode is enabled.
· On an Android device, Setting -> Developer options -> Turn on ‘USB debugging’
· On a host, run adb devices from a command line shell.
· On a Android device, confirm that the host is trusted.
· On a host, type adb shell and if that works, we’re ready.
Run VTS Tests
2.1. Run a VTS test plan
For Linux users,
$ vts10-tradefed
> run vts10
For Windows users,
please build on Linux. Then copy the following zip file to Windows and extract it.
out/host/linux-x86/vts10/android-vts10.zip
Launch the batch file in the extracted folder.
$ android-vts10\tools\vts10-tradefed_win.bat
> run vts10
Example stdout:
…
…
08–16 09:36:03 I/ResultReporter: Saved logs for device_logcat in …/out/host/linux-x86/vts10/android-vts10/logs/2016.08.16_09.17.13/device_logcat_7912321856562095748.zip
08–16 09:36:03 I/ResultReporter: Saved logs for host_log in …/out/host/linux-x86/vts10/android-vts10/logs/2016.08.16_09.17.13/host_log_2775945280523850018.zip
08–16 09:36:04 I/ResultReporter: Invocation finished in 18m 50s. PASSED: 18, FAILED: 0, NOT EXECUTED: 2, MODULES: 8 of 10
08–16 09:36:04 I/ResultReporter: Test Result: …/out/host/linux-x86/vts10/android-vts10/results/2016.08.16_09.17.13/test_result.xml
08–16 09:36:04 I/ResultReporter: Full Result: …/out/host/linux-x86/vts10/android-vts10/results/2016.08.16_09.17.13.zip
2.2. Test report for APFE (Android Partner Front-End)
The uploadable report xml file can be found at
out/host/linux-x86/vts10/android-vts10/results/
After Android O release, you will be able to upload that xml file to AFPE and obtain a certificate.
2.3. Check the test logs
$ vi out/host/linux-x86/vts10/android-vts10/logs/
Then select a directory which captures the time stamp of your test run (e.g., 2016.08.16_09.17.13).
Then select
host_log_<timestamp>.zip
and host_log.txt in that zip file for host log.
Then select
device_logcat_<timestamp>.zip
and device_logcat.txt in that zip file for device log.
Run Options for Advanced Users
3.1. List of VTS10 Plans
A VTS test plan consists of a set of VTS tests. Typically the tests within a plan are for the same target component or testing the same or similar aspect (e.g., functionality, performance, robustness, or power). There are three kinds of plans in VTS:
3.1.1. Official Plans
Official plans contain only verified tests, and are for all Android developers and partners.
- vts: For all default VTS tests.
- vts-firmware: For all default VTS System Firmware tests.
- vts-fuzz: For all default VTS fuzz tests.
- vts-hal: For all default VTS HAL (hardware abstraction layer) module tests.
- vts-hal-profiling: For all default VTS HAL performance profiling tests.
- vts-hal-replay: For all default VTS HAL replay tests.
- vts-kernel: For all default VTS kernel tests.
- vts-library: For all default VTS library tests.
- vts-performance: For all default VTS performance tests
3.1.2. Staging Plans
Serving plans contain experimental tests, and are for Android partners to use as part of their continuous integration infrastructure. The name of a serving plan always starts with ‘vts-staging-
3.1.3. Other Plans
The following plans are also available for development purposes.
- vts-camera-its: For camera ITS (Image Test Suite) tests ported to VTS.
- vts-codelab: For VTS codelab.
- vts-codelab-multi-device: For VTS codelab of multi-device testing.
- vts-gce: For VTS tests which can be run on Google Compute Engine (GCE)
- vts-hal-auto: For VTS automotive vehicle HAL test.
- vts-hal-tv: For VTS tv HAL test.
- vts-host: For VTS host-driven tests.
- vts-performance-systrace: For VTS performance tests with systrace enabled.
- vts-presubmit: For VTS pre-submit time tests.
- vts-security: For VTS security tests.
- vts-system: For VTS system tests.
- vts-vndk: for VTS vndk tests.
Debugging
Run VTS tests directly for debugging
First of all, if you have not done VTS setup, that is required here.
Download required Python packages to local host
$ . test/vts/script/pypi-packages-local.sh
Build Binaries
$ cd test/vts/script
$ ./create-image.sh <build target>-userdebug
or
$ ./create-image.sh aosp_salifish-userdebug
Copy Binaries
$ ./setup-local.sh <device name>
or
$ ./setup-local.sh sailfish
Run a test direclty
$ PYTHONPATH=$PYTHONPATH:.. python -m vts.testcases.<test py package path> $ANDROID_BUILD_TOP/test/vts/testcases/<path to its config file>
For example, for SampleShellTest, please run:
PYTHONPATH=$PYTHONPATH:.. python -m vts.testcases.host.shell.SampleShellTest $ANDROID_BUILD_TOP/test/vts/testcases/host/shell/SampleShellTest.config
More examples are in test/vts/script/run-local.sh.
Additional Step for LTP and Linux-Kselftest
Add 'data_file_path' : '<your android build top>/out/host/linux-x86/vts/android-vts/testcases' to your config file (e.g., VtsKernelLtpStaging.config).
Add a new test
In order to add a new test, the following two files needed to be extended.
test/vts/create-image.sh
test/vts/setup-local.sh
Optionally, the command used to add a new test can be also added to:
test/vts/run-local.sh
Where to find VTS test cases
In principle, VTS test cases are co-located with their respective projects.
- HIDL HAL Tests: test/vts-testcase/hal/<hal name>/<version> and hardware/interfaces/<hal name>/<version>/vts
- Kernel Tests: test/vts-testcase/kernel
- LTP: external/ltp
- Linux Kselftest: external/linux-kselftest
- VNDK (Vendor Native Development Kit) Tests: test/vts-testcase/vndk
- Performance Tests: test/vts-testcase/performance
- Fuzz Tests: test/vts-testcase/fuzz
- Security Tests: test/vts-testcase/security
Possible causes of VTS Fail
At the moment, I have summarized the following reasons why VTS tests may fail:
- Kernel interface problem
- No corresponding SEpolicy configuration added
- No binderized HAL is used
- There is an implementation in the HAL that does not match the interface specification.
