A Test Harness, also known as a testing framework or testing tool, is a software tool or library that provides a set of functions, APIs, or interfaces for writing, organizing, and executing tests. Test harnesses provide a structured way to write tests and automate the testing process.
Linux avocado test framework and Linux op-test framework are both open-source testing frameworks designed for testing and validating Linux-based systems. Both frameworks are widely used in the Linux community and have a strong user base. The choice between the two depends on the specific testing needs and requirements of the user.
The Linux avocado test framework is a modular and extensible testing framework that allows users to write and run tests for different levels of the Linux stack, including the kernel, user space, and applications. It provides a wide range of plugins and tools for testing, including functional, performance, and integration testing. The framework is easy to install and use and supports multiple test runners and reporting formats.
On the other hand, the Linux op-test framework is a set of Python libraries and utilities that automate the testing of hardware and firmware components in Linux-based systems. It provides a high-level Python API for interacting with hardware and firmware interfaces, as well as a set of pre-built tests for validating various hardware components such as CPU, memory, and storage. The framework is highly flexible and customizable, allowing users to create their own tests and integrate with other testing tools and frameworks.
While both frameworks are designed for testing Linux-based systems, the Linux avocado test framework provides a broad range of testing capabilities across different levels of the Linux stack, while the Linux op-test framework focuses specifically on automating hardware and firmware testing. The choice between the two depends on the specific testing needs and requirements of the user.
The Linux avocado test framework provides a plugin called "avocado-vt" which can be used to run tests that require a reboot between different test stages. This plugin enables the framework to run destructive tests, like kernel crash dump (kdump) testing, that require the system to be rebooted multiple times.
Similarly, the Linux op-test framework also provides support for testing scenarios that require system reboot. The framework includes a "reboot" library that allows users to reboot the system under test and wait for it to come back up before continuing with the test. This library can be used to test scenarios like kdump and fadump that require system reboot.
The community maintained avocado tests repository:
Avocado is a set of tools and libraries to help with automated testing. One can call it a test framework with benefits. Native tests are written in Python and they follow the unittest pattern, but any executable can serve as a test.
This repository contains a collection of miscellaneous tests and plugins for the Linux Avocado test framework that cover a wide range of functional, performance, and integration testing scenarios. The tests are designed to be modular and easy to use, and can be integrated with the Avocado test framework to extend its capabilities.
https://github.com/avocado-framework-tests/avocado-misc-tests
How to run avocado misc tests :
To run the Avocado Misc Tests, you first need to install the Linux Avocado test framework on your system. Once you have installed the framework, you can clone the Avocado Misc Tests repository from GitHub by running the following command in a terminal:
git clone https://github.com/avocado-framework-tests/avocado-misc-tests.git
git clone git@github.com:avocado-framework-tests/avocado-misc-tests.git
# git clone git@github.com:avocado-framework-tests/avocado-misc-tests.git
Cloning into 'avocado-misc-tests'...
remote: Enumerating objects: 18087, done.
remote: Counting objects: 100% (451/451), done.
remote: Compressing objects: 100% (239/239), done.
remote: Total 18087 (delta 242), reused 368 (delta 208), pack-reused 17636
Receiving objects: 100% (18087/18087), 6.15 MiB | 16.67 MiB/s, done.
Resolving deltas: 100% (11833/11833), done.
#
This repository is dedicated to host any tests written using the Avocado API. It is being initially populated with tests ported from autotest client tests repository, but it's not limited by that.
After cloning the repository, you can navigate to the avocado-misc-tests directory and run the tests using the avocado run command. For example, to run all the tests in the network category, you can run the following command:
cd avocado-misc-tests
avocado run network/
This will run all the tests in the network category. You can also run individual tests by specifying the path to the test file, like this:
avocado run network/test_network_ping.py
This will run the test_network_ping.py test in the network category.
Before running the tests, you may need to configure the Avocado framework to use the appropriate test runner, test environment, and plugins for your system. You can find more information on how to configure and use the Avocado framework in the official documentation:
https://avocado-framework.readthedocs.io/en/latest/
$ avocado run avocado-misc-tests/generic/stress.py
JOB ID : 0018adbc07c5d90d242dd6b341c87972b8f77a0b
JOB LOG : $HOME/avocado/job-results/job-2016-01-18T15.32-0018adb/job.log
TESTS : 1
(1/1) avocado-misc-tests/generic/stress.py:Stress.test: PASS (62.67 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
JOB HTML : $HOME/avocado/job-results/job-2016-01-18T15.32-0018adb/html/results.html
TIME : 62.67 s
There are a few more interesting things about the Avocado test framework and its usability and use cases:
- Flexible test design: The Avocado test framework is designed to be flexible and adaptable to a wide range of testing scenarios. It supports various test types, including functional, integration, performance, and stress tests, and can be used to test software at different levels of abstraction, from system-level to individual components. Avocado also provides a wide range of plugins and interfaces for integrating with other tools and frameworks, making it easy to customize and extend its capabilities.
- Easy to use: Avocado is designed to be easy to use, even for users who are new to testing or have limited programming experience. It uses a simple YAML-based syntax for defining tests and test plans, and provides a user-friendly command-line interface for running tests and viewing results. Avocado also includes detailed documentation and tutorials to help users get started quickly.
- Scalability and distributed testing: Avocado supports distributed testing across multiple systems, making it easy to scale up testing to handle large workloads. It includes a built-in job scheduler for managing test execution across multiple systems, and can be integrated with various cloud-based services for running tests in the cloud.
- Community support: Avocado is an open-source project maintained by a vibrant community of developers and testers. The community provides regular updates and bug fixes, and is actively involved in improving the usability and functionality of the framework. The Avocado community also provides support through various channels, including GitHub, mailing lists, and IRC.
- Use cases: Avocado is used by various organizations and companies for testing different types of software, including operating systems, virtualization platforms, container platforms, and cloud services. It is particularly well-suited for testing complex, distributed systems that require a high degree of automation and scalability. Some of the organizations that use Avocado include Red Hat, IBM, Intel, and Huawei.
License
Except where otherwise indicated in a given source file, all original contributions to Avocado are licensed under the GNU General Public License version 2 (GPLv2) or any later version. By contributing you agree that these contributions are your own (or approved by your employer) and you grant a full, complete, irrevocable copyright license to all users and developers of the Avocado project, present and future, pursuant to the license of the project.
================
The community maintained, op-tests repository:
https://github.com/open-power/op-test
git clone git@github.com:open-power/op-test.git
# git clone git@github.com:open-power/op-test.git
Cloning into 'op-test'...
remote: Enumerating objects: 8716, done.
remote: Counting objects: 100% (623/623), done.
remote: Compressing objects: 100% (275/275), done.
remote: Total 8716 (delta 416), reused 480 (delta 347), pack-reused 8093
Receiving objects: 100% (8716/8716), 23.89 MiB | 23.39 MiB/s, done.
Resolving deltas: 100% (6488/6488), done.
#
./op-test -c machine.conf --run testcases.RunHostTest --host-cmd ls
Testcase: https://github.com/open-power/op-test/blob/master/testcases/RunHostTest.py
where machine.conf :
[op-test]
bmc_type=OpenBMC /EBMC_PHYP/FSP_PHYP
bmc_username=abc
bmc_ip=w39
bmc_username=root
bmc_password=0penBmc
hmc_ip=a.b.c.d
hmc_username=hmcuser
hmc_password=hmcpasswd123
host_ip=x.y.x.k
host_user=hostuser
host_password=hostpasswd123
system_name=power10
lpar_name=lpar_name_1
lpar_prof=default_profile
CASE2:
./op-test -c machine_ltcever7x0.config --run testcases.RunHostTest --host-cmd-file cmd.conf
where :
# cat cmd.conf
echo "welcome SACHIN P B"
hostname
uptime
date
#
OUTPUT:
----------------------------
# ./op-test -c machine_ltcever7x0.config --run testcases.RunHostTest --host-cmd-file cmd.conf
Logs in: /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-20230829095648
2023-08-29 09:56:48,758:op-test:setUpLoggerFile:INFO:Preparing to set location of Log File to /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-20230829095648/20230829145648758035.main.log
2023-08-29 09:56:48,758:op-test:setUpLoggerFile:INFO:Log file: /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-20230829095648/20230829145648758035.main.log
2023-08-29 09:56:48,758:op-test:setUpLoggerDebugFile:INFO:Preparing to set location of Debug Log File to /root/fix_kdump_FalsePositives/op-test/test-reports/test-run-20230829095648/20230829145648758291.debug.log
[console-expect]#which whoami && whoami
/usr/bin/whoami
root
[console-expect]#echo $?
echo $?
0
[console-expect]#echo "welcome SACHIN P B"
echo "welcome SACHIN P B"
welcome SACHIN P B
[console-expect]#echo $?
echo $?
0
[console-expect]#hostname
hostname
myhost.com
[console-expect]#echo $?
echo $?
0
[console-expect]#uptime
uptime
09:58:15 up 7:50, 2 users, load average: 0.08, 0.02, 0.01
[console-expect]#echo $?
echo $?
0
[console-expect]#date
date
Tue Aug 29 09:58:15 CDT 2023
[console-expect]#echo $?
echo $?
0
ok
Ran 1 test in 7.510s
OK
2023-08-29 09:58:17,787:op-test:<module>:INFO:Exit with Result errors="0" and failures="0"
------------------------------------------------------------------------------------------------------------------
Example 2:
python3 op-test -c machine.conf --run testcases.PowerNVDump.KernelCrash_disable_radix
python3 op-test -c machine.conf --run testcases.PowerNVDump.KernelCrash_XIVE_off
python3 op-test --run-suite osdump-suite -c CR-machine.conf
python3 op-test --run testcases.RunHostTest -c CR-Machine.conf --host-cmd-file CR-Machine_command.conf --host-cmd-timeout 286400
- Testing hardware systems: The op-test framework is designed for testing hardware systems, particularly servers, using the OpenPOWER architecture. It includes a wide range of tests that cover different aspects of hardware functionality, such as power management, CPU, memory, and I/O.
- Integration with OpenBMC: The op-test framework integrates with the OpenBMC project, an open-source implementation of the Baseboard Management Controller (BMC) firmware that provides out-of-band management capabilities for servers. This integration allows users to control and monitor server hardware using the OpenBMC interface, and to run tests on the hardware using the op-test framework.
- UEFI and firmware testing: The op-test framework includes support for testing UEFI firmware and other low-level system components, such as the Hostboot bootloader. This allows users to test the system firmware and ensure that it is functioning correctly.
- Easy to use: The op-test framework is designed to be easy to use, even for users who are not familiar with hardware testing. It uses a simple command-line interface and provides detailed documentation and tutorials to help users get started quickly.
- Scalability: The op-test framework is designed to be scalable and can be used to test multiple systems in parallel. This makes it suitable for testing large server farms and data centers.
- Community support: The op-test framework is an open-source project with an active community of developers and testers. The community provides regular updates and bug fixes, and is actively involved in improving the usability and functionality of the framework. The op-test community also provides support through various channels, including GitHub, mailing lists, and IRC.
- Use cases: The op-test framework is used by various organizations and companies for testing hardware systems, including server manufacturers, data center operators, and cloud service providers. Some of the organizations that use the op-test framework include IBM, Google, and Rackspace.
2) cd kdump_xive_off_check
3) git clone git@github.com:SACHIN-PB/op-test.git
4) git config user.email
5) git config user.name
6) git branch
7) git remote -v
git push -f origin kdump_xive_off_check
No comments:
Post a Comment