Abstract
Software quality is of utmost importance to the financial sector. Software testing plays a critical role in achieving software product quality. Financial institutions benefit from rigorous testing by having confidence in the reliability and performance of the software. This can lead to improved customer experience, increased operational efficiency, and reduced risks of system failures or security breaches. A questionnaire-based survey was designed and subsequently an international survey was conducted involving sixty financial institutions, e.g., banking, insurance companies and pension funds, from across the globe to understand their level of test maturity. As a reference framework against which to measure their test maturity, the Test Maturity Model integration (TMMi) was used. In this paper their motivations for doing test process improvement and the benefits they achieved are discussed. Concrete examples of the benefits achieved are provided. The role of test automation with test process improvement at the financial institutions is also reported upon in this paper. The most common level of test maturity achieved, measured against the TMMi, is TMMi level 3 “Defined” which represents a more than average level of test maturity. Benefits are reported by the financial institutions, especially in the areas of software quality and testing productivity. The benefits achieved show a high level of correlation with the motivations for investing in test process improvement. Almost all of financial institutions also use test automation to improve their testing in parallel with process improvement, with test automation at system level being by far the most popular.
Keywords
Banking, Financial Domain, Insurance, Test Maturity Model Integration, TMMi, Test Process Improvement, Software Quality, Software Testing
1. Introduction
In today’s digital economy, software applications processing financial information must be reliable and operate without disruptive errors or incidents. It is also of utmost importance for public confidence that financial institution’s business operations have the capacity to continue uninterrupted. Driven by business needs, IT needs to ensure a high level of quality for the software applications in the financial sector. Software testing is the process to verify and validate that acquired or developed software products are reliable and comply with the institutions and regulated requirements.
To establish an understanding of test maturity in the financial domain, a survey was designed and thereafter conducted, involving sixty financial institutions from across the world. As a reference framework against which to measure their test maturity, the Test Maturity Model integration (TMMi)
| [1] | TMMi Foundation. TMMi model specification (reference model), release 1.3. 2022 www.tmmi.org |
[1]
was used. TMMi is a popular test framework in the financial domain, whereby banking, insurance companies and pension funds together represent 37% of TMMi users
| [2] | TMMi Foundation. TMMi 2nd worldwide user survey, version 1.2. 2023 www.tmmi.org |
[2]
. The financial institutions involved in the survey were previously either informally or formally assessed against the TMMi.
Although the study was primarily conducted to establish a view of test maturity in the financial domain, additional topics included in the survey were the costs and benefits associated with test improvement, which test levels and test types were in scope of the improvement process, reasons for investing in test process improvement and adopting TMMi, and the role of test automation. Based on the survey, a benchmark report has been published
| [3] | TMMi Foundation, TMMi Benchmark Report Financial Domain, version 1.1. 2024 www.tmmi.org |
[3]
that allows financial institutions involved in performing test process improvement using TMMi to validate their performance against the wider industry. Financial institutions that have not yet started carrying out test process improvement can learn from the costs and benefits reported, and can study how (e.g. at which test levels, test types, and degree of test automation) others are implementing test process improvement.
In this paper some of the results from the survey for the financial domain are reported and discussed, focusing on their test maturity status, motivations for practicing test improvement, benefits achieved and the role of test automation.
The paper is organized as follows, section 2 provides a brief overview of TMMi. Section 3 reports on test maturity status and TMMi levels achieved. Section 4 discusses motivations for test maturity. Section 5 presents the benefits achieved of adopting TMMi. Section 6 reports on the role of test automation. Finally, section 7 concludes the paper.
2. Overview of the TMMi
The TMMi framework has been developed by the TMMi Foundation as a guideline and reference framework for test process improvement, addressing issues important to test managers, test engineers, developers and software quality professionals. The roots of TMMi date back to Gelperin and Hetzel’s evolutionary testing model
, and an early test improvement model named Test Maturity Model (TMM)
| [5] | Burnstein. I. Suwanassart, I, Carlson C. R. Developing a Testing Maturity Model for software process evaluation and improvement. in: Proceedings International Test Conference Test and Design Validity. 1996. https://doi.org/10.1109/test.1996.557106 |
[5]
. The first version of the TMMi model was published by the TMMi Foundation in 2012. The latest version of the TMMi model is version 1.3, published in 2022. Whereas the TMMi was originally developed in the context of sequential software development, meanwhile studies have been performed and guidelines developed on how to also apply TMMi successfully in an Agile context
| [6] | Farid, A., Fathy, E., Abd, M. Towards Agile Implementation of Test Maturity Model Integration (TMMI) Level 2 using Scrum Practices. in: International Journal of Advanced Computer Science and Applications. 2015. https://doi.org/10.14569/IJACSA.2015.060931 |
| [7] | TMMi Foundation, TMMi in the Agile world, version 1.4. 2019 www.tmmi.org |
| [8] | Unudulmaz, A., Cingiz, M., Kalıpsız, O. Adaptation of the Four Levels of Test Maturity Model Integration with Agile and Risk-Based Test Techniques. in: Electronics. 2022. https://doi.org/10.3390/electronics11131985 |
[6-8]
.
The TMMi Foundation centrally (UK) is supported by the so-called TMMi Local Chapters that publish and organize TMMi-related services and activities locally in their country or region. At the time of writing, 29 TMMi Local Chapters, together covering 62 countries, are in existence, e.g. in Brazil, China, France, India, UK and USA. Many experience reports and case studies about the industrial application of TMMi have been published, e.g.,
| [9] | Rungi, K., Matulevicius, R. Empirical analysis of the Test Maturity Model Integration (TMMi). in: Skersys, T., Butleris, R., Butkiene, R. (eds.) ICIST 2013. CCIS, 403, pp. 376–391. Springer Heidelberg. 2013. https://doi.org/10.1007/978-3-642-41947-8_32 |
| [10] | Afzal W., Alone S., Kerstin, G., Torkar, R. Software test process improvement approaches: A systematic literature review and an industrial case study. in: Journal of Systems and Software. 2016. https://doi.org/10.1016/j.jss.2015.08.048 |
| [11] | Prastiti, R., Hidayanto, A. Evaluation of Maturity Level and Recommendations for Improvement of Software Testing Process Based on Test Maturity Model integration (TMMi): A Case Study. in: Jurnal Komtika (Komputasi dan Informatika), 6(2), pp. 145-154. 2023. https://doi.org/10.31603/komtika.v7i2.9628 |
[9-11]
. In a recent IEEE paper
| [12] | Garousi, V., van Veenendaal, E. Test Maturity Model integration (TMMi): trends of worldwide test maturity and certifications, IEEE Software, 39(2), pp. 71-79. 2022. https://doi.org/10.1109/MS.2021.3061930 |
[12]
, a status report about TMMi, and the trends of worldwide test maturity and certification was presented. Based on research
| [13] | Garousi, V., Felderer, M., Hacaloglu, T. What we know about software test maturity and test process improvement. IEEE Software, 35(1), pp. 84-92. 2018. https://doi.org/10.1109/ms.2017.4541043 |
[13]
and a follow-up study performed by the TMMi Foundation, the total number of TMMi model users is currently estimated to be around 10,000.
The TMMi model provides guidelines for assessing and improving testing capabilities. With TMMi, organizations can have their test processes objectively evaluated by accredited assessors, improve their test processes, and have their test processes and organization formally certified if they comply with the requirements.
Figure 1. Test Maturity Model integration (TMMi).
TMMi has a staged architecture for process improvement. It contains stages or levels through which an organization passes as its testing process evolves from one that is ad hoc and unmanaged to one that is managed, defined, measured, and optimized. Achieving each stage ensures that all goals of that level have been achieved and the improvements made provide the foundation for the next stage. There are five levels in TMMi that prescribe the maturity hierarchy and evolutionary path to test process improvement. For each maturity level, a set of process areas, goals and practices are defined. The process areas for each maturity level of TMMi are shown in
Figure 1. TMMi is freely available on the website of the TMMi Foundation (www.tmmi.org).
3. Test Maturity Status and TMMi Levels
Figure 2 shows the financial institutions’ test maturity status measured against the TMMi framework. It also shows the test maturity across “all” TMMi users as reported in the TMMi worldwide user survey 2023.
Figure 2. TMMi Levels Achieved.
Many of the financial institutions are operating at TMMi level 3 “Defined” (50%). An impressive number (29%) is at TMMi level 4 “Measurement”. The study reveals that most financial institutions meet the requirements of both TMMi level 2 and level 3. TMMi level 2 “Managed” mainly focuses on the implementation of a set of essential test practices for test management and test engineering required to run a successful project. At TMMi level 3 the focus is on institutionalization of testing in the organization, implementing a standardized way of working and early testing (often also referred to as shift-left).
Financial institutions are largely driven by (financial) numbers and data. This correlates with the process areas and objectives at TMMi level 4 “Measured” and could explain why many financial institutions are interested and have achieved this TMMi level. A major difference between the financial institutions and the overall TMMi market is that fewer financial institutions have achieved TMMi level 5 “Optimization” compared with the overall TMMi market. There seems to be less interest and drive towards achieving the objectives of TMMi level 5. Some market-driven organizations, e.g., companies offering testing as a service, aim to achieve TMMi level 5 for commercial reasons. This motivation is typically not (or less) applicable to financial institutions.
4. Motivations for Test Maturity
Figure 3 shows the financial institutions’ top 10 reasons and motivations for achieving higher levels of test maturity, for starting a test improvement program and adopting TMMi. Enhancing software quality, reducing product risk, achieving TMMi certification and increasing testing productivity were mentioned as the top four reasons. For example, 81% of the respondents stated that enhancing software quality is a motivation to invest in achieving higher levels of test maturity. In essence, one can observe here the project management triangle also often called the “golden triangle’. The “golden triangle” is a project management model showing that three constraints – time, scope and cost – must all be balanced in a project to deliver a quality product. This indicates that better management of testing (by, for example, having a good test strategy, test planning and test monitoring and control) is an important motivation to invest in higher levels of test maturity.
The motive of achieving TMMi certification (and probably associated with this, achieving standard compliance) indicates the importance of certification among financial institutions and their business operation. People benefits, such as an improved test engineering discipline and improved team morale, are of moderate importance to financial institutions as a motivation to adopt TMMi. People benefits almost seem to be considered a “nice to have” additional benefit.
Figure 3. Motivations for Test Maturity in the Financial Domain.
5. Benefits of Adopting TMMi
5.1. Survey Results
Figure 4 shows the results of benefits achieved by adopting TMMi. It also shows the correlation between motivations (reason) [in blue] presented in the previous paragraph, and benefits achieved (with percentages showing) [in red]. Enhancing software quality, reducing product risk, achieving TMMi certification and increasing testing productivity are not only high motivations but also observed benefits of adopting TMMi.
Figure 4. Benefits of Adopting TMMi in the Financial Domain.
An interesting finding is that many factors have almost closely matching percentages for motivations and benefits achieved. For reducing the number of defects and improving test morale, one can observe that both are achieved by substantially more (+10%) respondents than were stated as a motivation for investing in test maturity. Only with accelerating software delivery one can observe that the benefits are achieved by a lower percentage (27%) of the financial institutions, whereas 41% of the institutions stated it as a motivation. Improved testing capabilities typically implies finding more defects which thereafter need to be analyzed, solved and pass confirmation testing. This could well be a rationale for why the benefit of accelerated software delivery is not achieved as often as it is stated as a motivation.
5.2. Examples of Benefits Achieved
Survey participants were asked to provide concrete examples of the benefits achieved, some of which are summarize hereafter. These examples relate to either enhanced software quality (
Table 1) or increased testing productivity (
Table 2).
Regarding enhanced software quality, financial institutions investing in test maturity experienced an increase in defect detection ratios and fewer disruptions in production. The main objective of TMMi level 2 is to increase the defect detection capability of organizations, e.g. through product risk analysis, introducing test design and applying test design techniques.
Table 1. Examples of Software Quality Benefits Achieved.
Institution type | TMMi level | Software Quality benefits reported |
Bank | 3 | A 22% increase in defect detection effectiveness |
Insurance | 3 | Production defects (related to testing) close to zero |
Bank | 3 | Defect leaking to production decreased by 15% |
Bank | 4 | Defect leakage number decreased by 50% over 3 years. |
Clearing Center | 4 | Defect leakage rate reduced by more than 50% |
Insurance | 5 | Detection rate of production defects exceeds 99% (an increase of more than 30% over time) |
Credit Bank | 5 | Defect detection rate increased to 98.73% |
Regarding increased testing productivity, financial institutions investing in test maturity experienced reduced test cycle times and a more efficient execution of testing activities. At TMMi level 3, test efficiency is a main objective, this being achieved through, for example, standardization across projects and teams, the availability of supporting templates, a best practice library and early involvement of testing. The principle of early testing is often referred to as shift-left because it is an approach whereby testing is performed earlier (moved left) in the software development lifecycle timeline. Good practices that illustrate shift-left include reviews, more focus on component testing, testers involved at an early stage (e.g. at requirements sessions), static code analysis and writing test cases before the code is written.
Table 2. Examples of Testing Productivity Benefits Achieved.
Institution type | TMMi level | Testing Productivity benefits reported |
Insurance | 3 | Production cycle shortened from three months to less than one month |
Bank | 3 | Test execution lead time reduction from 19 to 5 weeks (through early involvement and early testing) |
Stock Exchange | 3 | Percentage of early defects found increased by 25% |
Bank | 4 | Test efficiency improved by 30% (effective test cycle time) |
Bank | 4 | Improved efficiency of test case development by 40% |
Insurance | 5 | Regression test execution cycle compressed from 4 hours to 30 minutes |
6. The Role of Test Automation
A highly popular framework in the business domain is the People, Process and Technology framework (also known as the PPT framework)
| [14] | Leavitt, H. J. Applied Organization Change in Industry: structural, technical, and human approaches, in: New perspectives in organization research, pp. 55-71. Wiley. 1964. https://doi.org/10.1037/e509852009-001 |
[14]
. It refers to and exhibits how the balance of people, processes, and technologies drives successful organizational change, improvements, and re-engineering. Technology provides the tools that people can use to implement
and perform the process. Technology is therefore an essential element to ensure the success of an improvement process. Translating the technology aspect to testing points towards test tools and test environments. There are many types of test tools, but the most popular type is the test automation tool, also referred to as test execution tool, supported by various methods, techniques and frameworks.
In this context, the financial institutions participating in the survey were asked whether they are using test automation for test improvement and, if yes, at which test level(s). The result of this investigation is shown in
Figure 5.
Figure 5. Test automation by test level.
Almost all of financial institutions (91%) also used test automation to improve their testing in parallel with process improvement, thereby accompanying the process improvement activities with supporting technology. Test automation at system level is by far the most popular (85%), followed by integration level (55%) and component level (42%).
This is not in line with the so-called Agile test pyramid
which advocates the idea that test automation should preferably start at component level. The test pyramid layers represent groups of tests. \t "_blank"Component testing serves as the foundation of the test pyramid. Component tests have a limited scope and ensure that isolated code components function as expected. Moreover, automation at unit level is typically both faster and more cost-effective.
The high incidence of test automation with system testing could be explained by analyzing at which test level test process improvement usually starts. In the survey, it was investigated which test levels are within the scope of a test process improvement effort. System testing was again very much in focus and scope at 80%. Both integration testing (47%) and component testing (25%) were substantially less often in scope.
7. Conclusions
In the context of their business needs, many financial institutions have invested in achieving a higher level test maturity. The most common TMMi level achieved by financial institutions is TMMi level 3 “Defined”, with TMMi level 4 “Measured” also having been achieved by 29% of the surveyed financial institutions. Results of the 2024 financial institutions benchmark survey, as reported in the paper, reveal that enhancing software quality, reducing product risk, compliance to an internationally used model, and increasing testing productivity (efficiency) are the main reasons to invest in test maturity and adopt the TMMi model. Most survey respondents reported observing those benefits after adopting TMMi, whereby test automation was largely used to support the test improvement process. The corresponding motivations and benefits were confirmed by the high satisfaction ratio reported in the survey. In answering the survey question, “In general, have the TMMi-based test process improvement efforts been successful?”, 95% of respondents stated that they are either satisfied, very satisfied, or extremely satisfied with the benefits achieved.
Abbreviations
CISA | Certified Information Systems Auditor |
IEEE | Institute of Electrical and Electronics Engineers |
ISTQB | International Software Testing Qualifications Board |
IT | Information Technology |
TMM | Test Maturity Model |
TMMi | Test Maturity Model Integration |
PPT | People, Process and Technology |
Acknowledgments
The author would like to thank all who responded and participated in the survey. A special word of thanks to Frank Frambach who performed the graphical design of the TMMi Benchmark Report Financial Institutions, and to Mattijs Kemmink, Adrian Howes, Anna Miazek, Alfonsina Morgavi, Paul Mowat, Meile Posthuma and Kelly Ye who all reviewed an early draft of this paper.
Author Contributions
Erik Van Veenendaal is the sole author. The author read and approved the final manuscript.
Funding
This work is not supported by any external funding.
Data Availability Statement
The data that supports the findings of this study can be found at: https://www.tmmi.org/tmmi-documents/
Conflicts of Interest
The author declares no conflicts of interest.
References
| [1] |
TMMi Foundation. TMMi model specification (reference model), release 1.3. 2022
www.tmmi.org
|
| [2] |
TMMi Foundation. TMMi 2nd worldwide user survey, version 1.2. 2023
www.tmmi.org
|
| [3] |
TMMi Foundation, TMMi Benchmark Report Financial Domain, version 1.1. 2024
www.tmmi.org
|
| [4] |
Gelperin, D. Hetzel, B. The growth in software testing. Communications of the ACM 31 (6), pp. 687-695. 1988.
https://doi.org/10.1145/62959.62965
|
| [5] |
Burnstein. I. Suwanassart, I, Carlson C. R. Developing a Testing Maturity Model for software process evaluation and improvement. in: Proceedings International Test Conference Test and Design Validity. 1996.
https://doi.org/10.1109/test.1996.557106
|
| [6] |
Farid, A., Fathy, E., Abd, M. Towards Agile Implementation of Test Maturity Model Integration (TMMI) Level 2 using Scrum Practices. in: International Journal of Advanced Computer Science and Applications. 2015.
https://doi.org/10.14569/IJACSA.2015.060931
|
| [7] |
TMMi Foundation, TMMi in the Agile world, version 1.4. 2019
www.tmmi.org
|
| [8] |
Unudulmaz, A., Cingiz, M., Kalıpsız, O. Adaptation of the Four Levels of Test Maturity Model Integration with Agile and Risk-Based Test Techniques. in: Electronics. 2022.
https://doi.org/10.3390/electronics11131985
|
| [9] |
Rungi, K., Matulevicius, R. Empirical analysis of the Test Maturity Model Integration (TMMi). in: Skersys, T., Butleris, R., Butkiene, R. (eds.) ICIST 2013. CCIS, 403, pp. 376–391. Springer Heidelberg. 2013.
https://doi.org/10.1007/978-3-642-41947-8_32
|
| [10] |
Afzal W., Alone S., Kerstin, G., Torkar, R. Software test process improvement approaches: A systematic literature review and an industrial case study. in: Journal of Systems and Software. 2016.
https://doi.org/10.1016/j.jss.2015.08.048
|
| [11] |
Prastiti, R., Hidayanto, A. Evaluation of Maturity Level and Recommendations for Improvement of Software Testing Process Based on Test Maturity Model integration (TMMi): A Case Study. in: Jurnal Komtika (Komputasi dan Informatika), 6(2), pp. 145-154. 2023.
https://doi.org/10.31603/komtika.v7i2.9628
|
| [12] |
Garousi, V., van Veenendaal, E. Test Maturity Model integration (TMMi): trends of worldwide test maturity and certifications, IEEE Software, 39(2), pp. 71-79. 2022.
https://doi.org/10.1109/MS.2021.3061930
|
| [13] |
Garousi, V., Felderer, M., Hacaloglu, T. What we know about software test maturity and test process improvement. IEEE Software, 35(1), pp. 84-92. 2018.
https://doi.org/10.1109/ms.2017.4541043
|
| [14] |
Leavitt, H. J. Applied Organization Change in Industry: structural, technical, and human approaches, in: New perspectives in organization research, pp. 55-71. Wiley. 1964.
https://doi.org/10.1037/e509852009-001
|
| [15] |
Cohn, M. Succeeding with Agile: Software Development Using Scrum. in: Journal of Object Technology. 2010.
https://doi.org/10.5381/jot.2010.9.4.r1
|
Cite This Article
-
-
@article{10.11648/j.ajcst.20240702.13,
author = {Erik Van Veenendaal},
title = {Test Maturity Model integration (TMMi): Test Maturity in the Financial Domain
},
journal = {American Journal of Computer Science and Technology},
volume = {7},
number = {2},
pages = {43-50},
doi = {10.11648/j.ajcst.20240702.13},
url = {https://doi.org/10.11648/j.ajcst.20240702.13},
eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.ajcst.20240702.13},
abstract = {Software quality is of utmost importance to the financial sector. Software testing plays a critical role in achieving software product quality. Financial institutions benefit from rigorous testing by having confidence in the reliability and performance of the software. This can lead to improved customer experience, increased operational efficiency, and reduced risks of system failures or security breaches. A questionnaire-based survey was designed and subsequently an international survey was conducted involving sixty financial institutions, e.g., banking, insurance companies and pension funds, from across the globe to understand their level of test maturity. As a reference framework against which to measure their test maturity, the Test Maturity Model integration (TMMi) was used. In this paper their motivations for doing test process improvement and the benefits they achieved are discussed. Concrete examples of the benefits achieved are provided. The role of test automation with test process improvement at the financial institutions is also reported upon in this paper. The most common level of test maturity achieved, measured against the TMMi, is TMMi level 3 “Defined” which represents a more than average level of test maturity. Benefits are reported by the financial institutions, especially in the areas of software quality and testing productivity. The benefits achieved show a high level of correlation with the motivations for investing in test process improvement. Almost all of financial institutions also use test automation to improve their testing in parallel with process improvement, with test automation at system level being by far the most popular.
},
year = {2024}
}
Copy
|
Download
-
TY - JOUR
T1 - Test Maturity Model integration (TMMi): Test Maturity in the Financial Domain
AU - Erik Van Veenendaal
Y1 - 2024/05/30
PY - 2024
N1 - https://doi.org/10.11648/j.ajcst.20240702.13
DO - 10.11648/j.ajcst.20240702.13
T2 - American Journal of Computer Science and Technology
JF - American Journal of Computer Science and Technology
JO - American Journal of Computer Science and Technology
SP - 43
EP - 50
PB - Science Publishing Group
SN - 2640-012X
UR - https://doi.org/10.11648/j.ajcst.20240702.13
AB - Software quality is of utmost importance to the financial sector. Software testing plays a critical role in achieving software product quality. Financial institutions benefit from rigorous testing by having confidence in the reliability and performance of the software. This can lead to improved customer experience, increased operational efficiency, and reduced risks of system failures or security breaches. A questionnaire-based survey was designed and subsequently an international survey was conducted involving sixty financial institutions, e.g., banking, insurance companies and pension funds, from across the globe to understand their level of test maturity. As a reference framework against which to measure their test maturity, the Test Maturity Model integration (TMMi) was used. In this paper their motivations for doing test process improvement and the benefits they achieved are discussed. Concrete examples of the benefits achieved are provided. The role of test automation with test process improvement at the financial institutions is also reported upon in this paper. The most common level of test maturity achieved, measured against the TMMi, is TMMi level 3 “Defined” which represents a more than average level of test maturity. Benefits are reported by the financial institutions, especially in the areas of software quality and testing productivity. The benefits achieved show a high level of correlation with the motivations for investing in test process improvement. Almost all of financial institutions also use test automation to improve their testing in parallel with process improvement, with test automation at system level being by far the most popular.
VL - 7
IS - 2
ER -
Copy
|
Download
Share This Article
Twitter
Linked In
Facebook
