Background: Organisational performance measurement (OPM) is an evidence-based tool for planning business improvement and creating sustainable competitive advantage. Despite its value, non-acute health charities under utilise OPM. This paper provides rationale for OPM and an OPM implementation model for non-acute health charities. Methods: The authors investigated the understanding and use of OPM in the non-acute health charity sector. A mixed-methods study, including PRISMA systematic reviews and two case study evaluations, identified factors and activities that are important for successful implementation of OPM in non-acute health charities. Findings were then integrated to form an implementation model. Results: The resultant methodological model - the Non-acute Health Charities Measurement Advantage Implementation Model (MAIM), is designed to respond to the specificities and strategic management needs of non-acute health charities. The model integrates five distinct yet inter-connecting components – a theoretical scaffold; guiding principles; an implementation framework; measurement domains; and a monitoring and improvement tool. Conclusions: MAIM is an evidence-informed approach to support the implementation of OPM in the non-acute health charity sector. Future studies should validate the model and the impact of OPM on health charity performance. The applicability of the implementation approach in aiding other management and process changes may also be tested.
Published in | Journal of Public Policy and Administration (Volume 7, Issue 2) |
DOI | 10.11648/j.jppa.20230702.19 |
Page(s) | 109-121 |
Creative Commons |
This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited. |
Copyright |
Copyright © The Author(s), 2023. Published by Science Publishing Group |
Organisational Performance Measurement, Performance, Implementation, Balanced Scorecard, Non-Acute Healthcare, Non-Profit, Charity
[1] | Griseri, P. (2013). An introduction to the Philosophy of Management. Sage Publications. https://doi.org/10.1177/1350507613510835 |
[2] | Jones, G. (2013). Organizational theory, design, and change. (7th edition). Pearson. |
[3] | Gomes, J. & Romão, M. (2019). Sustainable Competitive Advantage with the Balanced Scorecard Approach. In Khosrow-Pour, M., Advanced Methodologies and Technologies in Business Operations and Management (pp.1415-1428). IGI Globa. DOI: 10.4018/978-1-5225-7362-3.ch106. |
[4] | Sharma A. (2009). Implementing Balance Scorecard for Performance Measurement. The ICFAI University Journal of Business Strategy, VI (1), 7-16. |
[5] | Sen, D., & Vayvay, O. (2017). Strategic Enterprise Management for Innovative Companies: The Last Decade of the Balanced Scorecard. International Journal of Asian Social Science, 7 (1): 97-109. |
[6] | Bellante, G., Beradi, L., Machold, S., Nissi, E., & Rea, M. (2017). Accountability, governance and performance in UK charities. International Journal of Business Performance Management, 19 (1). https://doi.org/10.1504/IJBPM.2018.088493 |
[7] | Hyndman, N. & McConville, D. (2018). Trust and accountability in UK charities: Exploring the virtuous circle. The British Accounting Review, 50 (2), 227-237. https://doi.org/10.1016/j.bar.2017.09.004 |
[8] | Boateng, A., Akamavi, R. and Ndoro, G. (2015). Measuring performance of non-profit organisations: evidence from large charities. Business Ethics: A European Review, 25 (1), 59-74. |
[9] | Colbran, R., Ramsden, R., Stagnitti, K., & Toumbourou, J. W. (2019). Advancing towards contemporary practice: a systematic review of organisational performance measures for non-acute health charities. BMC Health Services Research, 19 (132). https://doi.org/10.1186/s12913-019-3952-1 |
[10] | Nalwoga, M., & van Dijk, M. (2016). Organisational performance measurement models, also for poverty alleviation. Int. J. Water, 10 (2/3), 122: 138. https://doi.org/10.1504/IJW.2016.075564 |
[11] | Richard, P., Devinney, T., Yip, S., & Johnson, G. (2009). Measuring organizational performance: towards methodological best practice. Journal of Management, 35 (3), 718–804. https://doi.org/10.1177/0149206308330560 |
[12] | Carneiro-da-Cunha, J., Hourneaux, F. & Correa, H. (2016). Evolution and chronology of the organisational performance measurement field. International Journal of Business Performance Management, 17 (2). https://doi.org/10.1504/IJBPM.2016.075553 |
[13] | Behn, R. D. (2003). Why Measure Performance? Different Purposes Require Different Measures. Public Administration Review, 63 (5), 586-606. DOI: 10.1111/1540-6210.00322. |
[14] | Bititci, U. (2015). Managing Business Performance: The Science and the Art. John Wiley & Sons. DOI: 10.1002/9781119166542. |
[15] | Kaplan, R. (2001). Strategic performance and management in nonprofit organisations. Nonprofit Management and Leadership, 11 (3), 353–70. |
[16] | Aboramadan, M., & Borgonovi, E. (2016). Strategic management practices as a key determinant of superior non-governmental organizational performance. Problems of Management in the 21st Century, 11 (2), 71-92. |
[17] | Seaman, B., & Young, D. (2018). Handbook of Research on Nonprofit Economics and Management. Edward Elgar Publishing. https://doi.org/10.4337/9781785363528 |
[18] | Clancey, G. & Westcott, H. (2017). “This rabid fight for survival”: Small NGO manager's experiences of funding reform. Australian Journal of Social Issues, 52 (2), 163-179. https://doi.org/10.1002/ajs4.9 |
[19] | Lecy, J., Schmitz. H., & Swedlund, H. (2010). Non-governmental and not-for-profit organizational effectivess: a structured literature review. Voluntas, 23, 434–57. |
[20] | Mosadeghrad, A. (2015). Developing and validating a total quality management model for healthcare organisations. The TQM Journal, 27 (5), 544-564. |
[21] | Alharbi, Atkins, Stanier, & Al-Buti (2016). Strategic Value of Cloud Computing in Healthcare Organisations Using the Balanced Scorecard Approach: A Case Study from a Saudi Hospital. Procedia Computer Science, 98, 332-339. https://doi.org/10.1016/j.procs.2016.09.050 |
[22] | Cacciatore, P. Kannengiesser, P., Carini, E., Di Pilla, A., Pezzullo, A., Hoxhaj, I., Gabutti, A., Cicchetti, A., Boccia, S., & Specchia, M. (2019). Balanced Scorecard for performance assessment in healthcare settings: a review of literature. European Journal of Public Health, 29 (4). |
[23] | Grigoroudis E, Orfanoudaki E, Zopounidis C. (2012). Strategic performance measurement in a healthcare organisation: a multiple criteria approach based on balanced scorecard. Omega, 40 (1), 104–19. |
[24] | Soysa, I. B., Jayamaha, N., & Grigg, N. (2016). Operationalising performance measurement dimensions for the Australasian nonprofit healthcare sector. The TQM Journal, (28) 6, 954-973. https://doi.org/10.1108/TQM-08-2015-0109 |
[25] | Mueller, J. (2007). When Doing Good Is Just the Start to Being Good: A Possible Tool to Improve the Organizational Effectiveness of Non-Profit Health Care Organizations. Journal Of Hospital Marketing & Public Relations, 17 (2), 45-60. |
[26] | Hardwick, R., Anderson, R. & Cooper, C. (2015). How do third sector organisations use research and other knowledge? A systematic scoping review. Implementation Sci, 10, 84. https://doi.org/10.1186/s13012-015-0265-6 |
[27] | Cortis, N., Lee, I., Powell, A., Simnett, R., & Reeve, R. (2016) Australian Charities Report 2015. Centre for Social Impact and Social Policy Research Centre. UNSW Australia. |
[28] | Park, Y., & Peng, S. (2019). Advancing Public Health Through Tax-Exempt Hospitals: Nonprofits’ Revenue Streams and Provision of Collective Goods. Nonprofit and Voluntary Sector Quarterly, 49 (2): 357-379. https://doi.org/10.1177/0899764019872007 |
[29] | Hung, C. & Hager, M. (2019). The Impact of Revenue Diversification on Nonprofit Financial Health: A Meta-analysis. Nonprofit and Voluntary Sector Quarterly, 48 (1): 5-27. https://doi.org/10.1177/0899764018807080 |
[30] | Hunter, J. (2007). A systems approach to revenue fluctuations in non-profit human services organizations. Digital Commons @ ACU, Electronic Theses and Dissertations. Paper 61. https://digitalcommons.acu.edu/cgi/viewcontent.cgi?article=1061&context=etd |
[31] | Laamanen, R., Øvretveit, J., Sundell, J., Simonsen-Rehn, N., Suominen, S., & Brommels, M. (2006). Client perceptions of the performance of public and independent not-for-profit primary healthcare. Scandinavian Journal of Public Health, 34 (6), 598-608. |
[32] | Colbran, Ramsden, Stagnitti, Adams. (2017). Measures to assess the performance of an Australian non-government charitable non-acute health service: A Delphi Survey of Organisational Stakeholders. Health Services Management Research. 2018; 31 (1): 11-20. doi: 10.1177/0951484817725681. |
[33] | Colbran, Ramsden, Stagnitti, Toumbourou & Pepin. (2021). A framework to implement organisational performance measurement in health charities. Journal of Public Policy and Administration. 5 (1), pp.13-23. doi: 10.11648/j.jppa.20210501.13. |
[34] | Colbran, Ramsden, Pepin, Toumbourou & Stagnitti. (2022/1). Journal of Public Policy and Administration. 6 (3), pp. 139-150. doi: 10.11648/j.jppa.20220603.14. |
[35] | Colbran, Ramsden, Pepin, Toumbourou & Stagnitti. (2022/2). Staff perceptions of organisational performance measurement implementation in a health charity. Health Services Management Research. 2022; 0 (0). doi: 10.1177/09514848221134403. |
[36] | Moullin, M. (2007). Performance measurement definitions: Linking performance measurement and organisational excellence. International journal of health care quality assurance, 20 (3), 181-183. |
[37] | Colldén, C., & Hellström, A. (2018). Value-based healthcare translated: a complementary view of implementation. BMC health services research, 18 (1), 1-11. |
[38] | Damschroder, L. J. (2020). Clarity out of chaos: use of theory in implementation research. Psychiatry research, 283, 112461. |
[39] | Li, H. & Fu, J. (2019). Application of Balanced Scorecard in Enterprise Strategic Management. International Academic Conference on Frontiers in Social Sciences and Management Innovation (IAFSM 2018). DOI: 10.2991/iafsm-18.2019.28. |
[40] | Gervais, M. (2008). A journey through five evaluation projects with the same analysis framework. Canadian Journal of Program Evaluation, 23 (2), pp. 165-190. |
[41] | Argote, L. (2013). Organizational learning: creating, retaining and transferring knowledge. (2nd Edition). Springer Science and Business Media. |
[42] | Hernaus, T., Škerlavaj, M., & Dimovski, V. (2008). Relationship between organisational learning and organisational performance: The case of Croatia. Transformations in business & economics, 7 (2), 32-48. |
[43] | Ricciardi, F., Cantino, V., & Rossignoli, C. (2020): Organisational learning for the common good: an emerging model. Knowledge Management Research & Practice, https://doi.org/10.1080/14778238.2019.1673676 |
[44] | Bergeron, K., Abdi, S., DeCorby, K., Mensah, G., Rempel, B., & Manson, H. (2017). Theories, models and frameworks used in capacity building interventions relevant to public health: a systematic review. BMC public health, 17 (1), 1-12. https://doi.org/10.1186/s12889-017-4919-y |
[45] | Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Sci 10, 53. https://doi.org/10.1186/s13012-015-0242-0 |
[46] | Field, Booth, Ilott & Gerrish. 2014. Using the Knowledge to Action Framework in practice: a citation analysis and systematic review. Implementation Science. 9: 172 https://implementationscience.biomedcentral.com/articles/a0.1186/s13012-014-0172-2 |
[47] | Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: systematic review and recommendations. The Milbank Quarterly, 82 (4), 581-629. https://doi.org/10.1111/j.0887-378X.2004.00325.x |
[48] | Rycroft-Malone. 2004. The PARiHS Framework – A Framework for Guiding the Implementation of Evidence-based Practice. Journal of Nursing Care Quality. 1 (4). 297-304. |
[49] | Garcia-Morales, V., Jimenez-Barrionuevo, M., & Gutierrez-Gutierrez, L. (2012). Transformational leadership influence on organisational performance through organisational learning and innovation. Journal of Business Research, 65, 1040-1050. https://doi.org/10.1016/j.jbusres.2011.03.005 |
[50] | Prugsamatz, R. (2010). Factors that influence organization learning sustainability in non-profit organizations. The Learning Organization, 17 (3), 243-267. doi: 10.1108/09696471011034937. |
[51] | Murray, P. (2002). Cycles of organisational learning: a conceptual approach. Management Decision, 40 (3), 239-247. doi: 10.1108/00251740210420192. |
[52] | Bisbe, J. and Barrubés, J. (2012). The Balanced Scorecard as a Management Tool for Assessing and Monitoring Strategy Implementation in Health Care Organizations. Revista Española de Cardiología (English Edition), 65 (10), 919-927. |
[53] | Geerligs, L., Rankin, N. M., Shepherd, H. L., & Butow, P. (2018). Hospital-based interventions: a systematic review of staff-reported barriers and facilitators to implementation processes. Implementation Science, 13 (1), 1-17. |
[54] | Morris, N. S., & Rosenbloom, D. A. (2017). Defining and understanding pilot and other feasibility studies. The American journal of nursing, 117 (3), 38-45. |
[55] | Eldridge, S. M., Lancaster, G. A., Campbell, M. J., Thabane, L., Hopewell, S., Coleman, C. L., & Bond, C. M. (2016). Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PloS one, 11 (3), e0150205. |
[56] | Bowen, D. J., Kreuter, M., Spring, B., Cofta-Woerpel, L., Linnan, L., Weiner, D.,... & Fernandez, M. (2009). How we design feasibility studies. American journal of preventive medicine, 36 (5), 452-457. |
[57] | Argote, L., & Miron-Spektor, E. (2011). Organizational learning: From experience to knowledge. Organization science, 22 (5), 1123-1137. |
[58] | Jyoti, J., & Rani, A. (2017). High performance work system and organisational performance: Role of knowledge management. Personnel Review. |
[59] | Graham, Logan, Harrison, Strauss, Tetroe, Caswell, Robinson. 2006. Lost in Knowledge Translation: Time for a Map? The Journal of Continuing Education in the Health Professions. 26. 13–24. |
[60] | Kitson, A., Harvey, G., & McCormack, B. (1998). Enabling the implementation of evidence based practice: a conceptual framework. Quality in Health Care, 7, 149-158. DOI: 10.1136/qshc.7.3.149. Retrieved from http://qualitysafety.bmj.com/content/7/3/149.abstract |
[61] | Metz, A. & Bartley. L. (2012). Active implementation frameworks for program success. Zero to One. 32: 11–8. |
[62] | Zimmerman, J. (2009). Using a balanced scorecard in a nonprofit organization. Nonprofit World, 27, 10–12. |
[63] | Lueg, R. and Vu, L. (2015). Success factors in Balanced Scorecard implementations – A literature review. Management Revue.26 (4): 306-327. |
[64] | Buchanan D, Fitzgerald L, Ketley D, Gollop R, Jones JL, Lamont SS and Whitby E. (2005). No going back: a review of the literature on sustaining organizational change. International Journal of Management Reviews.7 (3): 189-205. |
[65] | Smith, P., Mossialos, E., & Papanicolas, I. (2008) Performance measurement for health system improvement: experiences, challenges and prospects. WHO European Ministerial Conference on Health Systems, Tallin, Estonia. Retrieved October 10, 2018, from http://www.who.int/management/district/performance/PerformanceMeasurementHealthSystemImprovement2.pdf |
[66] | Schalm, C. (2008). Implementing a balanced scorecard as a strategic management tool in a long-term care organization. Journal of Health Services Research & Policy, 1 (13), 8-14. https://doi.org/10.1258/jhsrp.2007.007013 |
[67] | Tarigan, S., & Bachtiar, A. (2019, August). A REVIEW OF THE BALANCED SCORECARD IMPLEMENTATION IN HEALTH SECTOR ORGANIZATION. In Proceedings of the International Conference on Applied Science and Health (No. 4, pp. 942-949). |
[68] | Kaplan, R. & Norton, D. (1992) The Balanced Scorecard: Measures that Drive Performance Harvard Business Review, January-February, 71-79. |
[69] | Paranjape, B., Rossiter, M., Pantano, V. (2006). Performance measurement systems: successes, failures and future – a review. Measuring Business Excellence.10 (3), 4-14. |
[70] | Inamdar, N., & Kaplan, R. (2002). Applying the Balanced Scorecard in Healthcare Provider Organizations. Journal of Healthcare Management, 47 (3), 179-195. |
[71] | Odor, H. (2018). A literature review on organisational learning and learning organisations. International Journal of Economics & Management Sciences, 7 (1), 494. |
[72] | Guta, A, L. (2015). An analysis of factors that influence organisational learning: The case of higher education institutions. 15th International Academic Conference, Rome. doi: 10.20472/IAC.2015.015.067. https://www.iises.net/proceedings/international-academic-conference-rome/table-of-content?cid=10&iid=067&rid=3387 |
[73] | León, C., & Bousquet, B. (2018). A Performance Measurement Framework for NPOs. 2018 IISE Annual Conference. Retrieved October 25, 2019, from https://www.researchgate.net/profile/Hilda_Martinez_Leon/publication/325581735_A_Performance_Measurement_Framework_for_NPOs/links/5bc67a50299bf17a1c55d563/A-Performance-Measurement-Framework-for-NPOs.pdf |
[74] | National Collaborating Centre for Methods and Tools [NCCMT]. (2020). PARiHS framework for implementing research into practice. Retrieved October 18, 2019, from https://www.nccmt.ca/knowledge-repositories/search/85 |
[75] | Macmillan, R., Ellis Paine, A., Kara, H., Dayson, C., Sanderson, E. & Wells, P. (2014) Building capabilities in the voluntary sector: What the evidence tells us. TSRC Research Report 125, Birmingham: TSRC. Retrieved October 10, 2018, from https://www.birmingham.ac.uk/Documents/college-social-sciences/social-policy/tsrc/reports/research-report-125-building-capabilities.pdf |
[76] | Lowell S, Silverman L, and Taliento L. (2001). Not-for-profit management: The gift that keeps on giving. The McKinsey Quarterly. 147. |
[77] | Cairns, B., Harris, M. & Young, P. (2005). Building the Capacity of the Voluntary Nonprofit Sector: Challenges of Theory and Practice. Intl Journal of Public Administration, 28, 869-885. https://doi.org/10.1081/PAD-200067377 |
[78] | Schmitz, J. (2019). Is Charitable Giving a Zero Sum Game?-The Effect of Competition Between Charities on Giving Behavior. The Effect of Competition Between Charities on Giving Behavior. Management Science. Forthcoming. http://dx.doi.org/10.2139/ssrn.2862479 |
[79] | Archambault, E. (2017). The Evolution of Public Service Provision by the Third Sector in France. Political Quarterly, Wiley-Blackwell: No OnlineOpen, 88 (3), 465-472. |
[80] | Janus, S. S. (2016). Becoming a knowledge-sharing organization: A handbook for scaling up solutions through knowledge capturing and sharing. The World Bank. https://openknowledge.worldbank.org/bitstream/handle/10986/25320/9781464809439.pdf |
[81] | Duman, G. M., Taskaynatan, M., Kongar, E., & Rosentrater, K. A. (2018). Integrating environmental and social sustainability into performance evaluation: A Balanced Scorecard-based Grey-DANP approach for the food industry. Frontiers in nutrition, 5, 65. |
[82] | Beheshtinia, M. A., & Omidi, S. (2017). A hybrid MCDM approach for performance evaluation in the banking industry. Kybernetes. |
[83] | Neely, A., Adams, C., & Kinnerley, M. (2017). The Performance Prism: The Scorecard for Measuring and Managing Business Success. Cranfield School of Management |
[84] | International Academic Conference on Frontiers in Social Sciences and Management Innovation. https://www.atlantis-press.com/proceedings/iafsm-18/55915249 |
[85] | Gurd, B. & Gao, T. (2007). Lives in the balance: an analysis of the balanced scorecard (BSC) in healthcare organizations. International Journal of Productivity and Performance Management, 57 (1), 6-21. |
APA Style
Richard Colbran, Robyn Ramsden, Genevieve Pepin, John W Toumbourou, Karen Stagnitti. (2023). Creating Measurement Advantage for Health Charities – The Why and the How. Journal of Public Policy and Administration, 7(2), 109-121. https://doi.org/10.11648/j.jppa.20230702.19
ACS Style
Richard Colbran; Robyn Ramsden; Genevieve Pepin; John W Toumbourou; Karen Stagnitti. Creating Measurement Advantage for Health Charities – The Why and the How. J. Public Policy Adm. 2023, 7(2), 109-121. doi: 10.11648/j.jppa.20230702.19
AMA Style
Richard Colbran, Robyn Ramsden, Genevieve Pepin, John W Toumbourou, Karen Stagnitti. Creating Measurement Advantage for Health Charities – The Why and the How. J Public Policy Adm. 2023;7(2):109-121. doi: 10.11648/j.jppa.20230702.19
@article{10.11648/j.jppa.20230702.19, author = {Richard Colbran and Robyn Ramsden and Genevieve Pepin and John W Toumbourou and Karen Stagnitti}, title = {Creating Measurement Advantage for Health Charities – The Why and the How}, journal = {Journal of Public Policy and Administration}, volume = {7}, number = {2}, pages = {109-121}, doi = {10.11648/j.jppa.20230702.19}, url = {https://doi.org/10.11648/j.jppa.20230702.19}, eprint = {https://article.sciencepublishinggroup.com/pdf/10.11648.j.jppa.20230702.19}, abstract = {Background: Organisational performance measurement (OPM) is an evidence-based tool for planning business improvement and creating sustainable competitive advantage. Despite its value, non-acute health charities under utilise OPM. This paper provides rationale for OPM and an OPM implementation model for non-acute health charities. Methods: The authors investigated the understanding and use of OPM in the non-acute health charity sector. A mixed-methods study, including PRISMA systematic reviews and two case study evaluations, identified factors and activities that are important for successful implementation of OPM in non-acute health charities. Findings were then integrated to form an implementation model. Results: The resultant methodological model - the Non-acute Health Charities Measurement Advantage Implementation Model (MAIM), is designed to respond to the specificities and strategic management needs of non-acute health charities. The model integrates five distinct yet inter-connecting components – a theoretical scaffold; guiding principles; an implementation framework; measurement domains; and a monitoring and improvement tool. Conclusions: MAIM is an evidence-informed approach to support the implementation of OPM in the non-acute health charity sector. Future studies should validate the model and the impact of OPM on health charity performance. The applicability of the implementation approach in aiding other management and process changes may also be tested.}, year = {2023} }
TY - JOUR T1 - Creating Measurement Advantage for Health Charities – The Why and the How AU - Richard Colbran AU - Robyn Ramsden AU - Genevieve Pepin AU - John W Toumbourou AU - Karen Stagnitti Y1 - 2023/06/27 PY - 2023 N1 - https://doi.org/10.11648/j.jppa.20230702.19 DO - 10.11648/j.jppa.20230702.19 T2 - Journal of Public Policy and Administration JF - Journal of Public Policy and Administration JO - Journal of Public Policy and Administration SP - 109 EP - 121 PB - Science Publishing Group SN - 2640-2696 UR - https://doi.org/10.11648/j.jppa.20230702.19 AB - Background: Organisational performance measurement (OPM) is an evidence-based tool for planning business improvement and creating sustainable competitive advantage. Despite its value, non-acute health charities under utilise OPM. This paper provides rationale for OPM and an OPM implementation model for non-acute health charities. Methods: The authors investigated the understanding and use of OPM in the non-acute health charity sector. A mixed-methods study, including PRISMA systematic reviews and two case study evaluations, identified factors and activities that are important for successful implementation of OPM in non-acute health charities. Findings were then integrated to form an implementation model. Results: The resultant methodological model - the Non-acute Health Charities Measurement Advantage Implementation Model (MAIM), is designed to respond to the specificities and strategic management needs of non-acute health charities. The model integrates five distinct yet inter-connecting components – a theoretical scaffold; guiding principles; an implementation framework; measurement domains; and a monitoring and improvement tool. Conclusions: MAIM is an evidence-informed approach to support the implementation of OPM in the non-acute health charity sector. Future studies should validate the model and the impact of OPM on health charity performance. The applicability of the implementation approach in aiding other management and process changes may also be tested. VL - 7 IS - 2 ER -