PAUL J. AMANI¹*, RICHARD NGOWI¹, AND SANDRA DRATLER²
¹Department of Health Systems Management, Mzumbe University, Morogoro, United Republic of Tanzania
²School of Public Health, University of California, Berkeley, Berkeley, California, United States of America
The Mzumbe University MSc Health Monitoring and Evaluation (M&E) Program, launched in 2013, is aimed at building human resources capacity in Tanzania and East Africa by developing high-caliber M&E professionals to provide leadership in public health systems and the private sector. This case study aimed to describe the process of establishing the program and report results of an interim evaluation.
The program’s innovative curriculum focuses on enabling graduates to implement M&E systems that respond to organizational challenges. Program content was driven by results of a training needs assessment, which sought input from stakeholder organizations. The curriculum is competency-based with content determined by learning objectives established for each course. It is delivered over three academic semesters followed by a fourth semester during which students implement field projects that serve as their master’s dissertations. The program draws on outside faculty and M&E practitioners to assure that student learning is balanced between theory and practice. Graduates are expected to assume influential positions in their workplaces, including public and private health organizations, hospitals, central and local government health authorities as well as international health organizations.
The MSc Health M&E Program at Mzumbe University serves as a model for sustainable human resources for health M&E capacity and development in the East African region and beyond. The interim evaluation validated the success of the methods used in program development and implementation. Student feedback indicates that they have acquired the required skills to serve successfully in M&E leadership positions post-graduation.
In 1992 the Government of Tanzania undertook health sector reforms to ensure availability of qualified human resources for health, to facilitate effective management of health care systems, and to support decentralization of powers to local government authorities and sustainability in health care financing. The goals of the National Health Policy were to improve the health and well-being of all Tanzanians, with a focus on those most at risk, and to facilitate the provision of equitable, quality, affordable, gender-sensitive basic health care services. In order to sustain itself, the reform implementation process requires an effective monitoring and evaluation (M&E) system to track its performance and show its results. The M&E system’s goal is to ensure that services are delivered within a dynamic health system that is responsive to changing environments and the needs of the people who use the services (Ministry of Health and Social Welfare [MoHSW], 1992).
More recently, the Ministry of Health, Community Development, Gender, Elderly and Children (MOHCDGEC, formerly Ministry of Health and Social Welfare) recognized the importance of strategic information systems and the accompanying need for M&E and articulated this need in its health systems plan (United Republic of Tanzania, 2015). The focus as stated in the plan is on evidence-based decision-making and enhanced public accountability. Accomplishment of these goals requires a cadre of skilled professionals able to conduct M&E activities and to provide leadership in implementing strategic information systems at the organizational, district and national levels.
Numerous studies have reported a gap in the number and qualifications of M&E professionals for the health sector in resource-limited settings (United Republic of Tanzania, 2014; Mpofu et al., 2014). A training needs assessment conducted in Tanzania in May 2012 confirmed this shortage within the government and at public health-related non-governmental organizations (NGOs). The needs assessment also identified training gaps that would keep the health system goals from being fully realized.
The ongoing need for strong M&E is further affirmed by the Global Health Sector Strategies 2016-2021 being developed by the World Health Organization to support the newly-adopted health-related Sustainable Development Goals. They identify five strategic directions aimed at ‘eliminating diseases as public health threats’. Importantly, the first strategic direction calls for establishing and using strategic information systems that will provide focus and accountability and be used ‘as a basis for advocacy, political commitment, national planning, resource mobilization and allocation, implementation, and program improvement’ (World Health Organization [WHO], 2015a).
The use of strategic information gathered from program M&E efforts allows governments and other organizations to strengthen analysis, align reporting across programs and provide accountability for achievement of key indicators and the resulting outcomes (WHO, 2015b). Core to successful strategic information systems are robust M&E processes being conducted by well-trained professionals who bring a broad view of the health system to their work.
It is in this environment that Mzumbe University, in collaboration with the University of California, San Francisco (UCSF) in the USA and the former Ministry of Health and Social Welfare, Tanzania (current MOHCDGEC), decided to establish a Master of Science in Health Monitoring and Evaluation (MSc HME) program in October 2012.
In 2011, leadership of the School of Public Administration and Management at Mzumbe University understood the potential to provide graduate-level education as a means to fill the gap between available and required M&E personnel in Tanzania. They committed to creating a master’s degree program focused on graduating individuals to lead these efforts. The MOHCDGEC actively joined in the planning to establish the MSc HME program. The decision was taken to house the program in the Department of Health Systems Management with mentorship from the UCSF Global Health Sciences Group. Agreement was reached among the three collaborators in 2012 to move forward in establishing the program.
Mzumbe University is located in Morogoro, a city of 320,000 individuals about 200 kilometers inland from Dar es Salaam. The university was established by charter in 2007 and is a public university operating under the Ministry of Education and Vocational Training. The university and its predecessor institutions boast over 50 years of experience training professionals in the areas of management, administration and good governance. Mzumbe University’s School of Public Administration and Management was established in 2001 and has four academic departments: Human Resource Management, Local Government Management, Health Systems Management and Public Service Management.
Training needs assessment
To assure proper design for the envisaged MSc HME, Mzumbe University collaborated with UCSF to conduct a training needs assessment in May 2012. The primary aims of the needs assessment were (1) to assess the existing M&E training gaps; (2) to determine demand for master’s-trained M&E professionals by employers and employees in the health sector and other health-related projects; and (3) to seek input on the competencies that should be included in an M&E master’s degree program. Five regions were selected using convenience sampling to collect representative data from every zone in the country. Mwanza represented the western zone, Mbeya the southern zone, Dar Es Salaam the eastern zone, Dodoma the central zone, and Arusha the northern zone (Table 1). Interviews were conducted with participants from the five regions and included M&E personnel and potential employers of M&E personnel from the regions, districts, health management teams, health projects and NGOs. Two structured interview guides were developed, one to be used with current M&E personnel and the other to be used with potential employers of M&E personnel. The interview guides were used together with an M&E competency list. Of the 164 potential respondents identified, 84 were interviewed. The remainder could not be reached or were unavailable for interviews.
The results of the training needs assessment confirmed the need for well-trained M&E graduates, but also identified a system-level challenge. Almost all respondents conducted M&E activities as part of their responsibilities; however, the majority of them indicated that they had inadequate M&E training. When asked to state their understanding of what M&E is and how it is done, most of them expressed that training is highly needed as they are not sure if the way they did things was correct. Suggestions for training topics included preparation of M&E plans, methods to effectively monitor and evaluate programs, creating monitoring tools, setting indicators and milestones and monitoring shortage of health workers and preventive and curative services. At the time the interviews were conducted, there were no clearly designated roles or career ladders for M&E personnel in the public sector.
Competency-based training model
The results of the needs assessment were used to determine the program structure and method of delivery for the MSc HME, as well as to provide input when developing the curriculum. The collaborators identified the need for an innovative educational approach to train future M&E leadership. The decision was made to base the program on a competency-based education model, also known as outcomes-based education (Council on Education for Public Health, 2011). Research has shown this model to be successful in educating public health professionals (Gruppen, Mangrulkar, & Kolars, 2012; Calhoun, Ramiah, Weist, & Shortell, 2008). At its core, competency-based education focuses on what the students need to know and the skills they need to demonstrate to perform at a high level in complex public health environments. Systems-level thinking is emphasized (Fried, 2015). Table 2 displays the M&E competencies identified from the training needs assessment and the literature that were used as the basis for curriculum design (UNAIDS, 2010; Negandhi et al., 2015).
While many of the M&E competencies are technical in nature (e.g., research, analysis, information technology, project management), the desire to produce leaders and not just graduates with technical skills led the planning group to use a framework developed by the US-based National Center for Health Leadership (National Center for Health Leadership, 2016), which focuses at a strategic and behavioral level. This framework calls for developing competencies in areas that allow graduates to transform their organizations and institutions by translating vision and strategy into optimal organizational performance. Emphasis is placed on working through people to bring about change.
The design of the MSc HME curriculum shifts away from the traditional, lecture-heavy mode of instruction commonly found in the education system in Tanzania to a more student-centered learning approach that draws from students’ real world experiences and utilizes adult learning methods. The approach engages students in a dialogue with experts, which encourages both drawing on their existing experience and synthesizing this experience into a new understanding of a problem and actionable solutions. The method of teaching demands active learning and student engagement both in the classroom and in the community. Student-centered learning relies heavily on small group activities, the use of case studies and internalization of key issues through constant dialogue. These case studies, discussions of issues and collaborative teamwork require students to learn problem-solving skills and to relate theoretical learning to the experiences they have had and will have in the field. Experiential learning that brings the students in contact with M&E professionals also enhances classroom teaching.
To perform as leaders in health M&E, students need to come away from a graduate-level program with competence in the most relevant traditional core areas of public health, including biostatistics, epidemiology, health policy and management and social and behavioral health. In addition, research and evaluation skills are required. Workforce needs, as defined by the profession and potential employers, were used to determine the specific skills and educational outcomes within these areas that will be used to measure the graduates’ success. Objectives developed for each course within the curriculum measure the outcomes of student learning.
Curriculum development and structure
Faculty members from Mzumbe University and their counterparts from UCSF, along with representatives from MOHCDGEC and experts from other higher learning institutions within Tanzania, including the University of Dar es Salaam (UDSM) and Muhimbili University of Health and Allied Sciences (MUHAS), collaborated to develop the curriculum and course structure. The curriculum content is based on the requirements of the Tanzanian national M&E system and results of the training needs assessment and incorporates the competencies expected of graduates. In addition, the organizing framework takes into consideration the twelve main components of a strong and functional M&E system (UNAIDS 2010). Instructional materials for several of the courses addressing core competencies were developed in collaboration with faculty and consultants from UCSF.
The program consists of three semesters of classroom training spanning a twelve-month period followed by another twelve months dedicated to an M&E-related field project and preparation of the dissertation (Table 3). Candidates must complete a total of 180 credit points: 140 credit points for completing course work and 40 credit points for implementing the field project and completing the dissertation.
The program includes 14 courses that provide comprehensive coverage of key components of monitoring and evaluation plus a broader view of leadership and the health system. For each course, students are required to apply course concepts in a term paper, many of which are designed to help students prepare for their M&E field projects. Semester one courses provide students with a foundational understanding of the approaches to health monitoring and evaluation as well as basics in public health systems. The second, more complex, intermediate semester consists of courses in quantitative, qualitative and economic methods, along with data management and use. The third semester includes biostatistics and qualitative analysis as well as leadership-focused courses, including ethics.
The students take all classes together. All courses are required, with no electives. The curriculum is delivered primarily in the classroom and includes tests, university end-of-semester examinations, case studies, current events and various assignments. The classes are taught by Mzumbe University lecturers with the involvement of faculty from UCSF, MUHAS, Kilimanjaro Christian Medical University and UDSM. The objective of this co-teaching model is to ensure the success and sustainability of the program as well as to capacitate the Mzumbe University faculty as they transition to the competency-based educational model. Guest speakers and outside lecturers are invited based on the individual course requirements. In the initial phases of the program, the cost of bringing these experts to Mzumbe University has been supported by outside funding.
Following the third semester of coursework, students concentrate their efforts on their field projects, culminating in a dissertation. Each student is assigned a faculty mentor who works with them as they design their dissertation proposals, implement their research projects and complete their final papers. Proposal development is also integrated into and supported by the coursework students take throughout the first three semesters. The students have until June of the year following the completion of their coursework to complete and submit their M&E field project dissertations. Students are required to present their M&E field projects orally in September of their second year.
A resource center has been assigned to the program by the university. It includes a library as well as internet-linked workstations for the students. It is configured using round tables that facilitate group interaction.
Incoming students are required to have a degree in a health-related discipline and two years of relevant work experience to be admitted to the program. The courses are designed to build on their existing capacity and their exposure to implementing M&E components. Courses orient them to the application of these components in response to organizational problems related to strategic information and program planning and implementation, with a focus on using M&E to improve the quality of both. At the beginning of the program, students are organized into groups of 5-6 that study and work on projects together throughout the academic semesters.
Recruitment announcements for the first cohort admitted at the program inauguration in October 2013 were published on the Mzumbe University website as well as in local newspapers. The program received a total of 103 applications from individuals representing a variety of public and private organizations within Tanzania. All applications were reviewed, and 49 candidates (31 men and 18 women) were found qualified to join the program. However, a decision was made to keep the size of the first cohort small, with a total cohort size of 35 (22 men and 13 women). The students were employed in organizations ranging from local NGOs to the ministries of health in Tanzania and Zanzibar (Table 4). Half of the students (17) were sponsored (tuition fees paid) by their employers. The majority of those being sponsored were employees of one of the ministries of health.
The MSc HME program received funding from CDC-PEPFAR for initiation of the program, including paying honoraria to faculty and guest lecturers, paying program administrators’ salaries, refurbishing classrooms, covering costs of students’ fees and scholarships, purchasing books and funding students’ field trips. However, this external funding is only committed through a transition period and future funding for the program is uncertain.
An independent interim evaluation of the MSc HME program was commissioned towards the end of the first year of program implementation. The evaluation aimed to assess program performance, operations and results. The specific evaluation questions were: Do planned courses, programs, activities and learning opportunities as developed and organized produce desired results? How can the MSc program be improved?
A mixed method approach was used to collect data for the interim evaluation, including qualitative and quantitative methods. Data were collected through interviews with stakeholders, faculty members, and external lecturers; focus group discussions; document analysis; classroom observations; visits to physical facilities; and an email-based student survey.
The majority of students (92%) agreed that the program had prepared them to serve as M&E professionals in a health care setting. Students agreed that the faculty members were successfully implementing the competency-based education model on which the classes were designed. They cited the effective teaching methods that enhanced their learning and called for critical thinking. They noted the fact that they are encouraged to raise questions and make comments. They also indicated that faculty used technology effectively.
The evaluator concluded that the rationale for establishing the program was strong and evidence-based. Support by key stakeholders was cited as a strength. The process undertaken to create the curriculum was listed as a key success factor, with the highly participatory process involving Mzumbe University faculty, key stakeholders, M&E experts from within Tanzania and external technical assistance contributing to the strong content and delivery. The ongoing involvement of outside stakeholders is seen as contributing to strong and competent Mzumbe University faculty who are actively engaged in the program. The evaluator found that the program is viewed as innovative and rewarding by other university faculty members who are adopting competency-based education components in their own classes.
The evaluator cited the lack of a structured career path, as there are not established jobs within the central and local government authorities in Tanzania for graduates of the MSc HME program, despite the fact that the demand for M&E skills is very high. However, plans are underway within the Ministry of Health to establish M&E jobs so that graduates of the program can be hired in the government system. The evaluation also revealed a need for continuing education in M&E, as some faculty expressed a lack of experience in this field.
When queried regarding weaknesses, the students expressed a desire for more field visits and exposure to practice. Nearly half felt that the curriculum included too many courses, which created more burden than they had anticipated. A challenge noted by both faculty and students is the need to comply with Mzumbe University requirements for term papers and final examinations when other means of measuring student educational outcomes in a competency-based environment would be more appropriate.
In December 2015, Mzumbe University faculty emailed a survey to the first cohort of 35 students, of whom 34 successfully graduated from the program. Of the 22 who responded, most reported working within the public health system of Tanzania and nearly 20% had sought and found employment within organizations different than those that employed them before they entered the program. The graduates’ motivation for changing jobs was a desire to use their newly-acquired M&E skills. In addition, some of the MSc HM&E graduates have been contracted as M&E officers and consultants in both private and faith-based health care organizations. Over 80% of respondents indicated that they were conducting M&E activities within their organizations. Data analysis and research and evaluation were cited as the most valued skills acquired through their training. They also indicated that they were teaching the skills to others.
The implementation of the MSc HME program at Mzumbe University in Tanzania is a successful model for building capacity for applied health M&E in Tanzania and East Africa. The innovative competency-based education model has set an example for other departments in the university. The curriculum enables students to design and implement M&E systems that respond to the changing needs of their organizations and governments. The evaluation found that that the program had prepared graduates to successfully serve as M&E professionals in health care settings, as all graduates of the first cohort were working within the public health system of Tanzania.
Although there has been great success and progress in planning, launching and implementing the MSc HME program, the greatest challenge remains financial sustainability once outside funding for the project comes to an end. Mzumbe University will be required to identify internal funds to finance the costs for their own lecturers as well as guest lecturers and speakers. Maintenance of the high quality of teaching and learning cited by the students will depend on the program’s ability to retain faculty and continue to strengthen the capabilities required for student-centered teaching.
Because outside funding is only committed for a transition period, Mzumbe University faculty have developed a business plan with specific actions aimed at generating non-tuition revenue that will ensure the sustainability of the program going forward. At the core of this revenue model is the Centre of Excellence in M&E which was launched at Mzumbe University in September 2015. The Centre will contribute to research, dissemination, expertise and teaching in the field of M&E in Tanzania and beyond. This Centre of Excellence in M&E supports a network of M&E professionals in Tanzania through a hosted website; offers additional academic programs (e.g., short courses and executive education); supports an annual M&E conference; houses and supports faculty who conduct M&E research and who can serve as a pool of consultants for national and regional M&E consultancies; and is launching an M&E/Health Journal for East Africa in December 2016. These activities can generate revenue that will be used for faculty training and development and other program costs. Business planning efforts are ongoing to review potential opportunities continually.
Along with university financing, other factors that will ensure sustainability of the program and its graduates are: (1) hiring of the MSc HME graduates in the MOHCDGEC schemes of services as well as in the local government authorities and hospitals; (2) well-defined job functions that allow M&E professionals within the MOHCDGEC to practice skills acquired during the master’s training; and (3) a career ladder that rewards and recognizes M&E training as a professional and technical field in the central and local government authorities so that the graduates can be employed in the public health systems like other professions.
It is anticipated that MSc HME graduates will act as M&E change agents and, therefore, contribute greatly to improved health policies and strengthened health care systems in Tanzania, East Africa and beyond. The MSc HME graduates will help with improving the quality of data collected and therefore lead to improved quality of health care services. Finally, as Mzumbe University takes full ownership of the MSc HME program and institutionalizes it, the competency-based teaching model is increasingly being adopted for teaching other postgraduate programs within the university.
The successful implementation of the MSc HME Program at Mzumbe University is making an important contribution to human resources for health monitoring and evaluation in Tanzania and can serve as a model for other universities wanting to contribute to the strengthening of health systems in their countries.
The authors wish to acknowledge the hard work and commitment of the MSc HME students as well as UCSF Global Strategic Information, the CDC in Atlanta, Georgia, USA, faculty members in the Mzumbe University Department of Health Systems Management, and the MOHCDGEC Tanzania, who have been responsible for the success of the program. They would also like to give a special debt of gratitude to Issa Mbaga, Renatus Mashuri, Claud John, Anna Nswilla, Roger Myrick, Susie Welty, Christen Said, Karen White, Jenny Tiberio, Angela Makota, Gongo Ramadhani, Vamsi Vasireddy, Hannah Tesfayone, Elaine Baker, Jim Todd, Jenny Renju, Sia Msuya, and Tulli Tuhuma for their outstanding contributions to the MSc HME Program development and implementation. Thanks to Jan van Esch who conducted the external evaluation. Finally, the program is thankful for the funding provided by CDC-PEPFAR.
The authors declare that they have no competing interests.
All authors contributed to the conceptualization, drafting and finalization of this manuscript.
Calhoun, J.G., Ramiah, K., Weist, E.M., Shortell, S.M. (2008). Development of a core competency model for the Master of Public Health degree. American Journal of Public Health, 9, 1598-607. doi: 10.2105/AJPH.2007.117978.
Fried, L.P. (2015). Innovating for 21st-century public health education: A case for seizing this moment. American Journal of Public Health, 105, S5-S7. doi: 10.2105/AJPH.2014.302549
Gruppen, L.D., Mangrulkar, R.S., & Kolars, J.C. (2012) . The promise of competency-based education in the health professions for improving global health. Human Resources for Health, 10, doi: 10.1186/1478-4491-10-43
Mpofu, M., Semo, B., Grignon, J., Lebelonyane, R., Ludick, S., Matshediso, E., …Ledikwe, JH. (2014). Strengthening monitoring and evaluation (M&E) and building sustainable health information systems in resource limited countries: lessons learned from an M&E task-shifting initiative in Botswana. BMC Public Health, 14, doi: 10.1186/1471-2458-14-1032
Negandhi, H., Negandhi, P., Tiwan, R., Sharma, A., Zodpey, S., Kulatilaka, H., …Tikyani, S. (2015). Developing core competencies for monitoring and evaluation tracks in South Asian MPH programs. BMC Medical Education, 15. doi: 10.1186/s12909-015-0403-5
United Republic of Tanzania. (2014). Global AIDS response country progress report. Retrieved from http://www.unaids.org/sites/default/files/country/documents/TZA_narrative_report_2014.pdf
United Republic of Tanzania. (2015). Health sector strategic plan July 2015-June 2020 (HSSP IV), Retrieved from http://www.tzdpg.or.tz/fileadmin/documents/dpg_internal/dpg_working_groups_clusters/cluster_2/health/Key_Sector_Documents/Induction_Pack/Final_HSSP_IV_Vs1.0_260815.pdf
World Health Organization (2015a). Global health sector strategies 2016-2021, Briefing Note. Retrieved from http://www.who.int/reproductivehealth/GHSS_Briefing_Note_Final_October2015.pdf
World Health Organization. (2015b). Consolidated strategic information guidelines for HIV in the health sector. Retrieved from http://apps.who.int/iris/bitstream/10665/164716/1/9789241508759_eng.pdf?ua=1&ua=1
All inquiries can be sent via email to: firstname.lastname@example.org
IN COLLABORATION WITH
The Ministry of Health, Community Development, Gender, Elderly and Children.