JENNY TIBERIO¹*, CHRISTEN SAID¹, JOEL NDAYONGEJE², DELFINUS KIVENULE³, TEOPISTA KOMBA², AND SUSIE WELTY¹
¹Global Health Sciences, University of California, San Francisco, California, United States of America
²Global Programs, Dar es Salaam, United Republic of Tanzania
³Tanzania Commission for AIDS, Dar es Salaam, United Republic of Tanzania
Background: Quality data is critical for public health decision-making; however, adequate, high-quality data required to answer public health questions is not always available, particularly at sub-national levels. Triangulation is the practice of arraying different types of existing data from diverse sources to answer public health questions.The Tanzanian Triangulation Project was implemented to enhance understanding of the HIV epidemic at sub-national levels and promote a culture of data use among providers.
Methods:We conducted triangulation workshops with regional-, district-, and facility-level HIV/AIDS service providers. Workshops included building capacity in MS Excel, DHIS2, and mapping software to produce visual data displays; support in carrying out the triangulation process; and the development of regional HIV triangulation reports to assess and address challenges affecting service provision.
Results:Twenty-one workshops were completed covering 24 regions of Tanzania (mainland and Zanzibar) from 2009-2016. Participants were enthusiastic about seeing their data displayed visually in tables, charts and maps, often for the first time. Every workshop had data quality challenges and resulted in recommendations to improve data quality. Most (83%) participants reported learning new Excel skills, and among those, 86%reported using these skills after the workshop. Decision-makers at regional and district levels reported using their triangulation reports for strategic planning and decision-making.
Conclusion: Triangulation is an effective means of using existing data when exhaustive data is not available to understand the HIV epidemic at sub-national levels. While more work is needed, triangulation promotes a culture of data use for evidence-based decisions in the HIV response in Tanzania.
Quality data is the backbone of informed public-health decision-making, and responding to the HIV epidemic is no exception. However, “it is seldom the case that a high-quality study exists that can provide answers to everyday public health questions directly” (Rutherford, et al., 2010). In Tanzania, emphasis in the recent past has been put on large population-based surveys such as HIV Antenatal Care (ANC) Sentinel Surveillance and the Tanzania HIV/AIDS and Malaria Indicator Survey (THMIS), which are conducted roughly every two and four years, respectively. While these activities are critical for charting the course of the epidemic, it is not realistic to suspend all decision-making until such surveys are published. Timely decisions and more frequent course corrections are needed in the interim.
As data systems swell to meet donor and national program needs, accompanying growth in data use, analysis, and long-term capacity building has not kept pace (AbouZahr & Boerma, 2005;Rutherford et al., 2010). Instead, “diverse sources of data are rarely presented together and gathering, synthesizing and interpreting them has become increasingly challenging” (Rutherford, et al., 2010). Consequently, lack of data use may contribute to poor data quality, compounded by the heavy burden on health workers to provide services alongside extensive reporting requirements (Braa, Heywood, & Sahay, 2012; Gourlay et al., 2015).
Triangulation is one approach that aims to remedy this situation. Triangulation, or data synthesis, refers to comparing and contrasting different types of data from different sources that address the same topic. More specifically, public health triangulation is defined as “the process of reviewing and interpreting existing data and trends in those data from multiple data sources that bear on different facets of a broad public health question in order to identify factors that underlie the observed data and to assist with public health decision-making and actions” (Rutherford, et al., 2010). Triangulation is an iterative process; as new data sources are identified and analyzed, hypotheses are refined.Triangulation is best employed when data are plentiful and a rapid response is necessary. When there are urgent programmatic questions that need to be answered, but resources and time do not allow for a full study to be carried out in order to answer those questions, triangulation allows for decisions to be made quickly using whatever data already exist.
Tanzania is characterized by a multi-tiered,decentralized health system(Gilson, Kilima, & Tanner, 1994). It is a large country with distinct socio-cultural, economic, and geographic areas with different drivers of the HIV epidemic. Recent efforts to focus on these differences and address the epidemic at lower sub-national levels have required public health professionals to examineavailable data in depth to ensure rational decision-making for program and resource allocation. However, population-based surveys such as THMIS and the Demographic and Health Survey have historically targetedthe national level, with the regional level only receiving estimates. The Tanzanian Triangulation Project was developed to enhance understanding of the HIV epidemic at sub-national levels and to build capacity among local providers to promote a culture of data use and encourage evidence-based decision-making. In mainland Tanzania, the activity was implemented by a triangulation team consisting of the University of California, San Francisco (UCSF), the Tanzanian Commission for AIDS (TACAIDS), and the National AIDS Control Programme, and in Zanzibar, by UCSF and the Zanzibar Integrated HIV, Tuberculosis and Leprosy Programme (ZIHTLP). This paper outlines the process of conducting regional HIV triangulation workshops over eight years and the successes and challenges encountered.
The triangulation project can be broken down into three overarching steps: (I) plan triangulation, (II) conduct triangulation, (III) and communicate the results.
In 2009, an overarching protocol for triangulation and data synthesis was developed by UCSF and approved by the National Institute for Medical Research in Tanzania and UCSF’s Committee on Human Research and was given non-research determination by U.S. Centers for Disease Control’s (CDC) Associate Director of Science. The first triangulation activities took place from 2009 to 2011 and covered Mbeya and Iringa regions. All subsequent workshops were conducted between 2013 and 2016. The process was streamlined for each successive workshop with experience and lessons learned from past completed workshops. Preparations depended on the needs and capacity of the region. For earlier regions (Iringa and Mbeya [2009 to 2011]and Lindi, Kilimanjaro, Arusha, Mara and Shinyanga ), stakeholders’ meetings were held to identify questions for investigation that were actionable, answerable, and appropriate. The investigation questions developed in these initial stakeholder meetings were similar across all regions and continued to be used in subsequent workshops.
Regions were selected to include: regions of high and low HIV prevalence according to the most recent population-based HIV/AIDS indicator survey (THMIS 2011-12) and TACAIDS’ ten priority regions above the national HIV prevalence of 5.1%(THMIS 2011-12).
Workshop participants included professionals working in all HIV/AIDS program areas (e.g., prevention of mother-to-child transmission[PMTCT], HIV testing services,and HIV care and treatment), Regional and District AIDS Control Coordinators, and implementing partners in that region, as well as Council HIV/AIDSCoordinators and other TACAIDS representatives (and for Zanzibar, ZIHTLP staff). The average workshop size was 20 participants, although it grew to 25 by the final year to incorporate providers from all districts and program areas.
Before each workshop, facilitators conducted a brief literature search to ascertain whether special studies or surveillance activities had been conducted in that region. District-level monitoring data from all HIV/AIDS-related programs, along with data from national population-based surveys(including THMIS, Demographic and Health Survey, and Population and Housing Census, and special studies (e.g., Surveillance of HIV and Syphilis among ANC Attendees) were organized and compiled into an Excel workbook. The facilitators would examine the data, determine what gaps and data quality issues existed, and ask follow-up questions of regional and district authorities when needed.Planning visits were conducted with regions when necessary to fill in data gaps and resolve data quality issues.Flash drives were prepared with the complete set of data and reference materials and were distributed to participants at the workshops.All data sources used were publicly available data or programmatic data with no unique identifiers.
The five-day workshops began with two days of hands-on capacity-building activities. These included working with the MS Excel pivot table tool to learn how to organize and manipulate data, to analyze data, and to detect trends and patterns. Basic Excel skills (including sorting and filtering data and using formulas) were incorporated to support participants who possessed limited Excel or computer experience, and exercises with dummy data were utilizedto afford participantshands-on experience. Additionally, an introduction to triangulation and data synthesis was presented, and participants were taught about interpreting findings and effectively presenting them through the use of visual displays. The introduction included training on creating tables and charts in Excel and maps (previously using EpiInfo mapping module, most recently using QGIS 2.6.1, an open access more user-friendly software). DHIS2 was also incorporated to familiarize or refresh participants on how to access routine reports and display data visually with pivot tables and charts.Lessons also included the components of data quality and underscored the importance of having high-quality data for making strategic decisions.
For the remaining three days, participants were divided into four groups of 5-10 people according to program area to work on answering the investigation questions. Facilitators were present for support, but the group work was participatory, carried out by the regional-, district- and facility-level professionals, utilizing skills taughtduring the capacity-building activities of days one and two. The group workincluded detecting trends in the epidemic, determining coverage of HIV/AIDS services, and identifying challenges and proposing recommendations to address them. Routine monitoring data from health management information systems were arrayed alongside population-based survey and surveillance study data to get a complete picture of the epidemic and service provision in the region.
Early analyses typically generated more questions and often resulted in participants tracking down additional data to answer those questions. As new data sources were identified and analyzed, hypotheses were refined. The groups reconvenedto share new data sources, present findings, and harmonize recommendations. Communication between groups throughout the process ensured cross-pollination of the analysis and discussion.
After the close of the workshop, UCSF, TACAIDS, and National AIDS Control Programme facilitators compiled all group work into a regional triangulation report and shared a complete draft with participants for their input via email. The feedback was incorporated and the reports submitted to the U.S.CDC in Tanzania for review and approval to disseminate.
Participants completedan evaluation of the activity at the close of the workshop, and for 2013 and 2014 workshops, a capacity-building assessment was administered pre- and post-workshop (post-workshop by email) that aimed to determine the relevance and utility of the workshop once participants had returned to their posts.
III. Communicate Results
In 2014, the triangulation team started incorporating follow-up debriefingvisits at the close of each workshop to the Regional Administrative Secretariat, the second in command at the regional level, as well as senior regional authorities who are responsible for resource allocation and funding within the region. The triangulation team presented an executive summary of the findings and recommendations for action. Through these presentations, the triangulation team oriented authorities on the current HIV situation in their region with evidence of performance and gaps in the services provided to their constituents.
The final triangulation reports were distributed either through TACAIDS to the Regional Coordinator for TACAIDS or directly to the Regional AIDS Control Coordinator by email to be circulated to the participants and other stakeholders in the region.
From 2009 to 2016,the triangulation team conducted 21 workshops covering 24 regions of Tanzania(Figure 1).The workshops included 19 out of 25 regions from mainland Tanzania (Mbeya, Iringa, Lindi, Kilimanjaro, Arusha, Mara, Shinyanga, Ruvuma, Manyara, Kigoma, Katavi, Tabora, Rukwa, Njombe, Pwani, Dar es Salaam, Tanga, Kagera, and Mwanza) and all five regions of Zanzibar (Mjini Magharibi, Kusini Unguja, Kaskazini Unguja, Kusini Pemba, and Kaskazini Pemba). Iringa and Mbeya regions, which were covered during the first set of triangulation activities from 2009 to 2011, were updated with repeat workshops in subsequent years.Zanzibar was an exception, with one workshop for all five regions of the archipelago.
During the Workshops
The investigation questions to be answered during the workshops were narrowed to three main questions: (1) What are the current trends in HIV and what is the direction of the epidemic in this region?; (2) What are the current prevention efforts in this region?; and (3) What are the patterns of HIV service utilization in this region?. Routine monitoring data used for the analysis included data from the following programs: HIV care and treatment, PMTCT and HIV-exposed infants, HIV testing services, home-based care, and, where available, voluntary medical male circumcision, tuberculosis/HIV care, gender-based violence, cervical cancer screening, and condom distribution.
Reports were successfully completed in all regions (except Zanzibar, where a single report was developed for all five regions).Results varied by region, but workshops helped participants to identify programmatic challenges within their regions and districts, with support from their own data, and to develop recommendations to address the symptoms and causes of those challenges. Examples of the decisions borne out of the workshops included revising the voluntary medical male circumcision strategy to be more acceptable to older men, providing regular supportive supervision on forecasting and requesting of HIV commodities to prevent stock-outs (usually HIV test reagents), and expanding mobile services to provide greater access for hard-to-reach populations. Participants also wrote recommendations that were not programmatic in nature, such as: increasing the number of sites for ANC sentinel surveillance after the division of regions resulted in fewer sites, and conducting integrated bio-behavioral surveillance studies to improve knowledge of key populations at risk, including commercial sex workers, men who have sex with men, and people who inject drugs.
Every region had considerable data quality challenges. These challenges included inconsistent interpretations of indicator definitions across providers and districts (e.g., whether couples tested at ANC were a subset of partners tested), varying data aggregation practices(e.g.,the loss of disaggregated data,such as HIV testing modality as data were reported to higher levels or whether men tested for HIV through voluntary medical male circumcision should be counted as provider-initiated counseling and testing), and wildly discrepant values for service coverage depending on which data source was used (e.g.,treatment initiation for HIV-infected pregnant women under the Option B+ approach).In some cases, values varied up to 136% within a single district when comparing data accessed via DHIS2 to reports sent up the channels from facility to district to the Regional AIDS Control Coordinator or U.S. government implementing partners’ data. The recommendations drawn up by the participants during the workshops consistently included improving data quality and fostering a system of data use and feedback at all levels within the region.
Participants reported that the mode of facilitation and participatory nature of the workshops allowed them to develop skills, including data analysis, data interpretation, visual representation, and report creation, that are relevant to their work. Participants were enthusiastic about these new skills, and many expressed confidence that they wouldbe able to continue using them to be more efficient and effective in their jobs.
In a follow-up capacity-building assessment questionnaire, 83%of participants said they learned new Excel skills (only 42% of respondents knew how to use Excel before the training). Of those, 93% learned how to use pivot tables and 80% learned how to make charts and visual displays. In addition, 86% of respondents who said they acquired new Excel skills reported using these skills after the workshop. Examples of how they used these new Excel skills included: making tables and graphs for presentations to district and regional stakeholders; using tables and pivot tables to determine the trends of clients in care and receiving antiretroviral therapy; calculating frequencies and proportions; compiling monthly and quarterly reports; presenting trends on data related to HIV activities to stakeholders; and preparing the district budget for donors.
Use of the Reports
The regional triangulation reports that came out of the workshops were developed by the workshop participants who are district and regional experts, primarily for their use within the regions. Reports were distributed to all regions after being cleared by the U.S. CDC and dissemination activities varied by region.
There was overwhelming success in collaborating with authorities and garnering ownership of the triangulation process and reports. In a follow-up capacity-building assessment questionnaire, 66% of respondents reported that they used their regional triangulation reports to inform planning and decision-making. Participants used the reports in varying capacities, including to ordertest kits and antiretrovirals on time (including nevirapine for HIV-exposed infants); to present during data-sharing meetings; to estimate the number of clients in need of antiretroviral therapy; to develop Regional HIV/AIDS Strategic Plans; to inform the Comprehensive Council Health Plan; to show variation in the magnitude of HIV across districts; and to plan increases in the number of HIV testing sites.
In addition, someRegional Coordinators for TACAIDSreported that regional authorities were using the executive summaries from their triangulation reports as a source of information when conducting meetings and in developing regional health reports. Regional Coordinators for TACAIDS also reported using the triangulation reports to advise local government authorities on how to allocate funds intheir annual budgets. Furthermore, implementing partners working in the regions (e.g., Tanzania Health Promotion Support) requested copies of the triangulation reports for their own use.
With leadership from TACAIDS, the development of regional strategic plans started nationwide in 2014. Some regions used their triangulation reports as part of their regional strategic plan, including regions that underwent workshops in 2013. TACAIDS senior management noted that strategic plans that integrated triangulation reports were of much higher quality than plans submitted by regions where triangulation workshops had not yet been conducted.
The Tanzanian Triangulation Project demonstrated that using the data generated by routine program monitoring to make evidence-based decisions is critical to a well-functioning public health sector. Stakeholders are using the reports and visual displays generated during the workshops to make evidence-based decisions and to advocate for changes. All of this promotes a culture of data use and will result in higher-quality data with which to make decisions.
The challenge of missing data required for analyses was a universal experience across all workshops. Data gaps commonly resulted from incomplete reports, data that were inconsistent with other sources, or newly-appointed professionals not having access to historical data. Extra efforts were often required during the workshops to track down missing data, including traveling to or calling multiple sites or individuals and receiving data over the phone or via SMS. In addition, national data collection tools did not include data disaggregated by key populations, and surveillance studies had only been conducted in a limited number of regions. In many cases, the regional triangulation reports recommended conducting studies designed to understand the prevalence of HIV and risk factors among key populations in the regions.
Ensuring participation from key professionals serving in specific positions (particularly regional HIV coordinators) was an unanticipated challenge.Invitees often sent representatives or colleagues to take their places at the workshops. Last-minute competing priorities occasionally meant that key personnel were not present during crucial sessions of the workshops. Not having key personnel present compromised access to important data and the overall knowledge base at the workshop,such as knowledge of historical data and programs. While the triangulation team tried to ensure,during the preparation stage, that key personnel would be available to attend the workshops, this did not always happen.
Some workshop participants possessed limited computer skills, which was challengingas other participants were more competent with computers and Excel. In 2015 we expanded the capacity-building portion of the workshop to focus on introductory Excel skills, which helped to reduce this barrier. Furthermore, despite many participants having access to DHIS2, many did not know their logins or passwords or did not have them accessible, revealing that they were not being used. Reliable internet access was also a problem. As Tanzania migrates to an electronic health information system, ensuring that providers have regular access to DHIS2 (including internet connectivity, access, and IT equipment) is increasingly critical for routine data use. Users of the electronic system will need resources to manage these challenges at local levels.
The triangulation workshops imparted new skills, which likely increased participants’ efficiency and effectiveness in the workplace. For many participants, these workshops were the first time they were able to see their data displayed, and they found it very powerful to have their experiences in the field validated with visual displays. Participants occasionally denied the data they saw in the visual displays, asserting they did not reflect reality, which underlined the importance of health professionals using and analyzing the data they are collecting. For example, in one instance, data on PMTCT coverage differed by over 50% from what the district PMTCT leaders expected, highlighting major data quality issues in the region that made assessing program performance impossible. Understanding the value of collecting, reporting, and using data is a necessary step towards investing time and effort into ensuring data is of high quality. Theidea “that data quality and data use are interrelated” has been identified in many activities and contexts (Ndabarora, 2014), including Tanzania(Braa, Heywood, & Sahay, 2012).
Health information debrief visits with the Regional Administrative Secretariats at the close of the workshops occasionally resulted in the immediate implementation of recommendations. However, a system for tracking the status of recommendations to ensure their implementation was not developed and it is unknown to what extent recommendations have been executed.Delays in the triangulation report clearance process may have contributed to delays in implementation of recommendations as well as dissemination of the reports.
Evaluating the triangulation activity was difficult. An online assessment was administered to participants through their email addresses six months after the completion of two rounds of regions. However, the response rate was low (37%) due to lack of resources for participants, such as computers and internet connectivity, as well as people leaving or being transferred out of their positions. Nevertheless, the feedback received through the assessments was overwhelmingly positive.
The triangulation method has a number of documented limitations including ecological fallacy (drawing conclusions about individuals based on group-level data) and reproducibility of results (whether different investigators would arrive at the same conclusions)(Rutherford, et al., 2010).Although this project sought to identify the most plausible interpretations of the data viewed as a whole, alternative interpretations are possible, uncertainties remain anddata quality issues of varying degrees contribute to these uncertainties. Nevertheless, the triangulation activity was part of an ongoing process to instill a culture of data usefor evidence-based decisions in HIV programming in Tanzania.
The triangulation workshops have built on existing efforts in Tanzania to promote a culture and facilitating environment for data use to inform strategic planning and decision-making, even in contexts where exhaustive data are unavailable. Overall the activity was a success in developing the capacity for analyzing, interpreting, and communicating data and in promoting a culture of data use. While we saw an increase in capacity among those attending the workshops, there is a need to continue improvingaccess to data and bolstering capacity for using data nationally and sub-nationally.
We humbly thank all the health care providers for their service to their communities and their participation in the triangulation workshops. We would also like to acknowledge the generous financial support from the U.S. CDC for this activity (grant 5U2GGH000977).
The authors declare that they have no competing interests.
JT, SW, CS, JN and DK conceived and designed the study. JN and DK carried out the data collection. JT, SW, CS, JN, DK and TK carried out the data analysis. JT, CS and SW drafted the manuscript. All authors read, revised and approved the final manuscript.
AbouZahr, C., & Boerma, T. (2005). Health information systems: the foundations of public health. Bulletin of the World Health Organization, 83, 578-583.
Braa, J., Heywood, A., & Sahay, S. (2012). Improving quality and use of data through data-use workshops: Zanzibar, United Republic of Tanzania. Bulletin of the World Health Organization, 90, 379-384.
Gilson, L., Kilima, P., & Tanner, M. (1994). Local government decentralization and the health sector in Tanzania. Public Administration and Development.
Gourlay, A., Wringe, A., Todd, J., Michael, D., Reniers, G., Urassa, M., . . . Zaba, B. (2015). Challenges with routine data sources for PMTCT programme monitoring in East Africa: insights from Tanzania. Global Health Action.
Ndabarora, E. C. (2014). Systematic review of health data quality management and best practices at community and district levels in low and middle income countries. Information Development, 103-120.
Rutherford, G. W., McFarland, W., Spindler, H., White, K., Patel, S. V., Aberle-Grasse, J., . . . Stoneburner, R. L. (2010). Public health triangulation: approach and application to synthesizing data to understand national and local HIV epidemics. BMC Public Health, 10.
All inquiries can be sent via email to: email@example.com
IN COLLABORATION WITH
The Ministry of Health, Community Development, Gender, Elderly and Children.