Behind the Curtain of Mathematical Modelling : Inside a collaborative modelling project on public health strategies for syphilis management

This case study provides a window on the workings of a partnership between modellers and public health specialists.

Their story illustrates how mathematical modelling can provide timely evidence to guide decision-making by public health planners and practitioners throughout the implementation of a new intervention.

The lessons they share may help to demystify modelling and reveal the benefits of collaborations between modellers and public health personnel.

Readers will learn about what to expect from mathematical modelling, how the strengths of an interdisciplinary team can be applied to modelling projects to ensure they properly address public health challenges, and how to appraise and use the evidence from mathematical modelling.

This case study is for public health programmers and decision-makers who wish to become familiar with practical applications of mathematical modelling in public health. Planning effective and efficient screening programs for syphilis may be a timely example for public health specialists.

Behind the Curtain of Mathematical Modelling : Inside a collaborative modelling project on public health strategies for syphilis management

This case study describes the experiences of an interdisciplinary team associated with the Winnipeg Regional Health Authority (WRHA), the University of Toronto, and Harvard University who came together to apply mathematical modelling to assess the impact of a newly designed intervention to reduce the burden of syphilis in Winnipeg, Manitoba. Their story illustrates how mathematical modelling can provide timely evidence to guide decision-making by public health planners and practitioners throughout the implementation of a new intervention. The lessons they share may help to demystify modelling and reveal the benefits of collaborations between modellers and public health personnel.

Challenging times

Public health officials in Winnipeg, as in many other cities in Canada, have been struggling to control periodic outbreaks of infectious syphilis since its reappearance in the 1990s. In 2012, a new outbreak emerged in Winnipeg, primarily affecting men who have sex with men (MSM) (1). With an acceleration of the epidemic in 2014, the Healthy Sexuality and Harm Reduction team at the WRHA saw a need to develop new approaches. Despite sponsoring aggressive public awareness campaigns across the city, becoming more innovative in their use of social media and dating apps, supporting outreach efforts in bathhouses, and enhancing partner notification strategies, public health officials continued to see a growing number of cases at the city’s clinics every month (2).

The team was aware that reinfection among men who were previously diagnosed with syphilis was common in Winnipeg, a phenomenon increasingly reported in the scientific literature worldwide. In the studies the team consulted, one article suggested that a previous history of syphilis was a good marker for individuals belonging to sexual networks experiencing syphilis, and that those individuals were likely to return with an infection (3). Another article featured the use of mathematical modelling to explore how enhancing screening frequency could be an effective strategy to control syphilis in MSM populations, especially with interventions focused on core groups (4). Considering the combined evidence of the two studies, and thinking outside the box, the team sought to design a new intervention focused on increasing screening in clients with a previous history of syphilis. They were fully aware that this type of intervention would be time and resource intensive for public health nurses, but with a growing sense that the old strategies were no longer working and team morale challenged, they decided to move forward with implementing the “Assertive Syphilis Testing Strategy.”

Public health nurses would enroll and contact all men with a history of syphilis recorded since the declared start of the outbreak in 2012 living within the WRHA’s jurisdiction. The nurses would encourage clients to test every three months with their regular care provider, or with a partner clinic, and offer to book an appointment, if needed. They would also actively track participants’ medical records to ensure testing occurred and would call participants who had missed their last scheduled test.

As more participants were enrolled in the strategy, the time required of public health nurses for patient engagement increased, and the team had new questions: “Are we putting our efforts in the right place? How long will it take before we see benefits? Will this strategy really help curtail the epidemic?”

Arriving at this impasse, and recognizing the costs of inaction, the WRHA team considered what mathematical modelling could offer. The decision led them to reach out to experienced mathematical modellers at Harvard and the University of Toronto.

Coming together

For the WRHA team, as for many other public health officials, identifying the most effective course of action in the midst of an outbreak was a difficult exercise. They were aware that a complex system of interacting biological and social factors influence the spread of syphilis and that the information public health receives is imperfect. Many uncertainties remained for the WRHA about who was at risk or how fast the infection was spreading, which hampered the decision-making process.

It was important for the WRHA team to obtain answers quickly, given the time and resource constraints they faced, and they knew that modelling offered the best opportunity to shed light on some of their questions. Modelling had the potential to simulate intervention benefits or harms at a population level, and provide a body of synergistic evidence to inform program planning. For the WRHA team, mathematical modelling was a way to obtain timely answers to inform an implementation process that was already underway.

Three essential pillars came together to support this effort: 1) the WRHA program team, which included the Healthy Sexuality and Harm Reduction staff and two Medical Officers of Health, 2) the WRHA epidemiologist, and 3) three academic modellers based in Toronto and Boston. Every member of the team came with a unique set of knowledge and perspectives that complemented that of other members. The program team brought a pragmatic understanding of the local context, including knowledge on program operation, implementation considerations and affected populations. The team of modellers had experience with modelling infectious syphilis, and a thorough understanding of the natural history and transmission dynamics of the disease. The epidemiologist had a deep understanding of the surveillance program, with knowledge of what data were available, and their limitations. The diversity of expertise within the WRHA team proved to be beneficial to the project, though engaging the ‘right people’—who were amenable to applying modelling and informed on core group screening strategies for syphilis—did reflect some amount of luck.

With the team established, the project moved to focusing the scope. In the meantime the program team’s attention returned to implementation of the screening strategy, concurrently with the modelling work. Although each modelling undertaking is unique in purpose and progression, the path that unfolded for the WRHA team may serve to illustrate some useful lessons.

Framing the question

The first steps of the modelling project were to establish what research question mathematical modelling would be able to address, and to communicate expectations clearly among the team regarding what was feasible to achieve with the time and data available. Taking the time to develop consensus on these points helped focus the modelling endeavor on one simple question: “Can the syphilis epidemic in Winnipeg be curtailed by using prior syphilis infection as a marker of risk in a focused intervention to enhance syphilis screening among men?”

The team agreed that because this project was emerging from a new collaboration, its scope should be kept as simple as possible. The modelling team had previously developed a mathematical model that compared the effectiveness of different screening scenarios for reducing the burden of syphilis in Toronto’s MSM population (4). Adapting this model to capture the demographic and epidemiological features of Winnipeg was seen as the simplest way to address the WRHA’s research question. This model could examine the potential effectiveness of using previous infection of syphilis as a marker to focus screening and compare the results to other screening strategies, including universal screening and screening focused on behavioural assessment. Thus, the strategy was pragmatic, yet sufficient to respond to the program needs. An iterative process began between team members through a series of teleconferences and emails that helped refine the analysis plan, and inform the model’s parameters.

Adapting the model, interpreting the findings

Deciding on parameters

For modellers, setting model parameters is an enormous task. They must decide and define under what conditions virtual interactions occur to simulate transmission dynamics in a population. It takes some impressive mental gymnastics to identify and quantify relevant parameters. Every question answered leads to another question. For the WRHA project, a sequence of questions arose: Is everyone equally at risk of being infected by syphilis? No! Then, how do we define risk groups? Do we know the proportion of men in each of these groups in Winnipeg? Would an estimate from another country be representative of Winnipeg’s context?

To reflect Winnipeg’s context, the modelling team started collecting information from the surveillance program, behavioural data from national surveys, and rates reported in the literature. Because the model was simple, the surveillance data were readily available at the WRHA. Winnipeg’s enhanced surveillance meant the data could be used to define incidence and reinfection rates, and to provide information on infected individuals. Other behavioural and biological information had to be extracted from older national surveys and the literature.

Because the modellers had experience working with public health teams, they already had a certain level of awareness of the limitations and biases of surveillance data. In surveillance programs, data collection is biased toward infected individuals seeking care, and this leaves an important gap in knowledge about characteristics and behaviours of uninfected individuals, or individuals who do not present to care. Consultation with the epidemiologist and the program team was important at this juncture to understand optimal data sources to use, the best way to mitigate known data flaws, and to corroborate what information would be the most relevant to represent Winnipeg’s epidemic.

Understanding assumptions and uncertainties

Mathematical modelling is intended to help understand phenomena, and to test ‘what if’ experiments. The goal is not to reproduce reality, but to represent a simplified version of a complex system. To achieve this, modellers purposefully simplify some phenomena and keep some parameters stable or fixed in order to develop a ‘controlled’ environment to test a hypothesis. Assumptions are part of every mathematical model, and they are defined, to some extent, by the limitations in the available data and the understanding of a disease natural history. Therefore, assumptions put boundaries on how the model findings can be interpreted. For this reason, they need to be communicated effectively to program teams throughout the process.

To build their syphilis transmission model, the modellers elected to define three categories of risk according to sexual behaviours, and had to assume that the sexual behaviours of men would remain constant for the duration of the model simulation period (10 years). In reality, behaviours change over time; it is tacit knowledge that people enter and exit these risk groups as their lives evolve, but there are few data available to capture these changes. Assuming that behaviours remain unchanged simplifies the model but does not exactly reflect reality.

The team at WRHA understood theses nuances, and adjusted their expectations of the model predictions. It was clear for them that the model could provide information on the overall benefit of the intervention, but mainly in a qualitative manner (i.e. will it work or not?), with quantitative outputs needing to be taken with a grain of salt.

To lessen the effects of uncertainties, modellers calibrate their model. They fine-tune parameters and question their decisions until the model can effectively simulate the transmission dynamics and reproduce the prevalence and incidence captured by surveillance data. For the WRHA project, the modellers calibrated the model by identifying the parameter sets that reproduced Winnipeg’s syphilis incidence rates between 2011 and 2015, under the screening conditions in effect during that time. After calibrating this scenario, the modellers were ready to apply other scenarios and simulate what would happen if the screening conditions where changed in Winnipeg.

Refining and interpreting the findings

As the work progressed, the team undertook a second round of discussion on the preliminary findings. This iterative process allowed some flexibility for the program team to explore questions raised by the model findings. One assumption of the model was that 75% of the men with prior history of syphilis would be amenable to screening every three months. The program team knew that, in reality, having 75% of all men enrolled in the intervention and agreeing to screening every three months would be fairly challenging to achieve. Knowing this, they wanted to determine the extent to which clients needed to observe the tri-monthly screening schedule, and how long to sustain enhanced engagement with clients to continue observing benefits of the intervention.

The modellers were able to answer their query by conducting a series of new analyses with their model. When modellers suspect that a parameter has an important effect on a model outcome, they can recalculate the results using alternative assumptions in a process that is called sensitivity analysis. Knowing the concerns of the program team, modellers were able to assess if an intervention based on less frequent testing (i.e. annual or semi-annual) could still curtail Winnipeg’s epidemic. By running sensitivity analyses, the modellers were able to inform the team on how important the screening intensity was for their intervention, predicting that semi-annual screening would still be beneficial, and that a rebound in the number of cases could occur if the intervention terminated prematurely. For the program team, these findings were very important in informing the next steps.

A limitation in the model, however, resided in the fact that it was only able to provide information on population outcomes, and not on individual outcomes. As the intervention continued to enroll more and more clients, the program team wanted to know if the model could provide some guidance on the best way to keep the number of people enrolled in the intervention to a manageable size. Continuing to follow all participants with repeated infections every three months was putting more and more strain on their resources. The team asked, could the model predict when a specific man could be withdrawn from the intervention? When an individual had not had an infection for 2, 3 or 5 years, was he still at risk? Because the model was designed to look at interactions at a population level and not at an individual level, the modellers were unable to address this question.

Learning from the process

As the project came to an end, the team was able to reflect on factors that contributed to the success of their journey, and the lessons learned from this partnership. This experience was unique in its progression. As already noted, the modelling and implementation of the intervention were happening simultaneously, which allowed each to inform the other as the project progressed. The modellers worked in parallel with the WRHA team, informed by program questions, with the greatest value being their ability to simplify the model to accommodate the team’s need for timely answers to questions arising throughout the implementation process. The team also reflected on the value of their multidisciplinary team in generating a model of high quality that was relevant to the program.

The success of this partnership also resided in an open dialogue among the team’s members. This contributed to overcoming the initial skepticism encountered among some members of the team, and then tempered their growing expectations as the project moved along. It helped determine what was achievable that would satisfy all parties with the time and resources allocated, and balance the different roles and accountabilities of academia and public health. While the WRHA team sought rapid answers on the effectiveness of their intervention, the modellers’ pressure was to contribute to the broader body of knowledge, with robust and accurate results that would ideally be generalizable.

Overall, the model findings confirmed the hypothesis of the program team (5). The model predicted that focusing a screening strategy on men with prior history of infection had, indeed, the potential to curtail the epidemic in Winnipeg. The modelling exercise provided the program team with grounds to continue their work. Modelling demonstrated that the approach they took to identify clients at greater risk, using prior infection as an indicator of risk, had the potential to be effective. It was important for them to learn that their intervention could curtail the epidemic, even with imperfect compliance to the screening schedule, but that its premature termination could also result in a rebound in cases. This modelling exercise provided the team with supportive evidence for continuing their work and confidence that they could achieve results with their intervention if they stayed on course, acknowledging that the evidence was pertinent to a specific population at a given time. At that time, Winnipeg was experiencing an epidemiological shift with syphilis cases and the model was helpless in informing on the effectiveness of the strategy in the heterosexual population.

This collaborative project, though limited in scale, emerged as a very positive experience for everyone. It answered the questions brought by the program team, and did that well. By providing the team with synergistic evidence that their intervention had the potential to work, it encouraged buy-in of the larger team and management, and served as a tool to advocate for resources. By doing so, it supported long-term planning, opening the door for less reactive, and more strategic and pro-active planning.

Although outbreak and intervention characteristics may differ from one population and project to another, the lessons learned from the WRHA team’s experience may be appropriate to understand the power and limitations of mathematical modelling in many contexts.

Acknowledgments

The NCCID extends thanks to the following individuals for their contributions to this case study, including participation in interviews, editing, and providing thoughtful consideration to transferable lessons for others. Thanks go to the team at the Winnipeg Regional Health Authority: Souradet Shaw, epidemiologist; Joss Reimer, Medical Officer of Health; Craig Ross, former Program Specialist, Healthy Sexuality and Harm Reduction Program; Alicia Lapple, Communicable Disease Coordinator, Healthy Sexuality and Harm Reduction Program; and to mathematical modellers/researchers: Sharmistha Mishra, Assistant Professor, University of Toronto; and Ashleigh Tuite, former Postdoctoral Fellow Harvard University.

---

References

(1) Shaw, S. et al. (2017). Infectious syphilis in women: what’s old is new again? Int J STD&AIDS, 28(1), 77-87.

(2) Manitoba Health, Seniors and Active Living (2017). Epidemiological Update on Infectious Syphilis in Manitoba: Enhanced Data Analysis. January 1, 2016- February 15, 2017.

(3) Marcus, J. L., Katz, K. a, Bernstein, K. T., Nieri, G., & Philip, S. S. (2011). Syphilis testing behavior following diagnosis with early syphilis among men who have sex with men–San Francisco, 2005-2008. Sex Trans Dis, 38(1), 24–9. http://doi.org/10.1097/OLQ.0b013e3181ea170b

(4) Tuite, A. R., Fisman, D. N., & Mishra, S. (2013). Screen more or screen more often? Using mathematical models to inform syphilis control strategies. BMC Public Health, 13(1), 606. http://doi.org/10.1186/1471-2458-13-606

(5) Tuite, A. R., Shaw, S., Reimer N. J., Ross C. P., Fisman, D. N., & Mishra, S. (2018). Can enhanced screening of men with a history of prior syphilis infection stem the epidemic in men who have sex with men? A mathematical modelling study. Sex Transm Infect. 2018 Mar;94(2):105-110.