Decision analysis support for evaluating transmission risk of COVID-19 in places where people gather
This study (with plain-language summary) describes a framework for evaluating the relative transmission risk of COVID-19 between settings. This framework can help inform and identify settings where increased mitigation measures (non-pharmaceutical interventions) are required to drive down community transmission.
Glossary of Terms
COVID-19 Modelling Glossary of Terms
Model Summary
Public Health measures to contain COVID-19 and prevent transmission– physical distancing, case detection and isolation, face mask requirements and shutdowns, for example – have been implemented to varying degrees in Canada since early 2020. It has been a significant challenge for public health decision-makers to manage trade-offs between preventing COVID-19 transmission and substantial consequences of restrictions on other health, social and economic outcomes.
In late 2020, before the availability of new vaccines in Canada, a research team explored the available evidence on COVID-19 transmission to contribute a framework for evaluating transmission risk in different settings. The project objective was to help distinguish what it is about settings that can make one setting more risky for COVID-19 transmission than another, and develop a framework to assist with making decisions on where to focus restrictive measures. “Settings” included a broad range of places and events with a risk for COVID-19 transmission, including private gatherings in homes, and public places such as schools, grocery and other retail stores, and concerts, among others.
With the ongoing risk from COVID variants of concern and insufficient immunization levels in Canada, there is still a need to assess transmission risk where people gather and potentially apply restrictions and other nonpharmaceutical interventions. When the COVID-19 situation improves, this same framework can be used to help prioritize settings where restrictive measures can be lifted.
Methods
The research team used a multi-criteria decision analysis (MCDA) approach to create an evaluation framework which consists of stratified criteria to assess transmission risk in different settings. MCDA allows for systematically evaluating alternatives over numerous criteria and highlighting the strengths and weaknesses of the alternatives being considered. Participatory MCDA typically includes steps to define a problem, identify stakeholders, define and weight criteria ranking, followed by an analysis of sensitivity and interpreting the results of the evaluation.
In this project, a “rapid and light” version of the participatory MCDA process was used to build the framework, and to account for the very limited time experts had to review the criteria as well as the lack of published information on COVID-19 transmission risk available at the time. Four steps from a typical participatory MCDA process were adapted for this purpose (Table 1).
Table 1: Summary of steps in participatory and “light” multi-criteria decision analysis process
| Phase | # | Steps included in the participatory process | Steps included in the “light” process |
| Problem structuring | 1 | Definition of the problem of interest | ✓ |
| 2 | Identification of stakeholders | ✓ | |
| 3 | Identification of setting alternatives | – | |
| 4 | Definition of criteria | ✓ | |
| 5 | Weighting of criteria | ✓ | |
| 6 | Evaluation of setting alternatives based on criteria | – | |
| Decision analysis | 7 | Decision analysis | – |
| 8 | Sensitivity analysis | – | |
| 9 | Interpretation of results | – |
A quick scan of research literature identified 23 initial criteria based on common factors in settings where there were COVID-19 outbreaks. As early evidence showed droplet and aerosol transmission accounted for most cases, close contact in closed and crowded spaces was found to be a primary driver of transmission based on the literature. The initial criteria list was shared with 62 provincial public health experts who condensed the list to 15 criteria, including 10 related to sites and events, one participant level and four potential mitigation measures. The new criteria list was shared with members of the Pan-Canadian Public Health Network involved in the COVID-19 response, who were asked to rank these 15 criteria in order of importance for the evaluation of transmission risk. The expert-ranked lists were then combined using general Mallows models to produce a consensus ranking, that is, an ordered list of criteria (see the paper Appendix for more details on the general Mallows models), using the R package PerMallows to analyze the rankings. This consensus ranking was then organized into stratified sets of criteria to create the transmission evaluation framework.
Results
Fifteen experts from across Canada participated in ranking the criteria. A generally good consensus emerged among the experts on the relative importance of the criteria, with some individual variations in specific ranking positions. The consensus ranks resulting from the generalized Mallows models with the Kendall and Hamming distance are shown in Table 2. Based on the level of agreement, the researchers created four framework categories for the criteria: “critical”; “important”; “good to consider”; and “if time permits”. While there was broad agreement between the two ranking metrics (Kendall and Hamming), there were differences in respondent rankings for some criteria. Analysis of these differences were used to resolve final placement of criteria to each category. Location and ventilation and ease of contact tracing were the criteria on which experts most strongly agreed in terms of absolute rank ordering (i.e. most important and least important, respectively). Location and ventilation appears exclusively in the first seven ranks and almost always (n=14/15 times) in the first four ranks. Criteria with greater ranking disagreement among experts were number of households or individuals and level of expelled air.
Table 2: Consensus ranking (mode) of criteria under generalized Mallows models using the Kendall and Hamming distance
| Category | Criteria | Kendall | Hamming |
| Critical | Density of crowd | 1 | 2 |
| Contact between participants | 2 | 3 | |
| Location and ventilation | 3 | 1 | |
| Important | Number of households (or individuals) | 4 | 4 |
| Level of expelled air (of activity) | 5 | 10 | |
| Duration of event | 6 | 6 | |
| Personal protective equipment—use of masks or face coverings | 7 | 5 | |
| Mixing of participants | 8 | 8 | |
| Mixing of networks | 9 | 7 | |
| Good to consider | Engineering controls—use of physical barriers and environmental cleaning | 10 | 12 |
| Related activity (e.g. shared group travel) | 11 | 9 | |
| Administrative scheduling—use of cohorting to stagger participants and reduce contacts | 12 | 13 | |
| Age structure of participants in population | 13 | 14 | |
| Shared equipment or surfaces | 14 | 11 | |
| If time permits | Ease of contact tracing should an outbreak occur | 15 | 15 |
Criteria considered “critical” were density of the crowd, contact between participants and location and ventilation. The second set of criteria (“important”) were almost consistently ranked within the top half of the lists by experts, with some variability in specific rank: number of households; level of expelled air (of activity); duration of the event or activity; use of personal protective equipment; mixing of networks; and mixing of participants. Although level of expelled air was given a range of ranks by the experts, the researchers included it as important based on the available literature.
The “good to consider” category included five criteria ranked variously by the experts: the use of engineering controls and environmental cleaning; related activity; administrative scheduling; age structure of participants; and shared equipment or surfaces. The final category, “if time permits”, includes only ease of contact tracing, which was, as noted, almost consistently ranked last by the experts.
Discussion and Conclusion
The consensus-ranked list of transmission criteria and corresponding categories create a framework for ranking settings for COVID-19 transmission risk based on both the literature and expert opinion in late 2020. Using generalized Mallows models with Kendall and Hamming metrics allowed the researchers to obtain a consensus ranking among experts on the relative importance of different transmission risk factors.
The framework is intended to assist with evaluating transmission risk and should be used with contextual information, including local epidemiology and considerations of gathering size or specific activities, preferably with participation of local public health stakeholders. Involving public health experts will encourage on-going assessment of the framework and criteria for making decisions on public health measures. Many of the criteria are inter-related and thus may be difficult to evaluate individually (e.g. the number of participants at an event and the density of the crowd). It may be beneficial to consider different scenarios of transmission, gathering size, or vaccination rates, for example.
With ongoing emergence of variants of concern, the experts from this original project were once again consulted in March 2021 to see if their rankings of the criteria had changed. Although the experts did not change the rankings in the new context at the time, it would be essential to continue to monitor the framework and criteria as knowledge and situations shift during the pandemic. Furthermore, an assessment of transmission risk should be clearly defined in terms of scope and scale. For example, noting whether the assessment is evaluating individual visitors or employees at a site that will have accumulated exposure, or evaluating the general public’s risk of exposure at any given visit compared with evaluating the risk of exposure of vulnerable individuals (such as immunocompromised or elderly persons). A systematic, full participatory MCDA process can be used to conduct a full ranking of settings using the criteria defined in the current framework and better understand the relative transmission risk between settings by highlighting the strongest contributing factors as well as strongest protective factors for transmission risk between settings.
This project drew upon the latest evidence concerning transmission risk factors for COVID-19 in different settings, based on the available evidence. The expert rankings and resulting consensus list provide a generic framework of elements that can be applied objectively and transparently to assess transmission risk in any setting. Additional layers of information could be added to the participatory MCDA process to include economic, social and health criteria so that trade-offs could be more fully examined, allowing for more informed decisions about closures and re-openings to reduce the transmission risk of COVID-19.