V akademski reviji Vaccines je bil ta teden objavljen naš znanstveni članek na temo učinkovitosti cepljenja proti Covidu. Naša raziskava je bila narejena konec lanskega leta, na podlagi podatkov med 1. avgustom in sredino novembra 2021 (na podlagi dnevnih podatkov za 110 držav), s čimer je zajela učinkovitost cepljenja v času delta variante. V nadaljevanju objavljam uvod v našo raziskavo, ki ga je motiviral zavajujoč empirični pamflet dveh avtorjev (Subramanian & Kumar, 2021). Slednji je dosegel veliko popularnost (2.25 milijonov ogledov) ter služil kot argument anticepilcev, da cepljenje proti Covidu ni učinkovito. V nadaljevanju pokažemo na dve ključni metodološki napaki omenjenih avtorjev ter kako ti napaki vodita do povsem napačnih sklepov. Mi pokažemo, da če bi omenjena avtorja naredila resno ekonometrično analizo, bi seveda ugotovila, da je bilo cepljenje v času delta variante razumno učinkovito. Pokažemo tudi, da bi bilo v drugem valu Covida (druga polovica 2020) bistveno manj okuženih in mrtvih, če bi bilo cepivo že takrat na voljo. In drugače povedano, pokažemo, da so prav zaradi obstoja cepiv, države v delta valu epidemije lahko ostale bolj odprte oziroma uporabljale manj striktne zaščitne ukrepe (odprte trgovine in šole, brez policijske ure). Obstoj cepiva nam je omogočil bolj znosno življenje v času epidemije.

Celotno raziskavo lahko dobite na tej povezavi.

Apart from longitudinal medical observational cohort studies (see for instance Singanayagam et al (2021)), there are few cross-country empirical studies that use daily country-level COVID-related data to analyze the effectiveness of vaccination levels across countries on the dynamics of the number of new COVID cases, new hospitalizations, new ICU patients, and new deaths with COVID. To our knowledge, there are only two econometric studies that use cross-country data to analyze the effectiveness of vaccines. Aizenman et al. (2021) study the impact of vaccination on the ratio of mortality to infections. Using the data for the first half of 2019, Deb et al. (2021) examine the determinants of COVID-19 vaccine rollouts and their effects on health outcomes. They find that early vaccine procurement, domestic production of vaccines, the severity of the pandemic, a country’s health infrastructure, and vaccine acceptance are significant determinants of the speed of vaccination rollouts.

None of the papers reached a wide audience in terms of number of shares and citations. On the other hand, there is a much read and cited paper that uses very simple descriptive analysis to examine the effectiveness of vaccines on the incidence of new COVID -19 (hereafter: COVID) cases (see Subramanian and Kumar, 2021).[1] However, despite its broad popular appeal, this study is fraught with many methodological problems.

In examining the relationship between vaccination rates and the incidence of new COVID cases, Subramanian and Kumar (2021) use a simple static framework. First, they employ bivariate analysis by plotting a graph with the COVID cases per 1 million people for 68 countries on the vertical axis against the latest available percentage of the population that is fully vaccinated on the horizontal axis. No rigorous regression analysis was performed, only correlations were visually assessed to determine that there is no relationship between the two variables. Second, they choose only one cross-sectional observation per country (the last available) to show the relationship between the number of new cases and the vaccination rate. And third, all other factors that might influence the dynamics of epidemics except vaccination rate are ignored.

Let us first show why this approach is misleading and why it gives rise to potentially wrong conclusions. We illustrate two main concerns with this approach. The first issue is related to the use of only one cross-sectional observation per country. To illustrate this, let us take two extreme cases in terms of vaccination rates – Bulgaria and Denmark. As shown in Figure 1, Bulgaria is characterized by very low vaccination rates – between 15 and 22 percent in the period from August 1 to October 31, 2021, while at the same time it was affected by a dramatic increase in new COVID cases (see the steep blue trend line in the left part of the Figure 1). On the other hand, Denmark managed to vaccinate a large portion of its population – between August 1 and October 31, 2021, vaccination rates increased from 55 percent to more than 75 percent. At the same time, new COVID infections decreased (see the flat blue trend line in the right part of the Figure 1).

This suggests that the high (and increasing) vaccination rates might have dampened the spread of infections in Denmark as opposed to Bulgaria, as shown by a negative (red) trend line for the sample of both countries. In other words, differences in vaccination rates (and in their dynamics in terms of the rollout of vaccines) between countries can have a crucial impact on the dynamics of COVID epidemics over time. High (and increasing) vaccination rates may lead to reduced rates of increase in COVID infections. This can only be observed over a longer time period and in a large sample of countries. If we were to use only one observation per country, as Subramanian and Kumar (2021) have done, we would miss this fundamental insight on the dynamic relationship between vaccination and spread of infections. For this reason, an analysis based on single cross-sectional observation per country is misleading and easily leads to false conclusions that are not justified by the data.

Figure 1. New Covid cases and vaccination rates in Bulgaria and Denmark, by weeks. Notes: Data by weeks, period August 1 to October 31, 2021. Left part of the figure shows data for Bulgaria. Source: Our World in Data; own calculations.

The second issue is related to omitting all factors other than vaccination rates that might influence the dynamics of the pandemic from the analysis. This leads to what is commonly known as the omitted variable problem known in statistics. It occurs when one or more relevant variables are omitted from the empirical model, leading to bias in the results because the model attributes the effect of the missing variables to those that were included. In the economics literature, the problem was popularised, among others, by Barro (1991), who used a sample of 98 countries over the period 1960-1985 to try to uncover the factors that drive economic growth. Estimating the relationship between the growth rate of real GDP per capita and the initial (1960) level of real GDP per capita in a bivariate setting, Barro finds a positive but insignificant relationship between the two. Based on this, one might conclude that there is no empirical support for the Solow-Swan hypothesis of convergence across countries. However, after including additional factors in the model (such as differences in human capital, fertility rates, ratio of physical investment to GDP, share of government consumption to GDP, share of public investment, measures of political stability, and a proxy for market distortions) Barro finds that the relationship between the growth rate of real GDP per capita and the initial level of real GDP per capita turns to negative and significant. This means that when important factors are omitted, the model may attribute the effect of the missing variables to those that were included.

The same problem occurs in the study by Subramanian and Kumar (2021), who analyzed only the bivariate relationship between vaccination rates and the number of COVID cases but omitted large structural differences between countries in the factors that significantly influence the spread of COVID and the public health consequences. Let us illustrate what bias this might have in terms of the impact of vaccination on the spread of Covid infection. To show this, we estimate two models. First, we estimate a simple bivariate model by regressing the logarithm of the 7-day moving average number of new cases per 1 million population on the logarithm of the three-week lagged vaccination rate. In the next step, we estimate a full model that also includes other structural factors, such as the age structure of the population, population density, the prevalence of other diseases that increase the risk of developing severe COVID symptoms like diabetes or cardiovascular disease, etc., the availability of quality health care, the current state of the pandemic, and an indicator of countries’ response to epidemics (the stringency index). Figure 2 shows the data and linear regression lines for the bivariate relationship between vaccination rates and the number of new Covid infections. In the simplest model when all structural differences and policy responses of countries are omitted, the model yields a positive and significant relationship (see red regression line trending upwards), i.e. it may wrongly suggest that high vaccination rates lead to a higher number of Covid infections. However, when structural differences and country policy responses are included in the model, the relationship between vaccination rates and the number of new Covid infections becomes negative and significant (see orange regression line trending downwards).

Figure 2. Log Vaccination rate and log Number of cases per million. Notes: Based on results presented in Table A1 in the Appendix. Data by weeks, period August 1 to November 14, 2021, for 110 countries. Data is specified in natural logarithms; data on vaccination is lagged by 3 weeks. For the Full model the constant term was scaled up. Source: Our World in Data; own calculations.

Both issues illustrated above, show that investigating the impact of vaccination on the transmission of COVID in a cross-country setting requires careful analysis. First of all, there are large structural differences between countries in underlying factors that significantly influence the spread of COVID and the public health consequences. In addition, there is wide variation in the response of countries to the pandemic in terms of policies to reduce transmission of COVID, effectiveness in vaccination rollout, etc. There are also large differences between countries in terms of the state of epidemics. One should also consider the timing of vaccination, as immunity takes time to build up, while its effectiveness also gradually wanes after the vaccination. The paper by Subramanian and Kumar (2021) triggered many other responses to the journal, which also raised some major methodological issues (see Backhaus, 2021; Gianicolo et al., 2021; etc.)

In this large cross-country study, we consider the above factors in examining the impact of vaccination on the transmission of COVID and its public health consequences. To analyze the impacts of vaccination, we estimate two comprehensive models – one for the impact on the spread of COVID infections (in terms of number of new confirmed cases) and another one for the impact on severe COVID disease progression (in terms of number of new hospitalizations, admissions to intensive care and deaths). We take into account both the static cross-section and the dynamic aspects of the COVID pandemic. Our empirical models are designed to capture the differences across countries regarding the state of the pandemic (by including the initial level of infections per 1 million) and its dynamics (by including the most recent weekly reproduction rate). In addition to differences in vaccination rates and their dynamics, our model also includes country-specific factors that account for differences between countries that determine countries’ specific vulnerabilities or strengths to perform in dealing with the pandemic.

To examine the effectiveness of vaccination, we use the data for the latest wave of COVID outbreak (the Delta variant) in most countries (the period from August 1, 2021, onwards), while vaccines were also widely available. We use data for 110 countries (from Our World in Data) on the number of cases, deaths, hospitalized and intensive care patients, the percentage of people fully vaccinated, the reproduction rate, and the policy stringency index – all with daily frequency. The COVID-related daily frequency data (per million people) are then transformed into weekly averages using 7-day moving averages. As a point of reference, we use the data for the last day of the week capturing the average effects of the current week. We combine these data with a number of country-specific indicators accounting for demography, overall health of the population, state of the health-care infrastructure and economic development.

Our data are structured as a panel with a cross-sectional (country) and time (week) dimension. Both models are estimated using a pooled ordinary least squares (OLS) estimator. We first estimate our baseline models and then proceed with two alternative specifications to examine vaccination effectiveness from a different perspective.

The results can be summarized as follows. First, we show that, after controlling for the number of previous infections, the reproduction rate, the stringency of government protection measures and structural country-specific factors, the full vaccination rate is consistently and significantly negatively correlated with the number of new COVID cases. The regression coefficients indicate that, on average, at a lag of two weeks after the second dose, a 10 percent increase in vaccination rate is associated with a 1.3 percent decrease in new COVID cases. Extending the time elapsed after full vaccination results in a greater decrease in the number of new infections. The overall vaccination effectiveness, however, varies widely from region to region.

Second, our estimations show that magnitude of vaccination contributes significantly to reducing hospitalizations, intensive care (ICU) admissions and deaths with COVID. On average, after controlling for the number of previous infections and structural country-specific factors, a 10 percent increase in the rate of vaccination leads to a reduction of about 5 percent in the number of new hospitalizations, 12 percent decrease in the number of new intensive care patients and 2 percent reduction in the number of new deaths with COVID. Again, there are large differences in vaccination effectiveness between regions. The effects increase the longer the period after full vaccination.

Third, the estimates confirm that the moderating effect of vaccines on the number of cases and deaths occurs when the overall full vaccination coverage is sufficiently high.

Finally, by comparing the data for the same period between 2020 and 2021, we also check the viability of vaccination as a substitute for lockdowns or other, less stringent government protection measures. To do so, we test whether the availability of vaccines has helped countries curb infections and cases of severe disease progression compared to the same period in the previous year when vaccines were not available. More specifically, we check whether the dynamics of the COVID pandemic in the fall of 2021 as compared to the same period in 2020 is moderated in countries with high vaccination rate compared to countries with lower vaccination coverage. Our results suggest that vaccination does not appear to be an effective substitute for more stringent government safety measures to contain the spread of COVID infections until a certain vaccination coverage threshold has been achieved. The spread of infections is shown to be significantly reduced compared to the same period in 2020 only in countries with high vaccination coverage (more than 70 percent).

On the other hand, moderate vaccination coverage (between 40 and 70 percent) seems to be already a fairly effective tool and can serve in part as a substitute for more stringent government protective measures to reduce the pressure on the health care system. The availability and efficient rollout of vaccines allowed countries to adopt less stringent containment measures, which explains the favourable economic performance in the second half of 2021 despite the severe outbreak of the delta variant.

[1] As of March 2022, more than 2.25 million people have read this article online.

____________

Referenca na celotno raziskavo:

Damijan JP, Damijan S, Kostevc Č. Vaccination Is Reasonably Effective in Limiting the Spread of COVID-19 Infections, Hospitalizations and Deaths with COVID-19. Vaccines. 2022; 10(5):678. https://doi.org/10.3390/vaccines10050678