Skip to main content

Robustness of the BYM model in absence of spatial variation in the residuals

Abstract

Background

In the context of ecological studies, the Bayesian hierarchical Poisson model is of prime interest when studying the association between environmental exposure and rare diseases. However, adding spatially structured extra-variability in the model fitted to the data when such extra-variability does not exist conditionally on the covariates included in the model (over-fitting) may bias the estimation of the ecological association between covariates and relative risks toward the null. In order to investigate that possibility, a simulation study of the impact of introducing unnecessary residual spatial structure in the estimation model was conducted.

Results

In the case where no underlying extra-variability from the Poisson process exists, the simulation results show that models accounting for structured and unstructured residuals do not underestimate the ecological association, unless covariates have a very strong autocorrelation structure, i.e., 0.98 at 100 km on a territory of diameter 1000 km."

1 Background

Ecological regression studies investigate potential association between geographical variation in disease rates (or counts) and environmental covariates. For example, a recent study evaluated the ecological association between indoor radon concentration and acute leukaemia incidence among children [1]. For rare diseases and/or small areas, Bayesian hierarchical Poisson model is commonly used where within-area variability of disease is modelled at the first stage as a Poisson process and ecological relationships between disease and covariates are introduced at the second stage of the hierarchical model. Spatially extra-Poisson variability potentially due to aggregated effect of unknown confounders is commonly taken into account through spatially structured residuals added in the second stage of the model.

In that context, the BYM (Besag, York and Mollié) model [2] is a standard model for estimation of the ecological associations. The overall variability of a health indicator is broken down into a random Poisson component, a spatially structured area-specific random effect and an unstructured random term, across geographic units. It has been extensively shown that not accounting for an actual spatial variability may lead to major biases [3].

Conversely, if the spatial variability of a health indicator is completely explained by that of environmental factors and the other ecological covariates taken into consideration, regression residuals do not have spatial structure. Modelling the spatial structure of residuals could then lead to a biased estimate of the ecological association via a phenomenon of over-fitting [47].

To the author's knowledge, the quantitative impact of fitting a model including extra-Poisson variability to analyse data generated by a model where such extra-Poisson variability does not exist conditionally on the covariates included in the model (over-fitting) has not previously been explicitly and quantitatively investigated. Robustness of residuals modelling as BYM was studied in a different inferential context. In the frame of an extensive investigation of the statistical performances of a number of spatial models, Lawson et al. [8] studied the performance of such models on relative risks estimates in the case of mapping modelling, i.e. without covariates, where different true spatial structures of residuals were simulated. The authors showed that BYM model performed well on risks estimations except when the true residuals were resulting from a mixture structure. Different models were compared to detect effect from a putative source on a regular lattice [9]. They concluded in particular that the introduction of a spatially structured area-specific random effect leads to much less bias in the parameter estimate and that BYM model is the least biased. Notably, biases in parameter estimation appear when the random effects are not acounted for. In the present study we focused on ecological association estimate when covariates with spatially structure are introduced. Such covariates are often of interest in epidemiology when environmental exposures are studied. In our work we focus on a particular model with France mainland as study domain. The aim of our study was then to determine the robustness of the BYM model in the absence of residual spatial variation, i.e., the impact on the ecological association estimate on an irregular domain. The estimates performances were discussed and characterized according to the covariates structure. Simulations protocols assumed systematically a Poisson model at the first stage of the hierarchical model and log linear relationship between the incidence of the disease and exposure at the second stage without addition of extra-Poisson residuals. Various spatial structures of exposure were considered. The explained spatial variability is thus fully specified/attributable by the covariate structure.

First, ecological models that do or do not allow spatially structured or unstructured heterogeneity will be considered. Various simulation protocols for parameter values enabling balanced or unbalanced between/within area variability will then be presented. The results of the various simulation protocols will then be considered in terms of their performances with regard to the estimation of ecological associations. The paper will conclude with a discussion.

2 Methods

2.1 Statistical Models

Let D be the study area of interest, partitioned into m geographic areas. The data consist in the observed Y i and expected E i disease counts for each area i, (i = 1,...,m). Let X i be an ecological variable of interest in area i. The ecological Poisson M0 model is expressed in hierarchical form as follow:

Y i | R i ~ Poisson  ( E i R i ) log ( R i ) | α , β , X i = α + β X i ( M 0 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaafaqabeqacaaabaqbaeaabiWaaaqaaiabdMfaznaaBaaaleaacqWGPbqAaeqaaOGaeiiFaWNaemOuai1aaSbaaSqaaiabdMgaPbqabaaakeaacqGG+bGFaeaacqqGqbaucqqGVbWBcqqGPbqAcqqGZbWCcqqGZbWCcqqGVbWBcqqGUbGBcqqGGaaicqGGOaakcqWGfbqrdaWgaaWcbaGaemyAaKgabeaakiabdkfasnaaBaaaleaacqWGPbqAaeqaaOGaeiykaKcabaGagiiBaWMaei4Ba8Maei4zaCMaeiikaGIaemOuai1aaSbaaSqaaiabdMgaPbqabaGccqGGPaqkcqGG8baFiiGacqWFXoqycqGGSaalcqWFYoGycqGGSaalcqWGybawdaWgaaWcbaGaemyAaKgabeaaaOqaaiabg2da9aqaaiab=f7aHjabgUcaRiab=j7aIjabdIfaynaaBaaaleaacqWGPbqAaeqaaaaaaOqaaiabcIcaOiabd2eannaaBaaaleaacqaIWaamaeqaaOGaeiykaKcaaaaa@6476@

in which R i is the relative risk in area i. The second stage models the relationship between the relative risk and exposure variable. The ecological model M0 does not include any spatially structured or unstructured heterogeneity.

In order to account for those variabilities, the BYM model [2] was proposed :

Y i | R i ~ Poisson ( E i R i ) log ( R i ) | α , β , U i , V i = α + β X i + U i + V i U i | U j i ~ N ( j δ i U j / n i , 1 / τ U 2 n i ) V i | τ V ~ N ( 0 , τ V 2 ) ( B Y M ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaafaqabeqacaaabaqbaeaabqWaaaaabaGaemywaK1aaSbaaSqaaiabdMgaPbqabaGccqGG8baFcqWGsbGudaWgaaWcbaGaemyAaKgabeaaaOqaaiabc6ha+bqaaiabbcfaqjabb+gaVjabbMgaPjabbohaZjabbohaZjabb+gaVjabb6gaUjabcIcaOiabdweafnaaBaaaleaacqWGPbqAaeqaaOGaemOuai1aaSbaaSqaaiabdMgaPbqabaGccqGGPaqkaeaacyGGSbaBcqGGVbWBcqGGNbWzcqGGOaakcqWGsbGudaWgaaWcbaGaemyAaKgabeaakiabcMcaPiabcYha8HGaciab=f7aHjabcYcaSiab=j7aIjabcYcaSiabdwfavnaaBaaaleaacqWGPbqAaeqaaOGaeiilaWIaemOvay1aaSbaaSqaaiabdMgaPbqabaaakeaacqGH9aqpaeaacqWFXoqycqGHRaWkcqWFYoGycqWGybawdaWgaaWcbaGaemyAaKgabeaakiabgUcaRiabdwfavnaaBaaaleaacqWGPbqAaeqaaOGaey4kaSIaemOvay1aaSbaaSqaaiabdMgaPbqabaaakeaacqWGvbqvdaWgaaWcbaGaemyAaKgabeaakiabcYha8jabdwfavnaaBaaaleaacqWGQbGAcqGHGjsUcqWGPbqAaeqaaaGcbaGaeiOFa4habaGaemOta4KaeiikaGYaaabeaeaacqWGvbqvdaWgaaWcbaGaemOAaOgabeaakiabc+caViabd6gaUnaaBaaaleaacqWGPbqAaeqaaOGaeiilaWIaeGymaeJaei4la8Iae8hXdq3aa0baaSqaaiabdwfavbqaaiabikdaYaaakiabd6gaUnaaBaaaleaacqWGPbqAaeqaaOGaeiykaKcaleaacqWGQbGAcqGHiiIZcqWF0oazdaWgaaadbaGaemyAaKgabeaaaSqab0GaeyyeIuoaaOqaaiabdAfawnaaBaaaleaacqWGPbqAaeqaaOGaeiiFaWNae8hXdq3aaSbaaSqaaiabdAfawbqabaaakeaacqGG+bGFaeaacqWGobGtcqGGOaakcqaIWaamcqGGSaalcqWFepaDdaqhaaWcbaGaemOvayfabaGaeGOmaidaaOGaeiykaKcaaaqaaiabcIcaOiabdkeacjabdMfazjabd2eanjabcMcaPaaaaaa@A873@

in which δ i denotes the set of labels of the neighbours of area i, n i is the number of neighbours i, U i (i = 1,...,m) models the spatially-structured area-specific random effect based on the conditional autoregressive approach CAR [10], and V i (i = 1,...,m) is the unstructured random effect. The BYM model is the benchmark parametric model and is widely used in disease-mapping studies mainly because of the flexibility of the residuals.

2.2 Design of the simulation study

Processes X and Y are simulated on D under model M0 accordingly to parameters values. The simulation parameters were selected with reference to the overall variability of the estimated relative risks, thus enabling realistic and reasonable values for relative risks. More precisely, let R ^ i MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGsbGugaqcamaaBaaaleaacqWGPbqAaeqaaaaa@2F70@ = Y i /E i be the maximum likelihood estimate of the relative risk for the area. If X i ~ N(0, 1), then:

V a r [ log ( R ^ i ) ] E ( 1 E i R i ) + β 2 . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGwbGvcqWGHbqycqWGYbGCdaWadaqaaiGbcYgaSjabc+gaVjabcEgaNjabcIcaOiqbdkfaszaajaWaaSbaaSqaaiabdMgaPbqabaGccqGGPaqkaiaawUfacaGLDbaacqGHijYUcqWGfbqrcqGGOaakdaWcaaqaaiabigdaXaqaaiabdweafnaaBaaaleaacqWGPbqAaeqaaOGaemOuai1aaSbaaSqaaiabdMgaPbqabaaaaOGaeiykaKIaey4kaSccciGae8NSdi2aaWbaaSqabeaacqaIYaGmaaGccqGGUaGlaaa@4A98@

If the relative risks are spatially independent of the expected disease counts E i ,

E ( 1 E i R i ) 1 / E ¯ h × E ( 1 / R i ) , MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGfbqrcqGGOaakdaWcaaqaaiabigdaXaqaaiabdweafnaaBaaaleaacqWGPbqAaeqaaOGaemOuai1aaSbaaSqaaiabdMgaPbqabaaaaOGaeiykaKIaeyisISRaeGymaeJaei4la8IafmyrauKbaebadaWgaaWcbaGaemiAaGgabeaakiabgEna0kabdweafjabcIcaOiabigdaXiabc+caViabdkfasnaaBaaaleaacqWGPbqAaeqaaOGaeiykaKIaeiilaWcaaa@4664@

where E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ is the harmonic mean of E i (i = 1,...,m). The overall variance may be expressed as:

V a r [ log ( R ^ i ) ] 1 / E ¯ h × exp ( α + β 2 ) + β 2 . MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGwbGvcqWGHbqycqWGYbGCdaWadaqaaiGbcYgaSjabc+gaVjabcEgaNjabcIcaOiqbdkfaszaajaWaaSbaaSqaaiabdMgaPbqabaGccqGGPaqkaiaawUfacaGLDbaacqGHijYUcqaIXaqmcqGGVaWlcuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaOGaey41aqRagiyzauMaeiiEaGNaeiiCaaNaeiikaGIaeyOeI0ccciGae8xSdeMaey4kaSIae8NSdi2aaWbaaSqabeaacqaIYaGmaaGccqGGPaqkcqGHRaWkcqWFYoGydaahaaWcbeqaaiabikdaYaaakiabc6caUaaa@542C@

This variance may be broken down into a Within area variability term Wv = 1/ E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ × exp(-α + β2) and a Between area variability term Bv = β2. Let p denote the proportion of between area variance, p = Bv/Var[log( R ^ i MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGsbGugaqcamaaBaaaleaacqWGPbqAaeqaaaaa@2F70@ )], a high value of p corresponds to high between-area variability, that is a high amount of information with which to estimate the ecological link β. Hereafter, without any loss of generality, α will be considered equal to 0.

The geographic scale unit consisted in the 94 Departements of mainland France (Corsica excluded). The expected disease counts (E i ) consist in the expected cases of acute leukaemia in children aged less than 15 years for the period 1990–1998 in Departement i. The cases were retrieved from the French National Registry of Childhood Leukaemia and Lymphoma [11]. The expected numbers ranged from 4.2 to 204, with a harmonic mean of 23.35. Scenarios in which the within-area variance was either doubled ( E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 46.6) or divided by 10 ( E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 2.33) were also considered.

Given that X i ~ N(0, 1), within-area variance depends on 3 parameters, namely: the harmonic mean of expected disease counts E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ , the ecological link β and the autocorrelation structure of X i .

As X has a standardized normal distribution, the 2.5 (p2.5%) and 97.5 (p97.5%) percentiles of the relative risks are under model M0 exp(α ± 1.96β) and their ratio Q = p97.5%/p2.5% is exp(2 × β × 1.96). The quantile ratios were considered equal to 1.0, 1.5, 2.0 and 3.0, equivalent to no effect, weak, moderate and strong effects, respectively. The corresponding values for β were 0.00, 0.12, 0.21 and 0.33. The proportions of between-area variance, p, by ecological link β and E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ breakdown are summarized in Table 1.

Table 1 Between area variance proportion, p(3), according to ecological link β and E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@

For E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 23.35, the between area variance proportion ranges from 0 to 71%, with a balanced case for an ecological link when β = 0.21. For E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 46.69, p ranges from 0 to 83%, while p ranges from 0 to 19% when E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 2.33.

The autocorrelation of the exposure variable was also modulated. The following exponential autocorrelation structure was considered: cov(X i , X j ) = exp(-d(i, j)φ), in which d(i, j) is the distance between areas i and j. Let ρ xx = exp(-100φ) be the autocorrelation of two areas 100 km distant from each other. The following values for ρ xx = (0.40, 0.90,0.95, 0.98) were studied. That correlation structure is shown in Figures 1. High values of ρ xx may mimic a spatial bloc structure.

Figure 1
figure 1

Replicates of a gaussian covariate with autocorrelation strenght of ρ xx = 0.80 (left) and ρ xx = 0.98 (right) at 100 km, (in quartiles).

For each combination of parameters ( E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ , β, ρ xx ), 400 replicates of (X, Y) = ((X i , Y i ), i = 1,...,N) were generated using the M0 model. For each replication, the ecological link was estimated by both models (M0 and BYM) in a Bayesian framework.

The estimations were made with BRugs [12] software. For each data set, a burn-in of 5000 iterations was used and Bayesian inferences were based on 45000 iterations from Gibbs sampling giving Monte Carlo standard errors of less than 5% of the posterior standard deviation of each parameter [13]. The Monte Carlo standard error is an estimate of the difference between the mean of the sampled values and the true posterior mean. Non-informative priors were chosen for the parameters: α ~ U(-∞; +∞), β ~ N(0.0, 1.0E + 5), τ U ~ Γ(0.5, 0.0005), τ V ~ Γ(0.5, 0.0005) [14], in which Γ(a, b) denotes the Gamma distribution with expectation equal to a/b.

For each triplet ( E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ , β, ρ xx ), the ecological link was estimated using the M0 and BYM models. Let β ^ ( j ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaaiiGacuWFYoGygaqcamaaCaaaleqabaGaeiikaGIaemOAaOMaeiykaKcaaaaa@31A0@ be the posterior mean estimates of β at the jth replication, σ β ^ ( j ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaaiiGacqWFdpWCdaWgaaWcbaGaf8NSdiMbaKaadaahaaadbeqaaiabcIcaOiabdQgaQjabcMcaPaaaaSqabaaaaa@3396@ the posterior standard error of β ^ ( j ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaaiiGacuWFYoGygaqcamaaCaaaleqabaGaeiikaGIaemOAaOMaeiykaKcaaaaa@31A0@ and CIj(β) the 95% credibility interval of β. The following criteria were computed for 400 replications (j = 1,...,400):

• The empirical mean of the estimated posterior expectations of β ( β ¯ s i m = j = 1 400 β ^ ( j ) / 400 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaaiiGacqWFYoGydaqadaqaaiqb=j7aIzaaraWaaSbaaSqaaiabdohaZjabdMgaPjabd2gaTbqabaGccqGH9aqpdaaeWaqaaiqb=j7aIzaajaWaaWbaaSqabeaacqGGOaakcqWGQbGAcqGGPaqkaaGccqGGVaWlcqaI0aancqaIWaamcqaIWaamaSqaaiabdQgaQjabg2da9iabigdaXaqaaiabisda0iabicdaWiabicdaWaqdcqGHris5aaGccaGLOaGaayzkaaaaaa@47D5@

• The empirical mean of the estimated posterior standard deviations of β ( σ ¯ β s i m = j = 1 400 σ β ^ ( j ) / 400 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaaiiGacqWFYoGydaqadaqaaiqb=n8aZzaaraWaaSbaaSqaaiab=j7aInaaBaaameaacqWGZbWCcqWGPbqAcqWGTbqBaeqaaaWcbeaakiabg2da9maaqadabaGae83Wdm3aaSbaaSqaaiqb=j7aIzaajaWaaWbaaWqabeaacqGGOaakcqWGQbGAcqGGPaqkaaaaleqaaOGaei4la8IaeGinaqJaeGimaaJaeGimaadaleaacqWGQbGAcqGH9aqpcqaIXaqmaeaacqaI0aancqaIWaamcqaIWaama0GaeyyeIuoaaOGaayjkaiaawMcaaaaa@4BC1@

• The empirical standard deviation of the 400 estimated posterior means of β (sd(β sim ))

• The mean relative bias (MB) and its standard deviation (sd(MB)),

• The root mean square error (RMSE),

• The proportion of coverage, π(β): the percentage of time when β lay within its 95% credibility interval

• The proportion of non-coverage 1 - π(0): the percentage of time when 0 did not lie within its 95% credibility interval

When β ≠ 0, 1 - π(0) quantifies the ability of the estimation model to detect the existence of an association, which is analogous to the frequentist power. McNemar's test for comparing proportions from paired data (estimates of β from M0 and BYM based on the same replicated data set) was used to test whether the π(β) (or 1 - π(0)) values were significantly different. The over-fit of the BYM model (compared to the M0 model) was assessed via that criterion.

3 Simulation Results

Simulations results are structured as follow: firstly, we present results for an harmonic mean E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 23.35 of expected disease counts equal to those from acute leukaemia in children for 1990–1998 in France and a covariate X with null to moderate autocorrelation. Secondly, for the same harmonic mean E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ , we study the influence of strong autocorrelations for the covariate X and finally, variations of E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ (smaller and greater than 23.35) are explored.

3.1 Moderate covariate autocorrelation

The first scenario considered E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 23.35 and the autocorrelation equal to 0.0 or 0.4; the results are shown in Table 2. In that setting, the between-area variance varied from 0 to 71%. In the absence of any spatial structure for X (ρ xx = 0), the estimate of β was unbiased, irrespective of the estimation model. The mean bias was less than 1% and the RMSE was less than 0.02 for the four values of the ecological link (for both models). The coverage proportions π(β) were similar for the two models and greater than 94.5%. The coverage proportion of the BYM model was consistently slightly greater than that of the M0 model. For β = 0, π(β) was close to 95% and, equivalently, 1 - π(0) was close to 5%. The BYM model thus handles the scenario in which the covariate has no spatial structure.

Table 2 Estimation of the ecological link β when E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 23.35 and ρ xx = 0.0, 0.4 (400 replications)

When the autocorrelation was increased to 0.4, the results were similar to those with the previous setting. The mean bias was less than 1%. There was a slight increase in the RMSE but it remained less than 0.02. The β coverage proportion with the BYM model was greater than that with the M0 model. The non-coverage proportion was equal to 1, except when there was no association (β = 0). The non-coverage proportions were significantly different when β = 0. The proportion the closest to 5% was obtained with the M0 model. Irrespective of the value of β, the variability of β was always slightly over-estimated with the BYM model. The coverage proportion of the M0 model varied from 94.5 to 95.8% (96.8 to 97.5% for the BYM model). In the absence of, or with moderate, autocorrelation, the over-fitting effect was not observed. Both models provided an almost unbiased estimate of the ecological link.

3.2 Strong covariate autocorrelation

The scenario of strong autocorrelation for X was then considered: ρ xx = 0.90, 0.95, 0.98 at 100 km with E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 23.35. The results are shown in Table 3. When ρ xx = 0.90, the mean bias and the RMSE were low (0.03). The mean bias decreased as the value of the ecological link increased, reflecting an increase in between-area variability. The coverage proportion with the BYM model was higher than the coverage proportion with the M0 model for all values of β. The non-coverage proportions were significantly different for β = 0.00 and β = 0.12, in favor of the M0 model. For ρ xx = 0.95, the bias was still small and the RMSE increased to 0.04. The non-coverage proportions were again significantly different in the cases in which β = 0.00 and β = 0.12 in favor of the M0 model. Lastly, for ρ xx = 0.98 and for the first three values of β (β = 0.00, 0.12, 0.21), the non-coverage proportions were significantly different, again in favor of M0. When the ecological link was null, the non-coverage proportion was again smaller with the BYM model, a consequence of the over-estimation of parameter variability. The β coverage proportions were lower for M0 for 4 values of β. When the autocorrelation increased from 0.90 to 0.98, the RMSE increased from 2.99 to 6.48, mainly due to the decrease in independent information. A slight increase was observed for all the other criteria. The bias was weak, resulting in very small RMSE and sd(β sim ) in both models. At high autocorrelation values, the overall variability of the estimates increased. There was more variability for each β value with the BYM model. This is exemplified by the mean posterior standard deviation, which increased four-fold (for all β values) between the first (ρ xx = 0.00) and last (ρ xx = 0.98) autocorrelation scenario. This was also the case for sd(β sim ).

Table 3 Estimation of the ecological link β when E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 23.35 and ρ xx = 0.90, 0.95, 0.98

3.3 Variation of (harmonic mean of) expected counts

The next scenario consisted in strong autocorrelation of ρ xx = 0.95 at 100 km with variation in number of expected disease counts. The results for that scenario are shown in Table 4.

Table 4 Estimation of the ecological link β when ρ xx = 0.95 while E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ varying

For E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 46.6 (and ρ xx = 0.95), the mean bias and RMSE were smaller than in the scenario in which E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 23.3. The β coverage rate was greater than 94.5% for both models and the proportion was higher for the BYM model. The non-coverage proportions were significantly different, in favor of the M0 model, for β = 0.12. For β = 0, the non-coverage proportion was again smaller with the BYM model.

For E ¯ h MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWGfbqrgaqeamaaBaaaleaacqWGObaAaeqaaaaa@2F5C@ = 2.33, the bias increased (up to 6%) and the RMSE was the highest observed in the various cases (14% approx.). The coverage proportion was smaller than in the previous scenario but greater than 92%. The coverage proportion π(β) was higher with the BYM model than with the M0 model. While the non-coverage proportions of 0 were close to 1 (except for β = 0), the proportions decreased by 16% for M0 and 13% for BYM for β = 0.12. Moreover the non-coverage proportions of 0 for the two models were significantly different and in favor of the M0 model for β = 0.12, 0.21, 0.33.

High autocorrelations thus appear to influence the over-fitting effect of the BYM model. The expected disease counts, also modulates the overall accuracy of the estimation. In fact, with highly correlated spatial structure and low disease counts, the bias of the β estimate generated by the BYM increases. But, even with a highly autocorrelated covariate and adequate disease counts, when E i is doubled, the BYM model estimates the ecological link with little bias.

4 Discussion

A simulation study was conducted in order to assess estimation performance with respect to the ecological association between covariates and health indicators. Key parameters, such as the ecological link, expected disease counts and autocorrelation strength were selected to ensure that the simulation covered realistic situations. The choice of parameters enabled coverage of balanced and unbalanced between- and within-area variabilities.

For moderate autocorrelation structures, both the Poisson model and the BYM model performed well and the estimation performances were similar. Underestimation of ecological links was only observed for high autocorrelations. Overall, the posterior standard deviation of β was slightly over-estimated with the BYM model, resulting in conservative results when the true value of β was null.

The expected disease counts are also of interest because, with a high autocorrelation, the underestimation of the BYM model is present. In practice, this worst-case scenario can nonetheless be found. Except for the extreme scenario, strong spatial structure and low disease counts, both models perform well, even with strong spatial structure. As a consequence, the BYM model can be used to estimate ecological associations without fearing underestimation. The simulation results show that models accounting for structured and unstructured residuals do not underestimate materially the ecological association. The rational is the following: not accounting for an actual spatial variability leads to strong bias. Thus from a practical point of view, the BYM model should be preferred to the Poisson if spatial autocorrelation of covariate is suspected. Moreover, autocorrelation structure will be first investigated via Moran's I test [15].

References

  1. Evrard A, Hémon D, Billon S, Laurier D, Jougla E, Tirmarche M, Clavel J: Ecological association between indoor radon concentration and childhood leukaemia incidence in France, 1990–1998. Eur J Cancer Prev. 2005, 14 (2): 147-57. 10.1097/00008469-200504000-00011.

    Article  PubMed  CAS  Google Scholar 

  2. Besag J, York Y, Mollié A: Bayesian Image Restoration With Two applications In Spatial Statistics. Ann Inst Statist Math. 1991, 43: 1-59. 10.1007/BF00116466.

    Article  Google Scholar 

  3. Salway R: Statistical Issues in the Analysis of Ecological Studies. PhD thesis. 2003, Imperial College School of Medicine

    Google Scholar 

  4. Clayton D, L B, C M: Spatial Correlation in Ecological Analysis. International Journal of Epidemiology. 1993, 22: 1193-1202. 10.1093/ije/22.6.1193.

    Article  PubMed  CAS  Google Scholar 

  5. Best NG, Arnold RA, Thomas A, Waller LA, Conlon EM: Bayesian models for spatially correlated disease and exposure data. Bayesian Statistics 6. 1999, Oxford University Press, 131-56.

    Google Scholar 

  6. Richardson S: Spatial models in epidemiological applications. Highly Structured Stochastic Systems, Volume 27 of Oxford Statistical Science Series. Edited by: Green PJ, Hjort NL, Richardson S. 2003, Oxford: Oxford University Press, 237-259.

    Google Scholar 

  7. Wakefield J: Sensitivity analyses for ecological regression. Biometrics. 2003, 59: 9-17. 10.1111/1541-0420.00002.

    Article  PubMed  Google Scholar 

  8. Lawson AB, Biggeri AB, Boehning D, Lesaffre E, Viel JF, Clark A, Schlattmann P, Divino F: Disease mapping models: an empirical evaluation. Disease Mapping Collaborative Group. Statistics in Medicine. 2000, 19 (17–18): 2217-2241. [Comparative Study]

    PubMed  CAS  Google Scholar 

  9. Ma B, Lawson AB, Liu Y: Evaluation of Bayesian models for focused clustering in health data. Environmetrics. 2007, [DOI: 10.1002/env.850]

    Google Scholar 

  10. Besag J: Spatial interaction and the statistical analysis of lattice systems (with discussion). Journal of Royal Statistical Society, Series B. 1974, 36: 192-236.

    Google Scholar 

  11. Clavel J, Goubin A, Auclerc MF, Auvrignon A, Waterkeyn C, Patte C, Baruchel A, Leverger G, Nelken B, Philippe N, Sommelet D, Vilmer E, Bellec S, Perrillat-Menegaux F, Hemon D: Incidence of childhood leukaemia and non-Hodgkin's lymphoma in France: National Registry of Childhood Leukaemia and Lymphoma, 1990–1999. Eur J Cancer Prev. 2004, 13 (2): 97-103. 10.1097/00008469-200404000-00002.

    Article  PubMed  CAS  Google Scholar 

  12. Thomas A, O'Hara B: BRugs: OpenBUGS and its R interface BRugs. 2005

    Google Scholar 

  13. Roberts G: Markov chain concepts related to sampling algorithms. Markov chain Monte Carlo in Practice. Edited by: Gilks W, Richardson S, Spiegelhalter D. 1996, Chapman and Hall

    Google Scholar 

  14. Kelsall JE: Discussion of Bayesian models for spatially correlated disease and exposure data. Bayesian Statistics 6. 1999, Oxford University Press, 131-56.

    Google Scholar 

  15. Moran P: Notes on continuous stochastic phenomena. Biometrika. 1950, 37: 17-23.

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgements

This work was supported by grants AFSSE (RD2004004) and INSERM-ATC (A03150LS). The first author was supported by INSERM (poste Blanc)

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Aurélien Latouche.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Latouche, A., Guihenneuc-Jouyaux, C., Girard, C. et al. Robustness of the BYM model in absence of spatial variation in the residuals. Int J Health Geogr 6, 39 (2007). https://doi.org/10.1186/1476-072X-6-39

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1476-072X-6-39

Keywords