To our knowledge, this is the first study to have evaluated the validity of IDP or refugee population estimation based on satellite imagery in a variety of different sites and phases of displacement. Our findings suggest that a remote analysis approach relying on manual counting of structures and published occupancy estimates can achieve reasonable precision in sites where individual structures are distinguishable and neither clouds nor vegetation pose a significant barrier to visual analysis.
The method’s performance on the whole suggests that, rather than referring to it as a valid approach, one could consider it “good enough” for certain purposes, assuming that no robust ground estimation is possible within the same timeframe. Specifically, while inaccuracy of up to 30% is probably unacceptable in post-acute emergency scenarios where resources for on the ground population tracking are present, we believe that for the purposes of initial planning (e.g. vaccination, distribution of food and non-food items, emergency water and sanitation provision), this level of inaccuracy is a substantial improvement over no information or guesswork, which might be the case if the site is inaccessible or if expertise in ground estimation cannot be sourced. However, the expected level of inaccuracy of the method would have to be explicitly emphasised when presenting this as an option for rapid estimation.
Remote analysis appears feasible in terms of human resources and financial inputs: in our study, it required 2–5 days, two analysts and, apart from salary and office expenditures, only minimal imagery procurement costs (15 to 25 USD per km2, though these costs would be somewhat higher if images were commissioned).
Visual analysis of the imagery was not overly complicated, despite most analysts in this study having no prior GIS skills. However, the visual quality and complexity of the image were critical determinants of both speed and accuracy of counting. While experienced spatial analysts may be able to improve image quality by using various techniques that enhance the visibility of features, we wished to evaluate use of the method by analysts with limited GIS skills, and thus refrained from making such improvements to the images. Moreover, in many instances the very typology and layout of structures (e.g. multiple walls, structures connected to each other and removal or abandonment of structures) imposed a limit on accuracy that, given the present resolution of commercially available imagery, is likely to remain to some extent intractable (see Conclusions). However, having four or more bands in the multi-spectral image did help in a few cases to distinguish between vegetation and man-made structures when the latter were constructed out of different materials, and we believe therefore that these options should always be selected when obtaining imagery. Beyond these challenges, availability of cloud free images may be a serious constraint in some locations, particularly when a very short delay between the analysis and image time point is needed (i.e. in dynamic, evolving situations): for example, in DRC we excluded several candidate sites for analysis because no cloud free images were available.
While some occupancy data were available for each site, the literature search was onerous and had a low yield. For one site (Sherkole), the sole estimate available was clearly implausible and resulted in an under-estimate of population. In general, we found very few actual structure occupancy estimates, and had to rely instead on household size figures. These were sometimes fairly divergent within the same site (see Additional file 1), and their sparsity made it difficult to construct statistically meaningful confidence intervals around the population estimates. The hierarchy of evidence for structure occupancy information that we used to attribute weights to each report (Table 2) is an attempt to rely on all information available while minimising likely bias, but criteria and scores used in this hierarchy are ultimately arbitrary and can never fully reflect the actual validity of any individual estimate. Occupancy is known to fluctuate over time, particularly in situations of protracted crisis, and thus to some extent our decision to include data from fairly remote periods may actually have increased inaccuracy (of note, sites with the highest information score did not have the greatest accuracy). It is likely that this may partly explain why sites with quality images and simple structure layouts (Farchana, Bambu) did not perform as well as expected. This limitation could be addressed by carrying out a small, rapid structure occupancy survey to provide locally appropriate data, but this is only an option if the site is accessible and diminishes the method’s relative advantages over other options. We tested such an approach in Chad (paper forthcoming).
The above drawbacks are partly a result of our choice to investigate a simple, manual method designed to empower non-specialists to carry out population estimation. Automated counting methods would necessitate a far higher skills level and thus require input by centres of excellence in remote sensing. While a review of automated or semi-automated methods is beyond the scope of this paper, this is an area of vibrant research, and we believe that these methods have a considerably larger potential for improvement than manual analysis. Automation would prove particularly valuable in scenarios where population estimates need to be updated frequently to track displacement dynamics, and could perhaps provide a solution for urban areas in which the manual method may never perform as accurately as needed.
Our findings should be considered conservative, as they reflect application of the method in a more challenging set of conditions than would be the case in more current application by an agency with a recognised mandate. In prospective application of the method, it may be possible to commission new imagery, thereby ensuring a minimal time difference between the analysis and imagery time points, though again subject to constraints such as cloud cover. Contemporary sensors increasingly have wider spectral ranges, allowing various false colour combinations so as to maximise the contrast between structures and other landscape features. While costs of imagery are already reasonable, it is also likely that in future crises VHSR imagery will be reducing in cost and, in certain large scale emergencies, be available free of cost as was the case in Haiti. Photographs taken by unmanned drones or other aircraft could also be used, particularly if agencies pool together resources to obtain such images. However, it should be noted that coordination of agencies around procurement and use of satellite imagery, as well as, more broadly, sharing of resources for timely assessment and monitoring, has proven challenging in a variety of recent emergencies, and may remain so unless a clear mandate and resources are attributed to a lead agency.
In routine practice, it is likely that some real-time ground information may be available to the analyst, though data requests would have to be of limited burden to field workers (e.g. if email contact with anyone familiar with the site is possible, sample image screen-grabs could be shared with the field to help define the nature of certain areas or structures); in the early phase of displacement, it is likely that there would be few non-residential structures, thus simplifying the count; furthermore, the bank of occupancy reports available for any given site would probably be larger, as current emergencies benefit from more frequent assessments and household surveys, with a greater proportion of reports made available online; further reports could also be obtained through contact with field agencies, and the occupancy data bank could be built up progressively.
On the other hand, our results may not be fully reflective of the range of conditions found in contemporary IDP or refugee settlements. Despite a broad search, we could not identify a sufficient number of IDP, acute emergency and urban sites to analyse, mainly because of the lack of reference population estimates. A relative majority of displaced people currently are IDPs in urban settings . It is possible that the choice of sites we analysed may have led to overly optimistic conclusions regarding the method’s likely performance. Our method alone would not be useful for urban sites featuring displaced populations living alongside residents: a ground survey would be needed alongside it to estimate the number of displaced occupants per structure, as discussed above.
Lastly, despite investigating several sites, this study should not be seen as providing definitive answers. In particular, due to limited resources we did not fully explore the likely extent of inter-rater reliability among different analysts: future evaluations should test counting accuracy and agreement on a larger panel of analysts with varying expertise.