Rapid post-disaster damage assessment using multi-modal large language models

PI: Maryam Rahnemoonfar  

University: Lehigh University

Industry partners: Arch, Everest 

Frequent and increasingly severe natural disasters—such as hurricanes, floods, and wildfires—pose escalating risks to human life, infrastructure, and economic stability. It is essential for insurance companies to perform rapid, accurate, and scalable estimates of damaged buildings and infrastructure, as well as reliable attribution of damage causes (e.g., wind, flood, or fire). Timely post-disaster intelligence is critical for claims triage, loss reserving, reinsurance decisions, and efficient allocation of response and recovery resources. Delays or inaccuracies in early damage assessment can significantly increase operational costs and slow policyholder support.

Existing post-disaster damage assessment systems rely heavily on newly collected aerial or satellite imagery that must be manually annotated, resulting in substantial delays during the most time-sensitive phase of insurance response. Most current approaches fine-tune semantic segmentation or object detection models to classify damage levels; however, their accuracy depends on large, disaster-specific labeled datasets that are rarely available immediately after an event. As a result, insurers often face a trade-off between speed, accuracy, and geographic scalability.

Recent advances in multi-modal large language models (MLLMs) provide a new opportunity to overcome these limitations by combining visual perception with language-based reasoning. MLLMs can interpret complex visual damage cues alongside contextual information, enabling higher-level reasoning about damage severity and likely causes. When used with in-context learning, pretrained MLLMs can perform robust damage assessment without additional retraining, significantly reducing data requirements and deployment time.

We propose a hybrid framework that integrates a general-purpose pretrained remote-sensing object detector with the reasoning capabilities of MLLMs. The detector first localizes buildings in post-disaster imagery. Each target building and its surrounding context are then analyzed by the MLLM using visually similar in-context examples to estimate damage severity and infer probable damage causes. This approach enables rapid, scalable, and training-free damage assessment, supports flexible insurance-relevant damage taxonomies, and enhances robustness for real-world post-disaster insurance decision-making.