GSEIO.GLOBAL SEISMIC EARTH INFORMATION OBSERVATORY
OPEN EARTHQUAKE INFORMATION
LAMBDA-FIS RESEARCH PLATFORM

Statistical Testing and Forecast Evaluation

Evaluating the Predictive Performance of Physical Earthquake Models

A central objective of LAMBDA-FIS is not only to develop physical earthquake forecasting models, but also to evaluate their predictive performance through objective statistical testing. The system integrates two complementary methods: the Molchan Test and the Monte Carlo PI-test. Together, they investigate whether physical forecasting models contain statistically significant information about the future spatial and temporal distribution of earthquakes.

The central LAMBDA-FIS system supports global earthquake catalogues, archived forecast fields, reproducible model configurations, and statistical reference models derived from historical seismicity. These capabilities make it possible to compare physical forecasts against the seismic activity actually observed during selected testing periods.

METHOD 01 · MONTE CARLO

Monte Carlo Simulations – PI-test

Testing Forecast Performance Against Synthetic Earthquake Catalogues

The PI-test is a Monte Carlo statistical procedure designed to evaluate whether earthquakes preferentially occur in regions assigned higher forecast values by a selected model. The method compares the values of a previously calculated forecast field at actual earthquake locations with the values obtained from a large number of synthetic earthquake catalogues. These synthetic catalogues represent a statistical reference hypothesis derived from historical background seismicity.

How Does the PI-test Work?

The procedure consists of four principal steps.

01

Evaluation of observed earthquakes

LAMBDA-FIS selects earthquakes within a specified testing period and magnitude range. For each earthquake, the system evaluates the maximum value of the selected forecast field within three geographical distances from the epicentre:

100 km200 km500 km

The evaluation uses saved model fields from a configurable period preceding the earthquake, excluding the earthquake's own UTC day. The observed forecast values are then averaged across the earthquakes included in the statistical test.

02

Construction of the statistical reference model

A background seismicity model is constructed from the earthquake catalogue preceding the testing period. The catalogue is declustered to reduce the influence of dependent earthquake sequences. Its spatial distribution is subsequently smoothed using a spherical Gaussian kernel. The resulting statistical model describes the geographical distribution of background seismicity against which the physical forecast is evaluated.

03

Monte Carlo simulations

LAMBDA-FIS generates a large number of synthetic earthquake catalogues, typically using 1,000 simulations or more. Each simulation preserves the observed earthquakes' UTC occurrence times and magnitudes, while drawing new epicentral locations from the statistical background distribution.

The same saved physical forecast fields are evaluated at the synthetic locations using the same geographical distances and temporal windows applied to the real earthquakes. This produces a distribution of forecast scores expected under the selected statistical reference hypothesis.

04

Statistical comparison

The observed mean forecast value is compared with the distribution of mean values obtained from the synthetic catalogues. The standardized difference is expressed by the PI statistic:

Π = Observed Mean − Synthetic MeanSynthetic Standard Deviation

Positive PI values indicate that the observed earthquakes occurred in regions with higher forecast values than expected under the statistical reference model. Negative PI values indicate the opposite relationship. The absolute magnitude of PI describes the standardized difference between observed and simulated results.

LAMBDA-FIS also calculates an empirical upper-tail Monte Carlo p-value, measuring how frequently the simulated catalogue scores equal or exceed the observed score. A small p-value indicates that the observed result would be unusual under the selected reference hypothesis.

What Do the PI-test Results Show?

For each model, the results are displayed as three diagrams corresponding to the 100 km, 200 km, and 500 km evaluation distances. The diagrams compare:

Observed Mean
Mean forecast value at the observed earthquake locations.
Synthetic Mean
Mean forecast value averaged across the simulated catalogues.
Synthetic Variability
Standard deviation or the central 95% interval of the Monte Carlo results.
PI Statistic (Π)
Standardized difference between observed and simulated mean scores.
Monte Carlo p-value
Empirical statistical significance under the selected reference hypothesis.

The horizontal axis represents the cumulative minimum earthquake magnitude threshold, allowing the analysis of forecasting performance for progressively larger earthquakes. These diagrams help determine whether a physical forecasting model identifies future earthquake locations more effectively than the selected statistical background model. The PI-test evaluates relative predictive performance. It does not, by itself, establish the ability to predict the exact time, magnitude, or epicentre of an individual earthquake.

METHOD 02 · ALARM-BASED EVALUATION

Molchan Test

Evaluating Earthquake Forecasting Efficiency

The Molchan test provides another approach to evaluating the performance of earthquake forecasting models. Originally developed by Molchan (1990), the method investigates the relationship between the geographical and temporal extent of forecast alarm regions and the proportion of earthquakes that occur outside those regions. A forecasting model is considered more informative when it successfully identifies a large proportion of target earthquakes while designating a comparatively small portion of the testing region as an alarm.

How Does the Molchan Test Work?

LAMBDA-FIS evaluates a series of alarm thresholds applied to a selected forecasting model. At each threshold, geographical cells with sufficiently high forecast values are included within the alarm region.

The method calculates two principal quantities:

Alarm Fraction (τ). The proportion of the evaluated space-time domain classified as an alarm. In LAMBDA-FIS, the alarm fraction can also be weighted according to a reference seismicity model, such as Relative Intensity (RI). This accounts for the geographical nonuniformity of background seismicity.

Miss Rate (ν). The proportion of target earthquakes occurring outside the forecast alarm regions. A low miss rate indicates that a larger proportion of observed earthquakes occurred within the regions selected by the forecasting model.

0110Alarm fraction τMiss rate ν ReferenceExample curve
Conceptual diagram. A Molchan curve illustrates the relationship between alarm coverage and missed events. This example is schematic, not a measured LFIS result.

The Molchan Diagram

The Molchan diagram displays the relationship between alarm fraction (horizontal axis, τ) and earthquake miss rate (vertical axis, ν). By varying the forecast threshold, LAMBDA-FIS constructs a curve describing the trade-off between spatial alarm coverage and missed earthquakes. An effective forecasting model aims to obtain low miss rates with relatively small alarm fractions. A reference diagonal can represent the expected performance of a model with no predictive information relative to the selected reference distribution.

Area Skill Score

LAMBDA-FIS also calculates an Area Skill Score, summarizing the performance of the complete Molchan curve. Higher values indicate more favourable forecasting performance within the selected reference framework. The central system supports three evaluation distances: 100 km, 200 km, and 500 km. It can also use Monte Carlo simulations to compare the observed Molchan performance with results expected under a statistical reference model.

The Molchan test therefore provides a complementary perspective to the PI-test. Whereas the PI-test compares actual forecast values at observed and synthetic earthquake locations, the Molchan test examines the efficiency of identifying earthquake occurrences using geographical alarm regions.

PUBLIC RESEARCH RESULTS

Latest Published PI-test Results

Independent Statistical Evaluation of LAMBDA-FIS Physical Models

GSEIO publishes selected statistical evaluation results obtained from the central LAMBDA-FIS research system. Initially, the public Statistics page presents the latest available completed PI-tests for two physical forecasting models.

PHYSICAL FORECAST MODEL 01

Shear Traction

The Shear Traction model investigates normalized forecasting fields derived from weighted seismic wave structures and tectonic interactions. The PI-test evaluates whether observed earthquakes preferentially occur in regions characterized by higher Shear Traction forecast values compared with earthquake locations generated from the statistical background model.

Not yet published

Results have not yet been published from the central LFIS archive. No example values are shown.

PHYSICAL FORECAST MODEL 02

Normalized Shear Stress

The Normalized Shear Stress model investigates spatial patterns of normalized shear stress derived from reconstructed tectonic stress tensors and candidate fault-plane orientations. Its PI-test compares forecast values at observed earthquake locations with those obtained from synthetic catalogues and evaluates whether the model provides statistically detectable information beyond the selected statistical reference model.

Not yet published

Results have not yet been published from the central LFIS archive. No example values are shown.

Test Documentation and Reproducibility

Each published evaluation identifies the tested model, evaluation period, earthquake catalogue, magnitude threshold, geographical distance, number of observed earthquakes, number of Monte Carlo simulations, and statistical reference configuration. Where available, the publication also documents forecast issue times, model versions, input provenance, and whether the evaluation is retrospective or based on forecasts genuinely issued before the observed earthquakes. Only completed results exported from the central LAMBDA-FIS archive are presented as measured statistical results.

SCIENTIFIC CONTEXT

Scientific Interpretation

From Forecasting Models to Verified Predictive Information

The scientific value of an earthquake forecasting model depends not only on its physical interpretation or spatial agreement with observed seismicity, but also on its performance against clearly defined statistical reference hypotheses. A strong visual correspondence between a forecast field and earthquake epicentres is not sufficient evidence of predictive skill. Statistical testing is necessary to determine whether that correspondence could plausibly arise from the historical geographical concentration of seismic activity.

The PI-test and Molchan test provide complementary methods for investigating these questions. Positive statistical results must nevertheless be interpreted in the context of catalogue completeness, model configuration, evaluation thresholds, dependence among earthquakes, and the distinction between retrospective and prospective testing. In particular, statistical significance under a selected null hypothesis does not automatically establish deterministic earthquake predictability.

LAMBDA-FIS is designed to advance physical earthquake forecasting through the continuous interaction of theoretical modelling, observational data, and reproducible statistical verification.

The public GSEIO platform makes selected results accessible for scientific examination and informed public discussion.

EXPLORE THE RESEARCH

Models and Forecasts

Learn about the complete central LFIS architecture, or examine the selected published forecasting fields in the public viewer.