Predicting space weather extremes

Research background

Extreme geomagnetic storms — driven primarily by coronal mass ejections or strong high‑speed solar wind streams — can significantly affect both natural systems and modern technological infrastructure, damaging transformers and causing blackouts. In 1989, a geomagnetic storm knocked out power to more than 6 million people in Quebec.

At the University of Otago, undergraduate student Aimee Cody and supervisor Prof. Ting Wang are exploring ways to better predict the onset of extreme geomagnetic storm events.

Geomagnetic data from the Eskdalemuir Magnetic Observatory, specifically weekly maximums of rates of change of the Earth’s magnetic field, can be a reliable quantifier of geomagnetic disturbance.

Statistically, models aimed at predicting such events often rely on maximising a likelihood function, which has a high-dimensional multi-modal likelihood surface and means estimation greatly relies on initialisation.

The researchers developed an R package for the estimation and inference in hidden Markov models and hidden semi-Markov models, dynamic stochastic models to capture regime-switching behavior within a system that is assumed to be driven by a latent Markov chain.

The researchers apply a generalised extreme value (GEV) observation distribution, generally used for ‘extreme’ observations such as maximums over some time increment. To overcome this they cycle through 50-500 initialisations to explore the parameter space and ensure that a set of maximum likelihood estimators are reached. However this creates a computational bottleneck, which would be impossible for a regular computer to run in a reasonable amount of time.

Project challenges

Aimee and Ting developed a package written in the R and C++ programming languages that estimates parameters from Hidden Markov models and Hidden semi-Markov models.

The execution time of their package can be very long. For instance, running a six-state model with 200 initial values takes more than two days for a complicated distribution and a sample size of about 5,000.

Aimee and Ting approached REANNZ to get help with parallelising their package to shorten that processing time.

What was done

Alexander Pletzer, a Research Software Engineer at REANNZ, helped Aimee adopt FAIR (Findable, Accessible, Interoperable and Reproducible) software development practices.

This involved:

• Setting up a Git repository that tracks code changes. Aimee and her colleagues can now compare codes in different branches and create pull-requests to merge new features into the main trunk
• Adding continuous integration via GitHub actions to check that code changes do not break existing code
• Code profiling to identify performance hot spots, i.e. code regions and functions where most of the execution time is spent
• Parallelising two functions using R’s future package. This significantly reduces the wall clock time by running loop iterations concurrently.
• Improving C++ code for performance and easier maintenance

The above plot shows the speedup achieved after parallelising a loop in the findmleHMMnostarting function, as the number of compute cores is increased. Note that one core is used by the manager, the remaining cores are consumed by the workers.

Main outcomes

A reduced execution time, allowing for quicker turnaround time. A nearly 10x speedup was achieved using 10 CPU cores.

Researcher feedback

"I received a massive amount of support from REANNZ, they had the computational knowhow I was missing to massively increase the performance of my package, I learnt many valuable skills that will additionally help in the future. I am very satisfied with this collaboration and the outcome package of it."

— Aimee Cody, Department of Mathematics and Statistics, University of Otago

This case study shares some of the technical details and outcomes provided through our Consultancy Service. This service supports projects across a range of domains, with an aim to lift researchers’ productivity, efficiency, and skills in research computing. Get in touch to discuss how our Research Software Engineers and specialist support could help advance your project.