The Met Office – or Meteorological Office, to give it its full name – is the UK’s national weather service, responsible for simulating and predicting all weather events to reach our shores. That means spending a hefty amount of compute power on simulation and emulation, as well as needing a fundamental understanding of how climate change is affecting the planet. In this interview, Noel Anderson talked to CIO and Delta subscriber Charles Ewen about what other industries can learn from the Met Office’s simulation work, the role that AI and machine learning will play in the future, and industry changes driven by the coronavirus pandemic.
1. Digital twins as an exciting area of growth, especially for applications in engineering
2. Prepare for possible futures scenarios, and be ready to act quickly based on the level of threat (or opportunity)
3. Remote working: it hasn’t caused the world to fall apart, but extended isolation from colleagues can cause mental health issues
4. Political will and sentimentality should not get in the way of facts and data. We need to continue this renewed trust in science
5. A preferred future is one where society at large are more educated about uncertainty, how to interpret data and realise that things will not necessarily get bigger, faster and richer!
Watch the video now, or read the transcript below.
00:14 Noel Anderson
Hello! The weather’s going to be wet and windy in the UK this week, and later in the month there’s a threat of severe disruption to services from a weather front known as the Beast from the East. However, there’s a 70 per cent chance that this will be deflected from the warmth of today’s guest. He’s the CIO of the Met Office, I’m pleased to welcome Charles Ewen. How are you today, Charlie?
00:42 Charles Ewen
Hi Noel. Thanks for the intro, I’m not sure I can see off a sudden stratospheric warming, but we’ll have to see.
Excellent. This series is called Tech to the Future. There aren’t many people whose sole responsibility is to use technology to predict the future. How do you use tech processes and people to help you achieve this goal?
Interesting question, Noel, because obviously, prediction is kind of becoming mainstream – courtesy of advances in areas like machine learning, for example. But I think it’s fairly safe to say that the Met Office and the name it represents – weather and climate forecasting – we’re probably there earlier than most. That’s not to say that we’re best at it, of course, it just means we’ve been doing it for a long time.
What we do is really simulation: simulations as a class of prediction. What that really means is we understand things like Navier-Stokes equations, or the physics associated with the dynamics and physics of the atmosphere and how it evolves over time. Our approach is to take those known laws of physics – stuff that we all forgot from high school – and to apply those at massive scale. That’s, fundamentally, our weather forecast work, over many years. In parallel with that, using emulation, or statistical representations, of parts of that have always been the case, and they become more [important] because, once again, we – like everybody – make more and more use of machine learning, which emulates based on being able to recognise patterns in data. It’s not quite the same thing as understanding the fundamental drivers for a system, it’s a bit like the difference between AI and machine learning.
Artificial intelligence has many, many definitions, but one would be to create a machine that could act and behave and think and learn in a way that you and I can – or at least maybe I could have done 30 years ago! Whereas machine learning is really all about building an algorithm that can recognise a pattern in a dataset. If the link is robust between those datasets that feed the machine learning algorithm and the system that those datasets represent, then we’re all good. But people talk about drift, for example; there are many reasons why a machine learning algorithm might drift over time, but one of them is that the datasets being derived from the system no longer represent the system itself. So, you need to go back and retune and retrain an algorithm – [although] you’d like to think the laws of physics aren’t changing!
So, there’s not one [approach] that’s better than the other, I suppose. The kind of work we do necessitates very, very large supercomputers. Machine learning requires training very big machines of different architectures. I think the point is that what we’re well placed to do at the Met Office is to learn how to work with others to fuse together what the statistical emulation approaches can bring, with what representation of the raw system from a simulation point can bring. That’s very much what we’re working on as one of our lead priorities: how we can get better results for everybody by fusing together that world of emulation and simulation.
I saw somewhere that your predictions today for four days are as accurate as they were for one day, 30 years ago. Do you think with the use of machine learning that will accelerate even further?
Yeah – the run rate, if you want to see it that way, of those kinds of weather forecasts – and bear in mind, the Met Office does a lot in the seasonal climate, as well as the ‘nowcast’ – but that particular range does tend to improve by about a day a decade. In other words, we can push one more day out with every decade’s investment, and that’s been been the case for quite some time, there’s been quite stable improvement curve. All kinds of things may affect that. I’m not implying any of these are right or wrong, but you might, for example, think that you’re reaching the inherent limits of predictability with some of this stuff – that could be a feature that might change the run rate.
We’ve been fortunate, since 1950-something to the present day, to broadly have this exponential growth – exemplified, probably, by Moore’s Law. So, effectively, we’ve been able to get that – I won’t say free, because it’s not free – but that inherent exponential growth of compute capacity. Many would say that’s not looking the same way, [going] forwards. And then to counter that you’ve got new technologies and new innovations, such as machine learning. So, I’ve learned one thing being a technologist, and that’s to be very cautious about predicting the future, ironic as that may seem.
In this case, who knows? There are some pluses, just like the weather, just like your comment around the Beast from the East. Will there be one or won’t there? Well, the reality is there’s been a sudden stratospheric warming that took place in late December, that’s countered by other climate drivers that are trying to drive the weather the other way. It’s a question of which one of those will bear out, and your question is much the same. Weather [forecasting] will continue to improve at a day a decade, or more, or less – it remains to be seen, because there are drivers from both sides of that balance.
Are there any other industries that, if you were to go and work in them, you would be able to take some of those tools and technologies that you use and apply them? Are there any industries that you think could and should be revolutionised by the skills you have?
So, simulation definitely has a role to play in engineering, for example. It’s [already] well used in some applications: so, aviation, in the design of new engines, for example, have been using fluid dynamic simulations for some years. But the possibility now, with these really sophisticated codes on big machines, [is that we] have the ability to really simulate entire systems, as opposed to maybe [just] the blades of a jet engine. I think that’s what’s often implied when people start talking about digital twins. ‘Digital twins’ has become a vogue phrase over the last couple of years, and that’s really the ability to represent any big system: be it a whole aeroplane or the aeroplane engine, or a city and how a city works, and so on and so forth. In those applications, once again, you will get advocates of emulation and ‘Everything can be done with a machine learning algorithm’; equally, you have advocates of, ‘No, everything must be simulated from understood core principles’. Of course, the reality is almost certainly going to be some mix of those two approaches.
So, in short, I think what the Met Office is doing and learning how to best fuse those two areas together could have many, many applications: anywhere where it’s more cost-effective or safer to predict how something will behave in a future state than it is to actually do it. Of course, in weather terms, you can’t experience the weather before it’s happened, but if you think about building a battleship, or a helicopter, the ability to build that thing virtually, and test it out in some kind of simulated environment before you cut steel and do expensive things like that, of course, is going to have some economic benefits. And that is starting to happen.
2021 has kicked off where 2020 left us. What do you see as being the critical uncertainties for the next five years?
Yeah, the year that Mad Max was set in, so I’m led to believe, so that tells you a lot, doesn’t it?
I think the fact that there are uncertainties as an understanding, certainly [inaudible] leaders is probably the biggest uncertainty of all. Traditionally, organisations would try to reduce and mitigate risk, and work with things like scenario planning or whatever to try and reduce uncertainties to a bare minimum. I think what this last year has told us is that in some areas, these uncertainties are intrinsic.
Dealing with uncertainty in complex systems is the really big thing that’s come out of it for me, and the fact that everybody, in all walks of life – no matter whether it’s personal life or professional life – and businesses, big or small, are beginning to come to terms with the fact that, despite the fact it might appear that our world and our lives have been fairly linear, and there’s been this golden period for the last few hundred years whereby things have just got bigger, better, faster, our standards of life have improved, our standards of health improved – that’s not guaranteed. Not just in a health context, but in reality, the last few hundred years – the last 5,000 years – are a relatively brief snapshot of what is an uncertain system. There are lots of things that have happened historically.
So, not to get too prosaic about it, but in a business context, I think the learning and uncertainty is something you can’t always reduce to nothing, and planning to live with those uncertainties is in many ways quite welcome. This is a big part of where digital came from, and where DevOps came from, and agile came from – this knowing, from a technology point of view, that actually you’re arrogant, if not naive, to think that you can ever know how you can deploy technology to actually hit the value that somebody else has perceived, often your customer. The only real way to do that is to iterate and experiment. There’s a there’s a thing called cynefin, which I think in Welsh means place of significant importance. Cynefin is one of many examples of frameworks to help manage complex, unpredictable and therefore uncertain systems.
It’s something, again, at the Met Office we’re quite familiar with, because we do recognise that the weather and climate are to some extent chaotically driven. And therefore there is no linear path to the next step. [We need to be] always aware of that intrinsic uncertainty that’s there, and I’m quite pleased in many ways to see that being brought into day to day life.
Beyond that you asked a specific question, what things are particularly uncertain at the moment? Well, I think those things are well documented elsewhere: what makes a successful business model, how that business model may be applied, the degree to which you rely on people in a given place at a given time, any organisation that demands that or any market that demands that is now appearing somewhat vulnerable at the moment, because it’s very difficult to get people – be they customers or employees – in a given place in a given time to do anything. That brings a problematic [issue] to any kind of business model. We’ve seen, through this uncertainty, business models that just run like a well-oiled machine. Those kinds of organisations seem to have gone straight through with little problem. And then you’ve got the whole purpose-based theme, whereby organisations with great purpose – and I’m very proud to say the Met Office is one of those – it’s really easy, even in uncertain times to keep that focus on what’s important. Whereas I think if you’re unfortunate enough to work in an organisation that doesn’t carry that purpose, then you get knocked into second-order debates around profit and revenue. Of course, they’re really important, but organisations whose raison d’etre is around those things without any broader purpose or vision, if you like, they’ve also found it very problematic. So huge can of worms around all of the uncertainties, I think. Happy to go down any of those lines, if you’d like to.
We just did some futures work in the charity sector – one of the things that came out of that with them is there’s a number of things that they want to keep doing, as a result of the COVID crisis, and then there’s a lot of things that they will now throw away and won’t go back to, that were made possible by these events. Are there any things, any new ways of working, that you find are better, more stimulating, more motivating for the team? And are there things that you will throw away now, as well?
There are definitely some throw-away things – they’re probably the obvious unpalatable bits, but generally, I think there’s a lot to keep. So, from a broad perspective – and again, I know that we don’t look any different than anybody else in this regard – the confidence that’s engendered by really being forced to make everybody really work remotely, and finding that the whole world didn’t come to bits, is a confidence upon which we can build. We were very fortunate that our crisis management process is very well rehearsed. We use the three levels [of crisis management: pre-crisis, crisis response and post-crisis]. So, when there’s a significantly impactful weather event, for example, we will trigger our crisis management system at the Met Office. Because that’s so well-rehearsed, because obviously those kinds of things happen quite a lot, that was fundamentally the same process that we applied to COVID. So, that was really good, and we were lucky-slash-well positioned, with the right kind of tools. There was no last-minute rush to bolt on technology that allowed people to work remotely.
Nonetheless, despite being in quite good position, I think it was quite unthinkable that somebody would say, ‘Okay, all 2,000 of you, you’re now going to be doing your science slash your meteorology, your technology, or whatever else you’re doing, from home’. That was really not anticipated. Of course, there was some wrinkles, but by and large, we got away with it.
Obviously, like many others, the flexible and remote working aspects are definitely things that will persist into the future. In what form precisely, we don’t know. We’re having very open adult-to-adult facilitated debates with each other, to try and understand what that new or future normal situation might look like. We’re beginning to find ways to structure that conversation, based upon the kind of work that you do, the kind of work that’s being done, and the kind of person and context that you’re in, as being the three major dimensions that go together to dictate whether permanent working from home – if that’s the thing that you want to do – is going to work. There are some types of work that really don’t engender that, we all know the group-think team workshop, blue sky workshopping kind of thing; it’s really difficult to do that, so far, over the technology. There are some people that just have jobs that are hands on and require labs and those kinds of facilities – those things are hard to do [remotely]. And there are some people that just find it quite tricky.
For me personally, I work well [in this environment], but I do have a tendency to struggle with very long-term isolation. I had a mental health issue, I don’t mind saying it, about 15 years ago, and that was driven by the need to work from home for a couple of years, almost exclusively. Despite [the fact] that I’m not the most sociable character in the world, I found that isolation from my colleagues, in the end, really did give me some difficulties. So, I’m cautious of that for myself, and I’ve been taking any opportunity, prior to this most recent lockdown, to get together where that’s safe and appropriate.
Thanks very much for sharing that with us. It’s something that affects a lot of people, and the more people speak about that, the better, so thank you very much. Final question. A lot of visions of the future end up having a dystopian tinge to them. Do you have a preferred vision of the future, and how we use technology together?
I absolutely do. My preferred vision of the future is really that – the COVID experience has told us that in the end, and I’ve got to be ever so careful how I say this, no amount of political will or sentimentality is going to get in the way of the data, if the data generally represents the system. Obviously, the whole thing is lamentable, and in no way would you have wanted this kind of event to have happened – but the fact that it has means that I think it’s been a really good education and information piece for people to recognise that science and technology are to be trusted, within the boundaries of what science and technology are there to do. And I think it’s really done a good job in representing the difficulties that policymakers and politicians have, and how they are having to weigh up very, very difficult choices.
There’s an intrinsic uncertainty about this, and nobody knows. So, we’ve seen the term ‘U-turn’ used quite a lot. Well, it’s not a U-turn, clearly. It’s not as if anyone had an idea of how things were going to pan out and have then changed their decision based on the same understanding of how things are going to pan out. To your point, that future was always uncertain. And as we learn more, as science and technology allow us to better understand the situation, quite obviously, there are some times when not only have you got to change tack or course a little – sometimes you genuinely do have to do something almost the antithesis of what you anticipated. We all know that these things are the real world.
So, my vision for the future, really, is that people are that bit better educated about how to interpret data; that bit better educated and informed about uncertainty and its intrinsic nature. They’re a little bit more informed that our future is – I’m not scaremongering here, it’s not tenuous and hanging on a thread, but equally it’s not guaranteed, by any means, that things should always get bigger, better, faster, richer – whatever your measure of improvement is, over time. These things are somewhat fragile and somewhat brief. And again, not to get to prosaic, but we’ve got to look after ourselves, we’ve got to look after each other, and we’ve got to look after the finite world that we live in. So for me, climate change, obviously, is a huge, huge issue. And again, lamentable though it is, and in no way am I implying that it’s any kind of good thing that has happened, but the whole COVID experience is one that I genuinely hope will make us better-placed to deal with what I see to be the bigger issue – albeit slower and more insidious, perhaps – of climate change and its impact on humanity. Anything that we can do to be positive on that, fantastic. In terms of science and technology that can be applied to the problem? Yeah, there are some, but they all come with wrinkles and compromises, don’t they? And once again, it’s just like the kind of debate between fundamentally getting people back to work versus keeping them safe, and everybody recognising that’s a balance. We’re all trying to keep places open, but at the same time, keep people safe. This application of technology, so that we can continue to protect our planet, is going to look like that as well. It’s going to mean compromise; it’s going to mean balance. So my positive take, from all the lamentable and bad things that have happened, is that makes addressing that challenge more positive and likely than it would have done otherwise.
Charlie, that’s been hugely insightful, and thank you very much for your time today, and good luck with all of those things you have in the future.