When the screens went black
Published on 14 September 2017
Inside the Command and Control Centre, employees monitor the status of over 200 critical business applications, networks and servers around the world. Photo: Peter Elmholt
While it was the fourth day of the worst cyber-attack in company history, Tim Ferguson could not help but crack a smile at the present situation.
Recovery teams were finished preparing a critical piece of the recovery, a master image of the Microsoft Windows 10 Operating System, that would bring 50,000 employee computers back to life. But how to get it out to the company’s nearly 600 global office locations?
"Less than half of those locations have the bandwidth needed to download it from the internet. So, we needed 2,000 USB sticks and a postman, right away," says Ferguson, who on a normal day is Head of Programme & Service Transition but was taking a shift as Manager on Duty in charge of approving all decisions made by the recovery teams. "It turns out that shops carry no more than fifty USB sticks. So we sent people out in cars to buy every single one and these replicator machines so we could load them in big batches.”
The next morning Ferguson estimates they posted 300 DHL packages, with five or six USB sticks in each, to every corner of the globe. “We bought every USB in a 25-mile radius of the office. I’m sure someone out there got one shaped like Homer Simpson.”
An eerie silence
Three days before was Tuesday, 27 June 2017, a day that quickly went from normal to code-red for the more than 400 employees who work on six floors of an eight-storey building in Maidenhead, England. This is Maersk’s IT hub in the UK and Maersk Post visited in August to better understand what occurred in the last days of June.
While roughly half the people here work on maintaining the IT architecture and applications that run the global business, the other half work on developing new functionality and services. It is only now that they are beginning to be released from recovery teams back to their regular jobs.
Many Maidenhead employees were on holiday or working from home the day the malware hit. John Ashley was enjoying the view of his garden from his new home office when his screen went blank and the “notPetya” ransomware text appeared. After a phone call to his manager, he was in his car heading to the office.
It was a heck of a way to start a job, it was actually really exciting. We could see how much pressure everyone was under, so to be able to help out in any way was great.Says Michael Crawley, on his second working day of a two-year apprenticeship.
When he arrived, Maersk IT had consolidated on the fourth and fifth floors of the building. The Command and Control Centre is on the fourth floor. Resembling NASA’s Mission Control Centre, it constitutes the eyes of the Maersk IT estate. Here, employees monitor the status of over 200 critical business applications, networks and servers on a sweeping 10-metre array of screens. The screens were blank.
“It was eerie to come in and see everything powered down, blank screens. I immediately noticed the quiet, the hum from all the machines was gone, it was just voices,” says Ashley. “There was no panic, just an air of intense focus. There was a ‘war room’ and the analysis of what we were facing was underway. Work streams were put in place with leaders, shifts, and key supporting roles such as communication and administration. People just got to work, doing anything to help.”
Ashley and his colleague, Nallathambi Theogarajan, were among the technically skilled ones who were put to work right away on the task of safely restoring thousands of servers, 50,000 computers as well as all the business applications.
Tayler Cekalla and Michael Crawley were on their second day on the job as apprentices. They were scheduled to meet with their manager about initial tasks in business administration and internal communication, respectively. Instead, they became “runners,” doing whatever was needed for the teams which would operate on 14-18 hour shifts every day for the next three weeks.
“It was a heck of a way to start a job, it was exciting. We could see how much pressure everyone was under, so to be able to help out was great,” says Crawley. “It was inspiring to see the commitment of all the people at this company we had just joined,” he says.
“Senior people were sleeping on couches after 20-hour shifts just in case they were needed,” says Tayler Cekalla. “Michael and I don’t know what ‘normal’ is like here yet, so it will seem odd when everyone is back to their everyday jobs.”
Crises tend to bring out the best in people. From natural disasters to corporate crises such as this, studies and reports show that collaboration and creativity tend to rise and everyone pitches in to find solutions to problems. From talking to the employees at Maidenhead, this crisis was no different. Everyone has a story about a quiet colleague who took charge or of senior managers fetching coffee and taxis that it will surely become part of company folklore.
“This was the worst crisis I think any of us have experienced. And we were never alone, so many hands helped in this recovery. From the very first days, we got phone calls from all over the organization from people who wanted to fly in and help, but also from technology partners and other companies,” says Adam Banks, Chief Information Officer, A. P. Moller – Maersk. Photo: Peter Elmholt
“This was the worst crisis I think any of us have experienced. We were never alone, so many hands helped in this recovery. From the very first days, we got phone calls from all over Maersk from people who wanted to fly in and help, also from technology partners and other companies. Everyone pitched in. The level of support was a huge positive surprise for me,” says Adam Banks, Chief Information Officer, A. P. Moller – Maersk.
As reports rolled in from Maersk offices around the world that employees were making use of other tools like WhatsApp to communicate and do business, and local organisations were posting videos of employees fighting the good fight, CEO Søren Skou sent out a video message of gratitude, thanking them for their hard work, dedication and creativity.
Employees here say top management was supportive throughout the crisis. Tim Ferguson recalls that one of his first tasks as Manager on Duty was to contact CEO Søren Skou and ask to borrow the company jet. They were trying to fly in colleagues and others from outside Maersk to help with the recovery but the flights into London were completely full. “He said ‘Of course, that’s what it’s there for!” says Ferguson.
After the first couple of weeks, the emotional toll of the crisis was beginning to show on the faces of everyone, recalls Adam Goodall, Head of IT Operations. Caffeine-fuelled days followed by poor sleep on couches or in hotels away from family was wearing everyone thin.
“Every day we wanted to show our people how their efforts were having an impact. Usually it was in the shape of numbers, sites back online etc. Then we saw this video come up on our WhatsApp feed,” says Goodall.
“It showed this guy at a PC with his colleagues huddled around him as he starts up GCSS, a core application used to support bookings in Maersk Line. Everyone cheers and this guy is literally in tears. We showed that on our big screen here and it just gave this building a huge boost to keep going.”
Stephen Barraclough has been with Maersk for the past 40 years and while officially retired he was back in the building on a six-month contract as Head of Business Services Management.
“I haven’t really thought about it, about what happened. Looking back, what I think has struck me the most, is how people behaved. It didn’t matter what someone’s normal job consisted of, people just did absolutely everything they could to help. This is a young organisation and this experience, while not enjoyable, has certainly made it a heck of a lot stronger.”
Are we prepared for next time?
When the malware struck on 27 June 2017, the Maersk IT organisation was in the process of centralising control of the Group’s IT estate in Maidenhead. From the security of systems to upgrades and improvements, centralisation will lessen the likelihood of a similar attack happening again. At the same time, it will also enable the company to upgrade and improve (or remove) servers, applications and systems more easily.
The cyber-attack sparked some immediate improvements to security, most of which cannot be shared for security reasons. But the subsequent shutdown and reboot of global systems has helped speed up the process of centralisation and modernisation.
“Cyber-attacks are not going to go away and technology is becoming a more strategic asset in the future of our business,” says Adam Banks, Chief Information Officer at A.P. Moller – Maersk. “That means we need to continue what we’ve started and finish building a more secure and reliable infrastructure that can support the growth strategy of this company. We will have more to share about what this will look like and what it will mean for employees and the company when we announce the new IT strategy.”