A lot can go wrong when working in tech and digital transformation. Together, our advisors have over 100 years of experience in the field, so we decided to ask them if they have any horror stories from their careers. And wow, we got some really good stories. First up is a story about not underestimating the human factor even when working with the latest technology.
Part 1: Horror lesson in human error and technical security
In a successful and fast-growing technology company, where an efficient and functioning product is critical to the customers‘ business, a business-driven man called Karl works. The company is in an intense phase of development and is moving towards various certifications, which requires accurate and updated processes.
A chaotic night
It’s around 2 a.m. somewhere on the West Coast of the United States, Karl, the CTO of the company, is just about to solve a problem with their product that has been a long-time priority. However, he is unaware that he is connected to the production environment via his computer. Thinking he is working locally, he initiates a test process that starts by deleting all existing data. In a local test environment this is usually very fast as there is not much data. When Karl comes back to his computer after getting the fifth coffee of the night, he is a bit surprised that the test process is not finished. But by the time he realizes his mistake, it’s too late. A large amount of critical customer data has already been deleted, causing widespread production downtime and threatening to affect the company’s customers.
Proactive change
After several hours of work, the team manages to restore data from backup, but the damage is already done. The incident leads to a comprehensive review of the company’s processes. More stringent measures are put in place to prevent similar mistakes from happening in the future, including restricting access to the production environment for even the most senior people in the company. Finally, the company takes responsibility for the incident and compensates its customer base for the problems caused.
The balance between technology and people
This incident highlights not only the importance of having robust and well-followed processes in a technology-intensive company, but also the inevitable presence of the human factor. No matter how advanced or reliable the technology is, there is always a risk of human error. Humans need help not to screw up, which is where processes and actual technology implementation patterns come in and help. It is crucial to implement processes that minimize the risk of error and ensure that even if errors do occur, there are effective measures to deal with the consequences. Thus, it becomes a balancing act between exploiting the benefits of technology and recognizing the limitations imposed by the human factor.
Follow us on LinkedIn so you don’t miss the next two installments of our tech horror story series.
