Yesterday millions of people who day after day use Google as their search engine of choice suddenly found that you couldn’t directly search anywhere. Instead you were greeted with a warning suggesting that the place you wanted to go wasn’t a safe place – in fact anywhere you wanted to go wasn’t safe. What had apparently happened was that when applying a routine update of new malware sites human error caused all sites to be updated as such.
Now let’s examine what this means – an update that should have been applied for a few thousand sites was applied to millions of sites. Hmmm! That sounds like a massive update and depending how the data is kept must have been quite interesting to unpick!
Now I hear everyone shouting – what about change control and test? Well its my guess that great change control and testing exists……but is the process sound? What do I mean by this? Let me recount a similar experience from the SAP world.
In the world of SAP everyone know about transports of changes in configuration or code. And a lot of people will know about the practice of development instance, testing instance and productive instance. But how many know about the “lorry” button? Typically in an SAP environment there will be lots of projects with change underway and each project will be at different stages. For years all the change control documents managed the individual changes, all the sequencing and pre-requisites……and then one day simply by a slip of the mouse cursor the lorry icon was clicked whilst applying the weekly scheduled checked and tested individual changes. This little icon had no “are you sure?” question and so immediately it actioned every change however complete into the productive system. The lorry meant bulk everything up and apply immediately. A lot of pain ensued (particularly to the individual with the errant mouse).
So the process was flawed, all the change controls had been followed just the process wasn’t as complete as we thought. Sometimes, however well you may think you have instituted controls, – you can discover otherwise and that’s what Google discovered and learnt yesterday.