The most important lesson to figure out is why it is taking so long to restore services. That will tell us how to prevent such a calamity in other vital national institutions.
"A 170 million item catalogue is bound to be very complicated. A replica backup would no doubt be very expensive and hard to maintain"
The process of backing up & restoring large databases is a very old problem solved long ago. I don't know which database software the BL uses, but any database system worth its salt incorporates both full & incremental backups (& the ability to restore them) as a standard feature. I worked on developing such software in the early 1980s.
Backups are not "hard to maintain". The underlying complexity of the process should be hidden from the user by the database software. If backups are hard to maintain that suggests either poor choice of software or inadequate training of IT staff.
I hope that the BL now ensures that all its data is backed up in a way that enables it to be restored quickly in future.
I do sympathise with BL staff. It's not their fault if the systems provided to them are vulnerable or difficult to restore. Government needs to take BL IT infrastructure more seriously. It doesn't help that most government ministers appear to be IT illiterate.
I am very inadequately qualified in computing to pontificate on the cause of this disaster but my immediate thought was why wasn't the catalogue backed up? 170 million records seems a very small number. I have been playing around with about 2 million records of plant names on my very cheap (£500 ish) computer. I would very much like to read a more technical discussion (but not too technical or I won't understand it!) rather than broad brush political observations. My next immediate thought is why isn't the catalogue in the public domain anyway?? Surely if it is a catalogue us plebs wouldn't benefit that much from being able to see it and we might be able to send the BL a copy next time they loose it.
I've checked back here a couple of times but it seems to be a largely inactive Substack newsletter. I won't check again, at least for a while, but I'd like to abandon on a constructive note. I came here following a rec from Ian Leslie. That's a very strong rec ('The Ruffian' has a decent claim to being the best newsletter on Substack), but the author of this newsletter doesn't seem to have realised that or chosen to capitalise on it. So I'd say two things. First, Substack works by creating communities of interest; authors really have to work at that and engage with readers or it won't work for them. I see that the author has started off by making two apologies for engaging too little in the past, but then failed again to engage with posters at this very article; maybe it's time to call it a day here and maybe try another platform? Perhaps the occasional post on Medium would be a better option for this author? Second, Ian Leslie recommended because of the author's interesting and useful description of an area he (the author) has a professional background in. But I, and I think other readers (going by the unusually negative replies), found the article weaker than I'd expected. On reflection, I think this is because the author is neither a scholar nor a subject matter expert in the areas he writes about at the article. I guess the author is in a similar position to someone who's served a single appointment an ambassador to a lesser but now- important state in his last appointment for a few years; maybe a recent ambassador to Ukraine, or perhaps a uniformed head of Royal Navy's procurement branch or something? The appointment doesn't confer either deep subject matter expertise nor a true knowledge of the top-level politics, but it does provide, for a few years, an interesting and perhaps even unique perspective on some things. That tends to have a recent-historical flavour to it, because things change quickly, but that can help flesh out discourse about what's going on today. That's really what the author should write about, I think - I'd be interested to hear his thoughts about how he thought things might go when he was in the job, and perhaps compare that to how things might appear to be going now. Personal stories and broad inferences are useful and interesting. But if I want an academic analysis, I'll read people with a strong publishing record who cite relevant literature, if I want a techie analysis then I'll read people with a strong techie background and if I want a poltical/policy analysis I'll read former top level policymakers. The author here isn't constraining himself to where he might be useful and interesting, that over extension is perhaps most obvious when he refers to legislative priorities. Anyway, it's perfectly possible that this could be a decent newsletter; at present, I'll give it a miss for now. Genuine good luck with it, though!
Hi, came here on rec from Ian Leslie. it's a very interesting post on the tech and intra-organisational stuff. Thank you for that. There is this one thing I'd say, though: You haven't costed your solutions. We're in a proxy war with Russia; the biggest harms to the UK population are things like energy prices. In large part, people being told to accept those harms as collateral damage as long as there's a war on. The cost of hardening near-countless institutions like the BL against a risk which you correctly (in my view) suggest will reduce markedly when the proxy war is over would be, presumably, extraordinary. After the war, there'd be fantastically expensive yet wholly obsolete metaphorical pillboxes all over the place. More likely, such institutions will be told to simply do their best until the proxy war is over with a bit of extra funding here and there. I'd suggest, too, that there really is little or no comparison between ransomware and having your child kidnapped by BK in Northern Nigeria. The post is, to me, super interesting on detail but on the wider context tends towards special pleading. Is that harsh? :-)
You go out of your way to tell us you are not blaming the staff at BL and tell us how hard they are working but that (intentionally?) gives the impression that ultimately that is who you are blaming. You do not explicitly discuss where the fault lies which is a failing of the article. Perhaps the fault lies with the National Cyber Security Centre which you set up and has clearly not worked. This seems to be another example of the Post Office / Horizon situation where the top dogs push the underlings under the bus while pretending to be very caring.
Business services need to rally around support for the researchers who otherwise work at the British Library. Until the cyberattack the BL Business and IP Centre provided read-only access to various business databases and reports which gave readers market insights. Access was limited to readers who were physically in the Reading Rooms of the BL and there were strictly enforced limits on copying anything except a very small extract from a report. This research facility is gone - possibly forever.
We can only guess at this time, but it is likely that paying the ransom doesn't obviate the need to disinfect ALL the computers in the BL network, and sanitise backups, and test everything. The ransom would have only (perhaps) taken the data off the public dark web; it is unlikely that the attackers would have removed their malware thoroughly enough to just turn things back on.
Thanks for the interesting insights. One question: while it's clearly unrealistic to contemplate bringing Russian cyber criminals to justice, isn't there some way of tracing their equipment remotely and disabling it ?
You say
"A 170 million item catalogue is bound to be very complicated. A replica backup would no doubt be very expensive and hard to maintain"
The process of backing up & restoring large databases is a very old problem solved long ago. I don't know which database software the BL uses, but any database system worth its salt incorporates both full & incremental backups (& the ability to restore them) as a standard feature. I worked on developing such software in the early 1980s.
Backups are not "hard to maintain". The underlying complexity of the process should be hidden from the user by the database software. If backups are hard to maintain that suggests either poor choice of software or inadequate training of IT staff.
I hope that the BL now ensures that all its data is backed up in a way that enables it to be restored quickly in future.
I do sympathise with BL staff. It's not their fault if the systems provided to them are vulnerable or difficult to restore. Government needs to take BL IT infrastructure more seriously. It doesn't help that most government ministers appear to be IT illiterate.
I am very inadequately qualified in computing to pontificate on the cause of this disaster but my immediate thought was why wasn't the catalogue backed up? 170 million records seems a very small number. I have been playing around with about 2 million records of plant names on my very cheap (£500 ish) computer. I would very much like to read a more technical discussion (but not too technical or I won't understand it!) rather than broad brush political observations. My next immediate thought is why isn't the catalogue in the public domain anyway?? Surely if it is a catalogue us plebs wouldn't benefit that much from being able to see it and we might be able to send the BL a copy next time they loose it.
I've checked back here a couple of times but it seems to be a largely inactive Substack newsletter. I won't check again, at least for a while, but I'd like to abandon on a constructive note. I came here following a rec from Ian Leslie. That's a very strong rec ('The Ruffian' has a decent claim to being the best newsletter on Substack), but the author of this newsletter doesn't seem to have realised that or chosen to capitalise on it. So I'd say two things. First, Substack works by creating communities of interest; authors really have to work at that and engage with readers or it won't work for them. I see that the author has started off by making two apologies for engaging too little in the past, but then failed again to engage with posters at this very article; maybe it's time to call it a day here and maybe try another platform? Perhaps the occasional post on Medium would be a better option for this author? Second, Ian Leslie recommended because of the author's interesting and useful description of an area he (the author) has a professional background in. But I, and I think other readers (going by the unusually negative replies), found the article weaker than I'd expected. On reflection, I think this is because the author is neither a scholar nor a subject matter expert in the areas he writes about at the article. I guess the author is in a similar position to someone who's served a single appointment an ambassador to a lesser but now- important state in his last appointment for a few years; maybe a recent ambassador to Ukraine, or perhaps a uniformed head of Royal Navy's procurement branch or something? The appointment doesn't confer either deep subject matter expertise nor a true knowledge of the top-level politics, but it does provide, for a few years, an interesting and perhaps even unique perspective on some things. That tends to have a recent-historical flavour to it, because things change quickly, but that can help flesh out discourse about what's going on today. That's really what the author should write about, I think - I'd be interested to hear his thoughts about how he thought things might go when he was in the job, and perhaps compare that to how things might appear to be going now. Personal stories and broad inferences are useful and interesting. But if I want an academic analysis, I'll read people with a strong publishing record who cite relevant literature, if I want a techie analysis then I'll read people with a strong techie background and if I want a poltical/policy analysis I'll read former top level policymakers. The author here isn't constraining himself to where he might be useful and interesting, that over extension is perhaps most obvious when he refers to legislative priorities. Anyway, it's perfectly possible that this could be a decent newsletter; at present, I'll give it a miss for now. Genuine good luck with it, though!
Hi, came here on rec from Ian Leslie. it's a very interesting post on the tech and intra-organisational stuff. Thank you for that. There is this one thing I'd say, though: You haven't costed your solutions. We're in a proxy war with Russia; the biggest harms to the UK population are things like energy prices. In large part, people being told to accept those harms as collateral damage as long as there's a war on. The cost of hardening near-countless institutions like the BL against a risk which you correctly (in my view) suggest will reduce markedly when the proxy war is over would be, presumably, extraordinary. After the war, there'd be fantastically expensive yet wholly obsolete metaphorical pillboxes all over the place. More likely, such institutions will be told to simply do their best until the proxy war is over with a bit of extra funding here and there. I'd suggest, too, that there really is little or no comparison between ransomware and having your child kidnapped by BK in Northern Nigeria. The post is, to me, super interesting on detail but on the wider context tends towards special pleading. Is that harsh? :-)
You go out of your way to tell us you are not blaming the staff at BL and tell us how hard they are working but that (intentionally?) gives the impression that ultimately that is who you are blaming. You do not explicitly discuss where the fault lies which is a failing of the article. Perhaps the fault lies with the National Cyber Security Centre which you set up and has clearly not worked. This seems to be another example of the Post Office / Horizon situation where the top dogs push the underlings under the bus while pretending to be very caring.
Business services need to rally around support for the researchers who otherwise work at the British Library. Until the cyberattack the BL Business and IP Centre provided read-only access to various business databases and reports which gave readers market insights. Access was limited to readers who were physically in the Reading Rooms of the BL and there were strictly enforced limits on copying anything except a very small extract from a report. This research facility is gone - possibly forever.
We can only guess at this time, but it is likely that paying the ransom doesn't obviate the need to disinfect ALL the computers in the BL network, and sanitise backups, and test everything. The ransom would have only (perhaps) taken the data off the public dark web; it is unlikely that the attackers would have removed their malware thoroughly enough to just turn things back on.
Thanks for the interesting insights. One question: while it's clearly unrealistic to contemplate bringing Russian cyber criminals to justice, isn't there some way of tracing their equipment remotely and disabling it ?