Somewhere in all of this the question of what happened to abortive designs crops up. I think that references to the Admiralty meticulously retaining information on these designs is really references to the Ships' Covers, many of which are at the Brass Foundry out-station of the National Maritime Museum. This collection certainly includes a lot of abortive designs, such as a 1906 battleship, the Malta class carrier, the 1945 Battleship, the G (1945) class destroyer, the initial 1945 frigate, the postwar missile cruiser, and of course CVA01. But -- a big but -- a lot of Covers never made it to the Brass Foundry. In a few cases they were given numbers for what they didn't get (all the Covers are numbered). The example I can remember is the 1953 ASW submarine. Although in theory the Brass Foundry was supposed to get all Covers when they were declassified, in some cases the holders seem to have found it easier simply to destroy them -- I once heard that the Leander Cover died that way. The Covers were part of the way the DNC Department kept track of its work. They died when design authority went out-of-house, as is currently the case. My understanding is that the Type 23 Cover was the last of the lot.
To make things more interesting, there were a lot of designs which did not end up in Covers. Examples would be the bridge between the Iron Duke and Queen Elizabeth class battleships and a wartime anti-aircraft cruiser, a Legend for which you can find in the papers of the Future Building Committee. There are no Covers, I think, for the interwar sketch designs of midget battleships intended to explore Treaty limits (but there are papers, including drawings, in TNA).
There were also a lot of DNC drawings intended for the Board. Lots of them are referenced in Covers and elsewhere, but very very few have ever surfaced (the exception may be drawings in the Cover for the big missile cruiser killed in 1957). You'll notice that there are no drawings in the 1945 Battleship Cover, which has turned up on this site. And of course there are a lot of DNC-produced sketches in the papers produced to describe the future carrier-less fleet about 1966. But we haven't seen any relevant Covers.
So it isn't old disk drives that have killed this story, it is the much simpler and more depressing story of files destroyed to make space for newer ones. Once DNC and its successor DG Ships lost their authority over new designs, they really didn't need all that accumulated experience, meaning the Covers. I have no idea whether they were inherited by the current Design Authority, which is BAE, but I am skeptical.
On the US side, there are no Covers, but the old BuShips/NAVSEA Preliminary Design group kept files on each design. Typically there was a design history plus books of calculations, such as damage resistance. The design history was a compilation of relevant papers, including sketch designs. I think the Long Beach file has survived at College Park. Kidd is an interesting case. It was not a Navy design at all. Preliminary Design sketched a gas turbine destroyer as a feasibility study leading to a design competition among builders. Litton won with the Spruance. Kidd is a version of Spruance. It happened that Spruance was designed so that it could be converted into a DDG, because when it was conceived it seemed that the fleet might need more DDGs in future. There is a very good Spruance history by Potter (I forget his first name) published some years ago by USNI. Burke is, incidentally, a Navy design, as is the Perry class.