Before leaving Austin on Friday, I had lunch with a former student who is currently a graduate student at the University of Texas, working in an experimental AMO physics lab. I got the tour before lunch-- I'm a sucker for lab tours-- and things were pretty quiet, as they had recently suffered a catastrophic failure of a part of their apparatus.
Of course, there are catastrophic failures, and then there are catastrophic failures. Some dramatic equipment failures, like the incredible exploding MOSFET's when I was a post-doc, are just God's way of telling you to go home and get a good night's sleep before resuming in the morning. It takes a few hours, maybe a full day to patch everything back together, and then you go back to what you were doing before things went blooey.
Other failures are a little more... comprehensive. A one-of-a-kind piece of equipment is irreparably damaged, or something too expensive to replace fails and is destroyed. But even these aren't necessarily a bad thing.
Sometimes, an equipment failure can be the best thing that happens to an experiment. This is particularly true in labs that rely on short-term labor like post-docs (who are generally hired for about two years) and graduate students (who are in a given lab longer, but typically in charge of the experiment for only a few years), where kludgey short-term solutions implemented in order to get fast results can become locked in as new experiments build on the first one.
My favorite example of this was in a lab at NIST, where a high-voltage connection leading into a vacuum feedthrough had been made with a minigrabber lead clipped to a bare wire. This is ridiculously unstable, of course, but the person who implemented the quick-fix had thought of that, and held the minigrabber in place with heat-shrink insulation. It's not quite to the level of There, I Fixed It, but it's close.
Other classics included a couple uncovered when I was trying to reconstruct the BEC apparatus when I was a post-doc. I had a horrible time tracing one particular cable, which was attached to the output of a signal generator, then disappeared under the same signal generator, apparently headed toward something on the far side of the laser table, from which another cable came back to a second box six inches from the signal generator. I couldn't figure out what the damn thing was connected to, though, until I finally unstacked all the equipment and discovered that it was a ten-foot BNC cable coiled up under the signal generator, being used to make a six-inch connection.
The other one from that lab was a cable that came from the TTL logic output of a signal generator and went into a T connector, then both outputs of the T were connected to the same digital logic box, the output of which went to drive some other piece of equipment. This was a classic solution to an impedance matching problem-- the signal generator could not drive a 50-ohm input, but the logic box was buffered to drive 50 ohms. So, the signal was split in two, connected to both inputs of an AND gate, producing an identical logic signal with the necessary buffering to drive the 50-ohm device. I think I ended up leaving that as it was, once I figured out why it had been done. I did make a note of it in the lab book, though I'm sure that didn't help the next person to run across it.
A catastrophic equipment failure is often a good excuse to fix these sorts of kludges. Since things aren't working anyway, you might as well implement the more stable and safer solution that would've taken too long when you were in the middle of taking data. The rebuilt apparatus after the failure will be better and safer after the disaster than it was before things failed. Or just quieter and less messy, as in the case of my recent turbopump replacement.
The series limit of this sort of thing is the Davisson-Germer experiment, which showed that electrons have wave nature. The success of the experiment came about only because of an equipment failure-- Davisson and Germer were bouncing electrons off a nickel surface, and seeing nothing too remarkable, when a glass tube in their vacuum system cracked, venting the system to atmosphere. This created an oxidized layer on the nickel surface, which they cleaned up by heating the nickel up under vacuum. They overdid things a little, though, and ended up melting the nickel surface. When it cooled back down, it formed a single large crystal, rather than the large assortment of randomly oriented crystals they had had previously, and showed clear diffraction peaks when electrons bounced off it. Davisson shared the 1937 Nobel Prize in Physics for this demonstration of the wave nature of the electron.
So, sometimes the best thing that can happen to an experimental physicist is the destruction of a vital piece of apparatus. When it's re-engineered and rebuilt, it will be better than ever. That can be hard to keep in mind when you're standing next to the smoking ruin of your lab, though.
(The big word in the title is a literary term coined by J. R. R. Tolkien. It's not entirely appropriate, but it amuses me, and that's what really matters.)
I suppose it's related, though not exactly the same, but; I think the basic electronics skills I picked up in Instrumental Analysis (taught when I was an undergrad by a guy who was big on knowing *how* and not just *why* the machine you were using worked) have been some of the more useful skills I took away from my undergrad chemistry degree.
So far I've ended up making our fluorometer my pet piece of equipment (I don't know if this is how it is at bigger schools, but each grad student here has a pet piece of equipment they tend to and help with), and have managed to troubleshoot and fix both fluctuations in the lamp that caused noise in the signal (had to install a UPS; wiring in the room was crap), as well as using a voltmeter to trace the stir motor conking out back to a physical issue (some numbnuts [and I know who, but have no evidence] spilled acetone all over the sample housing and melted a plastic coating that fused the stir bar to the casing).
Still need to talk my advisor into letting me take some basic electronics courses when there are no more relevant courses I need to grab, and I'd really like to learn more so I can help more with the equipment.