Over at Curious Wavefunction, Ashutosh Jogalekar offers a list of great surprising results in physics. This is fairly comprehensive, but leaves out one of my favorites, which is the discovery of the muon. Muons are particles like electrons, but a couple hundred times heavier. When they were first detected in cosmic ray traces in 1936, physicists briefly thought they were the mesons that Hideki Yukawa had predicted as the carriers of the strong nuclear force. It quickly became clear, though, that while the mass was about right to be Yukawa's particle, the muon didn't have anything to do with the strong force.
The existence of a heavy version of an electron was a complete surprise. The eminently quotable I.I. Rabi famously responded to the news by asking "Who ordered that?"
It's interesting to look down Jogalekar's list of "surprises," though, and think a bit about what surprise really means in a physics context. Some of these are theoretical developments that people were led to by mathematical logic-- like the Dirac equation, or asymptotic freedom-- but other theoretical steps are desperate tricks-- like Planck's quantum hypothesis to get the black-body radiation formula, or Bohr's quantum model of hydrogen, or Pauli's proposal of the neutrino (which isn't on the list). I would argue that the latter are more surprising than the former, in the conventional sense of the word. I would also disagree with Jogalekar about which of Einstein's 1905 papers was genuinely "surprising"-- special relativity is weird, but follows naturally from a rigorous consideration of the foundations of physics. Henri Poincaré and Hendrik Lorentz very nearly had special relativity before Einstein. Einstein's photoelectric effect paper, on the other hand, was genuinely revolutionary, discarding classical ideas altogether, and that makes it the more surprising, to my mind.
It's also interesting to look at surprises on the experimental side, where you have a similar split between discoveries that genuinely come out of nowhere-- superconductivity, or the Marsden-Geiger experiment that discovered the atomic nucleus-- and ones that were anticipated, just not by the people who actually made them. The latter category would include Penzias and Wilson discovering the cosmic microwave background: they didn't set out after what they found, they were just trying to debug their big antenna. But just down the road in Princeton, Robert Dicke was setting up an experiment to deliberately seek exactly the background radiation that Penzias and Wilson found by accident (it had been predicted in the 40's by Ralph Alpher). When they called him to ask if their weird static might be related to the cosmic background, Dicke took the call, and when he hung up, turned to his team and said "We've been scooped."
There are also interesting cases of discoveries that were missed because they were too surprising, like the two Nobels the Joliot-Curies just missed. They might've discovered the neutron and the positron, but they weren't prepared to think about massive neutral particles or positively charged electrons, and so ceded those discoveries to others. Others were almost missed, like the Davisson-Germer experiment, which was a puzzling curiosity until Davisson ran into Max Born, who recognized what was happening in their experiment.
(This sort of thing, of course, serves as evidence for the Kuhnian "paradigm shift" picture of science, or maybe a "steam engine time" kind of view, where discoveries can't be made until the wider community is ready for them to be made. There's something to that, but it's often taken a good deal too far.)
I've never been part of any really world-changing surprises, but I did have one experience in grad school where a puzzling initial result turned out to be something really cool, when we started looking at time-resolved ionizing collisions. What I initially took to be a puzzling electrical problem turned out to be a rich source of data, and kicked off one of the best research experiences I ever had.
That kind of surprise, when something weird suddenly comes clear, and everything clicks into place, is the best feeling in science. Every odd result that comes out of an experiment or simulation creates a brief flash of hope that this is one of those surprises. The vast majority of these turn out to be boring results of loose cables or misplaced semicolons in computer code, but there's always hope...
(I.I. Rabi photo at the top of this post from this page.)
There's a great story about the discovery of the fractional quantum Hall effect. For those who don't know, the integer quantum Hall effect (Nobel for von Klitzing) becomes apparent (via a plateau in the Hall voltage and a dip to zero in the longitudinal resistance) when you have electrons confined in 2d, and you apply a perpendicular magnetic field that is so large the ratio of (number of electrons per unit area) to (magnetic flux in units of h/e) becomes a small integer (say 5, 4, 3....). The idea is that the magnetic field sets the energy scale for the spacing between the allowed energy levels (the cyclotron energy). With this expectation, as you crank up the magnetic field, that ratio I mentioned approaches 1, and all the electrons are now supposedly in the lowest "Landau" energy level. By that reasoning, cranking up the field more shouldn't do anything.
The story is that Dan Tsui and Horst Stormer were at the national high magnetic field lab at MIT doing these kinds of experiments, and because they could they kept cranking up the field even farther. They got to the ratio I mentioned (the "filling factor") equaling 1/3 and saw another surprise quantum Hall plateau show up on the strip chart recorder. Supposedly Tsui measured off the field scale with his fingers on the chart recorder and said "Huh. Quarks!" (because naively you could imagine this being an integer quantum Hall effect for particles with charge e/3). This showed Tsui's amazing intuition, because the low energy excitations of the 2d electrons under those conditions really do act like they have fractional charge e/3!
I forgot another example of an experimental surprise that turned out not to be thing thing the people doing the experiment were after, which is the Stern-Gerlach experiment that discovered spin-1/2. They thought they were measuring the orbital angular momentum predicted by Bohr's quantum model (and sent Bohr a letter saying so), but in fact were measuring the electron spin. As a bonus surprise, the experiment only worked because one of them smoked cheap cigars.
Bill Phillip’s discovery of sub-Doppler cooling has to be the most famous ‘recent’ unexpected find of atomic physics.
I'd add the discovery of charm to your list. Even though it had been predicted, no one believed that "charming idea" enough to look for it and Ting didn't believe he had it until the SLAC folks almost literally tripped over the peak. You know that story from Second Creation.
But my main reason for commenting is that I object to the historical description in a few of the cases in the article you cited.
1938: Not only did Fermi's group miss fission, he was awarded the Nobel Prize for mistakenly thinking he had made transuranic elements!
1905: Length contraction and time dilation were not surprising; Lorentz had proposed that they were actual real things that happened to ensure that the speed of light was constant as observed in experiment. (There were other things as well, an entire theory of the electron and the origin of its mass.) That had been around for decades. Einstein did the true paradigm shift with a bit of theoretical jujitsu that merely postulated that light speed was one of its fundamental properties (hence true in any inertial coordinate system) as Maxwell's equations also required, but the result was not surprising.
Deriving photons by assuming electromagnetic waves were actually an ideal gas, now that was surprising. Planck's work was what we now call phenomenology.