Battles of new ideas against conventional wisdom are common in science, aren't they?
It's very interesting how these certain negative principles get embedded in science sometimes. Most challenges to scientific orthodoxy are wrong. A lot of them are crank. But it happens from time to time that a challenge to scientific orthodoxy is actually right. And the people who make that challenge face a terrible situation. Getting heard, getting believed, getting taken seriously and so on. And I've lived through a lot of those, some of them with my own work, but also with other people's very important work. Let's take continental drift, for example. American geologists were absolutely convinced, almost all of them, that continental drift was rubbish. The reason is that the mechanisms that were put forward for it were unsatisfactory. But that's no reason to disregard a phenomenon. Because the theories people have put forward about the phenomenon are unsatisfactory, that doesn't mean the phenomenon doesn't exist. But that's what most American geologists did until finally their noses were rubbed in continental drift in 1962, '63 and so on when they found the stripes in the mid-ocean, and so it was perfectly clear that there had to be continental drift, and it was associated then with a model that people could believe, namely plate tectonics. But the phenomenon was still there. It was there before plate tectonics. The fact that they hadn't found the mechanism didn't mean the phenomenon wasn't there. Continental drift was actually real. And evidence was accumulating for it. At Caltech the physicists imported Teddy Bullard to talk about his work and Patrick Blackett to talk about his work, these had to do with paleoclimate evidence for continental drift and paleomagnetism evidence for continental drift. And as that evidence accumulated, the American geologists voted more and more strongly for the idea that continental drift didn't exist. The more the evidence was there, the less they believed it. Finally in 1962 and 1963 they had to accept it and they accepted it along with a successful model presented by plate tectonics....
Yeah, OK, and this is going to make mainstream theoretical physicists think about what they are doing? Yeah, right.
At a much more mundane level, "conventional wisdom" arises regularly in financial markets, and poses some similar problems/challenges/opportunities.
Michael Steinhardt, one of the early hedge fund pioneers, coined the term "variant perception." He would 1)identify the conventional wisdom/consensus; 2)develop/refine a non-consensus idea; 3)identify a triggering event to change the consensus. 4) Place his (huge) bet when #3 happens.
Getting #3 right is perhaps the most difficult thing -- as it requires getting the 2nd derivative correct -- even as the first derivative may be causing you to lose money as you enter your position. (For most market participants, the majority of profits are made from "riding the trend," and the largest losses occur at inflection points. Steinhardt would typically be on the other side of those investors.)
Are there optimal ways (in science and/or in the markets) to figure out how to discern when and if the "conventional wisdom" is changing???
Lipton's blog has a fine essay on this theme, titled Are Impossibility Proofs Possible?
He mentions the hilarious Far Side cartoon in which the indians---whose arrows are impotent against the cowboys wooden fort---have started shooting *flaming* arrows at the fort ... and one disconcerted cowboy asks another cowboy "Are they allowed to do that?" :)
Wegener's "flaming arrow of geology" was of course the idea that continents were not fixed in position, but rather were mobile on geological time-scales.
Krugmans "flaming arrow of economics" is the idea that humans do not always act in their own rational self-interest, but rather are often short-sighted and irrationally prejudiced.
My personal favorite "flaming arrows" of science include deliberately introducing noise into large-scale quantum system simulations (to make the system simulatable), and doing conformational biology by imaging-and-simulation (rather than experiments).
For sure, there are a lot of "flaming arrows" flying around nowadays ... we live in interesting times!
Over on the Lance/GASARCH blog, I posted an overview of what the "post-fort" world of biomedicine might look like (as we QSE folks conceive this world, anyway).
The point is, we quantum systems engineers are not passively awaiting this post-fort biomedical world ... instead we are working to bring about its advent as rapidly as feasible.
From our QSE point of view, the more QIT "flaming arrows" are flying around, the better! Because these QIT arrows are helping greatly to catalyze a "post-fort" world in which biomedicine and large-scale quantum simulation are conjoined disciplines.
Which is fun ... and in the long run, it's good news for young QIT people who will (someday) need family-supporting jobs.
Gell-Mann's perspectives are always valuable and instructive though I think he's characterized the situation regarding continetal drift in a less than accurate way. It's true continental drift wasn't accepted because there could be not shown any convincing mechanism for it. How could continents plow through the abyssal plain? It turns out of course that they don't, a few theories including an "expanding earth" and even "a shrinking earth" were proposed and they indeed are regarded as baseless (or as the popular media would describe it "bunk") but most serious geologists and others did not necessarily disparage the idea the the continents at some time in the past were joined. The IGY on discovering the spreading sea floor rifts now had good evidence not only that continents had spread but by a observable phenomenon; underwater volcanism and rifting as verified by paleomagnetic studies showing parallel geomagnetic reversals working in tandem with subduction zones and transversal faulting. This was consistent with theories of convective forces within the molten mantle. Interestingly the previous concepts of geosynclinal orogonies was put on a shelf as plate techonics took the center stage, but plate tectonics alone couldn't explain everything either and so the old theories have been revisited and found to have some merit in explaining some features that are hard to explain with plate tecktonics alone,and so we have now a more complex but comprehensive explanation as we'd expect for a natural system as vast and with as many realms and varaibles as our earth's geologic dynamics. But again, the idea of plate tectonics was not ridiculed per se,only the proposed mechanism which if I recall (hindsight being 20/20) were pretty vague, and that's putting it kindly.
Incidentally one of the most important aspects of the theory is that it completely inverted the previous concept of the age of the planet's two most significant features:ocean basins and continental landmasses. It was thought that the abyssal plains were passive accumulators of sediment and that continents were the relavely recent result of these deep deposits whose pressures from their accumulated weight caused weakened regions to be uplifted like some kind of cracke open poppin fresh dough cylinder. Of course we now know it's the reverse and that the abyssal plains of the ocean in no instances have survived more than a couple hundred million years without being subducted even when that rate is only a couple of inches a year, and we've found continental remnants that are thousands of millions of years old, although highly metamorphosed as one might expect having survived such tectonic forces over such long periods of time.
A similar transformation is taking place in geology now regarding the frequency of asteroid impact events which up until recently were considered one of the rarest of geological formations. With the release of once classified ocean floor mapping, and in some cases the use of Google Earth by amateurs looking for interesting features, the evidence is mounting for a greatly increased frequency and larger role for these game changing cataclysms in our history; some shockingly more frequent and severe that we dared to believe for fear of being discredited and labelled "catastrophists".
As to which interpretation of a phenomenon represents the orthodoxy and which the challenging view; it's hard to distinguish since to a degree it depends on which side one sees as their own and how they feel about it, but I'd think that the side that wishes to abide by what they consider unassailable evidence and have no wish to see the fine points debated I would call the orthodoxy, though I suppose that can flip in the wink of an eye or the flip a magnetic pole...or an unexpected and catastrophic impact.
This is not really a comment so much as a suggestion to go back and read Rocky Humbert's comment #2. In my opinion, someone with very different life experience is saying something important here. Unfortunately, I don't know Rocky from Adam but this is exactly the counter to the idea that great revolutions can't be 'managed'. They just have to be managed as 'long-volatility' investment programs.
I have no major quibbles with it except that Rocky is using shorthand. In point 4 change the word 'place' to 'structure'. In point 3, prefix it with 'evaluate the cost-of carry and'. That's still not perfect but it's closer (I suppose you should structure before you evaluate cost-of-carry, and you should manage your exposure with a risk program...etc...but this just detracts from Humbert's argument). The point is that there are constructive ways to manage contrariant positions.
[Rocky: http://bit.ly/Glashow which deals with your idea above at length, while http://sciencepond.com/EricRWeinstein/statuses/4047628906 deals with this issue superficially. If you want to talk more about this idea of managing long-volatility scientific exposures, I'd be happy to do it by phone.]