Peer review

[lawrocket 3]

jcd got me thinking a while back. I thought I need to explore with a more open mind the idea of anthropogenic warming of the atmosphere. that got me thinking back to college and my class on "Science and Technology Policy" (15 years ago). I recalled names such as Vannevar Bush (if responses confirm wha I think, he will be explored much more deeply), Richard Darman, and Thomas Kuhn.

Thomas Kuhn wrote a book that studied scientific revolutions, "The Structure of Scientific Revolutions." I read it back then. I should re-read it but I couldn't find it while on vacation in Las Vegas. Some things did stick out in my memory, though. Among these - paradigms exist, and changes in paradigms occur with opposition.

I recall the example of Copernicus. He had a theory that challenged the Ptolomeic theory that the earth was the center of the universe. He wrote that Copernicus was trying to do something that was contrary to thousands of years of understanding. Furthermore, Copernicus lacked legitimacy - "Who the fuck iss this guy?"

It wasn't until Galileo re-examined it that he gained more legitimacy. And not until Kepler and Newton went forward with the ideas did Copernicus get proven correct. Incidentally, Kun argued that it made sense to reject Copernicus, since the knowledge of the time dictated that he should be rejected!

Which leads me to peer review. At the time, Copernicus could not have withstood peer review. The state of knowledge at the time was such that Copernicus had holes in his theory. Only Galileo could provide the additional knowledge to force a "paradigm shift."

Now I go to global warming. GW as a scideentific paradigm is established in the same way that Ptolomeic thoery is established. The knowledge of today seems to support it (more on that below). However, "peer review" has been brought up numerous times here.

So I did some lookign into the "peer review" process. (My best friend has been a reviewer and has been reviewed - and he was there). And I did some more looking into the process.

Now, a problem I have with the process is that, in many or most occasions, "peer review" is a secret process. It seems that even if a study is "peer reviewed" the reviewer is not revealed, and comments are often not published.

I have a problem with that. Why are the names of the reviewers and their comments not published? If these are the "peers" wouldn't we want to know who they are and what they say?

And furthermore - do not the "peers" represent, in and of themselves, an inherent establishment line of thought? Say a "peer" has a 30 year history of being a proponent of the Ptolomeic theory? Then this guy Kepler comes along and pokes holes in a paradigm that he helped establish? A "peer review" would probably be negative - especially if comments aren't published with it.

Why is there a culture of secrecy behind this? And isn't there a sense of proponency behind the peer review process?

My wife is hotter than your wife.

[kallend 2,146]

Interesting argument, but flawed, I believe. The "scientific method" was in essence invented by Galileo, Hooke, Huygens and Newton, so the concept of a scientific "peer" was non existent prior to that.

Look at the speed with which "cold fusion" was debunked and HTC superconductivity confirmed by modern peer review methods.

...

The only sure way to survive a canopy collision is not to have one.

[masterblaster72 0]

Quote

Interesting argument, but flawed, I believe. The "scientific method" was in essence invented by Galileo, Hooke, Huygens and Newton, so the concept of a scientific "peer" was non existent prior to that.

Look at the speed with which "cold fusion" was debunked and HTC superconductivity confirmed by modern peer review methods.

I'm curious to hear what your opinion is on the question he asked about secrecy in peer reviews. Mind sharing?

Be humble, ask questions, listen, learn, follow the golden rule, talk when necessary, and know when to shut the fuck up.

[kallend 2,146]

Quote

Quote

Interesting argument, but flawed, I believe. The "scientific method" was in essence invented by Galileo, Hooke, Huygens and Newton, so the concept of a scientific "peer" was non existent prior to that.

Look at the speed with which "cold fusion" was debunked and HTC superconductivity confirmed by modern peer review methods.

I'm curious to hear what your opinion is on the question he asked about secrecy in peer reviews. Mind sharing?

Much the same as people who write job references or provide references for security clearances usually wish them to remain confidential.

BTW the comments ARE made available to the authors, who can do what they like with them. Usually they address any criticisms and re-submit the papers.

I've published over 70 scientific papers and never had a problem with the system. On the whole I've found the peer comments to be constructive and to improve the final product.

...

The only sure way to survive a canopy collision is not to have one.

[lawrocket 3]

Doctor:

I've never participated in "peer review" with the exception of motion practice. I understand the reasons amd logic of peer review.

You wrote that my argument is interesting but "flawed.". Don't go Sarah Palin on me. Explain why it is flawed.

My question is leading (sorry) but I akm curious. Does peer review reflect the establishment thinking? And why the secrecy? So much discretion is left to the editor!

My wife is hotter than your wife.

[lawrocket 3]

Do you get to kbow who reviewed it?

Also, have you published papers that take a controversial position? I.e., something to settle a debate?

Note: I am genuinely interested, John.

My wife is hotter than your wife.

[kallend 2,146]

Quote

Do you get to kbow who reviewed it?

No, but you get to see and respond to their comments.

Quote

Also, have you published papers that take a controversial position? I.e., something to settle a debate?

I've published stuff that went against the prevailing wisdom and pissed a few people off, but I never had a review I thought unfair.

...

The only sure way to survive a canopy collision is not to have one.

[lawrocket 3]

Quote

Quote

Do you get to kbow who reviewed it?

No, but you get to see and respond to their comments.

http://www.dropzone.com/cgi-bin/forum/gforum.cgi?post=3343232#3343232

This is my problem. Why is it anonymous? Sure, the comments can be useful, and make the paper better.

But why is anonymity involved? "Peer review" is intended, seemingly, to be an open process. Any process wherein the players are wholly anonymous creates, in my mind, an issue.

Yes, there are valid reasons why a person would want to remain anonymous. BUT - perhaps because of my profession - I find it to automatically give rise to questions. I want neither friend nor foe assessing something because they can be impeached.

How can an open process be closed? A security clearance, by it's very nature, is a closed process designed to vet a person's personality and character for veracity (I've been through it myself).

Peer review can be set up for problems. Would Luc Montagnier want Robert Gallo as an anonymous reviewer?

Would you see a problem with openness of peer review?

My wife is hotter than your wife.

[FallingOsh 0]

Quote

Would you see a problem with openness of peer review?

I always answer questionnaires a little more bluntly when they're anonymous. A simple answer, but probably true for most people.

--------------------------------------------------
Stay positive and love your life.

[masterblaster72 0]

Quote

Quote

Quote

Do you get to kbow who reviewed it?

No, but you get to see and respond to their comments.

http://www.dropzone.com/cgi-bin/forum/gforum.cgi?post=3343232#3343232

This is my problem. Why is it anonymous? ...

But why is anonymity involved? ..Any process wherein the players are wholly anonymous creates, in my mind, an issue. ...

How can an open process be closed? ...

Would you see a problem with openness of peer review?

You raise some good questions. Not to derail the thread, but do you think secrecy is more appropriate for government processes like elections (black box voting machines), 911 hearings testimony by our president and VP, energy meetings, etc., or do you find these things just as questionable?

Be humble, ask questions, listen, learn, follow the golden rule, talk when necessary, and know when to shut the fuck up.

[rasmack 0]

Quote

Peer review can be set up for problems. Would Luc Montagnier want Robert Gallo as an anonymous reviewer?

It is not uncommon for journals (at least in physics) to allow you to supply a list of people you definitely do not want to have as referees. Apart from that the main reason for the anonymity is that the reviewer will have a larger propensity for being "ruthlessly" honest if he knows he is not going to get an angry phone call the next day.

HF #682, Team Dirty Sanchez #227
“I simply hate, detest, loathe, despise, and abhor redundancy.”
- Not quite Oscar Wilde...

[Erroll 80]

Quote

..... that the reviewer will have a larger propensity for being "ruthlessly" honest if he knows he is not going to get an angry phone call the next day.

Does it not follow then that the reviewer could be equally ruthlessly dishonest for the same reason?


Furthermore, John claims that his 70-odd papers have been peer reviewed. I have no doubt that they were, but, in the light of this secrecy or anonymity, can he adequately demonstrate that?

[rasmack 0]

Quote

Does it not follow then that the reviewer could be equally ruthlessly dishonest for the same reason?

Of course. People are always free to be bastards. However, as John says, they rarely really are. I also fail to see which motivation should compel them. The dynamics of science are not (as seem to be a general misunderstanding) geared towards suppressing competing theories. I like to think that everyone in the game are driven by genuine curiosity.

Quote

Furthermore, John claims that his 70-odd papers have been peer reviewed. I have no doubt that they were, but, in the light of this secrecy or anonymity, can he adequately demonstrate that?

If his papers have been accepted to peer-reviewed journals that would be demonstration enough. If you're asking whether his reviewers did their job properly the answer is that we cannot really know that. Peer review is not a perfect process. It is just the least bad one.

HF #682, Team Dirty Sanchez #227
“I simply hate, detest, loathe, despise, and abhor redundancy.”
- Not quite Oscar Wilde...

[nerdgirl 0]

Quote

Furthermore, John claims that his 70-odd papers have been peer reviewed. I have no doubt that they were, but, in the light of this secrecy or anonymity, can he adequately demonstrate that?

Unquestionably yes. He can show the journals in which they were published and the protocol of the journals. And one could track back to editors and recover reviews, altho' the latter might require legal intervention and cause.

He can also show how many other papers have cited his (Scientific Citations).

VR/Marg

Act as if everything you do matters, while laughing at yourself for thinking anything you do matters.
Tibetan Buddhist saying

[nerdgirl 0]

Quote

Now I go to global warming. GW as a scideentific paradigm is established in the same way that Ptolomeic thoery is established. The knowledge of today seems to support it (more on that below). However, "peer review" has been brought up numerous times here.

You've inverted the respective roles - anthropogenic climate change is the Copernican model that overcame old Ptolomeic ideas. Anthropgenic climate change was the new idea to challenge the old ones.

Galileo's invention of the telescope (new instrumentation) validated heliocentric solar system model of Copernicus (that actually did have a Greek version by Samos, not unlike the first hypothetical ideas on anthropogenic climate change and 'greenhouse' gases go back to the 1800s). Similarly advances in instrumentation (computers and imaging) have validated climate observations, and like Kepler's new theoretical work explained (using mathematics and physics), new theory in climate change is advancing on those observations.

Like the challenges to anthopogenic climate change, most of the challenges to Galileo came from those outside of the science.

VR/marg

Act as if everything you do matters, while laughing at yourself for thinking anything you do matters.
Tibetan Buddhist saying

[lawrocket 3]

Nice points, and reasonable. But not really until Kepler pretty much founded modern scientific method did Galileo's ideas become accepted.

Now, the reason I brought up Vannevar Bush was because of the scientific political revolution he brought about. The government became the big dog in terms of funding dcientific research. When a political unit funds science, the funding becomes a political decision.

Thus, to get funding, the scientist must take a political position. Let's say a scientist (climatologist) believes that global warming is real and that there is an anthropogenic constituent to it. But, the scientist believes that the warming will be slight and that there is nothing that is economically feasible that can stop it. "Parts of Greenland are experiencing ice accretion but other parts are experiencing ice abatement. It is, on the whole, balanced. And, assuming that Greenland's ice is melting it'll take a couple hundred thousand years to happen, and we'll probably have at least a couple of ice ages in the interim."

But he needs funding. And he wants to go to Congress for money for his research to gather further data that can measure ice thickness, temperature, humidity and wind speed at 200 polar locations to aid with prediction models.

Compare him with a GW proponent and threats of impending disaster - he'll get funded before the other guy will.

The "denier's" views are contrary to the present paradigm, which is made stronger by the post WWII politicization of scientific funding. He points to no big danger or political interest. he doesn't want the data so he can stave off a disaster but merely to aid in understanding.

Politically speaking, he's got nothing to offer. It's like DARPA - its research is for the purpose of defense and security. I'll put it this way - had Kelly Johnson been born in 1950 would he be a legendary aerospace engineer? Probably not because he was in the right time. Why is the SR-71 still the fastest operational jet nearly a half century later? Because research and development priorities arw different now and satellites do a better job. We don't need am aircraft that'll wake up Putin once a week with a sonic boom.

To be a climatologist who isn't a GW "alarmist" is to be a Raytheon lobbyist who says, "ICBM's are not a big risk, and there isn't an economically feasible method to destroy them before detonation over the intended target. SDI is a waste of money".

Science is now political. In a sense it always has been. And it should be considered that peer review is susceptible to resistence of a paradigm, as well. We are human, after all.

My wife is hotter than your wife.

[kallend 2,146]

My experience is that the funding system** doesn't work the way you just described. I think your ideas are corrupted by the legal system in which you spend your time.

** excepting earmarks, which are not subject to peer review. I have a strong aversion to earmarks for science research.

...

The only sure way to survive a canopy collision is not to have one.

[jakee 1,594]

Quote

But not really until Kepler pretty much founded modern scientific method did Galileo's ideas become accepted.

Brilliant! So you confirm that your original example of how paradigm-challenging ideas are suppressed was pretty much irrelevant to modern science.

It's interesting how some people have such an aversion to GW theory that if they can't poke holes in any specific part of GW science they move onto "Hey, what if all of science is broken?" What other areas of research do you believe are suffering from this tyranny of consensus, or is it just GW?

Do you want to have an ideagasm?

[lawrocket 3]

So how does it work? (Obviously, the guy wanting to go to Greenland won't be in Washington for funding).

My wife is hotter than your wife.

[kallend 2,146]

Quote

So how does it work? (Obviously, the guy wanting to go to Greenland won't be in Washington for funding).

Why won't he? I've been on funding panels in Washington - that's where the money is (Willy Sutton Effect).

Kelly Johnson wasn't a scientist - don't confuse engineering with science.

While "alarmist" approaches may work with politicians and the tabloids, NSF review panels (to name one example) are not likely to be impressed.

...

The only sure way to survive a canopy collision is not to have one.

[lawrocket 3]

My suggestion is that of Kuhn - that any community will be resistant to ideas that challenge the status quo. These are both qualitative and quantitative.

GW proponents managed to change the paradigm successfully after resistance. Now the proponents are the paradigm and they will naturally, reasonably and understandably resist evidence contrary to their understanding. On the other side, the "deniers" will naturally, reasonably and understandably resist the opposite belief and evidence to the contrary.

Kuhn himself wrote that the Ptolemists were right to reject Copernicus. Einstein was resistant to Bohr. I cannot recall the name of the priest who first posited the concept of the Big Bang, but Einstein dismissed it by saying his math was good but his physics weren't. Present paradigm suggests that he was right.

There is always resistance and proponency with ideas. Paradigms shift. That is all.

And kallend has allayed most of my suspicions about peer review.

My wife is hotter than your wife.

[SpeedRacer 1]

The concept of prions defied the status quo for a time, but was soon accepted. By peer review, no less.

The concept of prions, in spite of how radical it was, did not meet the same amount of resistance that global warming does. (with much of the global-warming deniers coming from non-peers, ie, non-experts in the field.)

Speed Racer
--------------------------------------------------

[Andy9o8 2]

For everyone's edification and enjoyment, here's a presentation by a leading academic scholar who ... I am told ... has sat on a number of peer review boards.

http://www.youtube.com/watch?v=eaADQTeZRCY&feature=email

[nerdgirl 0]

Quote

The concept of prions defied the status quo for a time, but was soon accepted. By peer review, no less.

The concept of prions, in spite of how radical it was, did not meet the same amount of resistance that global warming does. (with much of the global-warming deniers coming from non-peers, ie, non-experts in the field.)

Excellent example!

/Marg

Act as if everything you do matters, while laughing at yourself for thinking anything you do matters.
Tibetan Buddhist saying

[nerdgirl 0]

Quote

For everyone's edification and enjoyment, here's a presentation by a leading academic scholar who ... I am told ... has sat on a number of peer review boards.

http://www.youtube.com/watch?v=eaADQTeZRCY&feature=email

I think I've reviewed some of his proposals as well.

Thanks for the morning chuckle.
/Marg

Act as if everything you do matters, while laughing at yourself for thinking anything you do matters.
Tibetan Buddhist saying