Reading the scientific literature for research ideas?

I'm not quite done with this semester, but I'm also starting to think about the courses that I'm teaching in the winter. In particular, I'm thinking about our department writing course. The course is in transition right now - in the past, it's been a writing-in-the-discipline course, but because of state-mandated changes to our general education program, students now have to take more outside-the-discipline writing courses (and the disciplinary writing courses are disappearing). We're not getting rid of the course completely, because we've also been using it to prepare students for their senior thesis work. But the focus of the course is changing, from perfecting the writing to putting together a good thesis proposal.



Our students take this course at the end of their junior year, after they've taken a lot of geology courses but before they start doing research for their senior thesis. When I started teaching the course, I hoped that the process of writing a proposal would help reduce the sense of panic that undergraduates can experience when they're first told "and now, you have to do something totally new - good luck!" But it didn't help - it just pushed the panic back a semester. Current seniors regularly tell the juniors that they need to decide on their project before they go into the writing class. So I need a different approach.



I'm considering starting the class by emphasizing reading papers - one paper per student per week for the first five weeks of class. I'm thinking about ways to make sure that students do the work, but which don't force me to read 22 papers every week. Right now, I'm leaning towards three assignments: a short written response each week (graded done/not done), a presentation and discussion of one of the papers in class (graded using some kind of simple, in-class rubric), and a short graded paper on one of the articles (possibly modeled after some of the better blog posts about peer-reviewed research).



That's all fine and good, but I've never been very good at getting ideas from reading the literature myself. So I've been trying to figure out what I should be getting out of articles. Here's what I've come up with:



Basics:

  • What did the authors conclude?
  • What alternate conceptual models, explanations, or hypotheses did the authors consider? Why did they prefer the explanation in their conclusion?
  • What methods did they use to approach the problem? (A few possibilities in the geosciences could be various numerical modeling approaches, sampling strategies, analytical techniques, ways of plotting field data, experiments...)
  • What's the context? How does this work fit with other work that's been done and questions being asked? Why does anyone care about this research?

Getting ideas for future work:

  • Do you accept the author's conclusions? If not, are there other approaches that could allow you to test their conclusion?
  • Does this research suggest new ways to interpret a different problem? (Could something like this model explain other areas? Other periods of time? Other types of processes?)
  • Are there other problems that could be studied using the same methods? (And what equipment or expertise is necessary to use these methods?)

Here's where I need some help. What am I missing? Is it really cheesy to steal my "basics" from the structure of a scientific paper? (I mean, all I've done is asked students to think about the paper backwards.) What kinds of questions do you find yourself asking when you read papers - especially when you get really productive ideas from reading a paper? ("What the &*^#^@ were the editors THINKING when they accepted this stinking pile of &*#%#" is not the kind of thought that inspires new research. At least, not for me.)


Tags

More like this

I also like the idea of having students "translate" the paper for a non-geoscientist. For upper-level students with some degree of fluency in the language and techniques of science, it's cool to let them put that expertise to work for the common good. Or at least allow them to start practicing being "citizen scientists." Next semester, I'm going to do something new (and exciting) with my Physical Geology students' field trip papers - I'm going to have them write (and illustrate) a "Travels in Geology" article for EARTH magazine, and then I will submit the best paper or two to EARTH editor Meg Sever for consideration to publish online. A similar sort of assignment could be put together for paper-reading and then published, say, here (or on another geoblog like the one UPJ runs for their undergrads).

Recently I read a paper that sad "we have decided that A caused short-term trend B1. Therefore, we find that C does not cause long-term trend B2." This kind of logical fail instantly made me think of ways to try and figure out how to show how much of long-term trend B2 is left unattributed even if the assumption is made that A is entirely the cause of B1, which could be a nice study... the next time I have a free month.

Mind you, I don't get idea flashes like this that often.

What am I missing?

You've emphasized regurgitation of the experimental data and conceptual arguments of the authors to the exclusion of critical assessment of the extent to which the conclusions of the manucript are justified by the experimental data and conceptual arguments.

In terms of your own workload, why wouldn't you just assign the same paper to all the students?

I would add one more specific question for the bacis:

What are the consequences of the authors conclusions outside of the purely scientific context?

CPP: But isn't understanding of the article a necessary prerequisite to critically evaluating whether the conclusions are justified? In my experience, there's such a huge step between textbook science and the peer-reviewed literature that students need to start by figuring out what on Earth the authors were trying to say. (There's a lot of prior knowledge assumed in most journal articles that isn't covered in coursework.)

That's a good point, though - I should add a question to the basics that asks whether the conclusion is justified. That's a reasonable place to end thinking about the previous work, and a good step towards thinking about one's own work.

As for your second suggestion, that would work if this were a seminar based around a particular scientific topic. But it isn't - it's preparation for all of the students for their own research project. (All of our students are required to do an independent, year-old undergrad research project. Some of the students won't have taken my junior-level course yet, so they wouldn't be prepared to read an article in my specialty at all. Other students didn't enjoy or do well in my class. I'm preparing other people's students for research, not my own students, for the most part. It would be a different course if I were preparing all these students to do research with ME.) My goal is to let the students figure out what THEY are interested in... which means a lot more work for me, unfortunately. (I'm pressuring my department to spread this responsibility around - I've been doing this course for 10 years, and I'm ready for a break from it.)

Callan: I thought about having students essentially blog about peer-reviewed research for a general audience, but that's a separate skill set from figuring out their possible research topic. (Also, I'm not sure I would want to throw my students into the fire of Sb. Elli's blog doesn't attract the death threats that we get here sometimes.) I think I want the course to be more focused than it's been in the past, so I want every assignment to help them figure out what they want to do next year.

ask them to find the flaws in the paper - assumptions which are insufficiently justified, weaknesses in the logical arguments... in the past i have had students draw a flow chart off the arguments in a paper.... maybe i will blog this soon because it works pretty well to force them to be critical

*What further questions are raised by the research? E.g. when I was reviewing the tectonic evolution of the Or-Wa Coast Range, one model of its rotation was a number of independent blocks rotating against each other to give an overall appearance of the entire terrane rotating. That would imply a number of strike-slip faults, which, as best I can tell, aren't there. But they may be... it's hard to say with fifty frakking feet of soil in a temperate rain forest.

*In what context is the research question being addressed? E.g. continuing the above, it was the rather bizarre finding that the rocks of the Coast Range had rotated 80-90 degrees in the past 60 my. Obviously, 'splainin was needed.

*What assumptions are being made? Are they duly noted and/or cited? Have they been tested in some way? Could different, perhaps better, assumptions be made instead?

*What other research exists on the problem/topic? What differing conclusions have been reached? Are the differences reconcilable? If not, what information would favor one over the other? The last is a direction for new research.

I think one thing to keep in mind is that a single research paper in and of itself is sort of meaningless- it is grounded in many other projects and papers. If it's a good one, or a contentious one, it will lead to more. It's like a single data point; by itself, it doesn't say much. It's not clear to me how you're planning to structure the assignment, but I would suggest that you have students pick a central topic to review, spend maybe a week skimming over a number of papers, then picking five related ones to read carefully and critically, briefly summarize each one, then synthesize their thoughts and unanswered questions.

An easy way to do this would be to simply get a recent article, then methodically going through its citations. I had to do this several times as an undergrad; it got to the point that after I read the abstract, the next thing I looked at was the citations, to make sure what they were talking about was in line with what I was looking for.

But isn't understanding of the article a necessary prerequisite to critically evaluating whether the conclusions are justified?

Of course. My experience with graduate students learning how to read the scientific literature is that they do a perfectly fine job of understanding what the authors are saying, but tend to forget that their job is to be skeptical about it. I can see where for undergrads it might take nearly 100% of the time and effort just to get them to comprehend what is being asserted.

In my studies, I found that focusing on a single paper per student works much better than having them report on a lot of different papers.

First week or two: Have the students write a broad, somewhat vague thesis proposal, with references.

Week 2: Have the students choose one reference from that thesis proposal that is representative of the kind of work they want to do. They will broadly outline this paper, its goals, its methods, its strengths, its weaknesses, etc. This is NOT a review, just an outline or similar; it should not have a lot of prose. This can be graded as completed/not completed.

Weeks 3&4: The students will reread the paper and present it in class. The summary of the presentation should be what they think the authors did right, missed, should have done, etc. Grading can be pretty simple.

Week 5: The students will propose a way to strengthen the paper. They will not do this work, but will have a detailed proposal that describes the weaknesses (with references) that they think can be overcome (with references). This will require work to grade.

Week 6 or 7: You will have reviewed their Week 5 proposal and they'll respond to the review with a stronger proposal. More work for grading.

Obviously the timeline is completely flexible and up to you (and doesn't need to be worked on straight through--you can give as much time between tasks as you deem necessary).

I did something similar as an undergrad and the result became my first peer-reviewed journal article and a chapter in my dissertation. I had a class in which I reviewed some literature on a topic that interested me, found a flaw in the work, proposed to correct that flaw with work of my own, re-proposed after review, then presented the main paper as a journal club presentation, and finally wrote up all the work; peer-review was surprisingly smooth. Since we're talking senior thesis and not dissertation, a single paper seems appropriate.

Three problems to watch out for:

1.If you give the wrong assignment, students will just paraphrase (or worse) the original text.
2.Students need quite a bit of knowledge (and/or lots and lots of feedback) before they can write a reasonable research proposal.
3.Not all papers are well-written, and some will be more accessible to students than others.

I would suggest choosing a few papers FOR themâpapers that are well-written and that can be understood with what your students currently understandâand giving them a choice from that list of papers. Then, you might want to scale back a bit, with something like this:

What is ONE specific question addressed by these researchers?

How do the authors make that question sound important, or worth addressing?

What data did they collect to address that one question, and how did they collect it?

What are two questions that you would ask next, based on this research? Or, What one question would you ask next, based on this research?

I donât think the students will be able to address these issues: âHow does this work fit with other work that's been done and questions being asked? Why does anyone care about this research?â

I donât think your students will be able to address these issues: âDo you accept the author's conclusions? If not, are there other approaches that could allow you to test their conclusion?â

By Jan Pechenik (not verified) on 08 Dec 2009 #permalink

I would suggest asking the students if the paper actually moved the research along in the subject. Too often I have read papers, expecially those by post docs looking to publish, that is nothing more that a regugitation of other peoples ideas and work. Feeling pressed to publish in volume rather than quality, I see this a lot. One tends to get more information from the papers the author cites.

I don't think you can expect college juniors to know whether a particular research paper moves a field forward or not, especially if this is the only paper they read. Perhaps you can deliberately select a few papers that WERE important in developing the field, and assign those papers. It should be clear to students why those papers are important from what they can read in their textbooks. Would that work for you?