I've railed against the rise of supplemental data and methods before, but, having just reviewed a paper where I spent more time reading the supplemental sections versus the actual fucking paper, what Scicurious wrote struck a chord with me:
Sci wishes she could make her own flowchart of supplemental data. It might look like this:
Does the journal have ridiculously short page/word limits compared to the ridiculously huge amount of data they require for publication? Yes.
Are your methods way too numerous and complicated due to the ridiculously huge amount of data required for publication? Yes.
Did a reviewer demand extra experiments which in no way help your paper's cause and which no one but that particular reviewer cares about? Yes.
Do you feel the need to publish ALL OF YOUR FIGURES. ALL OF THEM? Yes.Then you need SUPPLEMENTAL DATA!!!
Now, are any of the above items NECESSARY to the publication of decent science?
Nope.
Scicurious then explains what doing away with supplemental science will mean:
Of course, the final result of HAVING supplemental data in high end journals is that you end up with these tiny, tight little stories with ridiculously small word limits in Science, and with the "oh, yeah, here's what we REALLY did" in the fifty kabillion supplemental pages. It also means that you can get away with not particularly well crafted stories, as your many little side experiments get relegated to the supplemental data. So I'm glad that J Neuroscience is going to be getting rid of Supplemental data, but I also hope that they, and their REVIEWERS, keep this in mind. It's going to mean, maybe demanding tighter stories, and maybe ALSO not demanding stories that are as sweeping and huge as the big journals like to go for. And it's going to mean thinking carefully before reviewers suggest that you do fifty billion extra experiments to tie in to their little corner of the field. It may also mean, that if Journals really want to publish well-done, carefully crafted, high end experiments, with difficult methods, they may need to expand their sometimes rather paltry word limits.
We forget that, ultimately, whether journals have supplemental sections (and if we have to review and write them) is not up to the journals but us. Consider the publishers' business model:
What has always steamed me about the for-profit publishers is that they charge so much for something they have very little part in manufacturing.
They don't pay the salaries of the those who provide the product--the research and accompanying article. They don't pay the reviewers. They often don't pay most of the editors. To top it all off, they take publicly funded research and makes the results inaccessible to many citizens who, often, have supported that research.
Oh yeah, researchers in the developing world often can't afford the articles to the point that there was a non-profit, Satellife, that used to transcribe select articles and email them to universities and hospitals (email was used since, in the 1990s, bandwidth and reliability were serious issues).
But other than that, there are no problems at all.
So to get all Marxist and shit*, I see two options here:
1) Simply refuse to review for and, if your career can take it, publish in journals that have supplemental sections.
2) When asked to review supplemental sections, which is a pain in the ass (I agreed to review one paper, not two or three), 'reorganize' the paper. Make your 'grade' of the paper (e.g., accept with major revisions) contingent on moving what belongs in the paper into the paper, and then cutting the rest**. The worst case scenario is that editors don't send you manuscripts to review. They're pretty desperate for reviewers though, so they might just have to grin and bear it.
There's no reason to subject ourselves to this, and we can do something about it.
Discuss.
*I'm not a Marxist, not by a long shot, but they're never around to talk about ownership of the mean of production when you need them.
**There's nothing wrong with appendices--a place to put data that isn't to be reviewed (i.e., strains used, long lists of genes found in one genome and not the other, etc.), although I would argue the appendices should still be part of the paper, at least online, if they're worth communicating.
- Log in to post comments
I feel like my field is "the exception that proves the rule" - supplementary materials are widespread, but are almost exclusively video clips of whatever animal behavior we're analyzing, because while we can illustrate it pretty well, nothing goes as far as video, especially for complex movements like flight or maneuvering. Generally, there's little or nothing else in the supplementary materials, and all of the relevant materials, experiments, and results are in the paper.
IMHO, *that* is what supplementary material is for - stuff that cannot go into a physical paper because it's in dead-tree format.
We use supplements partly as tutorials. In a recent paper on modeling hunting, we described the model in English as well as in math. I write my supplement for people who want to repeat parts of what we did, not for general readers.
"there was a non-profit, Satellife, that used to transcribe select articles and email them to universities and hospitals (email was used since, in the 1990s, bandwidth and reliability were serious issues)."
AND - we still do! HealthNet News goes out once a week, we also send out HealthNet News - AIDS and - Community Health.
While we have access to 25-28 publishers of medical, public health and scientific journals, you're correct that affordable access is still extremely limited to all of this content.
I agree with Mokele that being able to put videos in supplementary material (in my field this may be optical data or animation of results from a simulation) is superficially reasonable. But, and this is an important caveat, it must be in a format that anybody can view with default software. Asking somebody to download and install (and possibly pay for) software to view supplementary material is, IMHO, an unreasonable request; some scientists I know are prohibited by workplace rules from doing so. (Unfortunately, I have had an editor overrule me on that point.) Supplementary material may also be reasonable for other things that cannot be reasonably displayed on the printed page, although I can't think of any other examples from my field offhand.
However, anything that *can* go on the printed page *should* go on the printed page if you want it to be considered part of the paper. As Mike pointed out, we have had for decades the technology to include wonky details that most readers will want to skip: it's called an appendix. Supplementary text/data/figures, to me, are not part of the paper.
The diverse results in a scientific publication will have different levels of value depending on the reader's personal research interests. The availability of supplemental data allows for the key discoveries in a comprehensive study to standout in the main paper without the potential loss of useful information to specialists who can benefit from the other results. This is particularly important for systems biology type investigation, which are data-rich and could be further investigated by alternative strategies.
Access to supplemental data can also instill greater confidence in the rigor undertaken in a published study. Ph.D. thesis examiners expect a high degree of detail in the descriptions of methodologies, results and discussion from a Ph.D. candidate. Should we expect less from experienced investigators? For pragmatic reasons, published scientific papers should obviously not be so detailed. However, I for one would like to have the choice of seeing more data from a study if it is available. I would not want to penalize those journals that provide such an option.
Supplementary material is good for dumping the original data (videos would count of course), modelling and analysis source code and other such background material.
But there should be nothing in the supplemental data that is required in order to understand the paper. The paper needs to be a self-contained unit - it's the paper that gets passed around to colleagues, presented in journal clubs, used in graduate seminars and so on.
If the paper is incomprehensible without the supplemental data, it needs to be rewritten. And if the journal page limits make it impossible to do so, then you're trying to publish in the wrong journal. Find another journal with a better fit to your work and publish it there.
Thanks for the link love, Mike! :)
I agree with Janne, I of course have no problem with video or other things in the Supplemental Data which cannot be presented in another manner. But the issue is this:
"But there should be nothing in the supplemental data that is required in order to understand the paper. The paper needs to be a self-contained unit - it's the paper that gets passed around to colleagues, presented in journal clubs, used in graduate seminars and so on."
That's where we run in to problems. Many of the major papers I have read recently in the big journals (Science, Nature, Nature Neurosci, etc), do not make any sense without the supplemental data. They put essential controls in the supplement, as well as the entirety of the methods section. But the problem is that, with the demands placed for things like word and page limits, and the very complicated methods that need to be reported in many fields, if we try to fit in all we need to fit in, we will be unable to publish in the big journals like Science or Nature (4,500 words and 3,000 word limits, respectively). While those word limits may be fine for some fields, why should fields like neuroscience be effectively barred from publication in those fields, because our methods alone exceed 3,000 words (you don't believe me, try describing optogenetics coupled with, say drug self-administration on four different schedules, and a viral mediated gene expression experiment)? In addition, even with the rest of the results and methods within the word limit, it is often extremely difficult to place the work in the context of a now highly-specialized sub-discipline in the few hundred words allotted in the discussion, let alone the fact that references (and you can only have 40 or less) are ALSO included within these word limits.
And of course, none of this addresses why scientists feel so much pressure to publish in these extremely high journals with the extremely paltry word limits. While we CAN go for a lower journal, having GlamourMag (TM) publications is often deemed necessary, or at least highly advisable, to obtain both funding and a job in your chosen field.
While we can rebel (and I think reviewers SHOULD rebel against excessive supplemental data), the final problem appears to be more multi-layered than just the supplemental data.
The only thing that maybe should not be posted within the article may be the original data, formatted for the web, not attached as PDF/Word/XLS/some other awful option.
However, data are essential, not a supplement. If you want a supplement, call the short version that gets printed the 'supplement' and the original, full-length article online the 'article'.
See also: http://friendfeed.com/mndoci/ed04168f/note-to-scientists-ultimately-it-…
One of the purposes of publishing your work in a professional journal is to give the reader enough detail so the research can be replicated. Word limits serve to help print journals keep their publishing costs low and their profits high. As long as journals publish print editions, first-line journals as well as specialist journals will limit the space available for each article. Since many journals offer web-available space for data, expanded methods and analysis sections, etc. publishing scientists can live with space limitations of the print editions.
Though I agree with Mike's analysis of the publisher's business model, I am unconvinced by arguments to do away with appendices and supplements in electronic versions. If a reviewer receives a supplemental package larger than the article itself, it is not like reviewing two or three papers for the price of one. The reviewer's obligation is to judge what part of the supplementary material is relevant to the paper (and not just to one reviewer's special little interest), and examine in detail just that material.
As an example of the value of electronically-published supplementary information, let's look briefly at some fields that rely heavily on statistical or mathematical modeling, principally the observational sciences, such as economics, cosmology-astrophysics or epidemiology. Evaluating the validity of conclusions in those and related fields depends on verifying the fit between the proposed models and the data. Sometimes the author can provide that information in a table and/or a figure, but most often not so simply. Any fool can open a statistical program, plug in variables in a multiple regression model, present a table of coefficients and standard errors, and then present conclusions based on the table. But all statistical models (and all mathematical models that deal with error and stochastic processes), require the data to meet certain assumptions of the used model. Presenting the diagnostics of those assumptions can occupy substantially more journal space than presenting the results themselves. As a reviewer, I sometimes review manuscripts that make absolutely no mention of diagnostics or use throw-away sentences "we performed all diagnostic tests...", where presented graphics show clear violations of assumptions. And, due to journal space limitations, there is often no graphic presentation of the data and results to guide the reader. I those cases, I have to return the manuscript to the editors, asking the authors to supply sufficient detail to verify the validity of their selected models.
This causes delay in publication. Lack of this information also diminishes the utility of the published work, as the users of the publication cannot properly evaluate where published coefficients are biased and standard errors are inefficient.
In contrast to some commentators' views, I would like to see the use of electronic publication of supplementary material increased and welcome manuscript review requests that include them.