How to Not Make a Scientific Journal by Accident

Most scientists agree the publishing system is broken. Journals cost billions, they delay discoveries by months or years, and they turn science from a messy search for truth into a polished performance of certainty. We mostly agree on all of this, and yet we can’t seem to get out of our own way long enough to fix it.

What I keep noticing is stranger than the broken system itself: the people working hardest to escape journals are quietly rebuilding them. You might hate journals and be certain you’d never make one, but it’s worth looking at what you’re actually asking for. If you want shortcuts to understanding research, or consensus before you’ll believe something, or curation to save you time, then you want a journal. You’re just building it under a different name.

And I’m watching it happen right now, among the most progressive champions of open science.

Nervous reformers make accidental gatekeepers

ArXiv liberated math and physics for more than thirty years. Researchers share work the moment it’s ready, without waiting months for editorial approval and without paying to be read, because the whole premise was that the community could evaluate the work directly rather than letting intermediaries decide what deserved attention.

So it’s worth sitting with their recent announcement that position papers and reviews will now go through pre-publication peer review. Only those, not original research. But watch where the line is moving. A preprint server is adding peer review, and the platform built to bypass editorial control is now adding controls of its own.

What’s driving it is anxiety about AI-generated content flooding the server, and that anxiety is really the whole story. When reformers hit a problem they don’t recognize, they reach for the tools they know, even when those tools are exactly what they set out to escape in the first place.

I saw the same pattern spread at a recent Chan Zuckerberg Initiative meeting, where open science advocates who’d spent years fighting journal gatekeeping started panicking about AI content, and every solution they reached for was another filter, more vetting, more controls. These weren’t old-guard editors protecting their territory. They were reformers rebuilding the thing they’d fought to tear down.

Even fresh builders fall into it, and the reason is worth understanding. Organizations bring in outsiders precisely because outsiders can see the pathologies with fresh eyes, but those same outsiders, careful not to be naive or to build in isolation, stay vigilant about user feedback. The catch is that users want whatever they think a journal is supposed to give them, and an outsider isn’t in a position to hold firm and say no. So they add ranking algorithms, quality filters, and consensus mechanisms, all in the name of helping people find what they want, when what scientists actually want is the warm blanket of journals: the shortcuts, the proxies, the pre-digested truth.

Build exactly what users ask for, even with the best intentions in the world, and you’ll rebuild what’s killing science.

The motives

This runs deeper than institutional capture or profit motives, which are the easy targets. The harder truth is that we, as scientists, hunger for the very features that make journals toxic. We want control over who gets to criticize our work. We slide into homogeneity in how research gets shared. We crave proxies when we’re reading outside our own wheelhouse, and we want gatekeepers standing between us and embarrassment. We want the safety net of other people’s opinions when we’re deciding who deserves the scarce jobs, the funding, the awards, the promotions. We understand exactly how messy science really is, and we still reach for simplicity and certainty over the chaotic, iterative reality of discovery.

We’ve got the whole thing backwards. Forget making science digestible for mass audiences. The real challenge is making sure a dataset buried in some obscure graduate thesis reaches the scientist at a national lab who can pull new insight out of it, and that dataset shouldn’t have to appear in a glossy magazine, funneled into fashionable framing, for the people who can actually use it to find it. Science advances through weird connections, through esoteric knowledge finding its match.

This is where AI changes things. AI can process raw notebooks, failed experiments, half-formed thoughts, and it can connect patterns across disciplines no human would think to bridge, and it’s only going to get better at all of it. Yet the response from the science community is vigilance, defensiveness, and alarm: pre-publication review to combat AI content, human gatekeepers to protect us from machine abundance.

It’s worth sitting with how strange that is. These platforms were built on a vision of near-universal open sharing, and their stated endgame is removing barriers and making far more content freely available, so when an additional surge of content triggers panic and new filtering, it tells you something real about whether those values were ever settled.

ArXiv argues that once the server becomes diluted it’s game over before any other solution can take hold, and I understand the concern. But human gatekeepers can’t keep up with scale either, and arbitrary filtering, prestige proxies, and overly stringent criteria end up being a greater threat than dilution ever was. If preprint servers become just another set of journals, that’s game over too.

What was the plan if the culture shift they want actually happens? If preprinting keeps spreading across disciplines, volume will grow far past today’s levels regardless of AI, and if the only way to cope with that future is heavier gatekeeping, then we’re drifting right back toward the artificial scarcity these systems were supposed to replace.

What scientists fear

Scientists complain endlessly about Twitter’s “For You” algorithm deciding what they see, and they flee to other platforms looking for more reader control. And then those same scientists turn around and demand that scientific publishing have editorial filters, curation committees, and quality rankings. They hate algorithms shaping their social media and desperately want algorithms shaping their research.

Every attempt to make science cleaner, clearer, and more consensus-driven strips away something essential. Unified formats erase the scribbled margin note that sparks a revolution. Significance filters bury the null result that would have saved a decade of wasted effort. Demands for immediate validation punish long-term thinking and the slow work of iterative refinement.

The fear of AI content points to a deeper discomfort, which is that we don’t actually like abundance. When we abdicate our judgment, our judgment atrophies and our calibration suffers, and we lose the independent critical thinking that defines us in the first place. We’re giving up the most valuable thing we do, which is grappling with uncertainty and separating signal from noise, all to avoid the discomfort of having to think for ourselves. That same abdication is what created the journal system to begin with.

The uncomfortable reality is that all content will be partly or fully AI-generated before long, and trying to filter it out is like trying to filter out anything written with a word processor. What matters is whether something advances knowledge and reaches the people it can help, and determining that takes engagement, not gatekeeping.

There’s a parallel here worth holding onto. We pour enormous energy into worrying that machine-learning models will become biased, overfit their training data, and fail to generalize to anything new, and then we go and engineer the exact same failure mode into human scientists. By forcing researchers to overindex on consensus views, editorial panels, and journal hierarchies, we train people on an artificially narrowed dataset. We constrain the variation they’re exposed to, suppress the outliers, hide the null results, and erase the rough edges of scientific exploration.

How to fix this

The fix means letting go of the journal mindset altogether. Stop asking for curated feeds of important research. Stop expecting three reviewers to settle what’s true. Stop believing science should be immediately comprehensible to everyone, and accept that most science will be irrelevant noise to most people, which is exactly how it should be at the leading edge of discovery.

Additional layers can serve broader audiences without holding back primary research, the same way journal front matter and popular science coverage always have. The problems of science at the leading edge have been muddied by conflating them with the problems of reaching non-experts, and we’ve sacrificed solving the first to chase the second.

Let research exist in its natural state, messy and contradictory and incomplete. Let scientists publish their notebooks, their doubts, their abandoned threads. Let machines work through that chaos and surface the unexpected connections, and let the right information find the right researcher through search and serendipity, through algorithms that empower people rather than decide for them.

The journal system fails at accessibility anyway. Peer-reviewed papers sit on every side of every debate, and meta-analyses lean on the false certainty of binary journal decisions that wash away the flaws and the nuance. It’s abstraction built on abstraction, creating an illusion of certainty where none exists.

If we separate the two goals, advancing science and communicating to broad audiences, we can serve both far better. We can build whatever layers we want for public understanding without forcing the actual practice of science through filters that were never meant for it. The real damage comes when the opinions of the people who control visibility decide what can be found at all.

What problem are we fixing

Progress happens when that one person with that one missing piece finds exactly what they need. Truly innovative work often takes many publications and downstream uses before anyone recognizes it as significant, and to suppress what can’t immediately win universal acceptance is to suppress progress itself.

The problem we actually need to solve is getting the right information to the right people, and every layer of curation, every consensus mechanism, every quality filter makes that problem worse. The esoteric needs to find the esoteric. The fringe needs to find the fringe. That’s where the breakthroughs live.

The next time you catch yourself wanting someone to tell you what research matters, or wanting consensus before you’ll believe something, or wanting a cleaner format for sharing work, recognize what you’re doing. You’re rebuilding the journal system, one perfectly reasonable request at a time.

If we can resist clinging to false certainty every time a new challenge makes us uncomfortable, something much more powerful opens up. Knowledge can contribute to human understanding without first being filtered, formatted, and approved.

The mess is the feature. We should stop trying to clean it up.