"Randomized trials cannot address all causal questions of importance in medicine and health policy and may have limited generalizability; thus, investigators may need to use observational studies as a source of evidence to address causal questions. The challenge, then, is to balance the importance of addressing the causal questions for which observational studies are needed with caution regarding the reliance on strong assumptions to support causal conclusions."
A challenge of our time truly
Source: "Causal Inference About the Effects of Interventions From Observational Studies in Medical Journals"
Folks outside of science may not be familiar with this but there is an absolutely massive struggle across so many fields to think about whether our scientific conventions around evidence from the 1900s are suitable to this century. I am firmly on the side of observational causal inference; the principle of "randomization" is almost never ACTUALLY implemented in the real world and our ecological validity suffers. We need all forms of evidence. RCTs are not possible for MOST questions.
it is also necessary to acknowledge the deep sociological realities that keep us from advancing: grad students who are massively underserved and undertrained in new methods, the devaluing of "practical statistics" education (despite all the hype about data); if we had healthier more inclusive pathways of learning for trainees of course we would have healthier methods. Many of us out here doing applied science have to entirely self-teach and un-learn poor statistics and poor methods training
It's really interesting to see health, achingly and terribly slowly, BEGIN to acknowledge the scale of the problem. Ironically in many ways health research has gone backwards in terms of not respecting or investing in high quality observations, as MDs churn out salami-sliced papers with stats they farm out to somebody else. The med school<>med publishing cycles are really something else. Can't tell you how many friends I've had consult on med papers' stats and get treated like shit by MDs.
I think what really gets me about this stuff is remembering how deeply I was interested in and motivated by the possibilities of large observational datasets in the world, and how soundly that was discouraged and made to feel lesser by the scientific establishment. How I was always more interested in properties of observed data and confounding and interactions, how it was never recognized that these are deep mathematical and technical skills I had, just because it wasn't in vogue posturing
We have to somehow thread this needle: we need to be hard on the way we do science but gentle as hell on the scientists because I know -- I KNOW -- we won't change or fix anything by making a scared grad student who's being yelled at by their PI feel lesser and like they're messing up their analyses. Instead we need to create pathways and acceptance for learning, for welcoming people who genuinely have been trying really hard to create evidence in this world.
Finally it's VERY freeing to embrace the reality that all scientific evidence is useful-but-wrong in some way and that this is the nature of it and that we are not defined as scientists by Being Right but by Trying To Be More Right and that is an incredibly human-centered philosophy that opens up your mind and heart to non-extractive, collaborative science, to seeing yourself as a collaborator not a ruler in this world and ecosystem of which we are all such a tiny part. So many fears fall away.
Oh, it's not just a massive intellectual struggle within the scientific community. It's become a hot "identity politics" issue in the general population. (… of those who argue online.)
"Deniers" shout "Double Blind Randomized Controlled Trials are The Gold Standard of Science! I will not accept any 'evidence' that 'falls short' of that!!!"
…
@JeffGrigg yeah totally. Very wild how these things became denialist catch phrases
But, ...
Doing an DBRCT of firefighter protective equipment is impossible because, …
1. It's unethical to knowingly subject study participants to harm. No ethics board would ever approve such a study.
2. No sane firefighter would agree to participate in such a study.
3. There's no practical way it could be "Double Blind," as the study participants would experience severe burns, and/or inability to breathe, with fake equipment.
.
(… same today's "hot button" issues.)
@grimalkina I regularly point out to people that the right way to think about science training is that it is more about teaching to be ok with saying "I Don't Know". Science is more about accepting that there are some answers we cannot adequately answer right now and living with it.
It is not about being right, it is about being ok with saying that we may not have a right answer rn.
@grimalkina This might be really dumb or something that’s already being done but is it possible to separate the statistics part and the science part? Like, the scientist comes up with the hypothesis and the experimental design and makes the observations, then collaborates with a statistician who crunches the numbers? Hell, maybe even *assign* a statistician to the scientist to ensure they’re disinterested in the outcome.
@ratkins It's not dumb at all, this is very frequently a model that's used and absolutely exists for some labs/teams/fields HOWEVER there are lots of different ways to be a "scientist" and for many types of work you also want scientists to know the research statistics and make research statistical choices (it is a large part of our authorship. Disinterest is not the only way we safeguard results because informed knowledge and expertise in the area can also be really important. So...
@ratkins ...it is better to have pre-registration of analysis plans and other things like that that ensure we commit to a path rather than "hack" the results; I don't personally believe good science comes from just reducing all the pieces to vendors and contractors. This is kind of how business school research works a lot of the time which many of us feel VERY negatively about because it's like uuuh, you're just paying for someone else to do your science and then getting authorship??
@ratkins But for departments, labs, and teams to sometimes or often have a dedicated "statistician" role? That is a model I believe in (assuming it is supported, respected, well staffed etc) and in fact one that I just created for my own lab. And I have a number of friends who are "staff statistician" kind of roles within scientific departments. It can be really good to have this as a dedicated team member!
@grimalkina I agree outsourcing sounds like a terrible model. I more had in mind the “balanced team” approach we used at Pivotal Labs where engineers, designers and product managers worked together daily—often pairing across disciplines—in service of the project. I’m more interested in the things a scientist *doesn’t find* in the data (because they find the stats painful and want to get back to “the science”) than the p-hacking to get the result they thought they should.
@grimalkina I suppose it’s Sturgeon’s Law, 90% of everything is crap like any other field. Once in a while you get a team who fights against the overwhelming external incentives and engineers themselves an environment where everyone is respected and valued and they can do great work.
"Many of us out here doing applied science have to entirely self-teach and un-learn poor statistics and poor methods training."
*So true*.
I see recent graduates with the same faulty NHST-based statistical education that I received decades ago. It's disappointing how poorly education has kept up with new and better statistical methods.
@rdnielsen @grimalkina As one of those self-learners, I feel like the main benefit of the statistics courses I took in university was to introduce me to a variety of statistical methods—but anything to do with statistical thinking I’ve had to develop on my own through experience and self-study.
It’s sad because there are tons of high quality resources out there teaching these skills, but they weren’t covered in courses I took due to lack of awareness, politics, etc.
@grimalkina Do you have any recommendations for which statistical methods to learn? I have the space right now to learn more, and I'd love to spend that effort usefully.
@willyyam tell me more about where your statistical "home" feels like it is and what kind of problems in the world you are thinking about!!
@grimalkina I am a data scientist/librarian working in healthcare, but I spend most of my time cleaning data and building pipelines - but I would love to know more about how to analyze data to see latent signals. The kind of things you can't build a trial for, like progression of untreated delirium, or pressure ulcers. If you knew the condition existed, you would act; but you don't always know.
@grimalkina I am statistically homeless; I know enough clin epi to know to keep my mouth shut :-)
RCT = Randomized Controlled Trial
for those without the necessary background context.