Back

Source: "Causal Inference About the Effects of Interventions From Observational Studies in Medical Journals"

https://jamanetwork.com/journals/jama/fullarticle/2818746

it is also necessary to acknowledge the deep sociological realities that keep us from advancing: grad students who are massively underserved and undertrained in new methods, the devaluing of "practical statistics" education (despite all the hype about data); if we had healthier more inclusive pathways of learning for trainees of course we would have healthier methods. Many of us out here doing applied science have to entirely self-teach and un-learn poor statistics and poor methods training

It's really interesting to see health, achingly and terribly slowly, BEGIN to acknowledge the scale of the problem. Ironically in many ways health research has gone backwards in terms of not respecting or investing in high quality observations, as MDs churn out salami-sliced papers with stats they farm out to somebody else. The med school<>med publishing cycles are really something else. Can't tell you how many friends I've had consult on med papers' stats and get treated like shit by MDs.

I think what really gets me about this stuff is remembering how deeply I was interested in and motivated by the possibilities of large observational datasets in the world, and how soundly that was discouraged and made to feel lesser by the scientific establishment. How I was always more interested in properties of observed data and confounding and interactions, how it was never recognized that these are deep mathematical and technical skills I had, just because it wasn't in vogue posturing

We have to somehow thread this needle: we need to be hard on the way we do science but gentle as hell on the scientists because I know -- I KNOW -- we won't change or fix anything by making a scared grad student who's being yelled at by their PI feel lesser and like they're messing up their analyses. Instead we need to create pathways and acceptance for learning, for welcoming people who genuinely have been trying really hard to create evidence in this world.

Finally it's VERY freeing to embrace the reality that all scientific evidence is useful-but-wrong in some way and that this is the nature of it and that we are not defined as scientists by Being Right but by Trying To Be More Right and that is an incredibly human-centered philosophy that opens up your mind and heart to non-extractive, collaborative science, to seeing yourself as a collaborator not a ruler in this world and ecosystem of which we are all such a tiny part. So many fears fall away.

@JeffGrigg yeah totally. Very wild how these things became denialist catch phrases

@ratkins It's not dumb at all, this is very frequently a model that's used and absolutely exists for some labs/teams/fields HOWEVER there are lots of different ways to be a "scientist" and for many types of work you also want scientists to know the research statistics and make research statistical choices (it is a large part of our authorship. Disinterest is not the only way we safeguard results because informed knowledge and expertise in the area can also be really important. So...

@ratkins ...it is better to have pre-registration of analysis plans and other things like that that ensure we commit to a path rather than "hack" the results; I don't personally believe good science comes from just reducing all the pieces to vendors and contractors. This is kind of how business school research works a lot of the time which many of us feel VERY negatively about because it's like uuuh, you're just paying for someone else to do your science and then getting authorship??

@ratkins But for departments, labs, and teams to sometimes or often have a dedicated "statistician" role? That is a model I believe in (assuming it is supported, respected, well staffed etc) and in fact one that I just created for my own lab. And I have a number of friends who are "staff statistician" kind of roles within scientific departments. It can be really good to have this as a dedicated team member!

@grimalkina I agree outsourcing sounds like a terrible model. I more had in mind the “balanced team” approach we used at Pivotal Labs where engineers, designers and product managers worked together daily—often pairing across disciplines—in service of the project. I’m more interested in the things a scientist *doesn’t find* in the data (because they find the stats painful and want to get back to “the science”) than the p-hacking to get the result they thought they should.

@grimalkina I suppose it’s Sturgeon’s Law, 90% of everything is crap like any other field. Once in a while you get a team who fights against the overwhelming external incentives and engineers themselves an environment where everyone is respected and valued and they can do great work.

@rdnielsen @grimalkina As one of those self-learners, I feel like the main benefit of the statistics courses I took in university was to introduce me to a variety of statistical methods—but anything to do with statistical thinking I’ve had to develop on my own through experience and self-study.

It’s sad because there are tons of high quality resources out there teaching these skills, but they weren’t covered in courses I took due to lack of awareness, politics, etc.

@willyyam tell me more about where your statistical "home" feels like it is and what kind of problems in the world you are thinking about!!

@grimalkina I am a data scientist/librarian working in healthcare, but I spend most of my time cleaning data and building pipelines - but I would love to know more about how to analyze data to see latent signals. The kind of things you can't build a trial for, like progression of untreated delirium, or pressure ulcers. If you knew the condition existed, you would act; but you don't always know.