Problems with Academia and scientific studies (1 Viewer)

[MTskibum]

Please read the below comic although it is quite long. It might take a minute or two to read, this guy is good. He tackles scientific and economic issues as good as anyone on the internet. Today's comic is one of his better ones.

https://www.smbc-comics.com/comic/science-fictions

The DM of our DnD group is an assistant professor of economics at an prodigious school. He has to pump out economic studies like crazy to try and get tenure. To become a full professor they need to put out so many papers that they cannot spend a long time on a specific paper. Occasionally we even need to delay DnD when he gets behind on his papers. Even though he only teaches a few entry level economics classes he spends more time working than me, and it is not even close.

 

[IvanKaramazov]

I have to chide the author of this cartoon for conflating "science" and "social science."  My general sense is that research in the hard sciences is pretty decent.  I'm 100% positive that a large volume of research in the social sciences is junk. I say that as a social scientist.  Most of this fellow's criticisms are spot-on when it comes to those disciplines.

Just to pick one small example, a lot of people would be surprised to learn that social scientists can publish empirical articles in peer-reviewed journals and -- in many cases -- refuse to share their data with other researchers.  I doubt that there's a lot of outright fraud in social science research, although there's definitely some in all disciplines.  There's definitely a ton of p-hacking though.

 

[MTskibum]

I have to chide the author of this cartoon for conflating "science" and "social science."  My general sense is that research in the hard sciences is pretty decent.  I'm 100% positive that a large volume of research in the social sciences is junk. I say that as a social scientist.  Most of this fellow's criticisms are spot-on when it comes to those disciplines.

Just to pick one small example, a lot of people would be surprised to learn that social scientists can publish empirical articles in peer-reviewed journals and -- in many cases -- refuse to share their data with other researchers.  I doubt that there's a lot of outright fraud in social science research, although there's definitely some in all disciplines.  There's definitely a ton of p-hacking though.

Social science is definitely the worst off when it comes to this phenomenon. However it is not the only one effected. There have been more than a few studies that show that hard science studies cannot be reproduced often enough.

I am not anti science at all. The first web sites I visit in the morning are live science and ars technica. However, because of the reproducibility problem even when reading studies in highly thought of journals like , Nature, NEJM, etc you always have to be careful that what you are reading may not be technically accurate unless you can verify it through a completely independent study.

However, i think that reform needs to be done. I like one of the solutions that he proposes near the bottom. We should incentivize universities to try to reproduce previous studies. Then we will know if it truly was a breakthrough, or if they just hit up some outliers and did not know statistics.

:Edit: I linked a couple of the studies from my work computer and I can tell from my home computer that at least one of them is a paid site that my company pays for. Sorry about that.

 

[tri-man 47]

@IvanKaramazov raises valid points.  Having switched to a full-time tenure track position in my accounting field five years ago (a late-in-life shift), I now have publication requirements.  Fortunately, I'm at a small, private university where the focus is still on the classroom more than research, so the research requirements are light.

I'm amused when I go to the regional and national conferences and see the line-up of research-related sessions.  Often, the session has 3-4 faculty presenting their research with a faculty respondent on each.  Maybe 2-3 other faculty attend the session because of a topic of interest.  Very geeky stuff.    And often, very obscure topics ...and results that are not necessarily conclusive.  Also, these studies are often carried out by doctoral students or younger faculty - many of them international (far east in particular) - with no business experience.  I just can't picture a CEO or a Board member reading the study and thinking, "hey, thanks!  I didn't know that green-eyed, purple-haired corporate officers are more likely to decline stock options!  Good to know!"

That said, a paper I submitted just last week took me deep down the rabbit hole of cognitive learning theories *.  It was fascinating stuff, and it's already a game changer in terms of how I approach the classroom.  But it's really a commentary, not a statistical study ('cause I never mastered those concepts).  Yet, that's the benefit of research ...when it informs better teaching.

* e.g., students' approaches to learning; cognitive load theory; achievement goal theory; self-regulated learning; universal design for learning     

 

[Doug B]

I doubt that there's a lot of outright fraud in social science research, although there's definitely some in all disciplines.  There's definitely a ton of p-hacking though.

Term of art that needs defining for a general audience:

In our explainer on statistical significance and statistical testing, we introduced how you go about testing a hypothesis, and what can be legitimately inferred from the results of a statistical test. In this post, we will look at a way in which this process can be abused to create misleading results. This is a technique known colloquially as ‘p-hacking’. It is a misuse of data analysis to find patterns in data that can be presented as statistically significant when in fact there is no real underlying effect.

Also see here:

Data dredging (also data fishing, data snooping, data butchery, and p-hacking) is the misuse of data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives. This is done by performing many statistical tests on the data and only reporting those that come back with significant results

 

[The Z Machine]

I have to chide the author of this cartoon for conflating "science" and "social science."  My general sense is that research in the hard sciences is pretty decent.  I'm 100% positive that a large volume of research in the social sciences is junk. I say that as a social scientist.  Most of this fellow's criticisms are spot-on when it comes to those disciplines.

Just to pick one small example, a lot of people would be surprised to learn that social scientists can publish empirical articles in peer-reviewed journals and -- in many cases -- refuse to share their data with other researchers.  I doubt that there's a lot of outright fraud in social science research, although there's definitely some in all disciplines.  There's definitely a ton of p-hacking though.

I've published engineering research (15+ years ago) in peer reviewed journals.  It's not much better in that field.  Maybe it's changed, but i didn't have to make my data publicly available.

My wife is an academic in public health.  She does mostly qualitative research, but she gets way more scrutiny  on the rigor of her research methods than I ever did. 

She's chosen to be "research faculty" and not tenure track, which means a) she has less pressure to publish to retain her position, and b) she cannot be the PI on a very large grant and must partner with a tenure track (typically full prof) faculty. It suits her, although there is tension with the second class status of research faculty vs. tenure track. Plus the ranks do not have equal gender balance.