Our conclusion was to never trust psychology majors with computer code. And like with any other expertise field they should have shown their idea and/or code to some CS majors at the very least before publishing.

From the perspective of the academic community, there will be lower incentive to publish incorrect results if data and code is shared.

Now I'm not saying that everything in M-S is junk, but the small subset I was exposed to was.

They make a lot of claims on how much faster they are than MASON, Netlogo, and Mesa. But in practice I am not finding that to be the case. Also they arent counting the Julia compilation step which takes an absurdly long time, and by the time that gets done similar simulations are already done, then they start the clock on their own benchmark.

Agents.jl and Mesa have the selling point of having better languages / libraries for numerical computation. But thats really a subset of msor ABM I think.

I was an undergraduate at the University of Maryland when you were a graduate student there in the mid nineties. A lot of what you had to say shaped the way I think about computer science. Thank you.

I recommended that the journal not publish the paper, and gave them a good list of improvements to give to the authors that should be made before re-submitting. The journal agreed with me, and rejected the paper.

A couple of months later, I saw it had been published unchanged in a different journal. It wasn't even a lower-quality journal, if I recall the impact factor was actually higher than the original one.

I despair of the scientific process.

How sad. Admitting and correcting a mistake may feel difficult, but it makes you credible.

As a reader, I would have much greater trust in a journal that solicited criticism and readily published corrections and retractions when warranted.

Ranking 1 to 3 - 1 being the best - 3 the bare minimum for publication.

3. Citations only

2. Citations + full disclosure of data.

1. Citations + full disclosure of data + replicated

Straight-up replications are rare, but if a finding is real, other PIs will partially replicate and build upon it, typically as a smaller step in a related study. (E.g., a new finding about memory comes out, my field is emotion, I might do a new study looking at how emotion and your memory finding interact.)

If the effect is replicable, it will end up used in other studies (subject to randomness and the file drawer effect, anyway). But if an effect is rarely mentioned in the literature afterwards...run far, FAR away, and don't base your research off it.

A good advisor will be able to warn you off lost causes like this.

When a junior researcher, e.g. a grad student, fails to replicate a study, they assume it's technique. If they can't get it after many tries, they just move on, and try some other research approach. If they claim it's because the original study is flawed, people will just assume they don't have the skills to replicate it.

One of the problems is that science doesn't have great collaborative infrastructure. The only way to learn that nobody can reproduce a finding is to go to conferences and have informal chats with people about the paper. Or maybe if you're lucky there's an email list for people in your field where they routinely troubleshoot each other's technique. But most of the time there's just not enough time to waste chasing these things down.

I can't speak to whether people get blackballed. There's a lot of strong personalities in science, but mostly people are direct and efficient. You can ask pretty pointed questions in a session and get pretty direct answers. But accusing someone of fraud is a serious accusation and you probably don't want to get a reputation for being an accuser, FWIW.

When you added it up, most of the hard parts were Engineering, and a bit Econ. You would really struggle to work through tough questions in engineering, spend a lot of time on economic theory, and then read the management stuff like you were reading a newspaper.

Management you could spot a mile away as being soft. There's certainly some interesting ideas, but even as students we could smell it was lacking something. It's just a bit too much like a History Channel documentary. Entertaining, certainly, but it felt like false enlightenment.

https://en.wikipedia.org/wiki/Addiction_Rare_in_Patients_Tre...

I've also seen the resistance that results from trying to investigate or even correct an issue in a key result of a paper. Even before it's published the barrier can be quite high (and I must admit that since it's not my primary focus and my name was not on it, I did not push as hard as I could have on it)

The replication crisis is largely particular to psychology, but I wonder about the scope of the don't rock the boat issue.

Schools should be using these kinds of examples in order to teach critical thinking. Unfortunately the other side of the lesson is how easy it is to push an agenda when you've got a little bit of private backing.

Personally, I would agree with you. That's how these things are supposed to work. In practice, people are still people.

This actually doesn't surprise much. I've seen a lot of variety in the ethical standards that people will publicly espouse.

These people are terrible at their job, perhaps a bit malicious too. They may be great people as friends and colleagues.

> Vonnegut is not, I believe, talking about mere inauthenticity. He is talking about engaging in activities which do not agree with what we ourselves feel are our own core morals while telling ourselves, “This is not who I really am. I am just going along with this on the outside to get by.” Vonnegut’s message is that the separation I just described between how we act externally and who we really are is imaginary.

https://thewisdomdaily.com/mother-night-we-are-what-we-prete...

For example, here's an article that argues (with data) that there is actually little publication bias in medical studies in the Cochrane database:

https://replicationindex.com/2020/12/24/ioannidis-is-wrong/

> because they know they can get away with it

the point is that the paved paths lead to bad behavior

well designed systems make it easy to do good

> Exactly the opposite is taught.

"trained" doesn't mean "taught". most things are learned but not taught

1. There are bad people

2. We know bad people are bad because they do bad things

3. There does not exist any set of bad actions that one could do to qualify one for the label of "bad person."

I've just come to the conclusion that a "bad person" is just a term for someone who does bad things, and for whom their extenuating circumstances don't count because they are the member of the wrong tribe.

I read the submitted version and told her it wasn't OK. She withdrew the paper and I left her lab shortly after. I simply could not stand the tendency to juice up papers, and I didn't want to have my reputation tainted by a paper that was false (I'm OK with my reputation being tainted by a paper that was just not very good).

What really bothers me is when authors intentionally leave out details of their method. There was a hot paper (this was ~20 years ago) about a computational biology technique ("evolutionary trace") and when we did the journal club, we tried to reproduce their results- which started with writing an implementation from their description. About half way through, we realized that the paper left out several key steps, and we were able to infer roughly what they did, but as far as we could tell, it was an intentional omission made to keep the competition from catching up quickly.

That's reference-stealing, some other paper I read cited this so it should be OK, I'll steal their reference. I always make sure I read the cited paper before citing it myself, it's scary how often it says something rather different to what the citation implies. That's not necessarily bad research, more that the author of the citing paper was looking for effect A in the cited reference and I'm looking for effect B, so their reason for citing differs from mine, and it's a valid reference in their paper but wouldn't be in mine.

IMHO this should be expected for any, literally any publication. If you have secrets, or proprietary information, fine - but then, you don't get to publish.

The number appears to be from Google Scholar, which currently reports 6269 citations for the paper

If the flow of tax, student debt and philanthropic money were cut off, the journals would all be wiped out because there's no organic demand for what they're doing.

They are pushed to publish a lot, which means journals have to review a lot of stuff (and they cannot replicate findings on their own). Once a paper is published on a decent journal, other researchers may not "waste time" replicating all findings, because they also want to publish a lot. The result is papers getting popular even if no one has actually bothered to replicate the results, especially if those papers are quoted by a lot of people and/or are written by otherwise reputable people or universities.

This is one of the reasons you should never accept a single publication at face value. But this isn’t a bug — it’s part of the algorithm. It’s just that most muggles don’t know how science actually works. Once you read enough papers in an area, you have a good sense of what’s in the norm of the distribution of knowledge, and if some flashy new result comes over the transom, you might be curious, but you’re not going to accept it without a lot more evidence.

This situation is different, because it’s a case where an extremely popular bit of accepted wisdom is both wrong, and the system itself appears to be unwilling to acknowledge the error.

Yes, the complicity is normal. No the complicity isn't right.

The banality of evil.

When the good thing is easier to do and they still knowingly pick the bad one for the love of the game?

you'll just get replication rings in addition to citation rings.

People who cheat in their papers will have no issues cheating in their replication studies too. All this does, is give them a new tool to attack papers they don't like by faking a failed replication.

“That’s a bad thing to do…”

Maybe should be: “That’s a stupid thing to do…”

Or: reckless, irresponsible, selfish, etc.

In other words, maybe it has nothing to do with morals and ethics. Bad is kind of a lame word with limited impact.

https://blog.plan99.net/replication-studies-cant-fix-science...

The idea failed a simple sanity check: just going to Google Scholar, doing a generic search and reading randomly selected papers from within the past 15 years or so. It turned out most of them were bogus in some obvious way. A lot of ideas for science reform take as axiomatic that the bad stuff is rare and just needs to be filtered out. Once you engage with some field's literatures in a systematic way, it becomes clear that it's more like searching for diamonds in the rough than filtering out occasional corruption.

But at that point you wonder, why bother? There is no alchemical algorithm that can convert intellectual lead into gold. If a field is 90% bogus then it just shouldn't be engaged with at all.

Still I'm skeptical about any sort of system trying to figure out 'trust'. There's too much on the line for researchers/students/... to the point where anything will eventually be gamed. Just too many people trying to get into the system (and getting in is the most important part).

[1] https://en.wikipedia.org/wiki/Replication_crisis#In_medicine

Judging from PubPeer, which allows people to post all of the above anonymously and with minimal moderation, this is not an issue in practice.

I only needed the Spanish translation. Now I am proficient in spoken and written Spanish, and I can perfectly understand what is said, and yet I still ran the English through Google Translate and printed it out without really checking through it.

I got to the podium and there was a line where I said "electricity is in the air" (a metaphor, obviously) and the Spanish translation said "electricidad no está en el aire" and I was able to correct that on-the-fly, but I was pissed at Translate, and I badmouthed it for months. And sure, it was my fault for not proofing and vetting the entire output, but come on!

That's not right; retractions should only be for research misconduct cases. It is a problem with the article's recommendations too. Even if a correction is published that the results may not hold, the article should stay where it is.

But I agree with the point about replications, which are much needed. That was also the best part in the article, i.e. "stop citing single studies as definitive".

Pushing for retraction just like that and going off to private sector is…idk it’s a decision.

> We need to throw all of this out by default. From public policy to courtrooms, we need to treat it like any other eyewitness claim.

If you can't trust eyewitness claims, if you can't trust video or photographic or audio evidence, then how does one Find Truth? Nobody really seems to have a solid answer to this.

God gave us free will to choose good or evil in various circumstances. We need to recognize that in our assessments. We must reward good choices and address bad ones (eg the study authors'). We should also change environments to promote good and oppose evil so the pressures are pushing in the right direction.

People are on average both bad and stupid and function without a framework of consequences and expectations where they expect to suffer and feel shame. They didn't make a mistake they stood in front of all their professional colleagues and published effectively what they knew were lies. The fact that they can publish lies and others are happy to build on lies ind indicates the whole community is a cancer. The fact that the community rejects calls for correction indicates its metastasized and at least as far as that particular community the patient is dead and there is nothing left to save.

They ought to be properly ridiculed and anyone who has published obvious trash should have any public funds yanked and become ineligible for life. People should watch their public ruin and consider their own future action.

If you consider the sheer amount of science that has turned out to be outright fraud in the last decade this is a crisis.

However if we stop teaching people that villains are bad and they shouldn't be villains, we'll end up with a whole lot more problems of the "yeah that guy is just bad" variety.

1: https://en.wikipedia.org/wiki/Separable_verb

As I said, harder from a research perspective, but if you can show, for instance, that sustainable companies are less profitable with a better study, you have basically contradicted the original one.

She was just done with it then and a pharma company said "hey you fed up with this shit and like money?" and she was and does.

edit: as per the other comment, my background is mathematics and statistics after engineering. I went into software but still have connections back to academia which I left many years ago because it was a political mess more than anything. Oh and I also like money.

This is true though, and one of those awkward times where good ideals like science and critical feedback brush up against potentially ugly human things like pride and ego.

I read a quote recently, and I don't like it, but it's stuck with me because it feels like it's dancing around the same awkward truth:

"tact is the art of make a point without making an enemy"

I guess part of being human is accepting that we're all human and will occasionally fail to be a perfect human.

Sometimes we'll make mistakes in conducting research. Sometimes we'll make mistakes in handling mistakes we or others made. Sometimes these mistakes will chain together to create situations like the post describes.

Making mistakes is easy - it's such a part of being human we often don't even notice we do it. Learning you've made a mistake is the hard part, and correcting that mistake is often even harder. Providing critical feedback, as necessary as it might be, typically involves putting someone else through hardship. I think we should all be at least slightly afraid and apprehensive of doing that, even if it's for a greater good.

A blameless organization can work, so long as people within it police themselves. As a society this does not happen, thus making people more steadfast in their anti-social behavior

I happen to agree that labeling them as villains wouldn’t have been helpful to this story, but they didn’t do that.

> It obscures the root causes of why the bad things are happening, and stands in the way of effective remedy.

There’s a toxic idea built into this statement: It implies that the real root cause is external to the people and therefore the solution must be a systemic change.

This hits a nerve for me because I’ve seen this specific mindset used to avoid removing obviously problematic people, instead always searching for a “root cause” that required us all to ignore the obvious human choices at the center of the problem.

Like blameless postmortems taken to a comical extreme where one person is always doing some careless that causes problems and we all have to brainstorm a way to pretend that the system failed, not the person who continues to cause us problems.

On the one hand, it is possible to become judgmental, habitually jumping to unwarranted and even unfair conclusions about the moral character of another person. On the other, we can habitually externalize the “root causes” instead of recognizing the vice and bad choices of the other.

The latter (externalization) is obvious when people habitually blame “systems” to rationalize misbehavior. This is the same logic that underpins the fantastically silly and flawed belief that under the “right system”, misbehavior would simply evaporate and utopia would be achieved. Sure, pathological systems can create perverse incentives, even ones that put extraordinary pressure on people, but moral character is not just some deterministic mechanical response to incentive. Murder doesn’t become okay because you had a “hard life”, for example. And even under “perfect conditions”, people would misbehave. In fact, they may even misbehave more in certain ways (think of the pathologies characteristic of the materially prosperous first world).

So, yes, we ought to condemn acts, we ought to be charitable, but we should also recognize human vice and the need for justice. Justly determined responsibility should affect someone’s reputation. In some cases, it would even be harmful to society not to harm the reputations of certain people.

1. Who is responsible for adding guardrails to ensure all papers coming in are thoroughly checked & reviewed?

2. Who review these papers? Shouldn’t they own responsibility for accuracy?

3. How are we going to ensure this is not repeated by others?

> If we systematically tie bad deeds to bad people, then surely those people we love and know to be good are incapable of what they're being accused.

A strong claim that needs to be supported and actually the question who’s nuances are being discussed in this thread.

I think perhaps blackball is guaranteed. No one likes a snitch. “We’re all just here to do work and get paid. He’s just doing what they make us do”. Scientist is just job. Most people are just “I put thing in tube. Make money by telling government about tube thing. No need to be religious about Science”.

You guys are saying that drink driving does not make someone a bad person. Ok. Let's say I grant you that. Where do you draw the line for someone being a bad person?

I mean with this line of reasoning you can "explain way" every bad deed and then nobody is a bad person. So do you guys consider someone to be actually a bad person and what did they have to do to cross that line where you can't explain away their bad deed anymore and you really consider them to be bad?

1) Anyone publishes anything they want, whenever they want, as much or as little as the want. Publishing does not say anything about your quality as a researcher, since anyone can do it.

2) Being published doesn't mean it's right, or even credible. No one is filtering the stream, so there's no cachet to being published.

We then let memetic evolution run its course. This is the system that got us Newton, Einstein, Darwin, Mendeleev, Euler, etc. It works, but it's slow, sometimes ugly to watch, and hard to game so some people would much rather use the "Approved by A Council of Peers" nonsense we're presently mired in.

It has 0 comments, for an article that forgot "not" in "the result is *** statistical significative".

I read the paper as well. My background is mathematics and statistics and the data was quite frankly synthesised.

Next, we need to understand why that is, which should be trusted, and which can't be. Also, what methods to use in what contexts. We need to develop education for people about how humanity actually works. We can improve steadily over time.

On my end, I've been collecting resources that might be helpful. That includes Christ-centered theology with real-world application, philosophies of knowledge with guides on each one, differences between real vs organized science, biological impact on these, dealing with media bias (eg AllSides), worldview analyses, critical thinking (logic), statistical analyses (esp error spotting), writing correct code, and so on.

One day, I might try to put it together into a series that equips people to navigate all of this stuff. For right now, I'm using it as a refresher to improve my own abilities ahead of entering the Data Science field.

Certainly, you are aware we literally had more crime back then, right? Additionally, we heaped shame on people who did not deserve it, like women and black people and gay people.

So what the fuck good does that do?

You know what actually changed? White collar crime stopped being a thing.

It's both obviously. To address the human cause, you have to call out the issues and put at risk the person's career by damaging their reputation. That's what this article is doing. You can't fix a person, but you can address their bad behavior in this way by creating consequences for the bad things.

Part of the root cause definitely is the friction aspect. The system is designed to make the bad thing easier, and when designing a system you need the good outcomes to be lower friction.

> This hits a nerve for me because I’ve seen this specific mindset used to avoid removing obviously problematic people, instead always searching for a “root cause” that required us all to ignore the obvious human choices at the center of the problem.

The real conversations like that take place in places where there's no recordings, or anything left in writing. Don't assume they aren't taking place, or that they go how you think they go.

In a blame focused postmortem you say “Johnny fucked up” and close it.

When you are about accountability, the responsible parties are known or discovered if unknown and are responsible for prevention/response/repair/etc. The corrective action can incorporate and number of things, including getting rid of Johnny.

1) Basic morality (good vs evil) with total agency ascribed to the individual

2) Basic systems (good vs bad), with total agency ascribed to the system and people treated as perfectly rational machines (where most of the comments here seem to sit)

3) Blended system and morality, or "Systemic Morality": agency can be system-based or individual-based, and morality can be good or bad. This is the single largest rung, because there's a lot to digest here, and it's where a lot of folks get stuck on one ("you can't blame people for making rational decisions in a bad system") or the other ("you can't fault systems designed by fallible humans"). It's why there's a lot of "that's just the way things are" useless attitudes at present, because folks don't want to climb higher than this rung lest they risk becoming accountable for their decisions to themselves and others.

4) "Comprehensive Morality": an action is net good or bad because of the system and the human. A good human in a bad system is more likely to make bad choices via adherence to systemic rules, just as a bad human in a good system is likely to find and exploit weaknesses in said system for personal gain. You cannot ascribe blame to one or the other, but rather acknowledge both separately and together. Think "Good Place" logic, with all of its caveats (good people in bad systems overwhelmingly make things worse by acting in good faith towards bad outcomes) and strengths (predictability of the masses at scale).

5) "Historical Morality": a system or person is net good or bad because of repeated patterns of behaviors within the limitations (incentives/disincentives) of the environment. A person who routinely exploits the good faith of others and the existing incentive structure of a system purely for personal enrichment is a bad person; a system that repeatedly and deliberately incentivizes the exploitation of its members to drive negative outcomes is a bad system. Individual acts or outcomes are less important than patterns of behavior and results. Humans struggle with this one because we live moment-to-moment, and we ultimately dread being held to account for past actions we can no longer change or undo. Yet it's because of that degree of accountability - that you can and will be held to account for past harms, even in problematic systems - that we have the rule of law, and civilization as a result.

Like a lot of the commenters here, I sat squarely in the third rung for years before realizing that I wasn't actually smart, but instead incredibly ignorant and entitled by refusing to truly evaluate root causes of systemic or personal issues and address them accordingly. It's not enough to merely identify a given cause and call it a day, you have to do something to change or address it to reduce the future likelihood of negative behaviors and outcomes; it's how I can rationalize not necessarily faulting a homeless person in a system that fails to address underlying causes of homelessness and people incentivized not to show empathy or compassion towards them, but also rationalize vilifying the wealthy classes who, despite having infinite access to wealth and knowledge, willfully and repeatedly choose to harm others instead of improving things.

Villainy and Heroism can be useful labels that don't necessarily simplify or ignorantly abstract the greater picture, and I'd like to think any critically-thinking human can understand when someone is using those terms from the first rung of the ladder versus the top rung.

It's a paradox. We know for an absolute fact that changing the underlying system matters massively but we must continue to acknowledge the individual choice because the system of consequences and as importantly the system of shame keeps those who wouldn't act morally in check. So we punish the person who was probably lead poisoned the same as any other despite knowing that we are partially at fault for the system that lead to their misbehavior.

The obedience to authority that we must be able to challenge to stand a chance.

Milgram's or Zimbardo's tests we're somewhat flawed yes, but still WW2 and Germany kinda proved this tendency IMO. And that's why I brought it up. Perhaps not the best comparison, I admit, but a comparison that seems more and more relevant ni many cases.

The whole "bad vs good person" framing is probably not a very robust framework, never thought about it much, so if that's your position you might well be right. But it's not a consideration that escaped me, I reasoned under the same lens the person above did on intention.

Anyone can do a bad deed.

Anyone can also be a good person to someone else.

If a bad deed automatically makes a bad person, those who recognize the person as good have a harder time reconciling the two realities. Simple.

Also, is the point recognizing bad people or getting rid of bad science. Like I said, choose your victories.

One thing I would very much like to see is personal financial disclosures about grant awards, salaries, and funding sources of the main authors of a paper.

The main author received a $400,000 grant from the Save Our Turtles foundation and a $2 million dollar grant from the John "Turtle Lover" Heisenberg foundation when writing his peer-reviewed paper revealing that more public funding for turtle sanctuaries unlocks massive local economic benefits in the Upper Mississippi Delta.

In terms of solutions, the practice of 'preregistration' seems like a move in the right direction.

Whatever happens, avoid direct confrontation at all costs.

Well, I'd argue the system failed in that the bad person is not removed. The root is then bad hiring decision and bad management of problematic people. You can do a blameless postmortem guiding a change in policy which ends in some people getting fired.

Not necessarily, although certainly people sometimes fall into that trap. When dealing with a system you need to fix the system. Ejecting a single problematic person doesn't fix the underlying problem - how did that person get in the door in the first place? If they weren't problematic when they arrived, does that mean there were corrosive elements in the environment that led to the change?

When a person who is a cog within a larger machine fails that is more or less by definition also an instance of the system failing.

Of course individual intent is also important. If Joe dropped the production database intentionally then in addition to asking "how the hell did someone like him end up in this role in the first place" you will also want to eject him from the organization (or at least from that role). But focusing on individual intent is going to cloud the process and the systemic fix is much more important than any individual one.

There's also a (meta) systemic angle to the above. Not everyone involved in carrying out the process will be equally mature, objective, and deliberate (by which I mean that unfortunately any organization is likely to contain at least a few fairly toxic people). If people jump to conclusions or go on a witch hunt that can constitute a serious systemic dysfunction in and of itself. Rigidly adhering to a blameless procedure is a way to guard against that while still working towards the necessary systemic changes.

Post-mortems are a terrible place for handling HR issues. I'd much rather they be kept focused on processes and technical details, and human problem be kept private.

Dogpiling in public is an absolutely awful thing to encourage, especially as it turns from removing a problematic individual to looking for whoever the scapegoat is this time.

I've read multiple times that a large percentage of the crime comes from a small group of people. Jail them, and the overall crime rate drops by that percentage.

Both views are maximalistic.

Ioannidis' work during Covid raised him in my esteem. It's rare to see someone in academics who is willing to set their own reputation on fire in search of truth.

“Most Published Research Findings Are False” —> “Most Published COVID-19 Research Findings Are False” -> “Uh oh, I did a wrongthink, let’s backtrack at bit”.

Is that it?

I don't think that that line can be drawn exactly. There are many factors to consider and I'm not sure that even considering them will allow you to draw this line and not come to claims like '99% of people are bad' or '99% of people are not bad'.

'Bad' is not an innate property of a person. 'Bad' is a label that exists only in an observer's model of the world. A spherical person in vacuum cannot be 'bad', but if we add an observer of the person, then they may become bad.

To my mind, the decision of labeling a person to be bad or not labeling them is a decision reflecting how the labeling subject cares about the one on the receiving side. So, it goes like this: first you decide what to do with bad behavior of someone, and if you decide to go about it with punishment, then you call them 'bad', if you decide to help them somehow to stop their bad behavior, then you don't call them bad.

It works like this: when observing some bad behavior I decide what to do about it. If I decide to punish a person, I declare them to be bad. If I decide to help them stop their behavior, I declare them to be not bad, but 'confused' or circumstantially forced, or whatever. Y'see: you cannot change personal traits of others, so if you declare that the reason of bad behavior is a personal trait 'bad' then you cannot do anything about it. If you want to change things, you need to find a cause of bad behavior, that can be controlled.

The system ends up promoting an even more conservative culture. What might start great will end up with groups and institutions being even more protective of 'their truths' to avoid getting tainted.

Don't think there's any system which can avoid these sort of things, people were talking about this before WW1, globalisation just put it in overdrive.

We are just back to the universities under the religious control system that we had before the Enlightenment. Any change would require separating academia from political government power.

Academia is just the propaganda machine for the government, just like the church was the tool for justifying god-gifted powers to kings.

Yet, I believe there hasn't been much progress as compared with STEM. But it is just a belief at the end of the day. There might be some study about this out there.

With the above, I think we've empirically proven that we can't trust mathmeticians more than any other humans We should still rigorously verify their work with diverse, logical, and empirical methods. Also, build ground up on solid ideas that are highly vetted. (Which linear algebra actually does.)

The other approach people are taking are foundational, machine-checked, proof assistants. These use a vetted logic whose assistant produces a series of steps that can be checked by a tiny, highly-verified checker. They'll also oftne use a reliable formalism to check other formalisms. The people doing this have been making everything from proof checkers to compilers to assembly languages to code extraction in those tools so they are highly trustworthy.

But, we still need people to look at the specs of all that to see if there are spec errors. There's fewer people who can vet the specs than can check the original English and code combos. So, are they more trustworthy? (Who knows except when tested empirically on many programs or proofs, like CompCert was.)

Scientists that have studied this over long periods of times and diverse population groups?

I've done this firsthand - remembered an event a particular way only to see video (in the old days, before easy video editing) and find out it... didn't quite happen as I remembered.

That's because human beings aren't video recorders. We're encoding emotions into sensor data, and get blinded by things like Weapon Focus and Selective Attention.

But the article is generally weird or even harmful too. Going to social media with these things and all; we have enough of that "pretty" stuff already.

Also if that doesn’t work, “Hey Bro I notice you like to give a lot of detail in standup. That’s great, but we want to keep it a short meeting so we try to focus on just the highlights and surfacing any key blockers. I don’t want to interrupt you, so if you like I can help you distill what you’ve worked on before the meeting starts.”

If one party decides that they don’t want to address a material error, then they’re not acting in good faith. At that point we don’t use blameless procedures anymore, we use accountability procedures, and we usually exclude the recalcitrant people from the remediation process, because they’ve shown bad faith.

One strategy for correcting the institution is to start holding individuals accountable. The military does this often. They'll "make an example" of someone violating the norms and step up enforcement to steer the institutional norms back.

Sure it can feel unfair, and "everyone else is doing it" is a common refrain, but holding individuals accountable is one way to fix the institution.

One problem is that if you behave as if a person isn't the cause, you end up with all sorts of silly rules and processes, which are just in place to counter "problematic individual".

You end up using "process" as the scapegoat.

It's way too easy to pretend the system is the problem while sticking your head in the sand because you don't want to solve the actual human problem.

Sure, use the post mortem to brainstorm how to prevent/detect/excise the systematic problem ("How do we make sure no one else can make the same mistake again"), but eventually you just need to deal with the repeat offender.

On the other hand, it sounds like this workplace has weak leadership - have you considered leaving for some place better? If the manager can’t do their job enough to give you decent feedback and stop a guy giving 10 min stand ups, LEAVE.

Reasons for not leaving? Ok, then don’t be a victim. Tell yourself you’re staying despite the management and focus on the positive.

In theory maybe, but in my experience the blameless postmortem culture gets taken to such an extreme that even when one person is consistently, undeniably to blame for causing problems we have to spend years pretending it’s a system failure instead. I think engineers like the idea that you can engineer enough rules, policies, and guardrails that it’s impossible to do anything but the right thing.

This can create a feedback loop where the bad players realize they can get away with a lot because if they get caught they just blame the system for letting them do the bad thing. It can also foster an environment where it’s expected that anything that is allowed to happen is implicitly okay to do, because the blameless postmortem culture assigns blame on the faceless system rather than the individuals doing the actions.

1) the immediate action _is more important immediately_ than the systemic change. We should focus on maximizing our "fixing" and letting a toxic element continue to poison you while you waste time wondering how you got there is counterproductive. It is important to focus on the systemic change, but once you have removed the person that will destroy the organization/kill us all.

2) I forgot. Sorry

This is just a proxy for "the person is bad" then. There's no need to invoke a system. Who can possibly trace back all the things that could or couldn't have been spotted at interview stage or in probation? Who cares, when the end result is "fire the person" or, probably, "promote the person".

This is exactly the toxicity I’ve experienced with blameless postmortem culture:

Hiring is never perfect. It’s impossible to identify every problematic person at the interview stage.

Some times, it really is the person’s own fault. Doing mental gymnastics to assume the system caused the person to become toxic is just a coping mechanism to avoid acknowledging that some people really are problematic and it’s nobody’s fault but their own.

is answered by:

> any organization is likely to contain at least a few fairly toxic people

This had been assigned many times previously. When my friend disproved the lemma, he asked the professor what he had done wrong. Turns out the lemma was in fact false, despite dozens of grad students having turned in "proofs" of the lemma already. The paper itself still stood, as a weaker form of the lemma was sufficient for its findings, but still very interesting.

Much of what many learned about life came from their parents. That included lots of foundational knowledge that was either true or worked well enough.

You learned a ton in school from textbooks that you didn't personally verify.

You learned lots from media, online experts, etc. Much of which you couldn't verify.

In each case, they are making eyewitness claims that are a mix of first-hand and hearsay. Many books or journals report others' claims. So, even most education involves tons of hearsay claims.

So, how do scientists raised, educated, and informed by eyewitness claims write reports saying eyewitness testimony isn't reliable? How do scientists educated by tons of hearsay not believe eyewitness testimony is trustworthy?

Or did they personally do the scientific method on every claim, technique, machine, circuit, etc they ever considered using? And make all of it from first principles and raw materials? Did they never believe another person's claims?

Also, "scientists that have studied this over long periods of times and diverse population groups" is itself an eyewitness claim and hearsay if you want us to take your word for it. If we look up the studies, we're believing their eyewitness claims on faith while we've validated your claim that theirs exist.

It's clear most people have no idea how much they act on faith in others' word, even those scientists who claim to refute the value of it.

If IFR is low then a lot of the assumptions that justified lockdowns are invalidated (the models and assumptions were wrong anyway for other reasons, but IFR is just another). So Ioannidis was a bit of a class traitor in that regard and got hammered a lot.

The claim he's a conspiracy theorist isn't supported, it's just the usual ad hominem nonsense (not that there's anything wrong with pointing out genuine conspiracies against the public! That's usually called journalism!). Wikipedia gives four citations for this claim and none of them show him proposing a conspiracy, just arguing that when used properly data showed COVID was less serious than others were claiming. One of the citations is actually of an article written by Ioannidis himself. So Wikipedia is corrupt as per usual. Grokipedia's article is significantly less biased and more accurate.

However there are two problems with it. Firstly it's a step towards gamification and having tried that model in a fintech on reputation scoring, it was a bit of a disaster. Secondarily, very few studies are replicated in the first place unless there is a demand for linked research to replicate it before building on it.

There are also entire fields which are mostly populated by bullshit generators. And they actively avoid replication studies. Certain branches of psychology are rather interesting in that space.

If Joe dropped the production database and you're uncertain about his intentions then perhaps it would be a good idea to do the bare minimum by reducing his access privileges for the time being. No more than that though.

Whereas if you're reasonably certain that there was no intentional foul play involved then focusing on the individual from the outset isn't likely to improve the eventual outcome (rather it seems to me quite likely to be detrimental).

I'm not saying you shouldn't eventually arrive at the conclusion you're suggesting. I'm saying that it's extremely important not to start there and not to use the possibility of arriving there as an excuse to shirk asking difficult questions about the inner workings and performance of the broader organization.

> Doing mental gymnastics to assume the system caused the person to become toxic

No, don't assume. Ask if it did. "No that does not appear to be the case" can sometimes be a perfectly reasonable conclusion to arrive at but it should never be an excuse to avoid confronting uncomfortable realities.

It’s a good thing to take a look at where the process went wrong, but that’s literally just a postmortem. Going fully into blameless postmortems adds the precondition that you can’t blame people, you are obligated to transform the obvious into a problem with some process or policy.

Anyone who has hired at scale will eventually encounter an employee who seems lovely in interviews but turns out to be toxic and problematic in the job. The most toxic person I ever worked with, who culminated in dozens of peers quitting the company before he was caught red handed sabotaging company work, was actually one of the nicest and most compassionate people during interviews and when you initially met him. He, of course, was a big proponent of blameless postmortems and his toxicity thrived under blameless culture for longer than it should have.

It could also well be that Joe did the same thing at his last employer, someone in hiring happened to catch wind of it, a disorganized or understaffed process resulted in the ball somehow getting dropped, and here you are.

Your customers would prefer to have the enterprise doing stuff rather than hiring and firing.

I am unfamiliar with the reasons to which the varying murder rate is ascribed. If I had to guess, I would guess economics is #1.

True evil is different.

Negative consequences and money always work!

I don’t think the general idea of co-opting is hard to understand, it’s quite easy to understand. But there is a certain personality type, common among people who earn a living by telling Claude what to do, out there with a defect to have to “prove” people on the Internet “wrong,” and these people are constantly, blithely mobilized to further someone’s political cause who truly doesn’t give a fuck about them. Ioannidis is such a personality type, and as you can see, a victim.

https://statmodeling.stat.columbia.edu/2020/04/19/fatal-flaw...

That said, I'd put both his serosurvey and the conduct he criticized in "Most Published Research Findings Are False" in a different category from the management science paper discussed here. Those seem mostly explainable by good-faith wishful thinking and motivated reasoning to me, while that paper seems hard to explain except as a knowing fraud.

I hope this was sarcasm.

Maybe, I cannot say, but what I can say is that CS is in the midst of a huge replication crisis because LLM research cannot be replicated by definition. So I'd perhaps tone down the claims about other fields.

I think there are too many people that actually like "blaming someone else" and that causes issues besides software development.

If someone wants to be bad or shitty in a way that makes their lives own better while making the lives of everyone around them worse, that's evil and parasitic, and I'm not going to wring my hands about labeling it as such.

In hindsight, I can't see any plausible argument for an IFR actually anywhere near 1%. So how were the other researchers "not necessarily wrong"? Perhaps their results were justified by the evidence available at the time, but that still doesn't validate the conclusion.

In rhetoric, yes. (At least, except when people are given the opportunity to appear virtuous by claiming that they would sacrifice themselves for others.)

In actions and revealed preferences, not so much.

It would be rather difficult to be a functional human being if one took that principle completely seriously, to its logical conclusion.

I can't recall ever hearing any calls for compulsory public interaction, only calls to stop forbidding various forms of public interaction.

There's the other angle of selective outrage. The case for lockdowns was being promoted based on, amongst other things, the idea that PCR tests have a false positive rate of exactly zero, always, under all conditions. This belief is nonsense although I've encountered wet lab researchers who believe it - apparently this is how they are trained. In one case I argued with the researcher for a bit and discovered he didn't know what Ct threshold COVID labs were using; after I told him he went white and admitted that it was far too high, and that he hadn't known they were doing that.

Gellman's demands for an apology seem very different in this light. Ioannidis et al not only took test FP rates into account in their calculations but directly measured them to cross-check the manufacturer's claims. Nearly every other COVID paper I read simply assumed FPs don't exist at all, or used bizarre circular reasoning like "we know this test has an FP rate of zero because it detects every case perfectly when we define a case as a positive test result". I wrote about it at the time because this problem was so prevalent:

https://medium.com/mike-hearn/pseudo-epidemics-part-ii-61cb0...

I think Gellman realized after the fact that he was being over the top in his assessment because the article has been amended since with numerous "P.S." paragraphs which walk back some of his own rhetoric. He's not a bad writer but in this case I think the overwhelming peer pressure inside academia to conform to the public health narratives got to even him. If the cost of pointing out problems in your field is that every paper you write has to be considered perfect by every possible critic from that point on, it's just another way to stop people flagging problems.

Yes. One's past behavior is a strong predictor of future behavior.

> so they should be preemptively jailed even if they didn't do a crime this time?

No, it means that each successive conviction should result in a longer prison sentence.

https://sites.stat.columbia.edu/gelman/research/unpublished/...

I don't think Gelman walked anything back in his P.S. paragraphs. The only part I see that could be mistaken for that is his statement that "'not statistically significant' is not the same thing as 'no effect'", but that's trivially obvious to anyone with training in statistics. I read that as a clarification for people without that background.

We'd already discussed PCR specificity ad nauseam, at

https://news.ycombinator.com/item?id=36714034

These test accuracies mattered a lot while trying to forecast the pandemic, but in retrospect one can simply look at the excess mortality, no tests required. So it's odd to still be arguing about that after all the overrun hospitals, morgues, etc.

It's also hard to determine whether that serosurvey (or any other study) got the right answer. The IFR is typically observed to decrease over the course of a pandemic. For example, the IFR for COVID is much lower now than in 2020 even among unvaccinated patients, since they almost certainly acquired natural immunity in prior infections. So high-quality later surveys showing lower IFR don't say much about the IFR back in 2020.

But then in the P.P.P.S sections he's saying things like "I’m not saying that the claims in the above-linked paper are wrong." (then he has to repeat that twice because in fact that's exactly what it sounds like he's saying), and "When I wrote that the authors of the article owe us all an apology, I didn’t mean they owed us an apology for doing the study" but given he wrote extensively about how he would not have published the study, I think he did mean that.

Also bear in mind there was a followup where Ioannidis's team went the extra mile to satisfy people like Gellman and:

They added more tests of known samples. Before, their reported specificity was 399/401; now it’s 3308/3324. If you’re willing to treat these as independent samples with a common probability, then this is good evidence that the specificity is more than 99.2%. I can do the full Bayesian analysis to be sure, but, roughly, under the assumption of independent sampling, we can now say with confidence that the true infection rate was more than 0.5%.

After taking into account the revised paper, which raised the standard from high to very high, there's not much of Gellman's critique left tbh. I would respect this kind of critique more if he had mentioned the garbage-tier quality of the rest of the literature. Ioannidis' standards were still much higher than everyone else's at that time.

Epidemiology tends to conflate IFR and CFR, that's one of the issues Ioannidis was highlighting in his work. IFR estimates do decline over time but they decline even in the absence of natural immunity buildup, because doctors start becoming aware of more mild cases where the patient recovered without being detected. That leads to a higher number of infections with the same number of fatalities, hence lower IFR computed even retroactively, but there's no biological change happening. It's just a case of data collection limits.

That problem is what motivated the serosurvey. A theoretically perfect serosurvey doesn't have such issues. So, one would expect it to calculate a lower IFR and be a valuable type of study to do well. Part of the background of that work and why it was controversial is large parts of the public health community didn't actually want to know the true IFR because they knew it would be much lower than their initial back-of-the-envelope calculations based on e.g. news reports from China. Surveys like that should have been commissioned by governments at scale, with enough data to resolve any possible complaint, but weren't because public health bodies are just not incentivized that way. Ioannidis didn't play ball and the pro lockdown camp gave him a public beating. I think he was much closer to reality than they were, though. The whole saga spoke to the very warped incentives that come into play the moment you put the word "public" in front of something.

> The point is, if you’re gonna go to all this trouble collecting your data, be a bit more careful in the analysis!

I read that as a complaint about the analysis, not a claim that the study shouldn't have been conducted (and analyzed correctly).

Gelman's blog has exposed bad statistical research from many authors, including the management scientists under discussion here. I don't see any evidence that they applied a harsher standard to Ioannidis.

The current effective IFR (very often post-vaccination or post-exposure, and of with weaker strains) is much lower. But a 1% IFR estimate in early 2020 was entirely justified and fairly accurate.

For what it's worth, epidemiologists are well aware of the distinction between IFR, CFR, and CMR (crude mortality rate = deaths/total population), and it is well known that CFR and CMR bracket IFR.