Understanding the rationale behind a rule when trying to circumvent it

Am reminded reading this of an esteemed and since passed away colleague who had written windows driver code since the dos days and may have had decades of insanely archaic knowledge die with him - when working on a difficult piece of windows driver code years ago, he said to me in a thick eastern europe accent as best i can remember “you make the primary mistake of thinking anything in windows makes sense. once you abandon this bias, you may someday hope to get where i am”

I feel like a lot of documentation falls into this kind of style, where the motivation behind design is not communicated, just the design itself. Sometimes it's because the documentation writer doesn't want to leak internal details to the end user (closed source libraries are an especially bad source of this). Sometimes it's that the writer is too close to the project, and is struck by the curse of knowledge (can't properly identify which details are actually self-evident, and which are only obvious because they already know them).

Another example of why technical writing is difficult, I think.

This seems adjacent to Chesterton's Fence, though maybe not the canonical form of it.

For anyone not familiar with the term, Chesterton's Fence is the idea that you should understand why a rule exists before trying to remove it or work around it: https://fs.blog/chestertons-fence/

Here the issue is not that the rule was removed, but that the code followed the wording while missing the reason the rule existed.

There is this famous experiment with 9 monkeys in a room with a banana attached to the ceiling and a scale.

First day, a monkey climbs the scale, gets the banana and is happy.

Second day, they start spraying whomever gets on the scale. Monkeys hate this. They learn not to climb.

Third day, they take a monkey out and replace with another. The new monkey sees a banana up there and tries climbing the scale. He literally gets beaten out by the others, like "seems like you're new here".

Days 4-12, they've replaced one monkey per day, so that no monkey was here when it was possible to get the banana. None of them have ever been sprayed either. Still, they enforce the rule not to climb up there.

I am putting this example because in our society as well, there are many rules that are enforced without anyone questioning the "why". Yet the "why" is often more important to know than the rule itself.

Designers know this dichotomy between the "why" and the "how". Most people don't.

I really wish Windows had a way to show what kernel callbacks are registered and execute on each call. Even better would be to have actual timing data on these callback. Using that one could much more easily to debug and potentially uninstall misbehaving software.

How many of these winapi callbacks should have been just messages?

Microsoft is so broken that an employee finds it easier to write a blog post about a documentation improvement than simply making that improvement? Explains a lot. "Conway's Flaw?"

of course they had to add dont wait for your worker thread in 2020

How many of these winapi callbacks should have been just messages?

of course they had to add dont wait for your worker thread in 2020

This seems adjacent to Chesterton's Fence, though maybe not the canonical form of it.

For anyone not familiar with the term, Chesterton's Fence is the idea that you should understand why a rule exists before trying to remove it or work around it: https://fs.blog/chestertons-fence/

Here the issue is not that the rule was removed, but that the code followed the wording while missing the reason the rule existed.

Agreed that on first blush it looks like Chesterton's Fence but it may be more of a cargo culting situation.

Came here to say this. How can you write an entire article about Chesterton’s Fence without mentioning Chesterton’s Fence?!

Another example of why technical writing is difficult, I think.

I worked on a team where the responsibility of maintaining the onboarding guide was always passed to whoever joined right after they finished going through it (with the current maintainer being the "onboarding buddy" of the new team member and passing the torch at the end). The rationale was that the longest-tenured members of the team would be the most likely to accidentally rely on implicit knowledge when updating it and forget to put in something that wouldn't be obvious to a newcomer, and the person who had needed to learn the processes from scratch most recently would be the most likely to notice gaps like that (and could reach out to the rest of the team to have the gaps explained to them so they could fill them in).

I always liked that mindset, and it helped teach me the important lesson that sometimes being the person who asks very basic questions because I don't understand things can make me valuable to the teams I work on. There are certainly times when the answer makes me go "oh, duh!", and I feel a little sheepish, but the feeling doesn't last very long, and no one has ever held it against me, whereas the times when it leads somewhere interesting and potentially to better documentation for people coming after me are far more frequent and memorable to everyone involved.

This isn't to say that the burden should fall on the ones who are reading the documentation rather than writing it, but to encourage people who are in the position of being frustrated and confused to take advantage of those moments, because they can make you very valuable to your team in addition to helping you learn. If you're on a team that operates in good faith, the burden for documenting things well can be shared, and in the long run it will matter less who's job it is to keep it updated and more about whether everyone is contributing however they can to maintain the quality (and if you're on a team that operates in bad faith, you have my permission to keep quiet and do whatever you can to get through that experience, not that you need it from me!)

There is this famous experiment with 9 monkeys in a room with a banana attached to the ceiling and a scale.

First day, a monkey climbs the scale, gets the banana and is happy.

Second day, they start spraying whomever gets on the scale. Monkeys hate this. They learn not to climb.

Designers know this dichotomy between the "why" and the "how". Most people don't.

Microsoft is so broken that an employee finds it easier to write a blog post about a documentation improvement than simply making that improvement? Explains a lot. "Conway's Flaw?"

I've heard that story. I've yet to see evidence it actually happened though. I don't the experiment with pass a modern ethics panel either.

It's funny how this monkey experiment is often trotted out as "people blindly follow the rules without knowing why," when the rule learned by the monkeys is a really good rule (it prevents a malevolent entity beyond your understanding from attacking you!), and the only reason the monkeys don't understand the "why" at the end of the experiment is because they don't have language.

The lessons I'd take away from the experiment would be 1) be sure to tell people why the rules exist, but also 2) follow the rules even if there's no apparent reason for them, otherwise you might get smacked down by some unimaginably powerful entity you're barely even aware of.

Raymond's important and high up enough that he probably only needs the approval of three AIs and five managers to publish a blog - updating documentation likely needs twice that.

the blog says the documentation improvement was made in 2020? presumably not by the post author either

He used his blog to queue the work to be executed asynchronously by another MS worker that reads it

Always been that way

If there's anything I've learned from reading Raymond's blogs, it's that Windows internals actually make a fair bit of sense. Every weird thing I've ever seen him address turns out to be something that has had real thought and engineering put into it, and if the solution is ugly that seems to be more due to external constraints placed upon the engineers than it is the engineers doing weird stuff shooting from the hip.

We all knew this kind of guys. Bless them

Agreed that on first blush it looks like Chesterton's Fence but it may be more of a cargo culting situation.

Raymond's important and high up enough that he probably only needs the approval of three AIs and five managers to publish a blog - updating documentation likely needs twice that.

We all knew this kind of guys. Bless them

I've heard that story. I've yet to see evidence it actually happened though. I don't the experiment with pass a modern ethics panel either.

parables that didn't happen can still be useful

the blog says the documentation improvement was made in 2020? presumably not by the post author either

From the article:

> The documentation should open with something like this:

>> The callback function must perform its work quickly without blocking. If you need to do complex work or synchronize with other threads or processes, do the work asynchronously, such as by using System Worker Threads.

A change was made, but not the change that Raymond thinks would explain why the list is there anyway.

Came here to say this. How can you write an entire article about Chesterton’s Fence without mentioning Chesterton’s Fence?!

Chesterton's fence has two parts:

1. I want to remove a rule

2. Understand why that rule is in place before proceeding

This article deals with the second part, but not the first. So it is only about about half of Chesterton's fence at best.

In these examples, a rule (avoid blocking calls) is in place to guide the programmer to a performant system. Programmers apparently thought that if they found a way to avoid directly blocking calls, but managed to indirectly block, they had still obeyed the rule. And strictly by the most narrow reading of the rule they had obeyed it. But they had defeated the purpose of the rule.

So definitely Chesterton's Fence adjacent, but not Chesterton's Fence itself.

It's not about Chesterton's Fence, though I thought so at first due to the title wording. The developers Raymond is referring to actually weren't trying to circumvent a rule. They followed the rules to the letter, but whoever made the rule didn't adequately document the spirit of the rule. So when the letter of the rule turned out to be insufficient, the devs couldn't really know that their solution violated the intent behind the rule.

He used his blog to queue the work to be executed asynchronously by another MS worker that reads it

And, since he isn't waiting on the reply, he is following the rule for the callback.

Always been that way

Chesterton's fence has two parts:

1. I want to remove a rule

2. Understand why that rule is in place before proceeding

This article deals with the second part, but not the first. So it is only about about half of Chesterton's fence at best.

So definitely Chesterton's Fence adjacent, but not Chesterton's Fence itself.

And, since he isn't waiting on the reply, he is following the rule for the callback.

parables that didn't happen can still be useful

The parable-replication crisis is real though

They can be, but you need to take care as sometimes they are misleading and can even drive wrong decisions.

From the article:

> The documentation should open with something like this:

A change was made, but not the change that Raymond thinks would explain why the list is there anyway.

So we can call this one Chen's Fence

The parable-replication crisis is real though

So we can call this one Chen's Fence

They can be, but you need to take care as sometimes they are misleading and can even drive wrong decisions.

Aesop wrote a lot of fables that had contradictory lessons.

I have more than once in life realized that one of the two contradictory lessons was correct for this situation, but in a different one the other was correct...

There's a meta-lesson there.

I have more than once in life realized that one of the two contradictory lessons was correct for this situation, but in a different one the other was correct...

In the documentation for best practices for implementing process and thread-related callback functions, it calls out

Keep routines short and simple.

Don’t make calls into a user mode service to validate the process, thread, or image.

Don’t make registry calls.

Don’t make blocking and/or Interprocess Communication (IPC) function calls.

Don’t synchronize with other threads because it can lead to reentrancy deadlocks.

So far so good. It seems that these callback functions need to operate quickly and cannot block. These are callbacks that are invoked when a process starts or exits, when a thread starts or exits, when a DLL or EXE is loaded or unloaded, and various other low-level events.

The various prohibitions above suggest that these callouts are called during the process creation/termination sequence, so if you take a long time to deal with them, you are slowing down the entire system. And the rather extreme requirements, like “Don’t make registry calls,” suggest that they might even be called while the system holds internal locks.

The list of best practices continues:

Use System Worker Threads to queue work especially work involving:

Slow APIs or APIs that call into other process.

Any blocking behavior that could interrupt threads in core services.

Okay, so this is providing a suggestion on how you can offload expensive work to code running outside the callback. This once again highlights that the callback itself needs to be fast with minimal blocking.

My colleagues in enterprise support often run into cases where the reason for a system hang is a driver violating the rule that these callbacks must return quickly. For example, a common anti-pattern is a driver whose callback starts by following the guidance above to queue work to a System Worker Thread, but then they block until the work item completes.

This is a case of following the rules without understanding why the rules are there.

The rule is that the callback needs to be fast and return quickly. The driver followed the letter of the law by delegating the work to a System Worker Thread, and there’s no rule that says “Don’t wait for work items”, so they must have figured that this gave them a loophole for executing synchronous long-running work.

But the rules “Don’t make blocking and/or Interprocess Communication (IPC) function calls” and “Don’t synchronize with other threads because it can lead to reentrancy deadlocks” make it clear that you shouldn’t be blocking in your callback for extended periods of time. The “Don’t”s are just calling out some common ways that your callback can block.

And it looks like the documentation was updated in 2020 to call out this specific case:

If you use System Worker Threads, don’t wait on the work to complete. Doing so defeats the purpose of queuing the work to be completed asynchronously.

One could argue that this rule is already covered by the “Don’t synchronize with other threads” rule, but I guess the driver vendor interpreted it as “But I’m not synchronizing with another thread. I’m synchronizing on an event!” But of course, the event is set by another thread, so you are effectively synchronizing with another thread.

My colleague in enterprise support describes this as the “It wasn’t me, it was my brother” excuse. You are told by your parents not to turn on the television set, so you tell your brother to do it. Technically, you didn’t turn the television set on, but in effect, you did because your brother is acting under your instructions. (This is why contracts often contain wording like “may not disclose or cause to be disclosed,” so that you can’t say “No, I totally didn’t disclose it. I gave the information to Bob, and it was Bob who disclosed it!”)

The documentation should open with something like this:

The callback function must perform its work quickly without blocking. If you need to do complex work or synchronize with other threads or processes, do the work asynchronously, such as by using System Worker Threads.

And then it can give a list of examples of things that count as blocking.

Some examples of blocking that is not allowed from the callback function:

And then it can follow up with additional constraints.

Furthermore, the callback function may not perform any of the following operations:

Author

Raymond Chen

Raymond has been involved in the evolution of Windows for more than 30 years. In 2003, he began a Web site known as The Old New Thing which has grown in popularity far beyond his wildest imagination, a development which still gives him the heebie-jeebies. The Web site spawned a book, coincidentally also titled The Old New Thing (Addison Wesley 2007). He occasionally appears on the Windows Dev Docs Twitter account to tell stories which convey no useful information.

Hacker Times

Hacker Times

Understanding the rationale behind a rule when trying to circumvent it

Discussion

Discussion

Author