Author makes up a lie.
Then lampshades it away with a colorful non sequitur.
---
The alternatives that people praise like golang, have other tradeoffs that are much worse because the async logic is now implicit. Your entire codebase is now a surface area that is at risk of being blocked by waiting on a channel; the the mitigation of this is through responsible use of coroutines, but then you're right back around to extra information about your code that is analogous to colring, except not as explicit as async/await.
All functions, even non-async functions, are colored. In any large system codebase you'll have functions that can only be called in certain situations, with the right setup, whatever, and if you're lucky this is communicated by types but regardless those restrictions can't be avoided. It's easy to call low-restriction functions from high-restriction ones and not the other way around.
Furthermore, it's not like the alternative to explicit await doesn't have issues too (that the article doesn't mention). There is inherent complexity, it's a tradeoff, you can't just syntax it away.
That makes it a pleasure to code concurrent stuff for IMHO.
It does have its own similar problems though - does a function return an error? If so you are going to need to plumb the error return through all the callers. Does a function need a context.Context? Ditto.
I guess you can't win them all :-)
Passing in the context as an argument or making it a global variable or returning a monad doesn't do anything to uncolor the function. What's the difference between `async function f()` and `function f(eventloop, callback)`? Only syntax.
Not to mention there's lots of colors unrelated to async, that most languages don't type at all. And if you use the wrong one, your program just doesn't work correctly at runtime. Thread-safe vs thread-unsafe. Blocking vs non-blocking. May throw/panic vs won't throw/panic. May fail/return null vs infallible.
The point of goroutines is that they can freely block when needed. It's not like async where you have to be paranoid at every moment about writing blocking code
Propagating errors up the stack is not the same, because the top-level function is not developing an error return because of the 10-level-nested function. It is developing one because the function it called has one, and apparently, it needs to return it to its local caller. It's a local consideration. It is true that it may be a recursive local consideration where this was true 10 times, but the reason it is different is that it doesn't have to be that way. It could have been the case that the function 7 layers down handled the error somehow and it stopped propagating up the stack. But at each point, the consideration was local, and as such, amenable to local solutions other than just tossing the error up. If you choose to "correctly" plumb the error through all your functions, well, good on you for apparently being willing to apply good software engineering practices even when it's annoying, but this is just normal day-to-day function activity stuff.
By contrast, in a function coloring situation, if the color is wrong 10 layers down, you must change the calling function. It's a non-local consideration. You don't get to decide not to change it. You can't encapsulate it. You don't get a choice. It pollutes the entire stack, forcibly.
Another way to look at it is, if the function 10 levels down developed what you think is a color, but there is a way for the function 9 levels down to hide the color from the rest of the stack, even via a hack like simply dropping an error you really need or hackily constructing an object of some type to pass in, then it is by definition not a color. A color change can't be stopped by any way of writing an intermediate function. It must be propagated all the way up the stack.
If you don't have this, you don't have "color". Like, some people will say that in their language that maybe there is some way to encapsulate "async". If you can, then you don't have an async color. Although I will say that if your "encapsulation" is basically to run it in a non-concurrent environment, that's really not encapsulation. It isn't really "encapsulation" if you're giving up an entire major feature of the language, because that is something very visible to the rest of the program.
Go's context.Context is similarly not a color. You can always just create a context.Background() and pass that down. If you didn't have any context already in hand, which means you must not care about any of the features context offers, then that is usually a fine thing to do. Context is trivially bypassed if you don't want it. It can be encapsulated within a portion of the stack without "polluting" the rest of the stack like any other function parameter.
The key aspect of color is that it is not optional. It isn't something that you can just decide to ignore and stop passing up, or trivially create a value for passing down to other functions. You have to change the "color". Async is a color in many environments. There aren't really that many colors in programming languages because they are very, very quickly inconvenient and we tend to squeeze them out. (Haskell really sticks out here as a language that is not only capable of creating arbitrary colors, but where this is an explicit tool used by the community rather than a limitation, and they even have ways of combining colors together deliberately.) Statement versus expression distinctions are another one, where a "statement" may not be usable in an "expression", and you'll note how languages have in general erased that one over time because it's really just a cost without much benefit.
Like, why can't my sync function await something asynchronous? If it has to lock up the whole thread while that function executes, that's fine because that's how it was going to work anyway 99% of the time
* Haskell: pure function and non-pure (IO monads) looks different. * Rust: unsafe functions (or block) requires special markers.
Thanks for my next horror shortfilm plot. Twist: he's the protagonist
This comes at a cost, namely that of reading five extra characters in a function signature, and I could kind of imagine (truly!) how that gets in the way for some people. There is a cost of writing the five characters as well (and like the author mentions, in a poorly designed codebase, this may have to go down the call stack), but code is read more often than written, so in a sense this is negligible.
Like the dynamic vs static typing debate, I feel like this ultimately boils down to context and personal taste, and some amount of intelligence as well. I'm impressed by the amount of stuff the dynamic typing / non-async crowd is able to keep in their working or long term memory while coding. I don't have that kind of mental bandwidth, sadly.
Having said all that, this argument is disingenuous in that it completely ignores the fact that the async keyword tells you something useful (rather than some made up nonsense like color), and most of the argument basically boils down to "if you ignore the benefits, this syntax has no benefits", and I really don't respect that as an argument.
Related, one of the former React maintainers wrote a primer on algebraic effects that's a good read: https://overreacted.io/algebraic-effects-for-the-rest-of-us/
Type classes can smooth over some of it but it's not unusual to have to do some plumbing.
The downside of goroutines is that you have no control when the goroutine context switches, so naively accessing a global value can lead to race conditions (which the language has no warnings for despite being such a concurrent language), while the same code works fine in JavaScript because context switches don't happen in synchronous code.
If you don’t depend on anything mutable that anyone else can modify then this is mitigated, but that’s a very specific discipline you have to abide by.
Plus, you probably don't want to lock up the whole thread if you're writing anything more than a quick script, like a web server or a GUI.
In Rust, unsafe code can call safe code, and safe code can call unsafe code. Calling unsafe code in safe code requires an explicit unsafe block, but that's fairly normal and not a hack to get around function coloring.
A better example could be Rust async, though unlike JavaScript, you have the option to block the thread on an async function in a sync function.
The answer, at least for Python, is that it is an intentional limitation because the alternatives introduce some quite bad trade-offs.
Option 1: your awaited promise goes into the main async event loop. This is bad because it means that your single-threaded sync function now needs to be thread-safe, and so does any sync code that calls your sync function despite it not even knowing that you're doing anything async. This is essentially unworkable without throwing away the option of writing non-thread-safe code.
Option 2: Your awaited promise goes into its own new event loop that only contains sibling and child promises. There's nothing technically stopping someone from doing this[1], but now you've lost a ton of the value of async because you will inevitably end up with a ton of siloed event loops that leave the process idle despite other async tasks existing that could run. Effective async code needs to share an event loop at as high of a level as possible, which means tainting as many methods with async as possible. At that point, you might as well enforce it at the language level and avoid the inevitable pain and fragmentation that comes from other devs across the ecosystem mixing sync and async code.
[1] https://pypi.org/project/nest-asyncio/
As explained by Guido: https://github.com/python/cpython/issues/66435#issuecomment-...
asyncio.gather is a lot less code than having to manage a thread pool or something like Celery with all it's underlying infrastructure.
If you're in an ecosystem where a lot of the async boilerplate is free/cheap (ex: FastAPI) then the developer overhead of sprinkling awaits on your I/O bound calls is pretty low IMO.
Unpopular opinion, but combining this with the other "no thanks" sentiments in this subthread is the right answer. Your app is so complicated you need async? Then it's complicated enough that you can benefit from infrastructure. I don't want to watch coworkers try to badly rebuild message queue or scheduling semantics in an application code base. Just use infrastructure that's made by people who know what they are doing. That was problematic in 2015, but in 2026 it's a bit of docker, and it's not just about web/microservices. Very easy for sufficiently complex apps to simply leverage a local sandbox of celery, redis, graphdb's and whatever. Stand-alone is overrated since we don't have to do it anymore.. app devs should get more comfortable working with ensembles like this so they have access to best-in-class solutions.
You don't like infrastructure AND have such a need for performance AND don't want threads or multiprocess? Consider using another language. Async is mostly a solution in search of a problem, and the enduring popularity of TFA goes to show this has been the right conclusion for ~10 years.
one, two = await asyncio.gather(callOne(), callTwo())
?
Every rich client-side experience in your browser is written using async code in Javascript or Typescript, as is every electron app. Every developer at my company is comfortable with this pattern, and frameworks like FastAPI make this a similarly smooth experience when using Python.
If async was a solution in search of a problem, it wouldn't have been stolen from C# and added to Rust, Python, Kotlin, etc. The engineering effort required to bring this solution to all these languages is immense, so I'm clearly not the only person seeing value in it.
I don’t know about you, but nothing gets me going in the morning quite like a good old fashioned programming language rant. It stirs the blood to see someone skewer one of those “blub” languages the plebians use, muddling through their day with it between furtive visits to StackOverflow.
(Meanwhile, you and I, only use the most enlightened of languages. Chisel-sharp tools designed for the manicured hands of expert craftspersons such as ourselves.)
Of course, as the author of said screed, I run a risk. The language I mock could be one you like! Without realizing it, I could have let the rabble into my blog, pitchforks and torches at the ready, and my fool-hardy pamphlet could draw their ire!
To protect myself from the heat of those flames, and to avoid offending your possibly delicate sensibilities, instead, I’ll rant about a language I just made up. A strawman whose sole purpose is to be set aflame.
I know, this seems pointless right? Trust me, by the end, we’ll see whose face (or faces!) have been painted on his straw noggin.
Learning an entire new (crappy) language just for a blog post is a tall order, so let’s say it’s mostly similar to one you and I already know. We’ll say it has syntax sorta like JS. Curly braces and semicolons. if, while, etc. The lingua franca of the programming grotto.
I’m picking JS not because that’s what this post is about. It’s just that it’s the language you, statistical representation of the average reader, are most likely to be able grok. Voilà:
function thisIsAFunction() { return "It's awesome"; }
Because our strawman is a modern (shitty) language, we also have first-class functions. So you can make something like this:
// Return a list containing all of the elements in collection // that match predicate. function filter(collection, predicate) { var result = []; for (var i = 0; i < collection.length; i++) { if (predicate(collection[i])) result.push(collection[i]); } return result; }
This is one of those higher-order functions, and, like the name implies, they are classy as all get out and super useful. You’re probably used to them for mucking around with collections, but once you internalize the concept, you start using them damn near everywhere.
Maybe in your testing framework:
describe("An apple", function() { it("ain't no orange", function() { expect("Apple").not.toBe("Orange"); }); });
Or when you need to parse some data:
tokens.match(Token.LEFT_BRACKET, function(token) { // Parse a list literal... tokens.consume(Token.RIGHT_BRACKET); });
So you go to town and write all sorts of awesome reusable libraries and applications passing around functions, calling functions, returning functions. Functapalooza.
Except wait. Here’s where our language gets screwy. It has this one peculiar feature:
1. Every function has a color.
Each function—anonymous callback or regular named one—is either red or blue. Instead of a single function keyword, there are two:
blue_function doSomethingAzure() { // This is a blue function... }
red_function doSomethingCarnelian() { // This is a red function... }
There are no colorless functions in the language. Want to make a function? Gotta pick a color. Them’s the rules. And, actually, there are a couple more rules you have to follow too:
2. The way you call a function depends on its color.
Imagine a “blue call” syntax and a “red call” syntax. Something like:
doSomethingAzure()blue; doSomethingCarnelian()red;
When calling a function, you need to use the call that corresponds to its color. If you get it wrong—call a red function with blue after the parentheses or vice versa—it does something bad. Dredge up some long-forgotten nightmare from your childhood like a clown with snakes for arms hiding under your bed. That jumps out of your monitor and sucks out your vitreous humour.
Annoying rule, right? Oh, and one more:
3. You can only call a red function from within another red function.
You can call a blue function from within a red one. This is kosher:
red_function doSomethingCarnelian() { doSomethingAzure()blue; }
But you can’t go the other way. If you try to do this:
blue_function doSomethingAzure() { doSomethingCarnelian()red; }
Well, you’re gonna get a visit from old Spidermouth the Night Clown.
This makes writing higher-order functions like our filter() example trickier. We have to pick a color for it and that affects the colors of the functions we’re allowed to pass to it. The obvious solution is to make filter() red. That way, it can take either red or blue functions and call them. But then we run into the next itchy spot in the hairshirt that is this language:
4. Red functions are more painful to call.
For now, I won’t precisely define “painful”, but just imagine that the programmer has to jump through some kind of annoying hoops every time they call a red function. Maybe it’s really verbose, or maybe you can’t do it inside certain kinds of statements. Maybe you can only call them on line numbers that are prime.
What matters is that if you decide to make a function red, everyone using your API will want to spit in your coffee and/or deposit some even less savory fluids in it.
The obvious solution then is to never use red functions. Just make everything blue and you’re back to the sane world where all functions have the same color, which is equivalent to them all having no color, which is equivalent to our language not being entirely stupid.
Alas, the sadistic language designers—and we all know all programming language designers are sadists, don’t we?—jabbed one final thorn in our side:
5. Some core library functions are red.
There are some functions built in to the platform, functions that we need to use, that we are unable to write ourselves, that only come in red. At this point, a reasonable person might think the language hates us.
You might be thinking that the problem here is we’re trying to use higher-order functions. If we just stop flouncing around in all of that functional frippery and write normal blue collar first-order functions like God intended, we’d spare ourselves all the heartache.
If we only call blue functions, make our function blue. Otherwise, make it red. As long as we never make functions that accept functions, we don’t have to worry about trying to be “polymorphic over function color” (“polychromatic”?) or any nonsense like that.
But, alas, higher order functions are just one example. This problem is pervasive any time we want to break our program down into separate functions that get reused.
For example, let’s say we have a nice little blob of code that, I don’t know, implements Dijkstra’s algorithm over a graph representing how much your social network are crushing on each other. (I spent way too long trying to decide what such a result would even represent. Transitive undesirability?)
Later, you end up needing to use this same blob of code somewhere else. You do the natural thing and hoist it out into a separate function. You call it from the old place and your new code that uses it. But what color should it be? Obviously, you’ll make it blue if you can, but what if it uses one of those nasty red-only core library functions?
What if the new place you want to call it is blue? You’ll have to turn it red. Then you’ll have to turn the function that calls it red. Ugh. No matter what, you’ll have to think about color constantly. It will be the sand in your swimsuit on the beach vacation of development.
Of course, I’m not really talking about color here, am I? It’s an allegory, a literary trick. The Sneetches isn’t about stars on bellies, it’s about race. By now, you may have an inkling of what color actually represents. If not, here’s the big reveal:
Red functions are asynchronous ones.
If you’re programming in JavaScript on Node.js, everytime you define a function that “returns” a value by invoking a callback, you just made a red function. Look back at that list of rules and see how my metaphor stacks up:
Synchronous functions return values, async ones do not and instead invoke callbacks.
Synchronous functions give their result as a return value, async functions give it by invoking a callback you pass to it.
You can’t call an async function from a synchronous one because you won’t be able to determine the result until the async one completes later.
Async functions don’t compose in expressions because of the callbacks, have different error-handling, and can’t be used with try/catch or inside a lot of other control flow statements.
Node’s whole shtick is that the core libs are all asynchronous. (Though they did dial that back and start adding ___Sync() versions of a lot of things.)
When people talk about “callback hell” they’re talking about how annoying it is to have red functions in their language. When they create 4,089 libraries for doing asynchronous programming, they’re trying to cope at the library level with a problem that the language foisted onto them.
Update 2021/12/03: 15,118 async libraries as of today.
People in the Node community have realized that callbacks are a pain for a long time, and have looked around for solutions. One technique that gets a bunch of people excited is promises, which you may also know by their rapper name “futures”.
These are sort of a jacked up wrapper around a callback and an error handler. If you think of passing a callback and errorback to a function as a concept, a promise is basically a reification of that idea. It’s a first-class object that represents an asynchronous operation.
I just jammed a bunch of fancy PL language in that paragraph so it probably sounds like a sweet deal, but it’s basically snake oil. Promises do make async code a little easier to write. They compose a bit better, so rule #4 isn’t quite so onerous.
But, honestly, it’s like the difference between being punched in the gut versus being punched in the privates. Technically less painful, yes, but I don’t think anyone should really get thrilled about the value proposition.
You still can’t use them with exception handling or other control flow statements. You still can’t call a function that returns a future from synchronous code. (Well, you can, but if you do, the person who later maintains your code will invent a time machine, travel back in time to the moment that you did this and stab you in the face with a #2 pencil.)
You’ve still divided your entire world into asynchronous and synchronous halves and all of the misery that entails. So, even if your language features promises or futures, its face looks an awful lot like the one on my strawman.
(Yes, that means even Dart, the language I work on. That’s why I’m so excited some of the team are experimenting with other concurrency models.)
C# programmers are probably feeling pretty smug right now (a condition they’ve increasingly fallen prey to as Hejlsberg and company have piled sweet feature after sweet feature into the language). In C#, you can use the await keyword to invoke an asynchronous function.
This lets you make asynchronous calls just as easily as you can synchronous ones, with the tiny addition of a cute little keyword. You can nest await calls in expressions, use them in exception handling code, stuff them inside control flow. Go nuts. Make it rain await calls like a they’re dollars in the advance you got for your new rap album.
Async-await is nice, which is why we’re adding it to Dart. It makes it a lot easier to write asynchronous code. You know a “but” is coming. It is. But… you still have divided the world in two. Those async functions are easier to write, but they’re still async functions.
You’ve still got two colors. Async-await solves annoying rule #4: they make red functions not much worse to call than blue ones. But all of the other rules are still there:
Synchronous functions return values, async ones return Task<T> (or Future<T> in Dart) wrappers around the value.
Sync functions are just called, async ones need an await.
If you call an async function you’ve got this wrapper object when you actually want the T. You can’t unwrap it unless you make your function async and await it. (But see below.)
Aside from a liberal garnish of await, we did at least fix this.
C#’s core library is actually older than async so I guess they never had this problem.
It is better. I will take async-await over bare callbacks or futures any day of the week. But we’re lying to ourselves if we think all of our troubles are gone. As soon as you start trying to write higher-order functions, or reuse code, you’re right back to realizing color is still there, bleeding all over your codebase.
So JS, Dart, C#, and Python have this problem. CoffeeScript and most other languages that compile to JS do too (which is why Dart inherited it). I think even ClojureScript has this issue even though they’ve tried really hard to push against it with their core.async stuff.
Wanna know one that doesn’t? Java. I know right? How often do you get to say, “Yeah, Java is the one that really does this right.”? But there you go. In their defense, they are actively trying to correct this oversight by moving to futures and async IO. It’s like a race to the bottom.
C# also actually can avoid this problem too. They opted in to having color. Before they added async-await and all of the Task<T> stuff, you just used regular sync API calls. Three more languages that don’t have this problem: Go, Lua, and Ruby.
Any guess what they have in common?
Threads. Or, more precisely: multiple independent callstacks that can be switched between. It isn’t strictly necessary for them to be operating system threads. Goroutines in Go, coroutines in Lua, and fibers in Ruby are perfectly adequate.
(That’s why C# has that little caveat. You can avoid the pain of async in C# by using threads.)
The fundamental problem is “How do you pick up where you left off when an operation completes”? You’ve built up some big callstack and then you call some IO operation. For performance, that operation uses the operating system’s underlying asynchronous API. You cannot wait for it to complete because it won’t. You have to return all the way back to your language’s event loop and give the OS some time to spin before it will be done.
Once operation completes, you need to resume what you were doing. The usual way a language “remembers where it is” is the callstack. That tracks all of the functions that are currently being invoked and where the instruction pointer is in each one.
But to do async IO, you have to unwind and discard the entire C callstack. Kind of a Catch-22. You can do super fast IO, you just can’t do anything with the result! Every language that has async IO in its core—or in the case of JS, the browser’s event loop—copes with this in some way.
Node with its ever-marching-to-the-right callbacks stuffs all of those callframes in closures. When you do:
function makeSundae(callback) { scoopIceCream(function (iceCream) { warmUpCaramel(function (caramel) { callback(pourOnIceCream(iceCream, caramel)); }); }); }
Each of those function expressions closes over all of its surrounding context. That moves parameters like iceCream and caramel off the callstack and onto the heap. When the outer function returns and the callstack is trashed, it’s cool. That data is still floating around the heap.
The problem is you have to manually reify every damn one of these steps. There’s actually a name for this transformation: continuation-passing style. It was invented by language hackers in the 70s as an intermediate representation to use in the internals of their compilers. It’s a really bizarro way to represent code that happens to make some compiler optimizations easier to do.
No one ever for a second thought that a programmer would write actual code like that. And then Node came along and all of the sudden here we are pretending to be compiler backends. Where did we go wrong?
Note that promises and futures don’t actually buy you anything, either. If you’ve used them, you know you’re still hand-creating giant piles of function literals. You’re just passing them to .then() instead of to the asynchronous function itself.
Async-await does help. If you peel back your compiler’s skull and see what it’s doing when it hits an await call you’d see it actually doing the CPS-transform. That’s why you need to use await in C#: it’s a clue to the compiler to say, “break the function in half here”. Everything after the await gets hoisted into a new function that the compiler synthesizes on your behalf.
This is why async-await didn’t need any runtime support in the .NET framework. The compiler compiles it away to a series of chained closures that it can already handle. (Interestingly, closures themselves also don’t need runtime support. They get compiled to anonymous classes. In C#, closures really are a poor man’s objects.)
You might be wondering when I’m going to bring up generators. Does your language have a yield keyword? Then it can do something very similar.
(In fact, I believe generators and async-await are isomorphic. I’ve got a bit of code floating around in some dark corner of my hard disc that implements a generator-style game loop using only async-await.)
Where was I? Oh, right. So with callbacks, promises, async-await, and generators, you ultimately end up taking your asynchronous function and smearing it out into a bunch of closures that live over in the heap.
Your function passes the outermost one into the runtime. When the event loop or IO operation is done, it invokes that function and you pick up where you left off. But that means everything above you also has to return. You still have to unwind the whole stack.
This is where the “red functions can only be called by red functions” rule comes from. You have to closurify the entire callstack all the way back to main() or the event handler.
But if you have threads (green- or OS-level), you don’t need to do that. You can just suspend the entire thread and hop straight back to the OS or event loop without having to return from all of those functions.
Go is the language that does this most beautifully in my opinion. As soon as you do any IO operation, it just parks that goroutine and resumes any other ones that aren’t blocked on IO.
If you look at the IO operations in the standard library, they seem synchronous. In other words, they just do work and then return a result when they are done. But it’s not that they’re synchronous in the sense that it would mean in JavaScript. Other Go code can run while one of these operations is pending. It’s that Go has eliminated the distinction between synchronous and asynchronous code.
Concurrency in Go is a facet of how you choose to model your program, and not a color seared into each function in the standard library. This means all of the pain of the five rules I mentioned above is completely and totally eliminated.
So, the next time you start telling me about some new hot language and how awesome its concurrency story is because it has asynchronous APIs, now you’ll know why I start grinding my teeth. Because it means you’re right back to red functions and blue ones.