LLM is going to change schools and universities a lot, teachers, tutors will have to find themselves in the new reality, as they have a strong competitor with infinite resources and huge knowledge, patient and ready to work with every student in a distinct way, according to student's needs, level, intelligence, etc.
Instruction-based tutoring is dead from that perspective, why should I follow someone reciting a book or online tutorial, while there is a tool that can introduce me into subject in a better and more interesting way?
Sure, there are great teachers, who are inspiring people, who are able to present the topic in a great way, the point is, they are minority. Now, everyone can have a great tutor for a few dollars a month (or for free, if you don't need generating too much data quickly).
it's best when you ask a well known problem/thing they can reference (vs. a niche way to solve exactly what you want to solve)
then you work backwards, I e. why is it like this, what is this for, what are the alternative ways to accomplish this etc...
it's a big query engine after all.
don't try to ask like "what is the exact right way" or etc. because it will try to generate that and likely hallucinate if there is no such answer in its training corpus.
instead ask what the model does know, or doesn't.
I'm kind of surprised they still do leetcode-style questions on remote interviews these days. I thought those types of interviews would be 100% gamed by now.
It is also odd that this article appears here after someone complained about vibe coding killing the interest in algorithms.
This game is played often. People have valid complaints, then someone posts a "rebuttal" ("LLMs are not bad for $X---they are good for $X").
Anyway, he uses LLMs more in the search capability, which is less controversial than generative AI and vibe coding.
it's been so uncool and harrowing for a while now, to deal with their leetcode BS. i mean obviously this guy is well meaning but didn't learn anything, other than for the paycheck and whatever desperate circumstances require that.
the LLM stuff being used to solve their interview process is an inflection point where it really steeply goes down to want to work for Google for any reason other than money
maybe this is why Google deepmind researchers keep leaving to start their own successful companies
> I generated a list of the most common interview tasks
How? I suppose they mean gathered, or searched for, not strictly generated?
Also a little light on details of the actual interview.
I'm also a little confused about the listing of "problems" - do they refer to some specific leet-code site's listing of problems?
It seems like half-way between naming an actual algorithm/problem and naming a concrete exercise.
As for:
> How is it that we do not use this "forgotten and forbidden" coding in our daily production code, even though all highly reusable, useful code is essentially an exploitation of the intersection between classical algorithmic thinking and real-world problems?
I'm not sure what to say - most of this stuff lives in library code and data structure implementations for any language in common use?
Indeed the one saving grace of leet code interview is arguably that it shows if the candidate can choose sane data structures (and algorithms) when implementing real-world code?
To answer your questions:
1. By "generated" I mean that I prompted the LLM incrementally to provide me the list of the next LeetCode problems to do (without the deep research/search function)
2. Yes, the problem names are the exact names from LeetCode. Initially, the LLM suggested this format, and I later forced it to stick to real LeetCode problems.
This allowed me to verify some output independently of the LLM (avoiding hallucinations), cross-check solutions with other materials, and track my progress.
Interestingly, I realized later that the LLM was mostly pulling from the standard Blind 75 problem set, and almost all the problems are from that list.
3. About the "forgotten and forbidden" code: I probably phrased it poorly in the article. As you said, this algorithmic logic is abstracted away in standard libraries and data structures. The disconnect for me (and I suspect for many "business logic" developers too) is that our daily production code rarely requires writing these fundamental structures from scratch, so we do not see the patterns that can also be applied in more high-level business logic. But this is still an in-progress hypothesis in my mind, without detailed examples.
>I’ve assigned this problem [binary search] in courses at Bell Labs and IBM. Professional programmers had a couple of hours to convert the above description into a program in the language of their choice; a high-level pseudocode was fine. At the end of the specified time, almost all the programmers reported that they had correct code for the task. We would then take thirty minutes to examine their code, which the programmers did with test cases. In several classes and with over a hundred programmers, the results varied little: ninety percent of the programmers found bugs in their programs (and I wasn’t always convinced of the correctness of the code in which no bugs were found).
>I was amazed: given ample time, only about ten percent of professional programmers were able to get this small program right. But they aren’t the only ones to find this task difficult: in the history in Section 6.2.1 of his Sorting and Searching, Knuth points out that while the first binary search was published in 1946, the first published binary search without bugs did not appear until 1962.
The invariants are "tricky", not necessarily hard but also not trivial to where you can convert your intuitive understanding back into code "with your eyes closed". Especially since most implementations you write will only be "subtly flawed" rather than outright broken. Randomizing an array is also one of the algorithms in this class, conceptually easy but most implementations will be "almost right", not actually generating all permutations.
In my experience, it’s the relatively basic questions that have the highest value — both because they’re what you run into programming most often, and because they’re less likely to overwhelm candidates in a high-stress setting.
The goal, at least from my point of view, isn’t to see if they can come up with the perfect algorithm, but about how they construct an algorithm, how they communicate about the decisions they’re making, how they respond to challenges about edge-cases, etc.
I’m also strongly in favour of picking out questions that are reflective of the actual codebase they’re being hired for — find something with some basic algorithmic complexity which has a relatively simple and easy to explain input and output, and use that as the problem.
In general, I think the best problems are those which any competent senior engineer could design a good solution for almost off the top of their head with little difficulty.
The rest is somehow orthogonal to the LLM and is just pure practice. It is very easy to procrastinate with an LLM without actual practice.
It allowed me to actually see the problem space and something like the "beauty of classical algorithms". It shifted my "unknown unknowns" into "known unknowns". I had failed so many times to achieve exactly that without an LLM in the past.
Then I had a technical interview when I was asked to implement a simple algo for the tris game (aka tic tac toe) and my mind was completely blurry.
I was tired, i'm in eu and this was for a San Francisco startup interviewing me at their lunch time, very late in Italy.
And generally don't like to be interviewed/tasked.
Of course the solution is beyond simple, but I struggled even at brute forcing it.
I can easily do these kind of exercises (and much harder ones obviously) for fun, but not when interviewed.
I struggled with the same thing in University. I graduated with 104/110 even though I was consistently among the most prepared, and I learned to learn, not to pass exams (plenty of stellar performers didn't remember anything few weeks after exams).
Once I asked a professor why did he grade me 27/30 even though I spent one hour answering with details on everything, including the hardest questions.
"Because you never appear convinced when you answer".
I get nervous, I don't like to prove my knowledge this way. I rethink constantly what I'm saying, or even how I sound.
I forget how to type braces or back ticks.
I did not have any issues when not interviewed, or in written exams, or during my research period when I published 3 papers that have been highly cited.
But I am just not a fan of these types of interviews they tell absolutely nothing about the candidate.
You interview me and you'll have the very wrong impression if you ask me to live code or white board.
Meanwhile I've seen leetcode black belts spend most of their time logged on Tekken 7 on discord, consistently creating work and providing negative value while somehow always selling their high skills.
I have found much more value in seeing personal projects, and OSS contributions.
Never asked a single of these bs questions and never failed at hiring anyone. Not once.
The best procrastination device ever built because it validates you the entire time. Great post, even beyond the algorithms example.
US layout and compose key on AltGr. You'll never look back.
Unfortunately this is wrong and I have seen tons of data at 5 companies showing this. These kinds of interviews really do correlate well with job performance
There is noise, but large companies in particular need a scalable process and this one works pretty well
Startups shouldn't do this though, but the reason is the opposite of what you're complaining about. It's too easy to accidentally waste your time on somebody who is good at leetcode
Knowing when and what to optimize is vital.
The most important thing about a candidate is whether he's gonna be professional and hard working.
Technical interviews tell me nothing about it.
Of course I can see large companies that have a high number of candidates relative to positions needing such methods, they can afford false negatives.
But outside those cases personal projects and OSS contributions say much more.
Maybe this is a Europe vs US thing?
Half the people I hired, I did so without any technical interview, I met them in coding communities and saw everyday their OSS contributions and skills.
In any case I'm not debating your method is wrong, I'm just saying there's individuals like me that don't do well in these kind of situations/exams and can easily be false negatives.
I'm also saying that this isn't rocket science, and in general trying to understand if the person is honest and hard working is much more important than his coding skills.
I've seen way too many stellar graduates leetcode ninjas being just terrible at their job or completely uninterested.
And in Europe it's hard to fire them.
About 2 months ago, an email from xwf.google.com dropped into my inbox, referencing an application from a year prior that I even forgot about. My initial classification was that it is not possible and that this is just spam. But after the screening call, the reality hit: I will have two online interviews (one technical, one behavioral) in just a week. And not just a regular interview to another company, these will be interviews for a company that I still consider as one of the top-of-the-world factory of engineers.
This was a critical state. I’ve worked as a software developer in telecommunications for a few years, focusing on high-level abstraction: routing, message processing, and writing business logic. In my hobbyist gamedev projects, even though sometimes I liked to make some pathfinding algorithm or to do a CPU 3D rasterizer by hand, at the end of the day my metric for success was simple: if it runs at >60 FPS without drops, it ships.
My data structures were pragmatic: flat vectors, statically sized arrays, sometimes simple maps, and for hard problems "SQLite". And all of my attempt at writing fancy algorithms or data structure was a cool learning journey, that unfortunately did not have a direct impact on my job skills, and there were always better ways to spend this time if I wanted to optimize for a telecommunications career as well as for a gamedev career.
Classical algorithms and strict LeetCode-style data structures were always outside my area of interest. My earliest attempts at learning them had failed completely. Even in my earliest memories on all primary school programming contests, I always failed at them.
I had one week, a day job, other regular obligations and a fundamental knowledge gap.
After some initial thinking (I mean procrastination on YouTube and online forums for at least a few hours for few days), I realized that the most successful strategy would be to learn as many patterns as possible - to at least be able to identify the core problems. I was aware that I had a very small chance of resolving it in the most optimal way during the interview, so I wanted to at least understand what was expected from me to do (for my brain that is trained on writing software and resolving useful problems, mathematical description is not clear in a lot of places, and for me, it hides the clue, especially behind artificially created, non-real problems).
The main problem at this point: I was unable to solve even the simplest LeetCode problem. But this time, I knew the goal was not just to solve the problem (like during a game jam), but to learn as many patterns and concepts as possible even blindly without fully understanding all quirks for now.
I started actual learning on friday. Based on previous thought I decided to give a Large Language Model a chance to be my algorithmic teacher, since I never had a human one before. I fed Gemini Pro chat with my exact situation (with my CV and preparation material provided by google in mail that i received) and set the following parameters for our session:
These attempts resulted in successfully completing this set of problems suggested by LLM: Best Time to Buy and Sell Stock, Contains Duplicate, Valid Anagram, Group Anagrams, Product of Array Except Self.
With hints, it occurred to me that these problems were simple usages of for-loops and arrays - but this gave me something that I never had before, courage that I am able to do even the simplest algorithm tasks. And I think that these are a very solid starting point because they bridged the gap between algorithms and problems that I had to do in "real" software, without being overwhelming. Moreover they showed me, not fully clearly for now but as a small hint, that sometimes a data structure like "unordered_set" in these examples can be used in a not direct way to solve a problem in a clever way by using its not obvious properties. This also provided me some insights that a simple indexed lookup table like in Valid Anagram maybe from this perspective is equal in some property to an unordered_set in such a way that I never thought before.
Also, I think that equally important is the fact that I started to show my full code solution to LLM, and ask for a better approach, and the pros and cons of each possible implementation.
This was Saturday. Assuming this might be my last deep-learning session, I changed the protocol to maximize throughput.
I started a new chat where I instructed the LLM:
The hard rule for me was to never blindly rewrite AI code: I had to write the code myself, using my own variable taxonomy and logic flow. I believe that forcing the solution into "my style" mapped the concepts deeper into my muscle memory, and (possibly I use this terms wrong) I believe that by doing this, I refactored my idiolect without forcing it to drop old behaviours. Later, I learned that forcing "my flow" actually made me write code patterns that were not always valid, whereas keeping a strict order (like in BFS) is very useful and frees up a lot of thinking space for thinking about the problem (like checking "business" conditions before adding to the queue). But now, looking at this from a wider perspective, I believe that this was a necessary part of the process.
I gave myself a maximum of 30 minutes for each problem: Reverse Linked List, Linked List Cycle, Valid Palindrome, Invert Binary Tree, Longest Substring Without Repeating Characters, Valid Parentheses, Maximum Depth of Binary Tree, Merge Two Sorted Lists, Same Tree, Number of Islands, Find if Path Exists in Graph, Flood Fill, Reverse Integer, Single Number, Climbing Stairs
In doing these problems, I learned that:
Somewhere in the midpoint of this list, I noticed that the LLM's context started to degrade after around five problems. So I separated one context for generating new problems, and a lot of smaller ones just to do 2-3 problems from the same domain (like graphs, or trees). Also, this was the hardest day, and to be honest, I rage-quit after Climbing Stairs. But during this day I spent something like 9 hours for learning, interrupted a lot by my other obligations.
Time started running out, so I assumed this was a hard stop for learning entirely new concepts. Because the chance of getting an "Easy" LeetCode problem during interview was small, I assumed I needed to feel comfortable with "Medium" problems.
I generated a list of the most common interview tasks, assuming there is a high probability of some variation of them, and I also provided all my previously solved LeetCode problems to avoid duplication. And also, I decided that due to the fact that I will not have a chance to click compile during the interview, to offload checking common errors to the compiler, I need to write the solution and send it to the LLM without compiling it and testing it in LeetCode.
The first three problems weren't so hard: Merge k Sorted Lists, Merge Intervals, Design Add and Search Words Data Structure. They required fewer hints than any easy ones.
I noticed that "Easy" problems are often the hardest because they introduce entirely new concept or pattern. The "Medium" problems I encountered were just trickier versions of the easy ones. The most important difference was that In most cases, I was able to write the code for medium problems without needing specific hints about the underlying concepts.
Course Schedule was a little bit tricky, but after understanding the need for "inverting directions" and keeping track of "visiting" and "visited" states, writing it wasn't so hard (it was hard, but not so much). And also my eyes opened widely because right now it seems to me like a very common pattern that I could use in a lot of situations. Also, I think that a lot of work that I need to track manually can be compacted to some variation of this problem, like dependency tracking.
Combination Sum was my first interaction with backtracking during this speedrun. At this point, it wasn't so obvious to me, but I think that nevertheless, the correct order of previous problems like Course Schedule before that one, helped me to make a crack in my ignorance, because at least I could see that this weird backtracking stuff was useful in Course Schedule and maybe this is the same pattern but written differently - I think that was an important breakthrough for me. Seeing a standard way to handle some combinatorial explosion was eye-opening, instead of inventing the wheel for each problem like I used to do.
Container With Most Water evoked strong emotion in me, because it seemed very similar to one from day one, but at this point, I was unable to write it myself... later it occurred to me that this is a similar but different, also powerful concept to attack the problem by going towards center from the corners (a variant of two pointers). It opens my eyes to various ways of iteration different from just forward and backward, I connected it with the inversion of iteration in anagram problem.
Sum of Two Integers was a very pleasant break from strong algorithmic problems. I wrote it as an iteration through binary operations that acts as a full binary adder.
I wanted to do at least one more dynamic programming problem LLM suggested Unique Paths. I assumed that I need to be aware of dynamic programming, but due to its incompatibility with my brain functioning at this point. I made the decision that I need to put everything on other cards, and I had hopes that even if there will be a dynamic problem during the interview, it will be an easy, very generic one.
In an algorithmic sense, this subset of problems seems to not be that hard, I was expecting exponential difficulty growth, or at least linear, and it seems that this was only multiplying difficulty by a constant. But my challenge of not compiling code before sending it to the LLM was very depressing. It occurred to me that I depend a lot on the compiler.
In previous problems, the lack of an LSP was a problem, but this one was a way harder. At the end of the day, my biggest worry was that I would submit code during interview which do not work properly. Also, edge cases like off-by-one errors or mistaking ">=" with ">" in iteration started to be a real concern for me, but this was fully dependent on my fresh (not fully developed) muscle memory. I accepted that I am unable to resolve this in such a limited time, and I need to overcome it with narration and dialogue with the recruiter.
It was weird that I started to make such mistakes, but it occurs to me that I used to write loops in a very repetitive way, and these tasks forced me to think about iteration more fully with all possible ways instead of the few "robust" ones that I used to do "in production".
I didn't have much time, so I allocated it to just practicing patterns similar to the previous ones. This is the list the LLM generated for me: Two Sum, Longest Consecutive Sequence, Clone Graph, Palindromic Substrings. I also watched a few videos about the STL and common interview coding patterns. I prepared for the behavioral interview and read some STL documentation. I still don't know if this was the right decision at that point, or if it was just my way of procrastinating.
In the meantime, I tried to describe the "divide and conquer" concept to my friends, who had a school assignment that required this concept. After that, I decided to do one more binary search problem. The LLM chose Find Minimum in Rotated Sorted Array (I did it in two suboptimal binary searches, instead of a single one—which seemed to be pure magic to me at this point).
The day before the interview, I decided not to look at any algorithms, and to just play games and watch casual YouTube videos. I only looked at each of my previous solutions and tried to "actively" recreate the writing process, but without focusing on one particular problem.
When the final day came, I was really shocked. One week earlier, I thought LeetCode problems were just stupid, academic, math-like problems where you needed to know weird tricks. But now, I still somewhat thought that way... but I had completed around 34 LeetCode problems (18 of them Medium and 1 Hard).
I started noticing what they actually meant by "data structures". In this context, they are understood more as specific concepts with tricky properties used through an API to handle edge cases, rather than how they actually work under the hood. Most of the problems were just tricky ways of describing a single issue where the right approach is to find some weird constant relation between the data and to exploit it as much as possible by using the right data structure and the right iteration method.
Dynamic programming was still black magic to me (it felt like using forgotten, forbidden, and dangerous ways to simplify problems) but I started to see them as way of some form of iteration with keeping the state. I was also scared of all combinatorial problems and iterative ones, because, there were a high chance that problem from this subset will require some math trick, that I do not know yet.
I cannot share the exact assessment. Nevertheless, the technical assessment hit exactly in the intersection of my preparation problem and interests: a hybrid of graph traversal and binary search. It was the first time I faced this kind of problem, but I liked it from the start because it felt very much like a gamedev problem. During the discussion phase, I felt like I was applying an innovative approach to a real problem (like I was making core mechanics for a new innovative RTS game). In the second part, I proposed an iterative increasing approach, realizing mid-sentence I could optimize it with binary search by slightly redefining the problem constraints without breaking them. The logic clicked.
Then, the critical failure. Under the stress of the running out of time, after successfully coding the graph traversal, my working memory dumped the standard implementation for iterative binary search. I knew the recursive version, but the iterative syntax vanished from my brain.
After some talk about that with the interviewer, we agreed that the iterative approach is the only valid one at this stage. The fact that I had only the last few minutes wasn't helpful in this situation. I fell back on the verbalization technique I used with my friends. I reconstructed the binary search aloud, explaining to the interviewer exactly how I was slicing the problem space, even though the final code had mistakes (a lot of them). I spotted them too late and I did not have time to refactor them.
After the interview, I was in a state of superposition: I simultaneously thought I had fully failed and somehow succeeded the interview. I realized that my speedrun was partially successful, but most of the learned knowledge wasn't available to me under stressful conditions. This waiting phase for the interview results was very interesting: it allowed me to observe how I (and my colleagues) worked in my current telecommunications job, by keeping algorithmic concepts fresh in my mind due to the tension related to impatience for the results. I was also able to calmly structure a lot of the newly acquired knowledge and observations in my head. For those few days, it was hard for me to write any code. For some reason, my brain had other plans and needed a cooldown.
After a few days, I received a phone call. I had already accepted that I had failed the interview and needed to return to my daily routine, which I had slightly neglected during that week.
But the call brought unexpected news: the recruiter mentioned I needed to pay more attention to code debuggability (whatever this means - I assume that under the corpo-language, they mean that I wrote invalid code), but nevertheless, they wanted to invite me to two on-site technical interviews.
For me, at this point, it was both good and bad news. I was stressed that this season in my life hadn't ended yet. At the same time, I wanted to change my job. But simultaneously, I was conflicted. I like my current job, and I had duties that I would inevitably neglect to some degree during the next preparation phase. Furthermore, I had only been working at my current company for about 10 months.
The continuation will be in Part Two, but there are some open questions, without clear answers, that live rent-free in my mind:
(Also, a last interesting fact: writing this post took me around 8 hours and I wrote it months after the interview - I think that I should count it as a part of this speedrun.)