I started a touch before this in London. I recall before Blaze and git5 - every morning we had a ritual of checking out google3 and making sure we could get some sort of build working for the day so that we could then attempt to write some software on top of it. The builds in play were “Mach” and “quickie” or something like that. It was so painful we used to agree that we wouldn’t grab food or coffee or anything until we’d worked out what CL we should sync to for the day to do some work on.

Pair programming was very in vogue and I used to get in a little later than some which was a great excuse to just hop on someone else’s machine who’d already gone through that pain

There are mercurial and jujutsu frontends now.

I still have nightmares about eclipse sometimes.

As an employee, I'm using Antigravity (CLI version) every day (because we can't use Claude) and it rules. I am way more productive than I was with CIDER-V, which itself was very nice.

One important piece of context that might make all these stories less confusing for non-googlers:

Code references are less important inside Google editors, because we have a code viewer tool inside the web browser.

Most people read, explore, follow references, and share permalinks to the view-only tool. It’s a lot better than viewing code in GitHub. It’s super fast, is connected to language servers and can actually trace referenced, and overall has a million little features optimized for reading code.

We also have a code reviewer tool, and a separate tool to run and view CI runs.

So what’s left for the editor? Syntax highlighting?

I would tend to view code, run tests and CI, and review in separate tools specialized for their specific use case. The code editor was just a place where I would type in my changes.

I’d imagine this workflow feels weird to people who learned in one-stop-shop IntelliJ and GitHub world. But I can’t emphasize how much better these other tools were compared to GitHib. So a code editor that also lets me read, review, and test code didn’t really matter for me when I had a collection of smaller tools specialized for each individual task.

We've moved on from Git5. Although it was a pain, I kind of liked that Git5 made the monorepo less monolithic to my editor.

It was a sad surprise last week when we tried to upgrade the workspace AI plan for some of our team members to Ultra and it was gone. We're moving to Claude/Codex.

Google employees can’t use antigravity. There is an internal version of it which has an agent which is shared between Cider and it.

Anyone care to speculate what the internal reasoning is?

Everything turns into the thing it was set out to replace.

Google has a rich history of product mismanagement. It would be a shame and legacy ruined if it were to change.

I can guess: I am 3 weeks into a 4 week Ultra subscription and the amount of Claude Opus and Gemini Pro tokens that they give you on the subscription is very generous - I feel like I have been gorging on tokens, tidying up 25 years of my open source projects. When my one month subscription runs out I will miss it.

It would be nice if they extended their external services the same behavior…

This is the other way people work at Google. You have a Vm and then connect IDE of choice to it via SSH. But honestly it’s a lot more effort that just using Coder

What tools available to the public would you say is similar to this workflow?

Do you mean local checkouts? There's a similar workflow with at least mercurial? Dunno about jujutsu.

It's the same thing with a different name and different default settings.

Hi Drew, I remember your "Show HN" from a while back and have been secretly rooting for you ever since! (I'm not a lawyer but for some reason I have many friends that are, and now I happen to do work for a firm in the legal publishing sector, so I often hear about how terrible "word processing" can be and think there've got to be better tools!)

May I ask, how are things going? Also, will your IDE always be focusing on transactional law or have you considered expanding to other legal areas and/or markets?

There was also the code search, uh, I forgot the name "quick change", I believe?

Very handy for seeing a problem, quickly solving it (sending out a CL) marking it autosubmit and just moving on.

That is because the Cider team did an amazing job of managing it, and spent tons of time going bug report by bug report to find and fix the blockers stopping people from preferring Cider-V over Cider, instead of the typical Google deprecation approach of "monkey knife fight"

You want bazel. Once you've internalized the bazel (blaze) system, you want all builds and tests to work that way.

I would imagine Android development, with its reliance on simulators for local UI testing, is pretty complicated to shoehorn into a web-based IDE? I think cloud-based IDEs would only really work for anything for which a text or web-based UI suffices. (Which is already quite a lot: that covers code, logs and web pages.)

For anything with native UIs, I suppose you could "remote desktop" into an app or a simulator running in the cloud but at that point you might as well run that locally and cut out all the issues introduced by networking.

Visibility doesn't always get you value though. See the many companies that unify their ticketing to something like Jira, and end up running reports on in. The actual accuracy of the aggregates is rarely great, and instead leads to people doing "jira optimization" to make reports look good.

I once worked at a place where VPs were looking at sprint burndown charts, and asked what happened if the line didn't look a lot like the line expected by JIRA. The telemetry is therefore often a curse, as any metric becomes a target. How many companies today have KPIs about having automated code reviews, which are then ignored by the devs, because said reviews are just wrong on almost everything?

The learnings of Seeing Like A State don't apply just to governments.

In my experience, antigravity IDE is much less seemless compared to Cider-V. I completely moved to using web-based antigravity for the agent and using cider-v to make manual changes and viewing code.

For most of my time here I used exclusively Chrome OS, and switched to it for personal use as well. My daily driver for years was a bright red Samsung Chromebook Galaxy (the first gen with the actual metal case). Literally none of my work is local, and it could run Secure Shell, Cider-V, and Docs as installed PWAs with their own taskbar items, etc. It was glorious.

When it finally failed in the most annoying way possible (the touch screen, which I do not use, started creating phantom clicks in the upper right corner of the display) I went looking for another Chromebook that was light, powerful, and well-built. Finding none, I now use MacBook Air and weep for the time I lose every time it needs an OS update.

Since it's mostly browser tabs, as long as you have ample memory (eg 16gb) it's good enough.

I do most of my development on a MacBook air and a Chromebook. The ~only thing I do from my local machine is ssh into a beefy workstation and use chrome.

From what I'd heard contractors get issued as little as a Pixel tab and dock? Everything else is in the cloud (either gLinux desktops or cloud shells) AFAIK.

I joined in late 2015. Cider was well-known by then.

I'm a UXE, so I tend to use the same tools an external developer might. But I never got the impression that Cider was a recent development.

I can't check when Cider got started. I was probably wrong (it wasn't much used in my circles at that time), I'll update the post.

The only thing remotely close is a monorepo checkout ... with all the problems that come with that.

When I joined in 2016, it CitC would make it look (and still does) like you had the entire monorepo on your local filesystem on your machine.

Git5 would copy some directories but builds would still fallback to files from the monorepo if you didn't track them. It was convenient for me since I could just grep and do fuzzy matching from my editor. Now I have to do some extra work to avoid grepping the entire monorepo. LLMs sometimes still try to grep the entire repo lol.

Now, you could use a perforace, mercurial, or jj interface and it works fine.

You don't really need it anymore - CitC let you do views (mapping just part of the monorepo into your filesystem via FUSE) since about 2013, and then that functionality just got built into Piper. When I returned in 2020 you'd have a file at the top of your source tree that included all the relevant file mappings as well as any Blaze flags needed to build the project, and you could just point your IDE at that and it'd map in just what you need.

The history of Google's relationship to version control is even more interesting than editors - it went from CVS in 1998 to Perforce (P4) in 2000, then gcheckout and g4 in ~2006, then OverlayFS was invented in 2008, git5 came out in 2009, CitC obsoleted OverlayFS in ~2012, Piper built this all into the VCS in ~2013-2014, while I was gone from 2014-2020 apparently we got hg and jujutsu frameworks, and then when I got back in 2020 you'd just check out a .blazeproject from your IDE and everything would magically work. Many of these started as 20% projects (I used to have lunch with the guy who invented OverlayFS; interesting character and one of the best programmers I knew) and then got folded into the "official" way of doing things once grassroot adoption showed the execs that this was how people really wanted to work.

Are they actually cut from the same codebase? The internal version has workspace support and other features cut from Cider I assume

From the article: “ Cider was a light client that opened much faster than traditional IDEs. All the magic happened on a backend that indexes the entire codebase, so that all the data was ready whenever someone opened the webpage. ”

Sounds like all other editors were slow compared to Cider.

Very little development in Google3 happens locally. You aren't even allowed to keep the source code on your local disk, and this is true no matter what OS it runs. (Android and Chromium are different though.)

You have access to an extremely powerful remote workstation that from a UI perspective functions almost identically to a local workstation, via Chrome Remote Desktop. Plus, no one builds things locally, even on that machine. There is a huge, absolutely amazing distributed build system that everyone uses for everything. (Again, Android and Chromium are different.)

So you don't really need a powerful local machine. I held out for a long time--there were a lot of growing pains in the early days. But eventually it got really, really good.

How common? I'd wager most people still use a mac, followed second, but far by regular goobuntu laptops. Chromebooks goes 3rd because Windows is practically banned.

When I joined, I started with a MacBook and lost it within three months :(

Afterwards I was issued a 12" Pixelbook and it was surprisingly much more usable than I had expected! I could ssh into a Linux box for running builds and tests. Cider worked perfectly. It was snappy enough to serve as a thin client even on a 4K screen.

How many new googlers use vim or emacs do you think? I can imagine at least a small amount of new vim people since vim will always be popular, but I would love to know if more than a handful of new googlers a year use emacs

Is Antigravity a Cider-V fork?

Was there never a detour via Eclipse Theia? I thought it might have had some traction internally, since Cloud Shell is based on it.

> Windows is practically banned.

FWIW I don't think this is accurate (was kinda true in the 2010s?). I wouldn't be surprised if it's almost easier to get windows laptop than linux one now.

Famously Jeff Dean uses emacs. Emacs integration to internal systems (source code, code search, LSP, build, etc) was super solid when I was there ~2020.

Chromebooks are pretty much only good as thin clients, so much so that when I have the money I plan on building a powerful rackmount workstation and connecting to it via chromebook/box

There is an internal website that tracks statistics of tool use, where “tool” is defined liberally and includes emacs. It would be tracked if you just (require 'google) somewhere in your initialization code.

I've switched to emacs and I no longer use IDEs. This is because I do all my edits, as a personal policy, via LLM. I mostly use emacs for magit (I work on a git-on-borg repo).

No. Antigravity is the public version of jetski which is a VScode fork made by the windsurf team.

Okay, but what about the corgies?

Not on the developer side. The cloud side, yes, a bit.

OK, this was probable me telling other people I have never worked in a large repo without telling other people that...

I can understand Android (including the Linux kernel) being "too big" and "too separate" to go into Google3, but why Chromium? When it was forked from KHTML/WebKit it was probably not that big compared to the rest of Google's codebase.

> You aren't even allowed to keep the source code on your local disk

How is this enforced?

Could you even put all of google3 on local disk if it were allowed?!? You'd need quite a RAID array. I suspect it'd be almost impossible in practice.

What is Google3?

I joined gdm recently, and previously used (neo)vim exclusively. Begrudgingly Cider-V is very, very good. It might be possible to get by without it, but the system is so locked down you’re going to make a lot of sacrifices. (very few authorised extensions, codebase is so large it’s going to break whatever tools your used to using anyway, no git)

I’m well thinking I may as well trade my brick of an m5 pro for a 13” chromebook, it’s a strange time.

I don't think so, I think they forked VS Code directly or possibly forked Windsurf which forked VS Code. Hence the turf war and internal controversy; a lot of the effort on Cider-V got dropped on the floor, right at the height of Cider-V's popularity when they were getting large amounts of features.

Duckie does still exist, and is probably one of the most used (and useful) AI tools at Google. Yes, it's just a Gemini wrapper with access to all the internal documentation. I wasn't doing daily development when I left so I don't know if it ever got into Cider-V.

> For anything with native UIs, I suppose you could "remote desktop" into an app or a simulator running in the cloud

This does exist. The network isn't the main problem. The Emulator has to run under nested KVM. That + graphics rendering on the CPU makes it not so responsive. It's useable enough in many cases though.

"Android" here refers to the Android operating system, which (like Chrome) has its own separate development stack. Most of Google's Android apps are developed using the main google3 stack described in the post, or at least were when I was there.

You have to be very careful in management to not create perverse incentives. I like to use change control processes as an example. In theory, a super strict change process for every single change is great, because it'll ensure everything gets reviewed thoroughly. In practice, that leads to people flouting the change process as much as they think they can possibly get away with, because it becomes a serious impediment to getting work done. A more moderate change process would have higher compliance, and actually lead to more oversight, than a super strict one.

It's VSCode, so it's 90% similar to https://vscode.dev There are internal extensions, but they don't dramatically change the look and feel.

I think many VSCode users are not familiar with the Comments UI, but it's used in e.g. the "GitHub Pull Requests" extension. Apart from that, some changes in the list of directories/files (for performance reasons) and a redesigned SCM integration.

I will say that the latency of the filesystem is a different problem. Most of the google filesystem tooling is not built for command line tools that expect to index large subsets of the filesystem at once.

Pretty much. Also it’s not that slow and you can’t just checkout all of g3

All the developments are done in a virtual remote file system. From editing to compiling, everything is done remotely. Of course this does not fully stop people from doing manual c&p, but it still makes it hard enough to discourage it.

I have no knowledge of actual enforcement mechanisms, but it is way, way, way easier to do all development on the distributed file system that feels like a local disk than it is to copy things over locally.

If you need to do development locally, you are either doing something very wrong or extremely specialized.

So there is effectively no motivation to copy the sources over. And because everything is on this distributed file system and built from it in a very bespoke environment, I would imagine (with no inside knowledge at all), that it is easy for auditors to detect when someone starts copying things out.

For security reasons, the VSCode marketplace is not accessible, but many (in the 3-digit range) external extensions have been imported. One technical limitation is that some extensions are not designed for the web (e.g. try to run local things).

> codebase is so large it’s going to break whatever tools your used to using anyway, no git

There is Jujutsu (with Piper backend) officially supported, and that is better than git. But of course, you will not be grepping the source code, there is code search for that.

Think for a large tech company, they did a really good job with success in software. For exammple, they were probably the first large tech company to realize AI was actually working, and made it their focus:

https://www.businessinsider.com/sundar-pichai-wants-to-build...

And yeah, they did/do a lot through acquistions, but seems like most major companies screw up acquistions. Google has it's fair share of failed acquistions, but especially in the earlier half of the company's lifespan, they really did some great one: Youtube, Google docs, Nest...

maybe am biased, but have always thought Google in general does do it better than most tech companies. think it's their focus on the love of interesting ideas vs the love of money (although, that changes more and more as the company ages)

The thing to remember about Google and software is that consumers don't see the vast majority of the software it produces and uses, from the distributed filesystem colossus (https://cloud.google.com/blog/products/storage-data-transfer...) to an enormous number of other internal projects just as complicated as that.

It's user-facing stuff may or may not be great--and the consumer level flops are legendary--but that is only the tip of the software iceberg.

like a workstation under your desk? is the latency bad when you remote access it not in the office?

It is basically VS Code Web. Try https://vscode.dev/ to see how you feel. If you don't like it you won't like cider.

To be fair, that is a blessing. Large monorepos are a terrible idea.

Chromium is open source. As such it needs to be hosted via a publicly accessible stack (Git/Gerrit) so that external contributors can use it.

Size has nothing to do with it.

There's no reason to pull the entire repo just to build one project. Do you pull all of GitHub to your disk?

The mono repo that holds most source code (pronounced google-tree?) It's referenced in the OP.

It's a monorepo, which is a bunch of libraries (in this case, the code for most Google products) in a single repository. Those libraries can have dependencies on each other.

One is a framework called Wiz, which renders the frontend for a bunch of Google web apps. You can imagine that the Wiz team might want to refactor an API, but not have to worry about different apps using different versions. In a monorepo, they can just find all the callsites and update them in the same commit that makes the API change. There's no package.json in google3 - everything builds from HEAD. Therefore, the commit that makes a breaking change is also the commit that fixes the would-be breakage.

This architecture evolved. Google used to use Perforce, which was a common commercial version control system before Git. Google had to figure out how to express the dependencies between packages in the monorepo (which can be in different languages with different build tools). They eventually created Bazel, which expresses those dependencies and orchestrates their build tools.

Build orchestration took a few attempts. Google3 is the third version of the monorepo, that is, the one that uses Bazel for dependency management.

When I was there all the cool people used mercurial. Git5 was creaky and didn’t work well but hg worked brilliantly. The cool people used hg to do stacked CLs so they were productive even when blocked by code review.

Fun fact: This particular version of hg with its extensions actually originated from Meta.

Wow, thanks so much for taking the time to answer my questions in detail!

> As I mentioned above, it's super hard to even set up a hands on demo because we have to get the desktop app installed on their infrastructure. But I'm shocked to learn that Googlers are happy to work in a browser, and distributing Tritium via browser is trivial, so we're going to 180 on that right here and now.

"Trivial" in the sense you can just compile everything to WASM? I'd be curious to know what such an IDE would feel like in the browser. I think the only WASM-based GUI apps I've tried in the browser were Flutter apps and those were… weird.

> I should probably do a post on this, but there's actually a topic we're working on that perhaps the HN audience will find even more interesting... coming soon!

I'll keep an eye out for the next Show HN! :-)

The first thing is hermeticity and and what it implies: caching. That if targets are a strict function of inputs, and inputs can be hashed, then you can reliably cache them - including test results!

The second thing is distributed caching. Done right, not only are your test results cached, but CI's test results can be cached too.

The third thing is distributed builds. This only starts to matter in big projects, but compilation is inherently a spiky load and if you can share a big pool of compute between a big pool of engineers, you get higher hardware utilization and lower latency to build artifacts.

The fourth thing, something that isn't really feasible outside big tech, is you could be bazel all the way down in a big monorepo. One of the niftiest things at Google is to be able to put a printf inside a database server and run your client test, and blaze knows that it needs to rebuild the database server and it will do it automatically, so that you can get extra insight at almost any level in the stack.

I would recommend learning the various "bazel query" variants starting with a plain "bazel query" https://bazel.build/query/language

Certainly fair. But they have tried some amusingly ambitious projects that make it pretty easy to raise eyebrows. Stadia alone is enough to make me nervous on any efforts they announce that are ambitious.

I have used something like that (mentioned in my original reply ^). It was even worse -- there were about 3-4 plugins available.

Basically that company (a well known social media company, not FB) tried to implement everything on their own. Infra is their own (kinda makes sense because it is so huge), IDE is their own, communication is their own (which has an interesting feature that if someone screen shares an internal doc, other people can click a link to access that doc, too, very useful).

I was very jealous about their tooling team (that's what I call real programming), but nevermind I quit after a few months due to some unrelated reason.

I have always had this itch to work on some real life serious system programming projects, with the most recent wave OS kernels. I completed the MIT xv6 labs (a very small repo) and did a few Linux device driver labs (very large repo and it was the first time I experienced compilation time > 5 minutes).

I got burnt out after a while, so that kinda wrapped up my experience working on large repos.

Yeah, under my desk. I rarely remote which is a good excuse for me to disconnect from work anyways.

Large monorepos have tradeoffs that may or may not make sense for a particular use case. Google's monorepo--in its form as a monorepo, not just the software it contains--is one of its biggest assets and creates enormous leverage.

And an enormous set of problems that must be managed. But multirepos have their own set of issues, and which set of problems you want is highly situation dependent.

Because of high coupling dependencies between google3 projects, compiling just a single project usually pulls hundreds of thousands of different build targets.

From the user interface perspective though, it does essentially look like you've pulled all of google3 into your disk.

We can use jj now, thank goodness. But I still miss my old git workflows + lazygit

Cool - thank you for answering

Probably not in the way that you might mean it, but for me (Xoogler, 2010 - 2023) internalizing bazel means:

"Hey, where's your tool's code in $MONOREPO?" "<path/to/stuff>"

Cool:

    g4d my-citc-client # moral equivalent to `cd ~/repos/stuff`
    blaze run path/to/stuff:target

... and you get a running version of whatever $stuff is, immediately built from head, quickly - no matter the set of dependencies, or which language they were built in. I can just try your thing out immediately with a common interface for all the builds, and I don't need to understand the build at all, unless or until I do, and then OK, absolutely every single build is always expressed in exactly the same way, same idioms, same patterns...

My point on the acquisitions was that a surprising amount of their successful software was not made in house. Again, I don't mean it as a knock against them, necessarily.

AI is an odd example. For one, a lot of the research there is from acquisitions. Somewhat feeding back to my first point. They also were seen as tripping up on a lot of the current AI race, no?

This is not a complete answer.

There are many open source projects that are developed in google3.

I've never heard that pronunciation.

Probably it's just down to a different understanding of the word "internalize".

I know how to _use_ bazel effectively to do my work. I'm comfortable with its well-designed surface but whenever I've tried to understand the inner machinery I've given up - especially when presented with a bunch of custom skylark rules code.

It's like an anti-git in some regards - the surface of git (the CLI) is an abomination in many ways but the the mechanics of the tool are so ingrained and the model is so clear and simple - I never feel uncomfortable.

I've a need to have some comprehension of the inner machinery or the underlying model of my tools.

Good old capybara

> My point on the acquisitions was that a surprising amount of their successful software was not made in house.

First, that's just not true. Their biggest products by revenue (search/adwords) and biggest stock value driver (AI/Gemini/Datacenters) are clearly in-house creations.

But even then, the two biggest "acquisitions" you're probably thinking of are YouTube and Android, acquired in 2006 and 2005 respectively. What fraction of the software base of those products do you think has survived the intervening two decades? To be blunt: most of the software being shipped out of those groups is being authored by engineers who couldn't even read when the ancestral code existed outside of Google.

Honestly the "acquisition" thing is just a cope meme promulgated by Apple stans, as it were. It's not a serious point.

What did you think it was pronounced as?

I mean... First, I don't think acquisitions are automatically a bad thing. And I was largely riffing on the list the post I responded to started. Youtube, Google Docs, and Nest were all acquisitions. As noted, we can add Android.

Do these also take a lot of effort to keep going? Absolutely! But that doesn't change that they acquire a ton. They just acquired Wiz this year.

I do question a lot of the focus on a unified IDE when it comes to this strategy. It is not surprising that there is a specific "discontinued google acquisitions" page in wikipedia with that in mind.

"google-three" which it obviously is?

Ah, I misread the post he was replying to.