You can add Reddit to that list, IPO is coming so better price out the third party apps we encouraged developers to build.
If decentralization is ever going to win, it needs to be turnkey, explained without showing a network diagram, be basically free to run on your laptop, and actually have the content people want to see and not just be a bunch of people who speak lojban. [1]
While I'm not sure how much of a legal leg X has to stand on, I understand why they'd rather Nitter not exist. Twitter tried (valiantly, imo) to stay open. What ultimately began the API lockdown was the need to stop bleeding financially. Ads were inevitable, and there being no reliable way to do that via API access.
Twitter solved those three problems, and no one spends nearly as much time talking or caring about the technology used to do it than people who talk about decentralization. Myself included in my younger days. Now they're just defending their moat.
So the question is, are you okay with giving random people a mirror of your public posts? After all, they're public. It's like putting them in a repo on Github for anyone to clone.
Google's Go Module Mirror is a similar but more specialized service that has a copy of all the Go modules that anyone has published on the Internet.
- Semantic HTML
- Web indexing services
In this case, it's mostly indexing and a little of the other two if you want to add social features. I despite ATProto with all my soul for taking a problem with such a simple and standard solution and totally obscuring it behind hundreds of layers of JSON, faux federated services, and technical jargon, all in service of creating an inferior version of Twitter. I guess it wouldn't be as sexy to offer a web indexing service instead.
There are multiple sites that support follower semantics over RSS. Feedland tracks subscriptions publicly, so you can see the blogs I read (https://feedland.com/?username=robalexdev), and who reads my blog (https://feedland.com/?feedurl=https%3A%2F%2Falexsci.com%2Fbl...).
I run another variant which collects OPML blogrolls via crawling, so you can find out who else likes your favorite blog and what else they recommend. Here's the page for Simon Willison's blog (https://blogroll-network.alexsci.com/discover/feed-a34ee2a88...). Thinking of RSS and blogrolls as a network feels much more resilient than blueskys Jetstream api endpoint.
ATProto is attempting the "if we make it P2P it'll be popular!" strategy except not with P2P.
I agree it would be neat.
But, when did we conclude this "should" be possible and thus warrant such a whiny post?
Nothing to do with Bluesky services. There are many independent firehoses and relays. Here's a stream of standard.site blog posts coming in over a firehose hosted in Chennai. Every single one. No curator between me and the posts, and no work by me to crawl the whole network for them. https://pdsls.dev/jetstream?instance=wss%3A%2F%2Fchennai.fir...
Edit: ok there is an aggregator, the relay is scraping all of the PDSs out there to build the event stream. Notably this is not possible with RSS, where you need to build a large index with knowledge. PDSs request relays to crawl and that's that.
One thing I'd love to see with rss / ompl sharing sites is more ease of exploration. It feels a bit clunky browsing many of these feed sharing sites because you need to evaluate each feed by manually clicking through each one etc or worse you can only import the whole opml at once flooding your feeds. For example I've seen some sites with feeds directly showing posts as they come in for each persons feed list(s) so you can see what that batch of rss feeds looks like in action without clicking around much. There are some sites doing great work getting people to share their lists but I think there is room for qol improvements.
One site I've been liking that has feeds for each list and is trying to reduce friction on sharing is blogflock.com. You can also follow other peoples lists directly on the site too so they show up in your main feed. I believe I've also seen some people self host their blogroll on their personal sites with similar feeds but I'm having trouble finding that software atm.
Absolutely. One good thing going for this approach is that anyone can grab the OPML export (the URL is stable) and build their own frontend, I'd love to see more.
Could be a fun weekend project for frontend folks.
One can imagine making it a little more featureful, for instance indexing OPML blogrolls which would allow you to see whom a person follows and https://microformats.org/wiki/h-entry which would allow outgoing links to be categorised (like, reply, etc) so notifications filtered. With that, it would be possible to add a front-end site imitating one of the popular social media paradigms (Reddit-style or Twitter-style being the most obvious).
If there's one genuine design decision I would give for it beyond what is basically a cobbling together of existing interfaces, it would be to charge users per page uploaded. Likely very detrimental to growing the service, but I think one of the simplest ways to weed out spam and junk. A real problem with many web services is that the receiver of the message pays for it in terms of attention, where in other mediums the sender has to pay. Given that uploading crap is basically free, that's all you get. Increasing the cost of upload would weed out those endless AI summaries and lists of affiliate links.
The issue with ATProto is that it operates on too many layers. You can see lacking in what I have described here the concept of durable authorship, but this is a property of content not how the content is distributed. Perhaps someone will invent a standard way to sign HTML documents, in which case you could base user accounts on that instead of DNS. AT enforces this centrally but it does not need to. It is walling itself off from the common and decentralised software ecosystem of the web for no good reason.
I think it is generally bad to implement new features like this on the part of the indexer. It should just keep track of an existing web of documents rather than creating it's own format and walled garden.
Ah right, of course. Fair enough. I think Dave Winer is trying some similar ideas, though I haven't looked at it beyond knowing he's trying to extend RSS to support these capabilities.
In 2007, Tim Berners Lee write the essay The Giant Global Graph. Quote:
There are cries from the heart .. for my friendship, that relationship to another person, to transcend documents and sites. ..Then any other site or program can use that information.
It seems appropriate as X is sending cease-and-desist letters to Nitter to remember TBL's essay. Nitter is - was - a simple frontend to X which allows users to view tweets without logging in. Even that small use of proxying to the pages is enough to receive threats of legal action.
Twitter's API in 2007 was famously open, which meant thousands of developers building clients, tools, and analytics for free. So, what happened? Why was it pulled? Simple: the network won. The developers stopped being an asset, and the API progressively closed. Rate limits, pricing tiers, login requirements, then technical blocks on the workarounds, and now letters from lawyers. Meta ran the same playbook a decade ago and it's now hard to remember there was ever a Facebook or Instagram API worth building on.
This is why Brewster Kahle, founder of the Internet Archive, has been calling for over a decade for us to lock the Web open.
Nitter started off using X's APIs. When that closed, it read public web pages. And now that there's nothing left to close, the demand is that the source code come down. A program that displays public posts is being treated as a circumvention device under computer-crime statutes.
We have a walled garden problem. It isn't going to change, and the only option in front of us is to start fresh.
The good news is, atproto continues to grow, activitypub remains resilient, and our community is full of believers and builders in the open social web. Since I work on atproto, that's what I'll talk about next.
SELECT *SELECT * FROM internet.blogposts
The walled garden problem is downstream of a simple question: how do I SELECT * FROM internet?
If you've never written database code, SELECT * FROM users is how you ask a database for everything it knows about its users. Once you have it you can filter it, sort it, and join it against anything else you've got.
The web doesn't historically work that way. The web is a few dozen companies, each holding a filing cabinet, each with a receptionist posted out front. He'll read you one file at a time, but only files you can name, as fast as he cares to read, and as long as his boss allows.
Nitter was a lightweight X reader that worked fine right up until X turned off the access it depended on. Every API (the "receptionist") is a business decision that hasn't been reversed yet.
But Impermanence isn't the only problem. Even a permanent, free, generously rate-limited API wouldn't be enough. Applications need much more meaningful access than APIs can provide.
You can only ask questions someone already thought to answer. An API is a fixed menu. It gives you getPosts(user) and getFollowers(user). If your product idea needs "posts from people my followers follow, ranked by how often they get quoted," there is no endpoint for that, and there never will be, because nobody at that company is building for your product.
Even the right questions come back in the wrong shape. Followers come 100 at a time. A two-million-follower account is 20,000 round trips. At any polite rate limit that's hours of work to answer one question about one user — so anything interactive, anything that has to feel instant, is off the table before you start.
You can't join across "cabinets". The interesting questions are almost always cross-service: this person's posts against that person's photos against a third service's reviews. Two receptionists in two buildings can't cross-reference anything, and neither can you.
You can't index data you don't hold. Search, ranking, recommendations, feeds, moderation tooling — all of it is built on indexes over the whole corpus, laid out for the specific questions your product asks. You cannot build an index through a keyhole.
To actually build a service, we need the whole dataset rather than a view onto it; we need it live, arriving as it changes instead of polled for; we need to index it however my product demands; we need to write back into it; and we need all of that guaranteed in a way no single company's quarterly priorities can revoke.
Desktop apps handle this by sharing the filesystem. Internet apps don't use files; they use databases. We need to share the database.
As a user, I don't want to be locked into an app anymore than I'd want to be locked in the trunk of a car. I want an actual free market.
So then, here's another set of needs.
Persistence of identity.
The export of living (not dead) data between services.
Exporting archives of your tweets is useless as an account migration solution because data doesn't live in isolation.
If data is no longer operable - capable of additional operations by participants in the network - then it's a static archive and useless to another application.
You could always print your tweets and look at them I guess.
If we want data to remain operable even outside of its original service, then we need to share the database.
These are all issues atproto is designed to solve, including open data access, account migration, and a live firehose of network activity.
SELECT * FROM internet happenHow do we share the database? We don't. We share a lot of them. We create a whole network of personal data servers (PDS) which applications interact with.
How do we handle apps sending complex SELECT * queries to our personal data servers? We don't. We replicate the data on logs. We have each application aggregate copies of the data to query locally.
How do we have apps write to those databases? In this case- we do! We have the apps send writes to the PDS, which in turn replicate back out to the other apps.
This last one is the core of the intuition about atproto: the write/ingest loop. Almost every atproto app has code that looks like this:
// write
pds.putRecord(post)
// ingest
onPut(‘app.bsky.feed.post’, evt => {
mydb.put(‘posts’, {...})
})
Rather than waiting for the ingest to come back over the wire, you can use a "short circuit" so your app's database can update more quickly. The 200 OK from the PDS is a transactional go-ahead.
And so the more efficient pattern looks more like this:
// write
pds.putRecord(post)
mydb.put(‘posts’, {...}) // ← optimistic
// ingest
onPut(‘app.bsky.feed.post’, evt => {
mydb.put(‘posts’, {...})
})
Yes. The network exists. It's live, it's public, and you can read all of it right now — from a laptop, without asking anyone's permission. This is exactly how Bluesky, Tangled, Leaflet, and a bunch of others work now.
Let me hit you with some stats. At time of writing, there are:
46.1M accounts on atproto
24.5B records
3.15B of them are posts
17.4B of them are likes
500-1000 write-events per second
Over 5000 personal data servers
It's never been easier to tap into the data with the new jetstream service.
import { Jetstream, isCreate } from '@bsky/jetstream';
import { app } from '@bsky/sdk/lexicons';
const jetstream = new Jetstream('https://jetstream.us-east.bsky.network');
const collections = [app.bsky.graph.follow, app.bsky.feed.repost, app.bsky.feed.post];
for await (const event of jetstream.live({ collections })) {
if (isCreate(event, app.bsky.graph.follow)) {
console.log(`🌱 ${event.did} follows ${event.commit.record.subject}`);
} else if (isCreate(event, app.bsky.feed.repost)) {
console.log(`♻️ ${event.did} reposts ${event.commit.record.subject.uri}`);
} else if (isCreate(event, app.bsky.feed.post) && event.commit.record.reply) {
console.log(`💭 ${event.did} replies ${event.commit.record.reply.parent.uri}`);
}
}
If you want a fast way to get into it, try it out here.
SELECT * FROM internet.blogposts instead.And, oh, if you're looking specifically for blogposts on atproto, you probably want to use standard.site.