pow_server  http://127.0.0.1:8080   backend avx512-x16
  ──────────────────────────────────────────────────────────
  uptime   00:03:12
  solver   ● BUSY  difficulty 9, 0.3s
  queue    [####################............] 5/8   peak 12
  ──────────────────────────────────────────────────────────
  accepted 1240        solved 1180
  503 shed 48      504 timeout 2      4xx/5xx 10
  ──────────────────────────────────────────────────────────
  last     difficulty 5  nonce 645376  in 9 ms  (101.6MH/s, avx512-x16)
  hashes   3.90GH total   avg 65.3MH/s
  Ctrl-C to stop

It's still valuable to raise the cost of scraping of course. I don't think anything can really stop a determined scraper from impersonating a human. I wonder though if a system similar to Anubis but mining some crypto would make bots _welcome_ - since they're paying for their traffic.

The vBulletin and PHPbb style forums have issues with slowdown (I haven’t had a forum since 2015; even back then those forums were overrun with spambots), but static content on a nginx site can be served lightning fast.

https://commoncrawl.org/

The next tier up from free is $25/month or $240 per year.

https://www.cloudflare.com/plans/ https://www.cloudflare.com/plans/free/

But if you want them to actually stop, you can also just serve a little JavaScript page that sets a cookie and refreshes, to anyone who hasn't set the cookie. The DDOS attacker doesn't run JavaScript.

https://patronview.com/news/99-percent-of-my-website-traffic...

And you're totally right: I mostly post on X (nee Twitter) and I probably have ADHD or just a low attention span, so I prefer to read things broken up into paragraphs. But for a smarter audience like this, and that reads long-form blog posts, I should tighten it up.

Thank you for the suggestion. LMK any other edits and I'll be happy to tighten it up.

Are you speaking from experience, or inferring from articles like this?

I serve a static site on the second lowest Linode $5/month VPS and it is grotesquely overprovisioned for that use case. It is not the case that every site is getting slammed every second by hundreds of requests per second.

Now, if you have some sort of dynamically-computed website that is generated by a slow scripting language that is poorly optimized and hits the database too many times for a single page, yeah, it doesn't take many RPS to take you out. But that isn't the only option; it's the slowest of the slow options. Realistic, there are plenty of sites that match that description, but I concatenated that many clauses on purpose. Drop any one of them and your personal site will be fine.

While dedicated scrapers/attackers could work around it, and they could do so much more efficiently than the client-side js, almost none of them do. Unless you like paying additional hosting resource fees to serve bots, it's a worthwhile option, and is less annoying to typical human visitors than cloudflare's interactive captcha/challenge which is what most people use.

Don't let the perfect be the enemy of the good enough. For now.

Targeted attacks may not be repelled at all. That's not the point.

They also don't load CSS but for some reason the security theater PoW won the mindshare.

I survived handling the search search traffic generated by this thread[1] on PC hardware off a residential broadband connection without any sort of degradation. Only time I've gone offline from traffic was when Elon Musk tweeted a link to one of my blog posts, and that was just a short temporary blip.

[1] https://news.ycombinator.com/item?id=28550764

Best thing I've read on HN so far this year.

People trying to block bots end up keeping out all kinds of users. I get blocked frequently for using a regular browser, just with JS disabled. 99% of the time, I just close the browser tab and move on with my life.

What's worked for me is I block any client that don't support brotli compression and http2. Seems to work well enough for stopping scrapers.

What do you mean "this way". What other way is there to run the site?

Yes, that's the social contract. Bots are not a part of it

And this is a complaint? I find nothing in this statement to sympathize with whatsoever. Isn't the contract of, essentially everything, that effort is required to obtain something worthwhile?

Please let me know where I'm getting this long-standing, fairly fundamental understanding of the world, wrong.

(This isn't a condemnation. AI can often do a better job of representing thoughts than humans.)

Examples:

- The article organization

- The general language flow

- The bullet points with a bolded gist, a colon, and then elaboration (and bonus with details stats).

- The images are almost certainly AI generated. They look AI generated.

(i skimmed the whole post, there are none)

I think you've hit the nail on the head here. I think that's a big reason why people want to ban bots.

I have an old os (macos 10.11), running the highest firefox esr I can run, and I get denied by cloudflare.

But not always immediately - I get to enable javascript/cookies sometimes just to be denied.

they are not the folks we want gatekeeping the internet, they are opportunists increasing their OKRs

No; in this case you are not a user, you are a bot user.

And a lot of bad scrapers.

Nuance, complexity, and an awareness of 'other people' do not exist in their worlds.

It can't all be agents reacting to user queries. It's confounding how much CPU and bandwidth is getting flushed down the drain.

    write a implementation of the anubis proof of work in native c code, optimized for speed above all else. use every trick available to make the proof of work as efficient and fast as possible, including modern processor tricks on the x86 platform. your code should avoid using external libraries where possible, include tests, and be readable and concise. a reference for what needs to be met is in this repository. https://github.com/TecharoHQ/anubis

    let’s develop this more. turn this solver into a local HTTP server that can be given work in the request, and it returns solved work. make an end to end tester that sends test work to the solver and waits for a valid response. add support for solving with a GPU using cuda.

That's not absent from the article, it's right there in the section titled "The Claude ratio". ChatGPT, Claude, etc. use different user-agents for scraping vs user-initiated requests, and the author notes that user-initiated requests were an absolutely miniscule fraction of the total traffic.

Or maybe -- just maybe -- using dashes isn't a sign of content being AI written, just a style it picked up from the training data.

So this is how AI wins, humans kill each other off because we might be bots and the bots inherit the earth.

Using it slowly sucks the uniqueness out of you.

> A human on a VPN sees one CAPTCHA and passes, but a headless browser fleet sees a wall.

I do wish we’d stop complaining about em dashes though, that’s lazy criticism. The sentence structure is far worse, and a big tell is subheadings that are all variants of “The <adjective> <noun phrase>”.

Equating bot with bad actor in 2026 is Luddite behavior. Driving people to your site so you can serve them adspam or just make whatever operation they want to do 10x more difficult is the same.

MCP style APIs should eat the web. This doesn't mean the "open" web goes away. > 99% of the time I don't care for a domains particular FE at all. It's a complete waste not only of time, but resources and bloat. Pushing the contract into the agent should become good UX. Making things harder for good faith users should never be the goal.

Overall systems and different business models need to be created that forces AI to pay it's fair share! Elon says there will be an abundance thanks to AI and we wont have to work -- ok how's that gonna work without different systems in place paying us?

I feel strongly about this topic and proposed some systems in this Substack post https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans...

I didn't ask the not to use this site, my query was broad and complex and would take me days to do it myself. I wouldn't

I'm quite certain they earn a hearty commission off it, and I think it was mostly possible because the site was "friendly" to bots. Otherwise I probably wouldn't choose the site because it's never any of my top choices when I look for this myself.

So, maybe you monetise it like this? You asked, I answered. Doesn't fit every site or business profile.

USER AGENT

not user, an agent on behalf of the user.

the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT.

for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.

the alternative is all your in home information owned by corpos when the info should be from user agents not corpo feeds. if you want to vote this idea down, you might be a corpo. :-)

For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an extension of me clicking into every page and making a spreadsheet myself.

However, if it were to continuously poll, or just scrape or something with no intention of buying, I would be no better than a bot.

And if not, what if it was a custom browser and not Chrome/Firefox/Edge?

I think I must be missing something here.

They're just better at English now.

Like, all the comments of this post would go into my personal database simply because I loaded the page, and it would help me find old information I've read, and could also inform LLMs I use.

This should be built into browsers.

Yeah, googling it does seem like "Claude-User" is for user-initiated requests.

By "Claude sent me" I thought the author meant referer header in real browser requests showed that they came from.

I mean, that's what the section "Pages crawled per visitor referred" refers to, right?

It's outside the great firewall. No requirements to hand over SSL keys to China. I worked for companies that ran into these challenges and solutions.

Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.

Can we build other monetization? Maybe. There's certainly proposals. It's real hard when there's too many layers between the user and the output though. I suspect the solutions will be worse than what we have now. For now the answer is "just paywall", but given your invocation of "corpo" here I suspect that's not an outcome you'd be too keen on ;)

You mean that key whose only legitimate values also call every browser Mozilla? Arguing from the dictionary is so tedious. I'm going to scrape your site, but the idea that you have some obligation to like it because of the definition of the word "agent" is profoundly stupid.

A lot of people live off ads. I block ads. People who live off ads aren't obligated to like it - and are free to take any means to block it without moral hazard.

Identified by the user agent header. Which most bots fake or leave out to increase the chance of getting where they are not wanted.

Your bot is a good bot? Great, let us know when you've dealt with all the bad bots and we'll open the doors to the remaining (good) bots again.

https://www.ycombinator.com/legal

For example, making a website that only works in Edge and refuses to load for anything else would be a shitty practice. You are free to do it. There's no one stopping you, but you should expect people to bitch about it and they might think you're kind of a jerk or maybe just bad at making websites. It's the same thing when a site actively rejects traffic from curl, wget, or any other common utility that many people find useful.

I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic

The User Agent would get a HyperText document from a Web server, render it, and show it to the user. You could parse HTML, display images, and follow links - with the assumption the user would want to edit them like any other document. There was no encryption. There was no scripting for dynamic content. Styling was done via semantic markup. There wasn't even any POST when the User Agent term was initially applied. Everything was supposed to be like one big distributed document store instead of centralized hosting on few servers for many client devices.

All that is to say: Regardless what I think sounds nice myself, if the web was just the original vision & definitions instead of everything it changed and grew to become we wouldn't even be using it to have this conversation.

Interestingly, people seem glad to have telephony companies filter bots even though the original idea of the telephone did not call out robocallers as invalid clients. Probably because, unlike the modern Web, the average person gets a chance to see what it means to allow anything inbound even though the original concept never considered robocallers.

This is giving "why is this plane full? What in doing is important but what is everyone else doing here?"

Yes, but no.

I did that. I even printed and bound it in a bunch of dead trees and put it up for sale. The LLMs still stole it.

This is a "what if it's for your own good?" argument. What if I break into your house to clean your toilets? Although amusingly, if you're a "web alerts company" that I didn't contract and sends out alerts in batches on your own schedule, you will probably send a ton of your customers to my site at the same time and slashdot me off of the web entirely.

https://news.ycombinator.com/item?id=32141027

The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money.

I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am considering as well as staples/groceries and everything in between. I have this little scrape/ingest pipeline because there's no other way to obtain this data... not even an API that I can pay for access to.

One of the large appliance sellers that I have in the scrape queue has gotten _hyper_ aggressive with bot detection to the point where even my personal head-full chrome instance doesn't always get to load the page. Guess who I will never buy that ~$2000 appliance from.

I agree with one aspect of the narrative you're saying, his idea was held as being for «documents». To be sure, document was more abstract than document on a Mac.

Either way, I'm very sad it mostly turned into animated brochures instead.

But the reason styles were separate, the reason everyone talked about semantics, was always about the universal knowledge DAG and enabling machines to help us navigate and retrieve it. It's why so many of the early agents referenced navigation (Spyglass, Navigator) or the interconnectedness of things (Mosaic).

So, ok, big document “store”, but that's a lifeless noun. The information you are looking for, the dots connected for you surfacing what you didn't know, that's a living growing thing, an agentic retrieval savvy knowledge engine, distributed for curation and resilience.

The librarian was always to be a machine or many, on our behalf.

LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all. They do not display the page for a real person to read, they merely ingest its contents and then spit out something completely different.

There's a "look what I built!" thread every few days on the /r/homeassistant sub and the home assistant forums, too.

The data sources vary but it's usually a mix of devices that the user has locally and some remote data sources like weather/public transit schedules APIs or RSS and in some cases also utility rate info all integrated into a SPOG that is then displayed on some ambient screen (usually a hacked tablet or other cheap ARM device)

  - Bias, specifically commercial bias
  - Webpage formatting: every website looks different, hides the information I want in different places. Sure, it "looks pretty", but I don't want pretty; I want info
  - Scams/SEO/etc...

Google "How to make ham fried rice"; you'll get 10,000 articles, most pretty good recipes. But they're all different; most of them are just bait for ads. And most of them, the 10-line recipe is hidden between 50 useless paragraphs about how serving food is life's most important goal.

Now, ask an LLM to search for it, find the best combination, and list only the recipes. You get a perfect, 10-line recipe that doesn't waste your time.

Significant sales (beyond for excess stock clearance/discontinued items) exist largely to generate some excitement from potential customers and to lure them in to taking a look at the site/store and to hopefully get them to buy some extra stuff that's less discounted while they're there.

The actual on-sale item is potentially losing money for the business or at break-even.

------

A person who's running a series of bots like this to track the exact lowest price on the internet for items over time and to always only buy at the lowest price it's ever been sold for....is probably the least likely kind of customer to buy anything extra that's actually profitable for the business.

I have a business where I clearly don’t want any bots. A retailer, maybe they do.

Saying an API follows REST but doesn't implement HATEOAS (part of its definition), is like saying a database follows ACID but doesn't implement Atomicity. Or like saying a software is Open Source but doesn't allow use for commercial purposes.

Yet somehow for one of those 3 cases it became accepted to misuse the term that way ("[original meaning] but without this one constraint"), but if someone misuses the term for the other two cases then everyone suddenly cares and insist that people should use a different term to describe their thing.

Not a rhetorical question, because most companies would welcome that interest from customers.

It was useless in 99.9% of the cases. You don't need or want to have constant access to how much free ram your computer currently has, as if that even meant anything.

So, after a few steps, that leads to the question "what is useful information? How do you define that?"

A basic attempt at it might be: useful information is something that causes you to make or change a previous decision. Take an action you wouldn't have otherwise done.

If you're living in the american southwest, and it's between the months of like, march and, dunno, septemeber, you probably don't need to know the current outside temperature to decide if you're wearing shorts or a jacket when you get up in the morning (you're wearing shorts because it's always hot)

Theoretically there could be some kind of "extreme" weather, aka rain, that causes you to do something different, bring an umbrella, but it's pretty difficult to not notice it currently raining, even if you're inside a house.

So what other information do you want? There's all sorts of stuff that's considered "news", but I'd argue that at least 98% of that is basically just entertainment. We'd need some kind of article or perhaps even a book to hammer out all the nuances involved, but seeing a headline in the morning that trump has caved in to iran again or killed some more random people on a boat, while sort of important to know, doesn't really change what you're going to be doing that day.

Knowing ICE is trying to do some domestic terrorism near where you might be could be useful information, but there's not a real good way to track that.

So yeah, what kind of information is actually useful to have in the morning? Stuff like calendar appointments would count, knowing you have things scheduled at specific moments or tasks to do, go grocery shopping, etc, todo lists in otherwords, but what else? That's basically all self-generated stuff by you about you, you don't really need any outside data sources.

  > it mostly turned into animated brochures instead

No, not browsers, user agents (some of which would be browsers) and there was never an assumption of how a website would behave on the user's end, that's the job the user agent. All that HTML and CSS are only suggestions, but the power was always intended to be left to the user to decide if/how they wanted that data presented to them and it was always intended that the user be able to choose whatever tools they wanted to collect, process, and display content pulled down from the internet. That same principle is how we have ad-blockers. You are free to infest your website with ads, but as the person requesting the website I'm under zero obligation to display any part of that site I don't want.

> LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all.

LLMs are just another tool used by people to collect and process the information available on websites. Maybe there is a distinction to be made between people using LLMs to get web content and corporations scraping websites to take training data, but even scraping has always been a common and expected practice. It's the current scale that is making things different.

To be fair, I do not generally support wholesale banning of scrapers and non-browser clients, since this tends to have lots of ugly side effects.

But regardless, I'm interpreting your last comment as saying that you should be entitled to behave as you please because you are a paying customer. If that's the case, then you are in for a rude awakening one day

So why should I pick an LLM over google's inbuilt LLM or an actual cookbook on the desk? What wasted time is the LLM saving?

So ... most people.

> That's basically all self-generated stuff by you about you, you don't really need any outside data sources.

Other than the ones you just enumerated that most people are likely to use, I need to know - using external sources:

- When to expect the garbage truck (do I put the recycle out tonight, too?)

- Is it a school holiday?

- When it's cheapest for me to charge the car up; if there are any "grid stress, please turn non-essentials off or we're getting rolling blackouts" alerts.

- Same goes for nearby fires or other issues that are going to change the "when do you open/close the windows and switch to AC" math.

- If it's likely to rain in the next few days so I change what plants get watered / when

- Packages that I am likely to get in the post today

- Weather (for display, not just for use in the automations mentioned above)

- Where my power provider is currently getting its power from (so I can calculate estimated CO2 emissions, if any)

And that's about 1/2, probably. Can't be bothered to go spend 20 min pouring through my HA config to build the full list.

I also stream the npr newscast as an MP3 as part of the "good morning, while you wait for coffee..." automations, if that counts for anything.

Almost certainly! No eComm site wants the cautious shopper that's done their comparisons and knows price trends; they want the impulsive people with a line of credit. I doubt that my local grocery store is happy that I index all of my receipts so I know what I buy, how often and when the prices change so I know when a discount on something I buy every once in a while is actually a discount.

When I built this a few years ago, I had to write a lot of code to make all this work.

Now that's a simple "hey $agent, check the price of https://some-site.com/appliances/that-new-fridge.html and https://other-site.com/shop/appliances/that-new-fridge.aspx daily and notify me when it's on sale or has a promotion"

I would expect that more and more people will discover that their LLM of choice can do this, that their LLM of choice can help them comparison shop and do analysis of when - historically - is the best time to buy that new TV or whatever (yes, pricing on this stuff is seasonal!).

Maybe _most_ people will never do this, maybe I'm wrong. But if i'm not, blocking all bots "because bots are bad, mmmkay" is how you loose a sale.

Ignoring the specific price data example, more and more and more people _are_ browsing the web through an LLM. Assuming that continues, blocking bots is going to have the same effect on your traffic as doing everything in the "what not to do" section of the google SEO guide book.

That's the thing; they can't, really.

Best we can do is compromise; allow bots and rate limit them. Maybe someday there will be a relatively universal standard for fetching pages akin to the `index.html/index.md` pattern that is emerging now. Put the bot filters on the .html path and some sort of micro transactions system on the .md endpoint.

In basically every cat/mouse game defense is forever one step behind...

To give a concrete example: I am a physician and have a site for my private clinic. I am not only unopposed to being scraped by bots/LLMs, I prefer it. The only value I gain from users being on my site is referral to my practice, and that same information being surfaced by search engines and LLMs just means I'm getting in front of more potential patients.

I've also been known to include some ground up rocks in my meals. I'm pretty picky though about them, I mostly just want a couple specific varieties.

The bot was also being generally courteous, only scraping the 2-3 pages needed to get the information for the human who spun up the LLM.

I don't think GP was at all suggesting he should be able to act however he wants, just because he's a paying customer.

I run an ecommerce site and I'm actually more worried about bots being rejected than accepted because of stuff like GP. We get a decent amount of business through ChatGPT.

If you use google (sans AI), you're putting some trust in their page ranking algorithm. If you use it with AI, you're trusting the same algorithm (since that's how the model gets it's sources), but then you're trusting the model to evaluate the sources for credibility and extract the information you actually want.

No. Neither can be trusted as far as you can throw them. They're both incredibly invasive data brokers. Their customer facing products are just vehicles to show ads and Hoover up more PII and behavioral data on everyone.