The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0.
Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.
This is a good thing, thanks to this we have powerful open source LLMs.
> This activity overwhelms sites with traffic.
When LLMs get good enough, we won't need those sites anymore :)
[not satire, this is what I think, without self-censorship]
Backbone operators should not be allowed to knowingly maintain connections to networks that allow connections from China or Russia.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.
The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
EDIT: Lwn.net is perhaps not a fair target of my ire.
“There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.”
Is reasonable! Many others are not
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.
Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...
That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).
[0]: https://github.com/jow-/nlbwmon
[1]: https://www.maxchernoff.ca/files/grafana-network-bandwidth.p...
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
WTF. That's just botnets.
Source: https://www.fbi.gov/investigate/cyber/alerts/2026/evading-re...
Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet
Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
https://news.ycombinator.com/newsguidelines.html
Edit: If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future. They're here: https://news.ycombinator.com/newsguidelines.html.
I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...
Unfortunately whatever HN is using routinely blocks my login with "Sorry."
some websites just always give me 403.
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
I've only just started using it, and the existing config DSL doesn't let you do this, but it's just a patch away (in principle): instead of using system load as a signal to increase or decrease weights, use a combination of finalized weight and system load (or a load-equivalent or net-traffic-equivalent signal from some openmetrics agent) to select between difficulty buckets.
So, as an example, difficulty could have an arbitrary scale independent of load, and regardless of that scale, it could increase by 1 at 50% cpu, 2 at 80% cpu, etc. I'm also not sure currently if a challenge will re-appear if anubis determines that a client should get a difficulty N challenge, but they only have a token for an N-1 or N-2 challenge. Easy enough to implement in principle, I just haven't looked at the code at all. I shall test it to see if it already does that.
I believe that's the HN application itself, not a WAF in front of it.
Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.
Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough.
However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.
If they have consent, they're not really botnets. Botnets involve infecting devices without the owners knowing.
With consent, it wouldn't be much different from e.g. open WiFis at restaurants and hotels, companies using a single ISP and single public IPv4 address for all their employees, and most VPN services.
unethical yes but really raises the question as to what we see is real or not
We agree that it would be great if it was even more widely used.
Im not against the ban perse (single purpose accounts are bad), just curious if they had a chance to change their contribution style.
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
I wasn't aware of this project. Thanks for the heads up.
Really makes you think, what we're feeding them...
its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective
also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities
I did an experiment and linked from HN to my lame blog site and disabled all my anti-scraping measures. Even with all the bots I did not see that much traffic. I suspect some people are specifically being targeted by very poorly configured or very poorly written archiving scripts. Just one example thread discussing this with someone on HN [1]. Each case of being targeted will require looking at generalized characteristics but most are easy to stop in my opinion and experience.
I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...).
Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time.
Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today.
I reported on findings at the time and several times since:
<https://news.ycombinator.com/item?id=36078578>
<https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>
The cheapest way to get a VPN (and if you're a horny and broke teenager perhaps the only way) is to trade your clean but censored IP address for an uncensored IP address in another country. You accept the bot traffic in return, or externalize it to your parents or the owner of the internet connection.
Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same address after the initial one?
The main problem becomes managing lists of millions of individual addresses. My (only semi-reliable these days, due to lack of time for maintenance) little project has nearly 2.3 million addresses recorded - although only 590k are from 2026, and only 38 were probes on ports 80 and 443. So maybe more manageable than I thought (but my servers don't host anything beyond personal interest to me, and access is filtered via cloudflare, which is it's own "internet control issue").
> In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.
Overall, my gut feel on residential proxies is that they're an untrustworthy scourge. I'd be interested in any arguments for residential proxies by people who don't (intend to) profit from using it facilitating them.
In regards to Bright Data, one of the companies that attempts to appear legitimate, at minimum these domains should be blocked:
brdtnet.com
luminatinet.com
bright-sdk.com
luminati.io
As listed in this article, on HN's front page 34 days ago: https://news.ycombinator.com/item?id=48422993 (https://blog.includesecurity.com/2026/06/the-smart-tv-in-you...)
Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads.
The question is which resources typical consumer hardware has that large centralized compute power does not. In-browser POW to pay websites would only be possible if such a resource exists.
I am not familiar with the topic, but maybe CPU power and memory? Both seem significant in a typical consumer device.
Napkin math: If a consumer device can generate $100 per month, that would be 100/30/24/60/60=$0.00004 per second. If the user waits for 5 seconds before the first pageview, that would then make the website provider $0.0002 per visitor. Serving a million visitors per month is nowadays easily possible on a $10/month machine. So the $0.0002x1000000 = $200 would make the website a nice profit.
I use a datacenter-based IPv6 address because my local ISPs don't offer v6 connectivity and the Internet is already broken for me. And generally the entire idea of a "residential" IP address smells.
residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).
This is such a malicious interpretation. Do you think VPN operating are also trying to attack websites? Both offer the same kind of product.
>paid for hijacking their users' network connections
Nothing is being hijacked. Again the author is using wording to try and paint these people as malicious actors.
>Recently, LWN was subjected what was, by far, the heaviest scraper attack yet.
LWN is a static site. To me it seems more expensive to use Anubis than just serve the actual page.
>will now check for NetNut-infected apps
Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.
<https://news.ycombinator.com/item?id=32048148>
<https://news.ycombinator.com/item?id=32031243>
(AFAIK that specific failure mode has in fact been addressed.)
The users presumably don't know about this, or you know, they clicked, "I agree."
Nearly Half of LG Smart TV Apps Contain Residential Proxies
- i know many people who buy shady IPTV boxes from stores/markets for like 50€/year
- i know some people who use "smart lightbulbs" and other nonsense
- almost everyone i know plays free smartphone games, which as LWN reminded, may contain a shady SDK
I made it all the way back to https://news.ycombinator.com/posts?id=atomic128&next=4628060... (6 months ago) before seeing posts about anything else, only to find that there was a different agenda before that. Not cool.
Edit: and before all that, there was this: https://news.ycombinator.com/posts?id=atomic128&next=4164795.... This is obviously not using HN as intended.
I took a look at the most recent comments from both accounts and they don't look similar to me in this respect.
I think there are two questions here though:
1. Was the violation egregious?
2. Did it deserve an immediate ban, or did they deserve a warning etc.?
Seems to me the answer to (1) is yes, but the answer to (2) I'm less sure about.
It's just a sign that single-agenda accounts aren't allowed here—no more, no less. That's why I said "We ban such accounts regardless of what the single purpose happens to be".
In other words, they don't care at all. For them, residential bandwidth is completely free.
Is that a side effect of whatever you are doing?
They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.
No they really don't, dishonest founders do that.
You're one with the lower case shibboleth so I have no doubt you surround yourself with dishonest founders, but faking users is pretty damn low on the usecases for residential proxies.
I said they're unethical because they tend to be hidden in innocuous seeming apps or sprung on unwitting individuals via clickwraps on their smart devices.
At which point, millions of people will be forced to complain to their local representatives and... hey presto? :)
I use them to scrape closed sites to make the information more open. For example YouTube.
Instead, exchange web traffic for actual $. Say, some kind of tokens that are easily turned back into hard cash through a 3rd party.
Requesting a 100KB file? Okay, that'll be a $0.00002 token, please! (visitor's user agent provides it in a manner transparent to regular web users). Requesting a 3MB image? Okay, that'll be a $0.0005 token, please!
Result: niche websites earn hard cash. It doesn't matter much if you're hammered as long as the hammering comes with a corresponding flow of tokens (read: $). No token(s)? No service.
Regular web users would pay for those tokens through their normal internet service fees, and otherwise not be bothered. Massive scrapers would have to pay somehow for the tokens to be served web data at all.
In effect: put the bulk of public web sites behind a paywall. But with the bar low enough & in a manner that it's transparent for regular web users. Clicked "reload" by accident? Oops, internet service bill got upped by 2 micro-$.
Also all major browsers block crypto miners on webpages now (for good reasons) so it may prove difficult to allow "good" mining scripts while still blocking "bad" ones.
I don't think this is a practical solution
There's no easy answer here. The ephemerality and pseudonymity of VPS address usage screams untrustworthy, and the only way to reign that in is better identification of who is using the VPS/address or significantly more restrictive rules applied to data/port usage. And I'm not sure if I like the general direction that points towards - away from the "open internet".
This is about filtering out bad bots/actors who have no respect for your resources and will drain all of it causing bad experience for everyone. But because they know they don't respect robots.txt or even simple rate-limiting, they have to employ so-called residential VPNs. They're residential in that they route through real user connections, and so you can't block the IP/subnet without dropping a certain amount of legitimate human-driven traffic.
Personal example: some time ago, i had to disable a wordpress plugin on a site that was causing 100% CPU usage on the whole box (hosting dozens of wordpress instances). That plugin was a simple calendar, but a bot was repeatedly scraping non-existent (or rather, "no event planned for this day") pages for every date in the calendar that you can represent in the DB timestamp, clearing the cache as it went to try and find new events for 1000 years ago. Whoever operates this IP space doesn't matter to me, i'd just like to block them because they don't respect robot.txt… but i can't because they use a "residential proxy" and will change IP address every hour or so.
Probably as simple as the fact that there are unmetered residential proxy plans, which means once you're already paying for one, there's no reason not to use it for everything.
There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?
The article was called "The one we're commenting on"
but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks
then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection
HN's prime directive is "anything that gratifies one's intellectual curiosity": <https://news.ycombinator.com/newsguidelines.html> and many, many, many dang comments.
I'm pretty sure that the specific gripe is posting excessively (not even necessarily exclusively) on a single topic or theme. See <https://news.ycombinator.com/item?id=19392902> for a more detailed comment from dang.
Occasional alts are explicitly permitted, though not to engage in abuse (e.g., mutual admiration societies, sock-puppetry). See: <https://news.ycombinator.com/item?id=9963551> <https://news.ycombinator.com/item?id=9823379> (both against sock puppetry) and <https://news.ycombinator.com/item?id=9122086> and <https://news.ycombinator.com/item?id=7504621> (on where throwaways are/aren't permitted).
Where HN does favour persistent accounts the stated claim is to foster community, rather than for nefarious tracking purposes: <https://news.ycombinator.com/item?id=18082346> and <https://news.ycombinator.com/newsguidelines.html>. From that last:
Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to.
If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network.
It doesn't seem like monopoly abuse to exclude such malware from application stores, just like key loggers or apps intercepting other apps network traffic without the user being aware of it (say the banking app's network traffic and password entry).
I have such system for the registration form on one of my website to prevent the double validation of emails to be used to spam emails of victims. The PoW challenge prevents less than 10% of the bots.
Welcome to LWN.net
The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net!
Our article "Fighting the AI scraper bot scourge", published in early 2025, discussed the problem of widespread scraping of web sites in search of training data for large language models and related projects. This activity overwhelms sites with traffic. Over a year after that article is published, the problem is still growing. The hammering of sites by shadowy actors has reached new heights, and the open web is becoming increasingly difficult to maintain. Where is this traffic coming from, and what can be done about it?
As was described last year, scraper attacks come from a huge number of sources across the net. It is not unusual to see coordinated requests from millions of unique IP addresses over the course of a few hours, each of which hits the site at most two or three times. Attacker-controlled data, such as the user-agent field, is entirely fictional; each hit is meant to look like just another human with a web browser. There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
This traffic comes predominantly from residential and mobile networks, directed by central command-and-control nodes. Software is installed on ordinary systems that takes orders from a control node, fetches web pages on demand, and forwards the resulting data back to the controller. Much of the time, this activity occurs without the knowledge or consent of the owner of the device in question. The term "residential proxies" is used to describe systems that are used in this way.
There are a few different (on the surface, at least) types of operator running residential-proxy networks to attack web sites. One type is purely criminal, running scrapers on systems that have been compromised with some sort of malware. At the beginning of the year, Google acted to take down a bot network called IPIDEA and provided a lot of information about how these operations work. The shutdown of IPIDEA correlated with a significant reduction in scraper traffic here at LWN; things were relatively peaceful for a few months. That period of peace has since come to an end, though.
More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
The second sort of operator works more overtly, pretending to a degree of legitimacy and offering "ethically sourced" IP addresses. A company called Bright Data is one of the most prominent of these; it happily advertises its prowess at getting around web-site access controls and traffic limits. Bright Data offers a "free" VPN service; all that is needed is for the user to give Bright Data the ability to route traffic through the user's device — to become a part of the company's residential-proxy network, in other words. Every phone or other device that makes use of this VPN becomes yet another endpoint that will be used to attack web sites.
There are many other examples of this type of operator out there; often they offer a library that app developers can link into their offerings and be paid for hijacking their users' network connections. One of them even sent us a query about running an ad for its SDK on LWN; that was, it suffices to say, a short conversation. In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.
While these residential-proxy networks are used for web-site scraping, it is worth emphasizing that these operators have the ability to run code that accesses resources on whatever networks millions of devices happen to be connected to. To assume that this type of access would only be used for scraping would be naive at best.
Then, of course, there are the high-profile companies developing models as their core business. These companies do their own scraping; the traffic that can be easily attributed to them is clearly identified in the user-agent field and, as a general rule, observes measures like robots.txt. They, too, will scrape an entire site, repeatedly, seemingly on the theory that articles written in 2003 might somehow have changed in the last day, but they do not generate overwhelming amounts of traffic from millions of systems and are not the biggest problem.
What isn't clear is who is using the residential proxies; somebody is paying them to run these attacks on web sites. There is no evidence (that I am aware of) that the frontier-model companies are using those networks. If it were to turn out that they are doing so, though, the increase in global astonishment would barely register. Those companies are feeding their models somehow, they are not forthcoming about how they get their training data, and they have not distinguished themselves with their level of respect toward content creators — or toward anybody who might have concerns about their operations.
For every public model, though, there must be a vast number of undercover models. Many companies are surely trying to build their own; after all, we are reliably informed that AI is going to take over the world and the companies that come out on top of that race will be worth untold amounts of money. There must be shadowy government agencies in many countries working on their own models and groping for training data wherever they can find it. Large-scale criminal organizations (to the extent that they are distinct from governments) probably also want to have their own models. These tools are seen as weapons, and there is an arms race underway. The Internet as a whole is caught in the crossfire.
In response to all of this, web-site operators have been scrambling to defend their sites while minimizing the effect on their actual users. Anubis, which attempts to fend off scrapers by requiring a proof of work, is now widespread. Other sites use commercial services, which sometimes make themselves known with a "prove you are human" button. Or sites force users to pick out squares containing streetlights (but only those with LED bulbs), place puzzle pieces, or hum a song while holding down the space bar. Many site features have been placed behind login gates or paywalls. Some sites attempt to actively poison the data sent to scrapers with tools like iocaine.
Both the need to set up and maintain these mechanisms, and the requirement that users cope with them to access a web site, constitute a heavy tax placed on the world as a whole by scrapers and those who pay them.
Recently, LWN was subjected what was, by far, the heaviest scraper attack yet. Thanks to the defenses that have been implemented, the site bore the traffic well enough that most actual readers probably did not even notice. There have been requests to describe the measures we have taken to defend the site; for obvious reasons we do not wish to discuss them in any detail. It is an arms race at this level too.
What we can say is that we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
There is also a desire to not impede the operation of legitimate search engines, the Internet Archive, and other such groups. Some sites may add explicit allowlists to, for example, give the dominant search engine access to the site. Such measures have the effect of further entrenching a monopoly that already serves us poorly and should be avoided. We have, thus far, succeeded in that.
We have aggressively optimized parts of the site, and found ways to minimize expensive operations during times when the site is under attack. Anonymous readers may occasionally encounter one of those measures; logged-in users will not. Amusingly, the response time when the site is under attack is often better than during the calm times, when the defensive measures are dormant. We have learned better than to think that the problem is solved, though; consideration must be given to our next steps once the current measures are no longer effective.
On July 2, Google announced that it had, in coordination with the US Federal Bureau of Investigation and others, taken down a residential-proxy network called "NetNut". For the time being, that action would, indeed, seem to have succeeded in reducing the level of scraper attacks somewhat. Experience shows, though, that this welcome peace will only last so long. Google takes pains to point out that its Play Store will now check for NetNut-infected apps, but all of the major vendors are silent on the topic of why it is so easy to put apps with residential-proxy functionality into their app stores.
It would be good to find a more lasting solution before the entire Internet is driven behind defensive walls, and the open network that inspired so much creativity is lost. The industry that is driving these attacks seems entirely at ease with turning independent web sites into smoking craters after having pillaged their contents — an attitude that extends to the planet and its economies as well. Some of us, though, object to that idea and will fight against it. Someday, with luck, the world as a whole will decide to hold the companies behind large language models and related technologies to a minimal ethical standard. Until then, though, this behavior will continue, and we will have no choice but to defend ourselves against it.
Does that sound like your typical self-hosted blog?
I've not worked with the API, and there's the blessing/curse (blurse‽) that HTML is a known, if poor, standard.
API always translates to "one more thing to learn, that's applicable to a single-use case". HTML scraping / sorting I can apply across multiple sites.
That said, a standard, say, JSON packaging of website contents available on request might be fun to have.
"ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks"
You've never seen this, making this non-sequitur? Or this "directionally correct by my priors" speak for something else?
-
Either way it has nothing to do with why the proxy is unethical (I already covered that) and everything to do with why their startup is unethical
(or not? if they're not being assholes about the scraping more power to them, I've used proxies + agents to let my users get their own data off other sites that felt it was their right to block people from accessing their own data easily)
That is not what the SDK was doing. The actual code in the SDK protects against this (simplified to take less space):
if (addr.isSiteLocalAddress() || addr.isLoopbackAddress()) {
LogUtils.e("PopaTunnelAsyncThread", "Hacking? The Host Resolved Ip is " + addr + " on tunnel id:" + tunnelId);
throw new IllegalArgumentException("Hacking? The tunnel host resolved ip is internal");
}
As long as the website gets paid more than the cost of serving the pages, it does not matter if a human or a bot did the POW.
Securing signup forms is another issue. Maybe related. But not what I was referring to.
> partly because it causes annoying delays for those trying to get to the site
This is true but usually a small issue. It’s further alleviated by cached tokens so you only have to solve the challenge once in a while per site, and a login token may let you skip it.
> partly because it seems inevitable that the scrapers will eventually find their way around it…A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
Solved by making money off it.
Also users might become part (victim?) of a police investigation because of illegal actions that seem to originate from their local residential connection.
So still good to take down such backdoors. Would be nice to go after the botnet operators as well...