🚨 𝐁𝐢𝐠𝐐𝐮𝐞𝐫𝐲’𝐬 𝐑𝐈𝐃𝐈𝐂𝐔𝐋𝐎𝐔𝐒 𝐩𝐫𝐢𝐜𝐢𝐧𝐠 𝐦𝐨𝐝𝐞𝐥… | Yingjun Wu | 681 comments
Agree & Join LinkedIn
By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.
[Skip to main content](https://www.linkedin.com/posts/yingjun-wu_%F0%9D%90%81%F0%9D%90%A2%F0%9D%90%A0%F0%9D%90%90%F0%9D%90%AE%F0%9D%90%9E%F0%9D%90%AB%F0%9D%90%B2%F0%9D%90%AC-%F0%9D%90%91%F0%9D%90%88%F0%9D%90%83%F0%9D%90%88%F0%9D%90%82%F0%9D%90%94%F0%9D%90%8B%F0%9D%90%8E%F0%9D%90%94-activity-7307736315079405569-F2Ng#main-content)
LinkedIn
Join now Sign in
Yingjun Wu’s Post
Yingjun Wu
Founder @ RisingWave. Stream processing, real-time lakehouse & AI.
1w
🚨 𝐁𝐢𝐠𝐐𝐮𝐞𝐫𝐲’𝐬 𝐑𝐈𝐃𝐈𝐂𝐔𝐋𝐎𝐔𝐒 𝐩𝐫𝐢𝐜𝐢𝐧𝐠 𝐦𝐨𝐝𝐞𝐥 𝐜𝐨𝐬𝐭 𝐮𝐬 $10,000 𝐢𝐧 𝐣𝐮𝐬𝐭 22 𝐬𝐞𝐜𝐨𝐧𝐝𝐬!!! 🚨 I am serious—𝐭𝐡𝐢𝐬 𝐢𝐬 𝐫𝐞𝐚𝐥. Read our experience, and maybe it could help you avoid getting overcharged silently!!! Last month, we were helping a customer build a data pipeline. We needed some sample data from a large public table. Knowing the data could be huge, we carefully used the 𝐋𝐈𝐌𝐈𝐓 statement, which should only scan 100𝐊 rows. The query ran fast, nothing seemed off. We ran it three times. Then the bill came. And 𝐰𝐞 𝐰𝐞𝐫𝐞 𝐬𝐡𝐨𝐜𝐤𝐞𝐝. 🔥🔥 𝐁𝐢𝐠𝐐𝐮𝐞𝐫𝐲 𝐜𝐡𝐚𝐫𝐠𝐞𝐝 𝐮𝐬 $9,847.24!!! 🔥🔥 🔥🔥 𝐓𝐡𝐫𝐞𝐞 𝐪𝐮𝐞𝐫𝐢𝐞𝐬 𝐬𝐜𝐚𝐧𝐧𝐞𝐝 1,576.56𝐓𝐁 𝐨𝐟 𝐝𝐚𝐭𝐚?! 🔥🔥 The breakdown was 𝐞𝐯𝐞𝐧 𝐜𝐫𝐚𝐳𝐢𝐞𝐫: for each query—with LIMIT—𝐁𝐢𝐠𝐐𝐮𝐞𝐫𝐲 𝐛𝐢𝐥𝐥𝐞𝐝 𝐮𝐬 𝐟𝐨𝐫 509.89𝐓𝐁 𝐨𝐟 𝐝𝐚𝐭𝐚, 𝐚𝐥𝐥 𝐰𝐢𝐭𝐡𝐢𝐧 22 𝐬𝐞𝐜𝐨𝐧𝐝𝐬. 𝐓𝐡𝐚𝐭’𝐬 23𝐓𝐁 𝐩𝐞𝐫 𝐬𝐞𝐜𝐨𝐧𝐝!!! How is this even possible?! We were speechless. Obviously, BigQuery is one of the most powerful data warehouses, and it definitely knows how to process queries efficiently. It must have already pushed down the LIMIT operator and returned results early. There must be something wrong with their pricing model. After checking with my friends at Google, I finally realized the trap: BigQuery 𝐜𝐡𝐚𝐫𝐠𝐞𝐬 𝐛𝐚𝐬𝐞𝐝 𝐨𝐧 𝐫𝐞𝐟𝐞𝐫𝐞𝐧𝐜𝐞𝐝 𝐝𝐚𝐭𝐚, not processed data! 💢 What does that mean? 💢 If you have a 1PB table, even if you only scan a small portion, BigQuery still charges you as if you scanned the entire 1PB. And guess what? Google explicitly states this in their pricing page!!! 𝐖𝐡𝐚𝐭 𝐚 𝐣𝐨𝐤𝐞!!! 🤡 🤡 🤡 For anyone with a non-GCP background, this is completely unexpected. I worked with Amazon Web Services (AWS) Redshift before, and Redshift and Athena never charge people like that. Even my friends at Databricks and Snowflake think this pricing model is ridiculous. Friends, cloud providers lure you in with seemingly cheap pricing models, but they are full of traps that go against common sense and silently drain your money. I have no idea why BigQuery’s GTM team would do this so openly, but if you use BigQuery, check your bill carefully. Share your experience. Cloud credits aren’t free—you will eventually pay them back. Evaluate third-party solutions, consider open formats like Apache Iceberg, and avoid vendor lock-in before it’s too late. 𝐓𝐡𝐢𝐬 𝐢𝐬 𝐧𝐨𝐭 𝐦𝐲 𝐬𝐚𝐥𝐞𝐬 𝐩𝐢𝐭𝐜𝐡. 𝐓𝐡𝐢𝐬 𝐢𝐬 𝐫𝐞𝐚𝐥 𝐦𝐨𝐧𝐞𝐲. 𝐃𝐨𝐧’𝐭 𝐠𝐞𝐭 𝐭𝐫𝐢𝐜𝐤𝐞𝐝 𝐛𝐲 𝐁𝐢𝐠𝐐𝐮𝐞𝐫𝐲!
4,692 681 Comments
Like Comment
Share
- Copy
- LinkedIn
- Facebook
- Twitter
Yingjun WuFounder @ RisingWave. Stream processing, real-time lakehouse & AI.
1w
Ippokratis Pandis any opinion on this? 🤔
LikeReply7 Reactions 8 Reactions
Yingjun WuFounder @ RisingWave. Stream processing, real-time lakehouse & AI.
1w
Folks - some disclaimers: - I'm not trying to convince you to ditch BigQuery - make your own decision - I used to work for AWS Redshift but now run my own stream processing company, RisingWave (risingwave.com) - I own stock shares in both Google and Amazon - AWS, Snowflake, and Databricks aren’t paying me commissions
LikeReply52 Reactions 53 Reactions
Yaron LeviCo-Founder & Chief Architect at YuviTal
1w
Now we all understand what makes Snowflake's pricing model so good. It's predictable!
LikeReply55 Reactions 56 Reactions
Ankur RanjanSoftware Engineer by heart, Data Engineer by mind
1w
That's why I feel SQL-Guadrail configs should be present for every query engine but I think the company simply doesn't want to build it only. They are well aware of these things and Guardrails are not NP-hard problems. It can be solved but no company wants to solve it. Yingjun Wu
LikeReply28 Reactions 29 Reactions
Emanuele Melis 💻Got Data? Ai Ai Ai! I build Data platforms by day and talk about data, GenAI, kdb+ and pizza by night
1w
It’s rough Yingjun Wu - speak with your SA, and hope for the best! Add reservation caps. Or Use tablesample instead of limit: SELECT * FROM dataset.my_table TABLESAMPLE SYSTEM (10 PERCENT)
LikeReply31 Reactions 32 Reactions
Balazs VajnaHead of Analytics at MarketingLens | BigQuery guy
1w
There are more AWS and Snowflake horror stories than BigQuery horror stories. But yes this could happen with enormous data sources. You seem to have picked one of the worst ones (it’s rare to find such huge data among the public datasets). Like Constantin Lungu said, try reaching out to Google and let them know this was a “beginner’s mistake”, they might reimburse you.
LikeReply1 Reaction 2 Reactions
Rishikeshan Sulochana/LavakumarAI Engineer
1w
Primitive technology: does row_number() work? People used to use it in place of LIMIT when LIMIT was not a thing, at least MSSQL...
LikeReply1 Reaction 2 Reactions
✨Shane GibsonAgile Data Coach. I help data teams change their Way of Working, to deliver more, in less time, while having more fun! | Reach out if I can help your team improve the way they work.
1w
Um in the old days the grumpy DBA used to tell us to RTFM. A cursory read of the doco would shown you that the limit statement does not limit your cost of queries. In BQ Studio you can see how much the query is going to query, before you run the query. So any unfavourable sayings I want to use right now. So ill say it another more positive way. We have run Big Query for over 6 years and we have run it at a cost that we could not run our workload for on AWS.
LikeReply5 Reactions 6 Reactions
See more commentsTo view or add a comment, sign in
10,000 followers
View Profile Follow
Explore topics
Sign in to view more content
Create your free account or sign in to continue your search
Sign in
Welcome back
Email or phone
Password
Show
Forgot password? Sign in
or
By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.
New to LinkedIn? Join now
or
New to LinkedIn? Join now
By clicking Continue to join or sign in, you agree to LinkedIn’s User Agreement, Privacy Policy, and Cookie Policy.