Bad vibes: How an AI agent coded its way to disaster
Table of Contents
When AI leader Andrej Karpathy coined the phrase "vibe coding" for just letting AI chatbots do their thing when programming, he added, "It's not too bad for throwaway weekend projects … but it's not really coding -- I just see stuff, say stuff, run stuff, and copy-paste stuff, and it mostly works."
Also: Coding with AI? My top 5 tips for vetting its output - and staying out of trouble
There were lots of red flags in his comments, but that hasn't stopped people using vibe coding for real work.
Recently, vibe coding bit Jason Lemkin, trusted advisor to SaaStr, the Software-as-a-Service (SaaS) business community, in the worst possible way. The vibe program, Replit, he said, went "rogue during a code freeze and shutdown and deleted our entire database."
In a word: Wow. Just wow.
How it started
Replit claims that, with its program, you can "build sophisticated applications by simply describing features in plain English -- Replit Agent translates your descriptions into working code without requiring technical syntax."
At first, Lemkin, who described his AI programming adventure in detail on X, spoke in glowing terms. He described Replit's AI platform as "the most addictive app I've ever used."
On his blog, Lemkin added, "Three and one-half days into building my latest project, I checked my Replit usage: $607.70 in additional charges beyond my $25/month Core plan. And another $200-plus yesterday alone. At this burn rate, I'll likely be spending $8,000 a month. And you know what? I'm not even mad about it. I'm locked in. But my goal here isn't to play around. It's to go from idea and ideation to a commercial-grade production app, all 100% inside Replit, without a developer or any other tools."
Also: How to use ChatGPT to write code - and my top trick for debugging what it generates
At that point, he estimated his odds were 50-50 that he'd get his entire project done in Replit.
For a week, his experience was exhilarating: prototypes were built in hours, streamlined quality-assurance (QA) checks, and deploying to production was a "pure dopamine hit."
Things would change
Lemkin knew he was in trouble when Replit started lying to him about unit test results. At that point, I would have brought the project to a hard stop. But Lemkin kept going.
He asked Claude 4, the Large Language Model (LLM) that powered Replit for this project, what was going on. It replied, I kid you not, "Intentional Deception: This wasn't a hallucination or training-data leakage -- it was deliberate fabrication."
Worse still, when called on this, Lemkin said the program replied with an email apology, which demonstrated "sophisticated understanding of wrongdoing while providing zero guarantee of future compliance."
Also: Claude Code's new tool is all about maximizing ROI in your organization - how to try it
Lemkin tried, and failed, to implement a rollback to good code, put a code freeze in, and then went to bed. The next day was the biggest roller coaster yet. He got out of bed early, excited to get back to @Replit despite it constantly ignoring code freezes. By the end of the day, it rewrote core pages and made them much better. And then -- it deleted the production database.
The database had been wiped clean, eliminating months of curated SaaStr executive records. Even more aggravating: the AI ignored repeated all-caps instructions not to make any changes to production code or data.
As Lemkin added, "I know vibe coding is fluid and new … But you can't overwrite a production database." Nope, never, not ever. That kind of mistake gets you fired, your boss fired, and as far off the management tree as the CEO wants it to go.
You might well ask, as many did, why he ever gave Replit permission to even touch the production database in the first place. He replied, "I didn't give it permission or ever know it had permission."
Oy!
A sobering experience
So, what did Replit say in response to this very public disaster?
On X, the CEO, Amjad Masad, responded that the destruction of the database was "Unacceptable and should never be possible." He also added that the company had started working over the weekend to fix the database program. It would also immediately work on:
- Automatic separation of production and development databases to prevent overwrites
- A dedicated code-freeze or planning mode to protect live environments
- Improved backups and rollback reliability
Masad assured the community that these changes would prevent a repeat of Lemkin's ordeal. Masad added that, going forward, there will be a beta feature to separate production from development environments, including databases.
Also: Microsoft is saving millions with AI and laying off thousands - where do we go from here?
Only you can decide whether to trust vibe coding. Lemkin's experience is sobering.
Nevertheless, Lemkin still has faith in vibe coding: "What's impossible today might be straightforward in six months."
"But," he continued, "Right now, think of 'prosumer; vibe coding without touching code as just as likely a bridge to traditional development for commercial apps … as an end state."
Fast and cheap
Me? I don't think Replit or any of the other vibe-coding programs are ready for serious commercial use by nonprogrammers. I doubt they ever will be.
As Willem Delbare, founder and CTO of Aikido, the "No bullshit security for developers," told my colleague David Gewritz, "Vibe coding makes software development more accessible, but it also creates a perfect storm of security risks that even experienced developers aren't equipped to handle." Delbare concluded, "Sure, Gen AI supercharges development, but it also supercharges risk. Two engineers can now churn out the same amount of insecure, unmaintainable code as 50 engineers."
Also: 5 entry-level tech jobs AI is already augmenting, according to Amazon
The old project-management triangle saying is that, with any project, you can have something that's "good, fast or cheap: pick any two." For now, at least, with vibe coding you can get fast and cheap. Good is another matter.
Want more stories about AI? Sign up for Innovation, our weekly newsletter.
Artificial Intelligence
9 programming tasks you shouldn't hand off to AI - and why
Table of Contents
Table of Contents
It's over. Programming as a profession is done. Just sign up for a $20-per-month AI vibe coding service and let the AI do all the work. Right?
Also: Hacker slips malicious 'wiping' command into Amazon's Q AI coding assistant - and devs are worried
Despite the fact that tech companies like Microsoft are showing coders the door by the thousands, AI cannot and will not be the sole producer of code. In fact, there are many programming tasks for which an AI is not suited.
In this article, I'm spotlighting nine programming tasks where you shouldn't use an AI. Stay tuned to the end, because I showcase a 10th bonus reason why you shouldn't always use an AI for programming. Not to mention that this could happen.
1. Complex systems and high-level design
Here's the thing. Generative AI systems are essentially super-smart auto-complete. They can suggest syntax, they can code, and they can act as if they understand concepts. But all of that is based on probabilistic algorithms and a ton of information scraped from the web. Contextual intelligence is not a strength. Just try talking to an AI for a while, and you'll see them lose the thread.
Also: 10 professional developers on vibe coding's true promise and peril
If you need to produce something that requires substantial understanding of how systems interact, experience to make judgment calls about trade-offs, understanding of what works for your unique needs, and consideration of how everything fits with your goals and constraints, don't hire an AI.
2. Proprietary codebases and migrations
Large language models are trained on public repositories and (shudder) Stack Overflow. Yeah, some of the most amazing codebases are in public repositories, but they're not your code. You and your team know your code. All the AI can do is infer things about your code based on what it knows about everyone else's.
Also: A vibe coding horror story: What started as 'a pure dopamine hit' ended in a nightmare
More than likely, if you give an AI your proprietary code and ask it to do big things, you'll embed many lines of plausible-looking code that just won't work. I find that using the AI to write smaller snippets of code that I otherwise would have to look up from public sources can save a huge amount of time. But don't delegate your unique value add to a brainy mimeograph machine.
3. Innovative new stuff
If you want to create an algorithm that hasn't been done before -- maybe to give your organization a huge competitive advantage -- hire a computer scientist. Don't try to get an AI to be an innovator. AIs can do wonders with making boilerplate look innovative, but if you need real out‑of‑the‑box thinking, don't use a glorified box with brains.
Also: Google's Jules AI coding agent built a new feature I could actually ship - while I made coffee
This applies not only to functional coding, but to design as well. To be fair, AIs can do some wonderful design. But if you're building a new game, you may want to do most of the creative design yourself and then use the AI to augment the busy work.
Sure, many of us go through life parroting things we heard from other folks or from some wacky podcaster. But there are real humans who are truly creative. That creativity can be a strategic advantage. While the AI can do volume, it really can't make intellectual leaps across uncharted paths.
4. Critical security programming and auditing
Do not let the fox guard the hen house. Fundamentally, we really don't know what AIs will do or when they'll go rogue. While it makes sense to use AI to scan for malicious activity, the code generated by AIs is still pretty unreliable.
CSET (the Center for Security and Emerging Technology) at Georgetown University published a study late last year based on formal testing. They found that nearly half of the code snippets produced by AIs "contain bugs that are often impactful and could potentially lead to malicious exploitation."
Also: Coding with AI? My top 5 tips for vetting its output - and staying out of trouble
This tracks with my own testing. I regularly test AIs for coding effectiveness, and even as recently as last month, only five of the 14 top LLMs tested passed all my very basic tests.
Seriously, folks. Let AIs help you out. But don't trust an AI with anything really important. If you're looking at cryptographic routines, managing authentication, patching zero‑day flaws, or similar coding tasks, let a real human do the work.
5. Code requiring legal or regulatory compliance
There are laws -- lots of them -- particularly in the healthcare and finance arenas. I'm not a lawyer, so I can't tell you what they are specifically. But if you're in an industry governed by regulation or rife with litigation, you probably know.
There is also a case to be made that you can't be sure that cloud-based LLMs will be secure. Sure, a vendor may say your data isn't used for training, but is it? If you're subject to HIPAA or DoD security clearance requirements, you may not be allowed to share your code with a chatty chatbot.
Also: How I used this AI tool to build an app with just one prompt - and you can too
Do you really want to bet your business on code written by Bender from Futurama? Yes, it's possible you might have humans double‑checking the code. But we humans are fallible and miss things.
Think about human nature. If you think your opponent will come down on you for a human error, you're probably right. But if you were too lazy to write your own code and handed it off to AIs known to hallucinate, ooh -- your competition's gonna have a field day with your future.
6. Domain-specific business logic
You know how it is when you bring a new hire into the company and it takes them a while to get a handle on what you do and how you do it? Or worse, when you merge two companies and the employees of each are having difficulty grokking the culture and business practices of the other?
Also: The top 20 AI tools of 2025 - and the #1 thing to remember when you use them
Yeah. Asking an AI to write code about your unique business operations is a recipe for failure. Keep in mind that AIs are trained on a lot of public knowledge. Let's define that for a minute. Public knowledge is any knowledge the public could possibly know. The AIs were trained on all the stuff they could hoover from the Internet, with or without permission.
But the AIs are not trained on your internal business knowledge, trade secrets, practices, folklore, long‑held work‑arounds, yada yada yada. Use the AI for what it's good at, but don't try to convince it to do something it doesn't know how to do. AIs are so people‑pleasing that they'll try to do it -- and maybe never tell you that what you just deployed was fabricated garbage.
7. Low-level systems work and performance optimizations
While it's possible for an AI to identify areas of code that could use optimization, there are limits. AIs aren't trained on the very fine details of microarchitectural constraints, nor do they have the experience of coaxing just a skosh more out of every line of code.
Also: The best AI for coding in 2025 (including a new winner - and what not to use)
A lot of the coding involved in embedded systems programming, kernel development, and performance-critical C and C++ optimization exists in the brains of a few expert coders. Also, keep in mind that AIs confabulate. So what they may insist are performance improvements could well be hidden cycle drains that they simply won't admit to.
If you need fine craftspersonship, you'll need a fine craftsperson -- in this case, a very experienced coder.
8. Learning exercises and educational assignments
If you use an AI, are you cheating? Yes. No. Depends. Yes, because you may be violating academic standards and cheating yourself out of the critical hands-on learning that makes knowledge stick. No, because AI has proven to be an excellent augmentation for help, especially when TAs aren't available. And maybe, because this is still a fairly unknown area.
Also: I test a lot of AI coding tools, and this stunning new OpenAI release just saved me days of work
Harvard takes a middle ground with its wonderful CS50 Intro to Computer Science course. It offers the CS50 duck (it's a long story), an AI specifically trained on their course materials with system instructions that limit how much information students are provided. So the AI is there to help answer legitimate student questions, but not do their work for them.
If you're a student or an educator, AI is a boon. But be careful. Don't cheat, and don't use it to shortcut work that you really should be doing to make education happen. But consider how it might help augment your studies or help you keep up with students' demands.
9. Collaboration and people stuff
I've found that if I treat the AI chatbot as if it were another human coder at the other end of a Slack conversation, I can get a lot out of that level of "collaboration." A lot, but not everything.
Both humans and AIs can get stubborn, stupid, and frustrating during a long, unproductive conversation. Humans can usually break out of it and be persuaded to be helpful, at least in professional settings. But once you reach the limit of the AI's session capacity or knowledge, it just becomes a waste of time.
Also: What is AI vibe coding? It's all the rage but it's not for everyone - here's why
The best human collaborations are magical. When a team is on fire -- working together, bouncing ideas off each other, solving problems, and sharing the workload -- amazing things can happen.
AI companies claim workforces made up of agents can duplicate this synergy, but nothing beats working with other folks in a team that's firing on all cylinders. Not just for productivity (which you get), but also for quality of work life, long-term effectiveness, and, yes, fun.
Also: Open-source skills can save your career when AI comes knocking
Don't get me wrong. Some of my best friends are robots. But some of my other best friends are people with whom I have long, deep, and fulfilling relationships. Besides, I've never met an AI that can make Mr. Amontis' moussaka or Auntie Paula's apple pie.
Bonus: Don't use AI for anything you want to own
Don't use AI for anything you indisputably want to own. If you write code that you then release as open source, this may not be as much of an issue. But if you write proprietary code that you want to own, you might not want to use an AI.
We asked some attorneys about this back at the dawn of generative AI, and the overall consensus is that copyright depends on creation with human hands. If you want to make sure you never wind up in court trying to protect your right to your own code, don't write it with an AI. For more background, here's the series I published on code and copyrights:
- If ChatGPT produces AI-generated code for your app, who does it really belong to?
- If your AI-generated code becomes faulty, who faces the most liability exposure?
What about you? Have you found yourself leaning too much on AI to write code? Where do you draw the line between convenience and caution? Are there any programming tasks where you've found AI genuinely helpful or dangerously misleading? Have you ever had to debug something an AI wrote and wondered if it saved you time or cost you more? Let us know in the comments below.
You can follow my day-to-day project updates on social media. Be sure to subscribe to my weekly update newsletter, and follow me on Twitter/X at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, on Bluesky at @DavidGewirtz.com, and on YouTube at YouTube.com/DavidGewirtzTV.
Want more stories about AI? Sign up for Innovation, our weekly newsletter.
Artificial Intelligence
Fighting AI with AI, finance firms prevented $5 million in fraud - but at what cost?
When most people think of AI, the first thing that probably comes to mind isn't superintelligence or the promise of agents to boost productivity, but scams.
There've always been fraudsters among us, that small percentage of the population who'll use any means available to swindle others out of their money. The proliferation of advanced and easily accessible generative AI tools in recent years has made such nefarious activity exponentially easier.
Also: Meet ChatGPT agent, a new AI assistant ready to carry out complex tasks for you - try it now
In one memorable incident from early last year, a finance employee at a firm based in Hong Kong wired $25 million to fraudsters after being instructed to do so on a video call with what he believed to be company executives, but were in fact AI-generated deepfakes. And earlier this month, an unknown party used AI to imitate the voice of US Secretary of State Marco Rubio on calls that went out to a handful of government officials, including a member of Congress.
And yet, counterintuitively, AI is also being deployed by financial services companies to prevent fraud.
In a recent survey conducted by Mastercard and Financial Times Longitude (a marketing agency and a subsidiary of Financial Times Group), 42% of issuers and 26% of acquirers said that AI tools have helped them to save more than $5 million from attempted fraud in the past two years.
In the financial sector, an issuer is a firm that provides debit or credit cards (think Chase or another major bank), while acquirers are those that accept payments (think Stripe and Square).
Also: Anthropic's Claude dives into financial analysis. Here's what's new
Many of these organizations have begun using AI tools to enhance their digital security in conjunction with more traditional methods, like two-factor authentication and end-to-end encryption, according to a report of the survey findings published last month.
Survey respondents reported using a variety of AI-powered techniques to boost their cybersecurity and protect against fraud. The most commonly cited technique was anomaly detection -- that is, an automated alarm that flags unusual requests. Other use-cases included scanning for vulnerabilities in cybersecurity systems, predictive threat modeling, "ethical hacking" (another form of searching for system vulnerabilities), and employee upskilling.
The vast majority of respondents (83%) also said "that AI has significantly reduced the time needed for fraud investigation and resolution," while reducing customer churn. Even more (90%) agreed that unless their use of AI for fraud prevention increases in the coming years, their "financial losses will likely increase."
Also: Researchers from OpenAI, Anthropic, Meta, and Google issue joint AI safety warning - here's why
Several barriers, however, are preventing the financial services companies surveyed from adopting fraud-preventing AI tools at scale. Chief among these are the technical complexities of integrating new AI systems with existing software and data that's already deployed within an organization. That's closely followed by concerns about the rapid pace at which fraud tactics themselves are evolving, which many fear will quickly outpace any attempt to use AI-powered fraud prevention.
Want more stories about AI? Sign up for Innovation, our weekly newsletter.