I'm going to take a contrarian view and say it's actually a good UI, but it's all about how you approach it.
I just finished a small project where I used o3-mini and o3-mini-high to generate most of the code. I averaged around 200 lines of code an hour, including the business logic and unit tests. Total was around 2200 lines. So, not a big project, but not a throw away script. The code was perfectly fine for what we needed. This is the third time I've done this, and each time I get faster and better at it.
1. I find a "pair programming" mentality is key. I focus on the high-level code, and let the model focus on the lower level code. I code review all the code, and provide feedback. Blindly accepting the code is a terrible approach.
2. Generating unit tests is critical. After I like the gist of some code, I ask for some smoke tests. Again, peer review the code and adjust as needed.
3. Be liberal with starting a new chat: the models can get easily confused with longer context windows. If you start to see things go sideways, start over.
4. Give it code examples. Don't prompt with English only.
FWIW, o3-mini was the best model I've seen so far; Sonnet 3.5 New is a close second.
I guess the things I don't like about Chat are the same things I don't like about pair (or team) programming. I've always thought of programming as a solitary activity. You visualize the data structures, algorithms, data paths, calling flow and stack, and so on, in your mind, with very high throughput "discussions" happening entirely in your brain. Your brain is high bandwidth, low latency. Effortlessly and instantly move things around and visualize them. Figure everything out. Finally, when it's correct, you send it to the slow output device (your fingers).
The minute you have to discuss those things with someone else, your bandwidth decreases by orders of magnitude and now you have to put words to these things and describe them, and physically type them in or vocalize them. Then your counterpart has to input them through his eyes and ears, process that, and re-output his thoughts to you. Slow, slow, slow, and prone to error and specificity problems as you translate technical concepts to English and back.
Chat as a UX interface is similarly slow and poorly specific. It has all the shortcomings of discussing your idea with a human and really no upside besides the dictionary-like recall.
That's such a mechanical way of describing pair programming. I'm guessing you don't do it often (understandable if its not working for you).
For me pair programming accelerates development to much more than 2x. Over time the two of you figure out how to use each other's strengths, and as both of you immerse yourself in the same context you begin to understand what's needed without speaking every bit of syntax between each other.
In best cases as a driver you end up producing high quality on the first pass, because you know that your partner will immediately catch anything that doesn't look right. You also go fast because you can sometimes skim over complexities letting your partner think ahead and share that context load.
I'll leave readers to find all the caveats here
Edit: I should probably mention why I think Chat Interface for AI is not working like Pair programming: As much as it may fake it, AI isn't learning anything while you're chatting to it. Its pointless to argue your case or discuss architectural approaches. An approach that yields better results with Chat AI is to just edit/expand your original prompt. It also feels less like a waste of time.
With Pair programming, you may chat upfront, but you won't reach that shared understanding until you start trying to implement something. For now Chat AI has no shared understanding, just "what I asked you to do" thing, and that's not good enough.
I think it depends heavily on the people. I've done pair programming at a previous job and I hated it. It wound up being a lot slower overall.
For me, there's
- Time when I want to discuss the approach and/or code to something (someone being there is a requirement)
- Time when I want to rubber duck, and put things to words (someone being there doesn't hurt, but it doesn't help)
- Time when I want to write code that implements things, which may be based on the output of one of the above
That last bucket of time is generally greatly hampered by having someone else there and needing to interact with them. Being able to separate them (having people there for the first one or two, but not the third) is, for me, optimal.
Pair programming is imo great when there is some sort of complementarity between the programmers. It may or may not accelerate output, but it can definitely accelerate learning which is often harder. But as you say, this is not what working with llms is about.
I would argue that is a feature of pair programming, not a bug. By forcing you to use the slower I/O parts of your brain (and that of your partner) the process becomes more deliberate, allowing you to catch edge cases, bad design patterns, and would-be bugs before even putting pen to paper so to speak. Not to mention that it immediately reduces the bus factor by having two people with a good understanding of the code.
I’m not saying pair programming is a silver bullet, and I tend to agree that working on your own can be vastly more efficient. I do however think that it’s a very useful tool for critical functionality and hard problems and shouldn’t be dismissed.
I guess it depends on a person. My experience is close to that of 'ryandrake.
I've been coding long enough to notice there are times where the problem is complex and unclear enough that my own thought process will turn into pair programming with myself, literally chatting with myself in a text file; this process has the bandwidth and latency on the same order as talking to another person, so I might just as well do that and get the benefit of an independent perspective.
The above is really more of a design-level discussion. However, there are other times - precisely those times that pair programming is meant for - when the problem is clear enough I can immerse myself in it. Using the slow I/O mode, being deliberate is exactly the opposite of what I need then. By moving alone and focused, keeping my thoughts below the level of words, I can explore the problem space much further, rapidly proposing a solution, feeling it out, proposing another, comparing, deciding on a direction, noticing edge cases and bad design up front and dealing with them, all in a rapid feedback loop with test. Pair programming in this scenario would truly force me to "use the slower I/O parts of your brain", in that exact sense: it's like splitting a highly-optimized in-memory data processing pipeline in two, and making the halves communicate over IPC. With JSON.
As for bus factor, I find the argument bogus anyway. For that to work, pair programming would've to be executed with the same partner or small group of partners, preferably working on the same or related code modules, daily, over the course of weeks at least - otherwise neither them nor I are going to have enough exposure to understand what the other is working on. But it's not how pair programming worked when I've experienced it.
It's a problem with code reviews, too: if your project has depth[0], I won't really understand the whole context of what you're doing, and you won't understand the context of my work, so our reviews of each others' code will quickly degenerate to spotting typos, style violations, and peculiar design choices; neither of us will have time or mental capacity to fully understand the changeset before "+2 LGTM"-ing it away.
--
[0] - I don't know if there's a a better, established term for it. What I mean is depth vs. breadth in the project architecture. Example of depth: you have a main execution orchestrator, you have an external data system that handles integrations with a dozen different data storage systems, then you have math-heavy business logic on data, then you have RPC for integrating with GUI software developed by another team, then you have extensive configuration system, etc. - each of those areas is full of design and coding challenges that don't transfer to any other. Contrast that with an example of breadth: a typical webapp or mobile app, where 80% of the code is just some UI components and a hundred different screens, with very little unique or domain-specific logic. In those projects, developers are like free electrons in metal: they can pick any part of the project at any given moment and be equally productive working on it, because every part is basically the same as every other part. In those projects, I can see both pair programming and code reviews deliver on their promises in full.
I agree, chat is only useful in scenarios that are 1) poorly defined, and 2) require a back-and-forth feedback loop. And even then, there might be better UX options.
At the same time, putting your ideas to words forces you to make them concrete instead of nebulous brain waves. I find that the chat interface gets rid of the downsides of pair programming (that the other person is a human being with their own agency*) while maintaining the “intelligent” pair programmer aspect.
Especially with the new r1 thinking output, I find it useful to iterate on the initial prompt as a way to make my ideas more concrete as much as iterating through the chat interface which is more hit and miss due to context length limits.
* I don’t mean that in a negative way, but in a “I can’t expect another person to respond to me instantly at 10 words per second” way.
Doing it in your native language can add an extra dimension to it, though. In a way, I would consider it like double checking your work on something like a math problem by solving it a different way. By having to express the problem and solution in clear language, it can really help you make sure your solution is a good one, and considers all the angles.
Someone else (future you being a distinct person) will also need to grok what's going on when they maintain the code later. By living purely in a high-dimensional trans-enlightenment state and coding that way, means you may as well be building a half-assed organic neural network to do your task, rather than something better "designed".
Neural networks and evolved structures and pathways (e.g. humans make do with ~20k genes and about that many more in regulatory sequences) are absolutely more efficient, but good luck debugging them.
> I focus on the high-level code, and let the model focus on the lower level code.
Tbh the reason I don't use LLM assistants is because they suck at the "low level". They are okay at mid level and better at high level. I find it's actual coding very mediocre and fraught with errors.
I've yet to see any model understand nuance or detail.
This is especially apparent in image models. Sure, it can do hands but they still don't get 3D space nor temporal movements. It's great for scrolling through Twitter but the longer you look the more surreal they get. This even includes the new ByteDance model also on the front page. But with coding models they ignore context of the codebase and the results feel more like patchwork. They feel like what you'd be annoyed at with a junior dev for writing because not only do you have to go through 10 PRs to make it pass the test cases but the lack of context just builds a lot of tech debt. How they'll build unit tests that technically work but don't capture the actual issues and usually can be highly condensed while having greater coverage. It feels very gluey, like copy pasting from stack overflow when hyper focused on the immediate outcome instead of understanding the goal. It is too "solution" oriented, not understanding the underlying heuristics and is more frustrating than dealing with the human equivalent who says something "works" as evidenced by the output. This is like trying to say a math proof is correct by looking at just the last line.
Ironically, I think in part this is why chat interface sucks too. A lot of our job is to do a lot of inference in figuring out what our managers are even asking us to make. And you can't even know the answer until you're part way in.
> A lot of our job is to do a lot of inference in figuring out what our managers are even asking us to make
This is why I think LLMs can't really replace developers. 80% of my job is already trying to figure out what's actually needed, despite being given lots of text detail, maybe even spec, or prototype code.
Building the wrong thing fast is about as useful as not building anything at all. (And before someone says "at least you now know what not to do"? For any problem there are infinite number of wrong solutions, but only a handful of ones that yield success, why waste time trying all the wrong ones?)
> Building the wrong thing fast is about as useful as not building anything at all.
SAY IT LOUDER
Fully agree. Plus, you may be faster in the short term but you won't in the long run. The effects of both good code and bad code compound. "Tech debt" is just a fancy term for "compounding shit". And it is true, all code is shit, but it isn't binary; there is a big difference between being stepping in shit and being waist deep in shit.
I can predict some of the responses
Premature optimization is the root of all evil
There's a grave misunderstanding in this adage[0], and I think many interpret it as "don't worry about efficiency, worry about output." But the context is that you shouldn't optimize without first profiling the code, not that you shouldn't optimize![1] I find it also funny revisiting this quote, because it seems like it is written by a stranger in a strange land, where programmers are overly concerned with optimizing their code. These days, I hear very little about optimization (except when I work with HPC people) except when people are saying to not optimize. Explains why everything is so sluggish...
[1] Understanding the limitations of big O analysis really helps in understanding why this point matters. Usually when n is small, you can have worse big O and still be faster. But the constants we drop off often aren't a rounding error. https://csweb.wooster.edu/dbyrnes/cs200/htmlNotes/qsort3.htm
> For any problem there are infinite number of wrong solutions, but only a handful of ones that yield success, why waste time trying all the wrong ones?
Devil's advocate: because unless you're working in heavily dysfunctional organization, or are doing a live coding interview, you're not playing "guess the password" with your management. Most of the time, they have even less of a clue about how the right solution looks like! "Building the wrong thing" lets them diff something concrete against what they imagined and felt like it would be, forcing them to clarify their expectations and give you more accurate directions (which, being a diff against a concrete things, are less likely to be then misunderstood by you!). And, the faster you can build that wrong thing, the less money and time is burned to buy that extra clarity.
> Tbh the reason I don't use LLM assistants is because they suck at the "low level". They are okay at mid level and better at high level. I find it's actual coding very mediocre and fraught with errors.
That's interesting. I found assistants like Copilot fairly good at low level code, assuming you direct it well.
I have a response to a sibling comment showing where GPT 4o and o1-preview do not yield good results.
> assuming you direct it well.
But hey, I admit I might not be good at this. But honestly, I've found greater value in my time reading the docs than spending trying to prompt engineer my way through. And I've given a fair amount of time to trying to get good at prompting. I just can't get it to work.
I do think that when I'm coding with an LLM it _feels_ faster, but when I've timed myself, it doesn't seem that way. It just seems to be less effort (I don't mind the effort, especially because the compounding rewards).
I can't get the prompt because I'm on my work computer but I have about a three-quarter-page instruction set in the settings of cursor, it asks clarifying questions a LOT now, and is pretty liberal with adding in commented pseudo-code for stuff it isn't sure about. You can still trip it up if you try, but it's a lot better than stock. This is with Sonnet 3.5 agent chats (composer I think it's called?)
I actually cancelled by Anthropic subscription when I started using cursor because I only ever used Claude for code generation anyway so now I just do it within the IDE.
> It has a great feature that grabs context from the codebase, I use it all the time.
If only this feature worked consistently, or reliably even half of the time.
It will casually forget or ignore any and all context and any and all files in your codebase at random times, and you never know what set of files and docs it's working with at any point in time
It probably isn't obvious in a quick read, but there are mistakes here. Maybe the most obvious is that how `replacements` is made we need to intelligently order. This could be fixed by sorting. But is this the right data structure? Not to mention that the algorithm itself is quite... odd
To give a more complicated example I passed the same prompt from this famous code golf problem[0]. Here's the results, I'll save you the time, the output is wrong https://0x0.st/8K3M.txt (note, I started command likes with "$" and added some notes for you)
Just for the heck of it, here's the same thing but with o1-preview
As you can see, o1 is a bit better on the initial problem but still fails at the code golf one. It really isn't beating the baseline naive solution. It does 170 MiB/s compared to 160 MiB/s (baseline with -O3). This is something I'd hope it could do really well on given that this problem is rather famous and so many occurrences of it should show up. There's tons of variations out there and It is common to see parallel fizzbuzz in a class on parallelization as well as it can teach important concepts like keeping the output in the right order.
But hey, at least o1 has the correct output... It's just that that's not all that matters.
I stand by this: evaluating code based on output alone is akin to evaluating a mathematical proof based on the result. And I hope these examples make the point why that matters, why checking output is insufficient.
Edit: I want to add that there's also an important factor here. The LLM might get you a "result" faster, but you are much more likely to miss the learning process that comes with struggling. Because that makes you much faster (and more flexible) not just next time but in many situations where even a subset is similar. Which yeah, totally fine to glue shit together when you don't care and just need something, but there's a lot of missed value if you need to revisit any of that. I do have concerns that people will be plateaued at junior levels. I hope it doesn't cause seniors to revert to juniors, which I've seen happen without LLMs. If you stop working on these types of problems, you lose the skills. There's already an issue where we rush to get output and it has clear effects on the stagnation of devs. We have far more programmers than ever but I'm not confident we have a significant number more wizards (the percentage of wizards is decreasing). There's fewer people writing programs just for fun. But "for fun" is one of our greatest learning tools as humans. Play is a common trait you see in animals and it exists for a reason.
I've changed my mind on that as well. I think that, generally, chat UIs are a lazy and not very user friendly. However, when coding I keep switching between two modes:
1. I need a smart autocomplete that can work backwards and mimic my coding patterns
2. I need a pair programming buddy (of sorts, this metaphor doesn't completely work, but I don't have a better one)
Pair development, even a butchered version of the so called "strong style" (give the driver the highest level of abstraction they can use/understand) works quite well for me. But, the main reason this works is that it forces me to structure my thinking a little bit, allows me to iterate on the definition of the problem. Toss away the sketch with bigger parts of the problem, start again.
It also helps me to avoid yak shaving, getting lost in the detail or distracted because the feedback loop between me seeing something working on the screen vs. the idea is so short (even if the code is crap).
I'd also add 5.: use prompts to generate (boring) prompts. For instance, I needed a simple #tag formatter for one of my markdown sites. I am aware that there's a not-so-small list of edge cases I'd need to cover. In this case I'd write a prompt with a list of basic requirements and ask the LLM to: a) extend it with good practice, common edge cases b) format it as a spec with concrete input / output examples. This works a bit similar to the point you made about generating unit tests (I do that too, in tandem with this approach).
In a sense 1) is autocomplete 2) is a scaffolding tool.
Oh yea, point 1 for sure. I call copilot regex on steroids.
Example:
- copy paste a table from a pdf datasheet into a comment (it'll be badly formatted with newlines and whatnot, doesn't matter)
- show it how to do the first line
- autocomplete the rest of the table
- Check every row to make sure it didn't invent fields/types
For this type of workflow the tools are a real time saver. I've yet to see any results for the other workflows. They usually just frustrate me by either starting to suggest nonsense code without full understanding, or its far too easy to bias the results and make them stuck in a pattern of thinking.
> I've changed my mind on that as well. I think that, generally, chat UIs are a lazy and not very user friendly. However, when coding I keep switching between two modes:
> 1. I need a smart autocomplete that can work backwards and mimic my coding patterns
> 2. I need a pair programming buddy (of sorts, this metaphor doesn't completely work, but I don't have a better one)
Thanks! This is the first time I've seen it put this clearly. When I first tried out CoPilot, I was unsure of how I was "supposed" to interact with it. Is it (as you put it) a smarter autocomplete, or a programming buddy? Is it both? What was the right input method to use?
After a while, I realized that for my personal style I would pretty much entirely use method 1, and never method 2. But, others might really need that "programming buddy" and use that interface instead.
I work on GenAI in the media domain, and I think this will hold true with other fields as well:
- Text prompts and chat interfaces are great for coarse grained exploration. You can get a rough start that you can refine. "Knight standing in a desert, rusted suit of armor" gets you started, but you'll want to take it much further.
- Precision inputs (mouse or structure guided) are best for fine tuning the result and honing in on the solution itself. You can individually plant the cacti and pose the character. You can't get there with text.
I think chat is a nice intermediary evolution between the CLI (that we use every day) and whatever comes next.
I work at Augment (https://augmentcode.com), which, surprise surprise, is an AI coding assistant. We think about the new modality required to interact with code and AI on a daily basis.
Beside increase productivity (and happiness, as you don't have to do mundane tasks like tests, documentations, etc), I personally believe that what AI can open up is actually more of a way for non-coders (think PMs) to interact with a codebase. AI is really good at converting specs, user stories, and so on into tasks—which today still need to be implemented by software engineers (with the help of AI for the more tedious work). Think of what Figma did between designers and developers, but applied to coding.
What’s the actual "new UI/UX paradigm"? I don’t know yet. But like with Figma, I believe there’s a happy ending waiting for everyone.
Yesterday, I asked o3-mini to "optimize" a block of code. It produced very clean, functional TypeScript. However, because the code is reducing stock option chains, I then asked o3-mini to "optimize for speed." In the JavaScript world, this is usually done with for loops, and it even considered aspects like array memory allocation.
This shows that using the right qualifiers is important for getting the results you want. Today, I use both "optimize for developer experience" and "optimize for speed" when they are appropriate.
Although declarative code is just an abstraction, moving from imperative jQuery to declarative React was a major change in my coding experience. My work went from telling the system how to do something to simply telling it what to do. Of course, in React—especially at first—I had to explain how to do things, but only once to create a component. After that, I could just tell the system what to do. Now, I can simply declare the desired outcome, the what. It helps to understand how things work, but that level of detail is becoming less necessary.
Nah, a Chat is terrible for development. In my tears of working, i have only had the chance to start a new codebase 3-4 times. 90% of the time is spent modifying large existing systems, constantly changing them. The chat interface is terrible for this. It would be much better if it was more integrated with the codebase and editor
Cursor does all of this, and agent chats let you describe a new feature or an existing bug and it will search the entire codebase and add relevant code to its context automatically. You can optionally attach files for the context - code files that you want to add to the context up front, documentation for third-party calls, whatever you want.
As a side note, "No, you're wrong" is not a great way to have a conversation.
This only works for small self-contained problems with narrow scope/context.
Chat sucks for pulling in context, and the only worse thing I've tried is the IDE integrations that supposedly pull the relevant context for you (and I've tried quite a few recently).
I don't know if naive fine-tuning with codebase would work, I suspect there are going to be tools that let you train the AI on code in the sense that it can have some references in model, and it knows how you want your project code/structure to look like (which is often quite different from what it looks in most areas)
What did you create? In my field, so far, I've found the chat bots not doing so well. My guess is the more likely you're making something other people make often, the more likely the bot will help.
Even then though, I asked o1-cursor to start a react app. It failed, mostly because it's out of date. It's instructions were for react 2 versions ago.
This seems like an issue. If the statistically most likley answer is old, that's not helpful.
The most recent one was a typescript project focused on zod.
I might be reading into your comment, but I agree "top-down" development sucks: "Give me a react that does X". I've had much more success going bottom-up.
And I've often seen models getting confused on versions. You need to be explicit, and even then then forget.
Totally agree. Chat is a fantastic interface because it stays out of my way. For me it's much more than a coding assistant. I get live examples of how to use tools, and help with boilerplate, which is a time saver and improvement over legacy workflows, but the real benefit is all the spitballing I can do with it to refine ideas and logic and help getting up to speed on tooling way outside of my domain. I spent about 3.5 hours chatting with o1 about RL architecture to solve some business problems. Now I have a crystal clear plan and the confidence to move forward in an optimal way. I feel a little weird now, like I was just talking to myself for a few hours, but it totally helped me work through the planning. For actual code, I find myself being a bit less interactive with LLMs as time goes, sometimes it's easier to just write the logic the way I want rather than trying to explain how I want it but the ability to retrieve code samples for anything with ease is like a superpower. Not to mention all the cool stuff LLMs can do at runtime via API. Yeah, chat is great, and I'll stick with writing code in Vim and pasting as needed.
Interesting to see the narrative on here slowly change from "LLM's will forever be useless for programming" to "I'm using it every day" over the course of the past year or so.
I'm now bracing for the "oh sht, we're all out of a job next year" narrative.
I think a lot of people have always thought of it as a tool that can help.
I don't want an LLM to generate "the answer" for me in a lot of places, but I do think it's amazing for helping me gather information (and cite where that information came from) and pointers in directions to look. A search engine that generates a concrete answer via LLM is (mostly) useless to me. One that gives me an answer and then links to the facts it used to generate that answer is _very_ useful.
It's the same way with programming. It's great helping you find what you need. But it needs to be in a way that you can verify it's right; or take it's answer and adjust it to what you actually need (based on the context it provides).
Maybe. My sense if we'd need to see 3 to 4 orders of magnitude improvements on the current models before we can replace people outright.
I do think we'll see a huge productivity boost per developer over the next few years. Some companies will use that to increase their throughput, and some will use it to reduce overhead.
Whenever I read huge productivity boost for developers or companies I shiver. Software sucked more and more even before LLMs, I don't see it getting better just getting out faster maybe. I'm afraid in most cases it will be a disaster
I would actually join you, as my longstanding view on coding is that it is best done in pairs. Sadly humans and programmers in particular are not so ready to work arms-by-arms, and it is even more depressing that it now turns AI is pairing us.
Perhaps there's gonna be post-AI programming movement where people actually stare at the same monitor and discuss while one of them is coding.
As a sidenote - we've done experiments with FOBsters, and when paired this way, the multiply their output. There's something about psychology of groups and how one can only provide maximum output when teaming.
Even for solo activities, and non-IT activities, such as skiing/snowboard, it is better to have a partner to ride with you and discuss the terrain.
Copilot, back when I used it, completely ignored context outside of the file I was working in. Copilot, as of a few weeks ago, the absolute dumbest assistant of all the various options available.
With cursor, I can ask it to make a change to how the app generates a JWT without even knowing which file or folder the relevant code is in. For very large codebases, this is very very helpful.
ya know what, after a couple times hearing this comment, I downloaded it literally yesterday. It does feel pretty different, at least the composer module and stuff. A bit improvement in ai tooling imo
Same for me. I wonder if Claude is better at some languages than others, and o models are better at those weaker languages. There are some devs I know who insist Claude is garbage for coding and o3-* or o4-* are tier 1.
> 1. I find a "pair programming" mentality is key. I focus on the high-level code, and let the model focus on the lower level code. I code review all the code, and provide feedback. Blindly accepting the code is a terrible approach.
This is what I've found to be key. If I start a new feature, I will work with the LLM to do the following:
- Create problem and solution statement
- Create requirements and user stories
- Create architecture
- Create skeleton code. This is critical since it lets me understand what it wants to do.
- Generate a summary of the skeleton code
Once I have done the above, I will have the LLM generate a reusable prompt that I can use to start LLM conversations with. Below is an example of how I turn everything into a reusable prompt.
The first message is the reusable prompt message. With the first message in place, I can describe the problem or requirements and ask the LLM what files it will need to better understand how to implement things.
What I am currently doing highlights how I think LLM is a game changer. VCs are going for moonshots instead of home runs. The ability to gather requirements and talk through a solution before even coding is how I think LLMs will revolutionize things. It is great that it can produce usable code, but what I've found it to be invaluable is it helps you organize your thoughts.
In the last link, I am having a conversation with both DeepSeek v3 and Sonnet 3.5 and the LLMs legitimately saved me hours in work, without even writing a single line of code. In the past, I would have just implemented the feature and been done with it, and then I would have to fix something if I didn't think of an edge case. With LLMs, it literally takes minutes to develop a plan that is extremely well documented that can be shared with others.
This ability to generate design documents is how I think LLMs will ultimately be used. The bonus is producing code, but the reality is that documentation (which can be tedious and frustrating) is a requirement for software development. In my opinion, this is where LLMs will forever change things.
To add to that, I always add some kind of debug function wrapper so I can hand off the state of variables and program flow to the LLM when I need to debug something. Sometimes it's really hard to explain exactly what went wrong so being able to give it a chunk of the program state is more descriptive.
I do the same for my QT desktop app. I’ve got an “Inspector” singleton that allows me to select a component tree via click, similar to browser devtools. It takes a screenshot, dumps the QML source, and serializes the state of the components into the clipboard.
I paste that into Claude and it is surprisingly good at fixing bugs and making visual modifications.
I had the beta as well and have been using it quite a bit. I like the concept a lot. This is exactly how I want to be using AI for writing purposes. Overall, it’s whatever content I want to write in, but then I can ask it, “Hey, help me improve this line,” or “Fix this paragraph,” piecemeal.
That said, it only worked for me maybe 70% of the time. It would either have trouble finding things, or it would just arbitrarily rewrite the entire document, even when it was supposed to be focused on a certain part of it.
I agree 100% with this approach, and I'd love to see some research to back it up.
I think a lot of software development is "way finding" - Experimentation to figure out an architecture, an implication, a performance improvement, etc.
We often don't a) call them experiments, and b) we don't constrain them well. I.e., we use the scope of the feature to bound the experiment, instead of taking a step back to figure out the right approach, we dive into implementation.
I'm curious if there's a more formal way to think about this all?
I played around with it a bit, here are some hot takes.
For context, I first tried this procession of searches on the Mac OS app.
1. "Who won the world series"
2. Who was the MVP?"
3. "Give me his bio"
My observations:
1. UX: The "search" button feels oddly placed, but I can't put my finger on it. But once I got it is a toggle, it wasn't a bit deal.
2. The first result had 3 logos, headlines and timestamps delineated, and easy to ready. The second one and third ones included a "Sources" button that opened a fly open menu. Clicking those opened a web link. The third result also included images in the fly open.
3. Citations were also inlined. The third result, for the bio, included a citation per paragraph.
4. It wasn't as fast as google. Which makes sense, given it's going through the LLM. But it will take a while to rewire my brain to expect slower responses to search.
5. Overall, I found the chat interface a very intuitive interface.
The second search I asked was "Give me a plan for a Thanksgiving meal."
I to a long response that felt like a weird mashup of LLM-generated content and search results:
1. A list of menu selections
2. Links to some recipes
3. Prepration timeline
4. Shopping list
5. Additional tips
There were 15 citations listed in the popup button, but only 3 inlined.
This was... not great. A traditional list of search results feels better here.
Overall, I like the direction. Innovation in search has been dead for close to 10 years, and this feels like I'd use it for certain inquiries.
I asked “Find me articles about the World Series MVP.” ChatGPT returned snippets and URLs from: MLB Reuters AP NYPost and FoxSports.
I asked “Please find articles about planning a Thanksgiving meal for a family reunion.” It returned links to: GatheredAgain FavFamilyRecipes TastesBetterFromScratch etc. I like that it is returning niche sites I do not know about.
> There can’t be any information about “truthfulness” encoded in an LLM, because there isn’t a notion of “truthfulness” for a program which has only ever been fed tokens and can only ever regurgitate their statistical correlations.
I think there are two issues here:
1. The "truthfulness" of the underlying data set, and
2. The faithfulness of the LLM to pass along that truthfulness. Lack of passing along the truthfulness is, I think, the definition of the hallucination.
To your point, if the data set if flawed or factually wrong, the model will always produce the wrong result. But I don't think that's a hallucination.
The most blatant whoppers that Google's AI preview makes seem to stem from mistaking satirical sites for sites that are attempting to state facts. Possibly an LLM could be trained to distinguish sites that intend to be satirical or propagandistic from news sites that intend to report accurately based on the structure of the language. After all, satirical sites are usually written in a way that most people grasp that it is satire, and good detectives can often spot "tells" that someone is lying. But the structure of the language is all that the LLM has. It has no oracle to tell it what is true and what is false. But at least this kind of approach might make LLM-enhanced search engines less embarrassing.
I read "A City on Mars" last year, and it opened my eyes to just hard space travel is. The government constraints on aerospace projects doesn't help. There's a reason SpaceX moves so much faster; they don't have to justify and explain things to taxpayers.
Beyond that, the book makes a good case for how unrealistic a long-term colony on Mars is... at least in the short term (Short being the next 50 to 100 years).
My biggest take away is: for all his talk, Musk basically just wants to be the Uber to Mars: shuttling people there and back. He don't seem serious about _actually_ solving the problems of how to stay alive and thrive once we get there.
I found it sort of depressing as first, as I'd love to see people loving on Mars in my lifetime. But when I thought about it, I saw that they outlined a bunch of really important problems we should be working on as a society. The sooner we work on those problems, the better.
> Boeing developed Starliner under the exact same set of constraints that SpaceX had for crewed dragon.
You could be right. My (limited) understanding is that SpaceX is doing most of their R&D internally, and therefore they don't have the same oversight requirements more NASA-centric projects require. But that was based on an article about Artemis, and not Starliner.
NASA officials have previously admitted to applying more oversight to SpaceX than they did to Boeing regarding commercial crew. SpaceX's success is in being self-motivated. The government money is nice, but they want to develop and commercialize the tech anyway and are willing to put their own money into it. As evidence we have all the private Dragon flights, extra capsules built for free-flying missions and their self-developed EVA capability that should launch some time in the coming week.
That significantly relaxes the controls compared to Boeing, which is structured around exploiting cost-plus contracting, so every bit of work needs to be tracked and billed, the more time it takes, the better. Starliner was fixed price and they had to put in their own money so they would've been doing the bare minimum to keep the program going. They've only built the bare minimum 2 vehicles they'd need to meet contract requirements, and can only launch on the few launches reserved on a now retired rocket, so no room to commercialize until someone pays them to make it work with Vulcan.
I'm a CTO, and this is one of my cardinal rules: no assholes. We hire by it, and we fire by it. I took me a while to become such a firm believer, but here's what I've learned:
1. The true, genius level assholes are very few and far between. If you do find one, isolate them from your broader team, and ideally put them on contract. They are going to build you a great organization. The corollary of this: if you're an asshole, you likely aren't a genius. So stop acting like one.
2. Most assholes don't actually create anything great, they just look like they do. Strong opinions and blistering personalities tend to hide a crappy work product.
3. You'll learn there are enough nice people out there that you don't need to settle for crappy behavior. It can take time to find capable, nice people, but it's worth the effort.
Finally: there are naturally asshole people, and then there are people who have been around so much institutionalized asshole-ism they just look like assholes. Learning the difference is critical.
Genius assholes are rare but so are geniuses. I think the odds of a genius being an asshole are higher than a below-average person being an asshole.
I also think asshole behavior is a subjective label in many cases. If your culture is not compatible with someone else's, they might be averse to you for no reason and assume you're an asshole, and then any minor encounter can turn into a dispute. Other people are so used to conformity that if you disagree with them, they'll shun you or else shout you down.
> I also think asshole behavior is a subjective label in many cases.
It is absolutely subjective, but I think the advice still applies: "No assholes" means no assholes, regardless of your definition.
That said, I also see a lot of assholes hide behind this reasoning. "I'm not a jerk, your culture is just weak." In my experience, 9 times out of 10, they really are just a jerk.
I know where you're going with that but some cultures really are better in some ways. I don't think cultures that overly promote conformity and are "strong" for example. Nor are ones that tolerate too much corruption or dysfunctional behavior. Neither are ones that have no sense of self-preservation and allow infinite outside influence.
I'm not even just talking about people who are way outside the norm when it comes to behavior, like the "I'm just blunt/honest" or contrarian type. Some people legitimately rub each other the wrong way, or else can't be read easily. They have different senses of humor and cultural backgrounds. When people start off with a bias or misunderstanding, it's easier to get frustrated and think the other person is an asshole.
Here's an example. I literally had to sit through a HR training video where some people were pissing and moaning about a guy using sports metaphors they didn't understand. Instead of pissing and moaning about it, you know they could just educate themselves about that tiny bit of someone else's life. I'm sure that the guy would love to explain it to them, and enough people needing an explanation would make him reconsider his metaphors. There are times when the metaphors people use may be inappropriate, but is a basic sports metaphor in the US really so unacceptable? The choice of actors in that video also paints a troubling picture about whose culture is welcome or not.
I just finished a small project where I used o3-mini and o3-mini-high to generate most of the code. I averaged around 200 lines of code an hour, including the business logic and unit tests. Total was around 2200 lines. So, not a big project, but not a throw away script. The code was perfectly fine for what we needed. This is the third time I've done this, and each time I get faster and better at it.
1. I find a "pair programming" mentality is key. I focus on the high-level code, and let the model focus on the lower level code. I code review all the code, and provide feedback. Blindly accepting the code is a terrible approach.
2. Generating unit tests is critical. After I like the gist of some code, I ask for some smoke tests. Again, peer review the code and adjust as needed.
3. Be liberal with starting a new chat: the models can get easily confused with longer context windows. If you start to see things go sideways, start over.
4. Give it code examples. Don't prompt with English only.
FWIW, o3-mini was the best model I've seen so far; Sonnet 3.5 New is a close second.
reply