Joshua Rogers sent us a *massive* list of potential issues in #curl that he found using his set of AI assisted tools. Code analyzer style nits all over. Mostly smaller bugs, but still bugs and there could be one or two actual security flaws in there. Actually truly awesome findings.
I have already landed 22(!) bugfixes thanks to this, and I have over twice that amount of issues left to go through. Wade through perhaps.
Credited "Reported in Joshua's sarif data" if you want to look for yourself
Here's a simple example where it reports that we considered a nread == 0 as reading a byte, when we shouldn't.
@bagder
so this is what an AI can do when wielded by a competent human?
@wolf480pl yes! and this after three competent code analyzers already say "no issues found" ...
@bagder Is Joshua Rogers a regular or a new contributor to cURL. Are these findings landing a lot of money in his pocket?
@bagder I wonder how many issues the AI tooling will find *after* those bugfixes are applied. Hopefully fewer!
@bagder Your run-in with AI + curl reports was on the YouTube channel Low Level, did you see it? https://youtu.be/-uxF4KNdTjQ
Sorry you have to deal with all that, that has to be frustrating... glad you're encountering good use of AI too
@bagder This is what I was hoping for when the ML stuff started taking off before the LLM apocalypse. Like, a model that can rummage through a given limited dataset like library source code or a car part shop's entire catalogue, and then be able to make inferences that are too laborious for a human. "Find me the cv joint boot that has these dimensions but isn't officially compatible."
But then what we got was this bullshit and it'll just make up a compatible part or function.
@bagder Nice, what tools was he using ?
@MrMagne this is his (long) blog post on his work: https://joshua.hu/llm-engineer-review-sast-security-ai-tools-pentesters
@bagder This is kind of happening in a lot of industries and should be expected, competent people will always be able to make the most of the tools they have available, but people who aren't will try to cut corners with new tools
@BrodieOnLinux indeed. In this case I'm almost blown away by the quality of some of this...
@BrodieOnLinux @bagder reminds me of my Calculus class. We were first taught how to solve the problems the hard way then taught the shortcuts to solving problems. I see a big push to use LLMs as not learning the hard part first.
@bagder Am I reading this right, this looks like it describes an sread function call, then displays a code snippet of the exact line and there's no sread call.
@chris the code snippet is off, but the description is 100% accurate
@bagder Oh, wow. Then I guess I misjudged this. So glad someone managed to make an LLM pay off and provide good code analysis.
@chris look at this one, where the tool "knows" lots of details of the protocol neg details and can report this masterpiece on the curl telnet code:
@bagder Yikes! What a find.
@bagder well, if the socket read returns ok and 0 length, we received the first reply from the server, eg that it closed the connection on its end.
That is what the senantics of „first_byte“ is supposed to track. The var would have been better named „first_reply“.
tldr
The code was correct, the naming was wrong.
@bagder in the alt text the nread equals copyright
@ondrejkolin yeah sorry, I did not proof read the alt text properly
@bagder a great example where tools can help humans, but it doesn't help when the humans are tools