Sometimes incorrect suggestions can be useful - if you already know what you're doing: a story of using ChatGPT
I am a pretty proficient computer-toucher (I am literally a professor of computer touching after all). Still, shockingly, I don't know everything about the devil's silicon.
Yesterday, I was trying to solve a really annoying problem that's been plaguing me for the past week: the video game controller that I use with my laptop would just stop working (usually while I was healing a party in #FFXIV, naturally) and it would take a reboot of my laptop to fix it.
I won't bore you with all the things I tried or all of the web searches I did, I bored myself plenty. But let's say this: I suspected it was a kernel problem because the device in `/dev/input` would vanish without anything showing up in `dmesg` and no amount of unplugging and replugging would make it come back.
I know kernels and I know device drivers but I know next to nothing about libinput and the kernel subsystems that support it. So, I thought, what the hell, let's see if ChatGPT has any suggestions on how to debug this.
It started with some sensible suggestions I had already tried: how about a different cable? How about a different USB port? This controller also supports bluetooth, have you tried using it that way?
But then, it suggested something I didn't know about: have you tried the xpadneo driver? Oh, I hadn't heard of that. And, because it had picked up on the fact that I'm on NixOS from some stuff I had pasted in, it helpfully gave me the nix expressions. It gave me clear instructions: here's where you put the nix expressions, make sure to rebuild and reboot, here is how you put this specific controller in xinput mode (required for that driver): hold these buttons then plug in the USB cable.
But it didn't work. The controller didn't get picked up by the new driver. ChatGPT led me in several rounds of: let's blocklist the old driver, here's where to check in the filesystem to make sure the xpadneo driver is actually there, etc.
It turned out, ChatGPT was wrong. No, it didn't hallucinate the xpadneo driver, that does exist. BUT it only works with Bluetooth controllers, and ChatGPT had specifically told me to use USB. Also, it turns out, it gave me incorrect information on where on disk to look for the driver (I've never bothered to learn what magic NixOS does with kernel modules). I finally figured both of these things out with good old fashioned web searches, myself.
But the thing is, I learned some things in this process: I learned about a new driver, and I was reminded of the fact that this controller has multiple modes, and these were the keys to solving my problem.
Once I got the controller connected to xpadneo via bluetooth, it was more stable than before, but I noticed that there would be periods of time - several seconds, every 20 minutes or so - when it would stop responding, but then it would come back.
That led me to what I *think* is the correct conclusion: that the problem is actually a hardware problem in the controller, and the old driver was silently dying while the new one was being a bit more graceful and recovering.
So, I pulled out an older controller of the same type. Before, when I plugged it in it had failed to work at all. So, dead controller, I thought. But no: now that I was reminded that these controllers have different modes, I recognized the actual problem - that it was just in the wrong mode.
Poof, back to healing.
So what I want to point out here is that it *is* possible to use these tools even when they give wrong information - IF you already more or less know what you're doing and have the skills and knowledge to figure out when they're wrong and how to find the right information. Hell, those of us who are used to diagnosing these kinds of problems through forums and reddit threads are very used to this process: there's *plenty* of wrong, outdated, and/or irrelevant information all *over* the Internet, the game has *always* been to know *enough* that you can make sense of what you read and to learn new things in the process.
This is why the expectation that non-programmers can vibe-code with these things is such bullshit: *if* you have a programmer who *fundamentally know what the hell they are doing* an LLM might be able to help them when they're at the boundaries of their knowledge. But dropping someone in the middle of the ocean without a life raft and telling them to vibe their way out of it is never, ever going to work.