New AI experiment this week 🔬
How fast can a large product change be designed and implemented, with AI? 🤔
End to end AI collaboration: plan, design, specs, and implementation.
Turns out that I am the bottleneck for AI. This is what happened 👇
AI not only for coding
AI can read the code. What better way to define new product features?
No need to explain "this is how the app works"/"this is what I want to add and where". The AI can just read the code and understand 💁🏼♂️
That's what I did, all before even defining database schema or code changes. The prompt was "[...] keep this as a discussion, no code changes."
Sure enough, we nailed the details rather quickly, and the AI had all the context to write down the full specs 🪄
One problem: I didn't 😬
Spec-driven, or being driven by the spec?
We moved to the specs.
I thought: "I have a general idea of how this will work, the AI will figure out the details, in code." 💁🏼♂️
A moment later, the AI wrote down 10 spec markdown files, each a long bullet list of changes, file references, and something else I guess 🤯
Again, I thought: "I can always reference the spec to understand, while the AI works on the code." 💁🏼♂️
My context window was full, 10 spec files too much. But, again, since we discussed requirements together, the AI probably figured out the details.
The specs must be mostly correct, I thought 🤦
Coding (slop) too fast
AI started coding, one spec file at the time.
The situation became critical after the first spec file implementation, no need anything more 🤦
The AI went super confident in implementing part of the spec. Yeah, just a slice, not the full spec, yet. It decided that it was too much, better implement a smaller part.
Meanwhile I was trying to figure out the full scope of the requirements, without success 🤯
Reading code did not help, at all.
Time was flying by, and we still had 9+ spec files to implement.
At this point, I decide to go all-in with this experiment: let's see where AI (mostly) alone can get with the specs 🤔
No big picture, no big change
Forget the specs, forget fully automated AI coding.
After the second spec file, the plan diverged completely, and it was all to throw away 🤦
I pushed AI a little more, a little more, and at the end I had 20k+ line changes in a mostly broken PR.
Lesson from this experience:
If you don't have the full picture in mind, no amount of AI coding will rescue you 🙌
When I use AI day-to-day, I always have an idea of the requirements.
Not always the details in code, but a general expectation of what will come out the other side.
In other words: I know what to code and how, it's just that AI is faster at typing ⚡️
If instead I am missing requirements in my head, prompting will be weak, AI code mostly slop, and the result broken.
Not sure how you can go full AI with confidence, without software engineering expertise.
Nearly all prompts and AI coding needs refinements. If you skip this clean up, over time the amount of slop with render your app first unstable, then unusable 🙌
See you next 👋