I love coding, so to me vibe coding (even ignoring the quality of the result) is not something I would do. I'm probably also faster and more accurate anyway. So for developing my own package I would not resort to vibe coding ever. I'd probably only use AI to do some chores or as a way to play with it.
At the moment I'm working on writing documentation for the Apie project. And as a package maintainer I should write my documentation for humans, but in the current world you also write documentation for AI agents. So I decided to see what would happen if a vibe coder would try out Apie and fix the documentation so an AI agent would understand how it works.
Setup
I had to decide where to start. Since the Apie library is still in release candidate status and not a very popular option like the Api Platform or Laravel, an AI agent would never pick my library for developing a new application. However I do have a composer create-project script that you could use to setup Apie very fast and it is also reused for the docker setup. So if I add AI agent specific documentation here it would be very easy to start vibe coding afterwards.
First documentation
I started with Gemini as for a non-company developer it is the friendliest with token usage etc. It would also sound a bit overkill to start using multiple AI agents at the same time to see if all of them work. It's also that all of the AI Models are not deterministic so I would have to run it so many times to see if was not just an accidental failure of the agent if it did not work.
I asked Gemini to install AI documentation. To give it a headstart I provided some details for gemini:
- The project is written to install Apie on Laravel or Symfony.
- I mentioned the class that renders the templates.
- Split the documentation in multiple markdown files.
So what did Gemini did? Gemini figured out the structure of the project quite well, but it picked to put the documentation in the core templates folder so I could not make it an option whether to include it or not. I did ask to move it to its own folder and Gemini did this perfectly.
It created the AI documentation in GEMINI.md which means it will only be used by Gemini, but I will probably fix this in the future. It did decide to split the information in multiple markdown files as a asked him to.
AI Stupidity
Another issue was that it created all coding examples to be written for Laravel and Symfony but this is not good for a LLM model to follow as you would always have to provide whether the application is in Symfony or Laravel, while the only difference is the location of the console command (artisan for Laravel and bin/console for Symfony). I asked him to make separate documentation for both frameworks, but Gemini decided to copy paste the documentation. That is a maintenance nightmare!
So I asked to reuse it by providing whether it is bin/console or artisan. It did so using the suggestion I gave, but decides that other difference between the frameworks by comparing the settings object directly in the template which is a bit of an anti-pattern and not in line with the suggestion I gave. I kept it for now, but maybe in the future I might modify this.
The first attempt of vibe coding
So I ran make test-project went into the folder and started Gemini. I asked it to make an Address resource and it worked! So my findings:
- The console commands you could use mentioned in the documentation were constantly executed by Gemini to verify his action worked, so it keeps seeding data and clean the code style
- It creates domain objects as instructed with bin/console apie:create-domain-object, but decides that it should set the id to uuid explicitly. By default it selects ulid. The fun part is that it got the value incorrect, so it decides to read the source code to see what it should write to get uuid id's.
- In my code documentation all identifier classes are suffixed with 'Id', but apie:create-domain-ibject creates identifier classes with the suffix 'Identifier', so Gemini decided to rename them all after creation.
- It uses actual value objects over using primitives. This was not documented, but somehow Gemini did decide this correctly.
- For some reason you could only order one product at a time.
- The code originally crashed because the User object did not implement isDisabled() as was requested by the interfaces. It also fixed this by adding a 'deleted' status to the user status enum, which is a terrible idea as you can not recover the original role once a user is deleted.
My second attempt of vibe coding
So I updated the documentations adding extra instructions for the odd behaviour, removed the test project and create a new project and tried it out again. The prompt I used to let Gemini make a REST API:
Can you make a webshop with Apie? Employees can log in and see all orders and have full CRUD over all products. Customers can see all published products and can place an order of multiple products as long inventories allow it. Customers can only see their own products. They do not need to login to place an order, but a prospect user account should be created.
The above description can be made with the Apie library, but without documentation it would be hard to know how. What comes out of it was an API and there were some attempts in adding authorization checks, but the result was actually worse:
It seems the documentation was not accurate enough for an AI agent to understand how Apie works.
- It used enums correctly.
- It was still calling bin/console apie:create-domain-object --help first and then decided to fill in the optional --id-type argument even though I explicitly ask him not to.
- It uses actions as controllers. While these are possible actions, action are meant to be used for non-CRUD actions, like logging in.
The weirdest thing that Gemini did was trying to add authorization checks. It added these to the AppServiceProvider:
I have no idea where this getApieUser() method comes from (in Laravel the Laravel user object is the ApieUserDecorator class, but to get the Apie entity you need to call getEntity() instead), but it tries to handle all authorization from AppServiceProvider.
The actions are also completely wrong and the #[Context] part as well:
More additions
So more additions were added. I also found a few bugs with Apie in Laravel (Normally I use Symfony as backend framework) and the agent even wrote me code to be able to login into Apie really fast wih a change in the Apie facade without having to call the method verifyAuthentication.
Did the code improve? Short answer: yes it did, but sometimes a previous instruction was no longer followed. Wrtiing automatic tests to see if it does everything as instructed also felt like a weird way to test as the AI could sometimes be wrong and sometimes be right.
Still it's cool to see that we write documentation for AI bots instead. It's also cool to see where AI does not get your documentation to improve your docs. In case you are really lazy you can ask AI to write a prompt for AI to understand the documentation. My experience is that AI Agents are not good in sorting on priority, so the results will not be good.
Conclusion
So my conclusion might surprise you:
- do you want to get your library used by others, make a good documentation. This is even more evident when a large group of developers use AI in their daily workflow.
- Most package developers do not test if AI understands their language models, but the group of 'vibe coders' is getting bigger.
- Let AI try to figure out your library to see where your documentation still sucks.
- AI agents get better. Last year I tried, but it could not figure out how to use value objects and relied heavily on primitive obsession. It was very much writing all codes in properties, constants and config vaul.
- The more structured your code the easier it is for Apie to find the lost entity body.
- It seems letting AI generate documentation often leads to text that is easy for other AI agents to understand.
- Mentioning url's for testing and how to run specific console commands also helps AI a lot.
- AI agents rarely write documentation for other AI agents for some reason. Every agents also have a differen timing related to reifistering them.
- Agents do read tests and how a class is being used, so good unit tests help AI very well.
Comments
Post a Comment