A few days ago, I saw this post about the Mintlify AI Doc Writer. It is an IDE extension that generates docstrings automatically for functions and classes. This sounds like a great plugin for saving developer’s time. Who wants to spend time on writing docstrings while you can build more interesting things!
In this story, I will first try with “normal” usage of the plugin. Then, I will test the AI with some extreme/confusing cases to see how far could it go.
Test it with a simplified Wordle
To try a normal use case, let’s create a simplified wordle game and ask Mintlify AI Doc Writer to generate the docstrings for them.
Here is the code (Sorry for the long Gist):
There is no complicated logic here and I try to name the function as declarative as possible. But having docstrings should make the function even easier to read. So, let’s try the AI Doc Writer out!
Working with a Named Tuple
The named tuple itself is already simple enough. In this case, the AI Doc Writer created a comment of “Why” this named tuple is created.
Understanding if-else statements
The generated docstring says:
If the answer is correct, return a list of green characters. Otherwise, return a list of yellow characters
:param word: The word that the user has guessed
:return: A tuple of a boolean and a list of characters.
Although the description is not 100% accurate, it correctly predicted the abbreviation of G and Y as colors. More interestingly, for the input parameter word, the AI was able to infer that it is a “Guess” from the user.
Comments for a simple class
Given the whole Answer class, AI Doc Writer generated the comment correctly. Indeed, we are trying to write a word game!
Working with a wrapper function
It takes a word as input and returns the result of the comparison between the word and the answer
:param word: The word that the player guessed
:return: A CompareAnswerResult object.
For the player_guess function, it is a wrapper for Answer.check_for_word function. To my surprise, it can spell out “comparison between the word and the answer”. However, this only appeared once in a while. The generated docstring for the same function is different across different trials.
For the first 4 trials, the docstring says something different:
The player_guess function takes a word and returns the result of the check_for_word function
Working with the prompt function
If the game is still in progress, print a prompt for the user to guess a word, and then pass the user’s word to the player_guess function
:param word: The word the player is trying to guess
:type word: Optional[str]
Again, it does a pretty good job describing what the code does.
Describing Test Cases
When dealing with the test cases, the AI Doc Writer is doing okay. At least half of them are impressive
Trying the AI Doc Writer with Extreme Cases
The normal use case looks good to me. Let’s try to trick the AI with extreme cases!
Adding Meaningless steps
The AI Doc Writer seems to be aware of the content of the code block. Let’s try to add some meaningless steps between the prompt function to see if we can confuse it!
To confuse the model, I added some random arithmetic calculations and non-existent conditions around the main logic. The docstring now says
This function is a function that is a function
That is so deep!
Meaningless function and variable names
This time, let’s mess with the function name and variable names but not change the logic. This time, I will also use the prompt function.
I hope you still recognize the function.
If the game is in the GniYalP state, print a message asking the user to guess the word and then call the pg function to check the user’s guess
:param shoooo: The Hangman object
:param dorw: The word to guess
:type dorw: Optional[str]
The result looks good! It can say something about states, user guesses, and checks. Although it might have guessed something wrong about Hangman, it is still a word game.
Misleading function name
For this final extreme case, I will test the AI with misleading function names with a simple Fibonacci sequence.
This is a textbook code for generating the n-th Fibonacci number. The docstring is generated correctly by the AI Doc Writer.
However, things don’t go quite well when the function name/signature is misleading
Final Thoughts
After all these tests, I think the Mintlify AI Doc Writer is doing well! No machine learning model can be perfect and the current result is impressive to me. However, misleading names and meaningless logic in the code will confuse the model. Even an AI could not save you if the code is written badly.
This tool will be very useful for me in 2 ways.
First, it will save me a lot of time and keystrokes from writing docstrings from scratch. The AI Doc writer can describe “What” is the function doing. I only have to do some modifications and probably add the “Why”s in the docstring.
Secondly, it would be a useful help for reading and understanding undocumented code by ghosts who left the company for a long time (or myself a few months ago).
Hope this is useful for you as well. If you like this article, please follow for more interesting articles in the future.
Thank you for reading!
