While it’s not many people’s favorite thing to do, communicating updates in release notes (or a more basic changelog) is important to let partners and customers know what’s going on with the product. At DevRev, we’ve had a repeatable monthly process to communicate user-visible changes and additions for about a year. But until now, there hasn’t been a similar process for APIs.
The lack of an API changelog is a hurdle for onboarding and supporting partners. Although no breaking changes would be introduced without issuing a new version, it’s still important for partners and customers to know about the new capabilities. At the same time, we have to figure out a process that’s sustainable with our small Knowledge and API release teams.
API management
The platform that we use for our developer documentation generates the API reference from the OAS spec that our API wrangler Brian uploads whenever there are changes. There’s a bunch more that happens behind the scenes, but this is the point at which changes come to my awareness, since I have to approve the PR. This is also the point at which we need the changelog, ideally as part of the same PR.
We have two versions of the API: public and beta. The changelogs should be separate because different things happen with each release.
Automation
It’s critical to have the changelog generation automated as much as possible. It’s just out of the question to use a quarter of a headcount to manually prepare changelogs, which is what I estimate it would take.
About a year ago, I asked one of my team members to investigate utilities for automatically generating a changelog, of which there are a few. The output from these is comprehensive but completely unusable.
Here’s an example of the OpenAPI Specification diff changelog entry for adding a new enum value to an attribute on an object.
POST /conversations.create
- Responses changed
- Modified response: 201
- Content changed
- Modified media type: application/json
- Schema changed
- Properties changed
- Modified property: conversation
- Property 'AllOf' changed
- Modified schema: subschema #2
- Properties changed
- Modified property: messages
- Items changed
- Property 'OneOf' changed
- Modified schema: subschema #1: timeline-change-event
- Property 'AllOf' changed
- Modified schema: subschema #1: timeline-entry-base
- Property 'AllOf' changed
- Modified schema: subschema #2
- Properties changed
- Modified property: object_type
- New enum values: [incident]
Twenty lines to say that an enum value was added! What’s more, only three or so of these lines are meaningful. To top it off, since this schema is used for many objects in DevRev and the generated changelog is organized by endpoint, these same twenty lines are duplicated dozens of times. The changelogs that I generated for the past three months beta updates were mostly in the range of 1-3,000 lines. One, however, was over 15,000 lines!
Even so, having this listing was progress over the raw YAML diff. It certainly wasn’t the final product but gave me something to work with. So set to work I did, taking the 15,000-line generated changelog and turning it into something comprehensible. It took a couple hours of copying, pasting, and chuckling with derision to come up with a pretty reasonable 25-line changelog–1/600th of the generated changelog.
Assistance from AI
Manually compacting the generated changelog was fine to do once as a research and design activity, but not something to do once or twice a month.
I was then reminded that the I’d Rather be Writing blog had a post on using AI for an API changelog. The technique described there uses file diffs, and I figured that a structured diff would be even better as input. Tom generously included the nearly 100-line prompt he uses. I customized it a bit and submitted it to an LLM.
The results were almost exactly what I wanted.
Frustration with AI
Then when I started in on refinements, they weren’t. Inexplicably, the same prompt on the same LLM on a different day yielded radically worse results. It just started ignoring 75% of the changes. Perhaps the nadir of my professional career was when I was arguing with it that I had given it everything and it had indeed not included everything. Finally, after days of back and forth with it, I somehow wrested everything out, and in my ecstasy asked what prompt I should use next time to get the same result, which it enthusiastically provided. Then when I used that prompt in a new chat, it was back to ignoring the majority of the content.
My ecstasy having turned to despair, I reached out to DevRev’s prompting wizard, Harpinder. He gave me two pieces of advice: first, include all the content in the prompt directly, not in an attachment; and second, use a much shorter prompt. I have no idea why the first one matters, but it helped and that’s good enough for me.
AI triumph
It’s astounding, but using a shorter prompt really made the difference. The main prompt (only about 5 lines) is just this:
Please provide an API changelog for the {version} API from the following OASDiff of OpenAPI spec changes. The output should be in markdown format grouping endpoints by use case/object type. For cases where some schema is modified, please also tell what endpoints it affects. Wherever an endpoint, property, or enum value is mentioned, surround it with backticks.
Then one last thing I wanted was for every mention of an endpoint to hyperlink to the API reference. The path of the reference page for an endpoint is deterministic from the spec, so I wrote a bit of Python to read the API spec and generate the list of links. That list is also included in the prompt along with the instruction:
Wherever an API is mentioned, include a hyperlink to the corresponding path from the
<api_links>section.
And well, it pretty much works. There’s still some variation that is confounding, but so far it hasn’t been unmanageable.
The solution
The overall solution is in a GitHub action. Whenever the specs are updated in the repo, oasdiff runs and generates the verbose changelogs. Then a Python script runs, which reads the OASdiff files, generates the API reference URLs, compiles the prompt, and sends it to the LLM. The resulting changelog is added to the same branch.
The SMEs and I just have to check it over and remove or rephrase anything that warrants it. When we merge the PR, the changelog is published at the same time the reference is updated.
This process is an example of AI enabling something that would simply not have been possible otherwise. You can see it on our developer docs site.