What is API-first development
In a distrusted system, such as a microservice architecture, inter-process communication plays a crucial role. But it comes at a price. Distributed systems are often harder to debug and to refactor. In a monolith, the IDE and the compiler help refactoring programming APIs such as an interface or abstract base class. A mismatch between the API provider and the consumer will usually be caught by the compiler. In a distributed system, usually no compiler helps when changing APIs and we cannot rely on it to find mismatches between consumers and providers. Without care and special testing techniques, errors will only surface at runtime, making the system more brittle and changes more costly. Therefore, it is important that:
- APIs are well-designed so that breaking or incompatible changes are infrequent
- if required, breaking or incompatible changes happen in a controlled way that prevents API consumers from suddenly failing without prior notice.
Moreover, one of the benefits or distributed systems – if done right – is the higher potential to develop in parallel and with different development teams. This is only possible if API providers and consumers have agreed on a stable API.
API-first development provides a solution to the aforementioned problems by making the definition of the API the first step in the development process. Before starting to implement, provider and consumer agree on the desired API (changes) by negotiating them using a formal specification of the API (independent of the programming languages used to implement the provider or consumer).
This process has several benefits:
- By placing the negotiation of the required API changes up front, including a review process, the API is most likely better designed and better fits the purpose [Vasudevan2006]. Therefore, costly and potentially insecure breaking changes are less likely after this step.
- Once defined, API provider and consumer can develop in parallel. API providers can make use of validators such as prism proxy to ensure that their new implementation complies with the defined API. Consumers can make use of mock servers such as prism mock or MockServer to emulate the yet to be completed API provider for development and testing purposes.
- Based on the formal specification of the API, linters can support the API definition process by alerting about potentially breaking changes or by enforcing API consistency rules.
- The API specification can become a source for code generation, later on simplifying the development process. [Bryzek2018]
Of course, like everything in software engineering, there are also downsides to using an API-first development process. Most notably, putting the API definition first delays the start of the implementation [Vasudevan2006]. Also, developers needs to learn the specification language. Finally, developing systems using the API-first principles requires changes to the development process. All participating developers and development teams need to agree that all API changes are performed using API-first methods. Otherwise, the benefits of this process are lost soon. But the same is true for many ideas in software development. Infrastructure as code also loses its value if you constantly have to battle infrastructure drift because someone changed the setup without adapting the code…
API-first development for REST APIs with OpenAPI
When developing RESTful APIs, the predominant specification format is OpenAPI. Despite some promising alternatives filling a few gaps that the current OpenAPI standard has (e.g., RAML or AsyncAPI), the most established format with the widest range of supporting tools and frameworks remains OpenAPI. OpenAPI is a general format for describing REST APIs via YAML or JSON documents. The petsore example gives a good overview of how an OpenAPI specification looks like.
API-first and code-first development with OpenAPI
Using OpenAPI specifications is not limited to API-first development. OpenAPI specifications can also be used in a process that is usually called code-first development. Many web development frameworks offer options to directly generate the OpenAPI specification from the implemented endpoints, thereby making the specification a byproduct of the implementation. To end up with usable specifications, this approach often requires enhancing the endpoint implementations with additional documentation or annotations to guide the OpenAPI generation process. My personal feeling is that it is pretty hard to end up with a specification that is of similar value and rigor as a hand-crafted one. Moreover, the required annotations often make the endpoint implementation more verbose and harder to read. See Required annotations for good OpenAPI specifications in Spring Boot for an example of how this might pollute an otherwise simple implementation. In contrast, Endpoint definition in an OpenAPI specification shows a comparable endpoint definition specified using OpenAPI. The specification is better readable and avoids polluting the implementation. In any case, if you decide against going the API-first approach, generated specification are often better than having nothing at all. However, you will also lose most of the aforementioned benefits of the general API-first process.
Here is an example of which annotations are required in Spring Boot to end up with a well-documented auto-generated OpenAPI specification.
@Operation(summary = "Get a book by its id")
@ApiResponses(value = {
@ApiResponse(responseCode = "200", description = "Found the book",
content = { @Content(mediaType = "application/json",
schema = @Schema(implementation = Book.class)) }),
@ApiResponse(responseCode = "400", description = "Invalid id supplied",
content = @Content),
@ApiResponse(responseCode = "404", description = "Book not found",
content = @Content) })
@GetMapping("/{id}")
public Book findById(@Parameter(description = "id of book to be searched")
@PathVariable long id) {
return repository.findById(id).orElseThrow(() -> new BookNotFoundException());
}Source: https://www.baeldung.com/spring-rest-openapi-documentation
paths:
/books/{book_id}:
get:
summary: Get a book by its id
parameters:
- $ref: "#/components/parameters/BookId"
responses:
"200":
description: Found the book
content:
application/json:
schema:
$ref: "#/components/schemas/Book"
"400":
description: Invalid id supplied
"404":
description: Book not foundA definition resembling what would be generated from Required annotations for good OpenAPI specifications in Spring Boot. While the total number of lines is larger, the definition visually less cluttered and easier to read.
Setting up an API-first development workflow in Quarkus
The remainder of this blog post describes how to set up an API-first development workflow in Quarkus (with Kotlin). All explanations given here are glued together in the quarkus-api-first-example on my GitHub profile. If in doubt, look into that project and experiment with it to understand how things work. The example project uses Maven as the build system but things should be transferable to Gradle pretty easily.
The most important decision taken here is that we use code generation with the OpenAPI Generator only for generating the classes directly related to implementing the REST API inside an existing Quarkus project. By default, most generation targets supported by OpenAPI generator would create a fully functional project including all boilerplate code including the build system, gitignore file etc. While this sounds like an easy path to start, such an approach makes iterating and production-readiness a lot harder. First, adapting such a project to new API changes is hard as the generator would overwrite all manual modifications and implementations done after the initial project generation. Second, the generated projects are very opinionated and only a few configuration options exist with respect to the resulting configuration. The opinions of the template authors might not match what you need and customizing these projects via template changes for the generator is not an easy task. Therefore, I think the better approach is to let the generator only handle the parts of the application that are concerned with actual REST API implementation by restricting code generation to REST resource interfaces and model classes.
Starting from a basic Quarkus with Kotlin project similar to the one bootstrapped in the official docs, I’ll now explain important dependencies and configuration changes to the project. The project later serves an implementation for the aforementioned petstore API.
Add the API definition to the project
The first step we need to do is to provide the API definition file in the project resources so that we can bootstrap the API-first workflow from the specification in that file.
For this purpose, the petstore.yaml file mentioned before is placed under src/main/resources/META-INF/openapi.yaml.
This is the canonical path for a pre-defined OpenAPI specification for some of the Quarkus extensions used later on.
Other locations would be possible but would require further configuration options.
Enable SwaggerUI
When working with an OpenAPI specification, one usually wants to provide a SwaggerUI based on the API specification for easier development and experimentation with the implemented API.
Quarkus can natively serve the SwaggerUI using an extension, which we add to the dependencies section in pom.xml:
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-smallrye-openapi</artifactId>
</dependency>With these changes in place, we can start the Quarkus project for the first time using:
$ ./mvnw quarkus:devAfterwards, the new SwaggerUI is available as part of the Quarkus Dev UI under http://localhost:8080/q/dev-ui/io.quarkus.quarkus-smallrye-openapi/swagger-ui.
Only serve the defined API in SwaggerUI
By default, SwaggerUI would serve a combined specification comprising the contents of the YAML file and further endpoints declared only in the application code.
In an API-first world, this is usually not desired, because any publicly available endpoint has to be defined in the OpenAPI specification first before being implement.
To avoid this behavior, we restrict SwaggerUI to only serve the statically defined specification by adding the following line to src/main/resources/application.properties:
mp.openapi.scan.disable=trueLet Quarkus determine the server in the OpenAPI specification
The OpenAPI standard allows defining servers, which are URLs under which the API is served for clients.
These servers are selectable in SwaggerUI and used when testing requests there.
This model is nice for testing publicly deployed APIS.
However, it doesn’t work well for local development purposes.
We’d have to anticipate all potential local testing environment of different developer systems and pollute the servers array with all of them (IPv4 vs IPv6, running on a remote test host vs. localhost, etc.) so that local testing is possible.
To prevent this configuration issue, it is better to let Quarkus determine a suitable server for us depending on how the application is launched and accessed.
The first thing we need to do is to get rid of any manually defined servers in the servers array of openapi.yaml:
servers: []The second thing to do is to instruct Quarkus to add a suitable server automatically by adding another option to src/main/resources/application.properties:
quarkus.smallrye-openapi.auto-add-server=trueAfterwards, we get an auto-generated server that’s suitable for local development:
Generate API stubs using the OpenAPI Generator
One of the main benefits of an API-first approach with an OpenAPI specification is that we can make use of the OpenAPI Generator to generate code from the specification. This ensures that our implementation complies to the implementation and does not drift from it over time. And it also avoids manual work for us. Therefore, the next thing we do is to use the OpenAPI Generator Maven plugin to generate REST resource interfaces and model classes for use. Afterwards, actually providing the endpoints boils down to implementing the generated interfaces using the generate model classes.
Setting up the OpenAPI Generator plugin is a bit involved and I’ll explain the different configuration options in detail after showing the full plugin configuration, which needs to be added to the plugins section in pom.xml:
<plugin>
<groupId>org.openapitools</groupId>
<artifactId>openapi-generator-maven-plugin</artifactId>
<version>7.7.0</version>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
<configuration>
<inputSpec>${project.basedir}/src/main/resources/META-INF/openapi.yaml</inputSpec> 1
<generatorName>kotlin-server</generatorName> 2
<modelPackage>${project.groupId}.adapters.rest</modelPackage> 3
<apiPackage>${project.groupId}.adapters.rest</apiPackage> 3
<generateApiTests>false</generateApiTests> 4
<generateModelTests>false</generateModelTests> 4
<generateSupportingFiles>false</generateSupportingFiles> 5
<configOptions>
<sourceFolder>src/main/kotlin</sourceFolder>
<library>jaxrs-spec</library> 6
<useJakartaEe>true</useJakartaEe> 6
<interfaceOnly>true</interfaceOnly> 7
<useCoroutines>true</useCoroutines> 8
<returnResponse>true</returnResponse> 9
</configOptions>
</configuration>
</execution>
</executions>
</plugin>- We instruct OpenAPI generator to generate code from our specification residing in the Quarkus default location, which OpenAPI generator isn’t aware of without this configuration option.
- As this example is a Quarkus Kotlin project, we want to generate Kotlin code for a server implementation of the API.
- We have to provide a package into which the API resource interfaces and data classes for the models are generated.
- We don’t need auto-generated tests for the server code. These are usually of low value and manually written tests better reflect needs of the implementation (we usually do grey-box testing).
- By default, the
kotlin-servergenerator target would generate a fully-functional server project. As explained above, we don’t want this and disable all supporting files that surround the actual API. - The
quarkus-restextension used in this project is based on JAXRS. Therefore, we want the generator to generate code complying with this specification. Moreover, we need to use the Jakarta EE namespace for the specification classes and annotations, which is the default in Quarkus. - To support iterating the API and the implementation we need a strong separation between generated code and the manual implementation. Therefore, we let the OpenAPI generator generate interfaces for the REST resources only. We can then provide the API implementation by implementing these interfaces in completely separate files (and a separate source tree). That way, a new invocation of the generator will never change manual implementations we have done and we can safely iterate.
- As this project uses Kotlin, it’s probably a good idea to generate route functions as coroutines in the resource interfaces so that we can freely make use of the couroutine-based ecosystem available in Kotlin.
- Be default, generated functions would be of the form
operationId(params): ModelClass. This results in concise API implementations but limits our options to control things like response headers. Therefore, we instruct the generator to expect instances ofResponseinstead. This is a trade-off, as we now lose type-safety in the function signatures. In case you never need to change response headers dynamically, not using this options is probably the better way to go.
With this configuration in place, a fresh run of ./mvnw compile (or ./mvnw generate-sources) results in the OpenAPI Generator creating two classes.
First, a model class for a Pet complying with the defined JSON format when serialized:
package de.semipol.adapters.rest
import com.fasterxml.jackson.annotation.JsonProperty
data class Pet (
@JsonProperty("id")
val id: kotlin.Long,
@JsonProperty("name")
val name: kotlin.String,
@JsonProperty("tag")
val tag: kotlin.String? = null
)Second, an interface for a REST resource providing the routes defined in the API specification:
package de.semipol.adapters.rest;
import de.semipol.adapters.rest.Error
import de.semipol.adapters.rest.Pet
import jakarta.ws.rs.*
import jakarta.ws.rs.core.Response
import java.io.InputStream
@Path("/")
@jakarta.annotation.Generated(value = arrayOf("org.openapitools.codegen.languages.KotlinServerCodegen"), comments = "Generator version: 7.7.0")
interface PetsApi {
@POST
@Path("/pets")
@Consumes("application/json")
@Produces("application/json")
suspend fun createPets( pet: Pet): Response
@GET
@Path("/pets")
@Produces("application/json")
suspend fun listPets(@QueryParam("limit") limit: kotlin.Int?): Response
@GET
@Path("/pets/{petId}")
@Produces("application/json")
suspend fun showPetById(@PathParam("petId") petId: kotlin.String): Response
}We can now go along and implement this interface using a separate class.
Implementing a generated API resource
Implementing the API boils down to creating a class that implements the generated PetsApi interface.
Before I can show how this looks like, we first need a dummy backend to store and request instances of Pet from.
Here’s a simple stub repository implementation [Fowler2003] for pets that’s used in the example project:
@ApplicationScoped
class PetRepository {
private val pets = mutableListOf<Pet>()
fun clear() {
pets.clear()
}
fun addPet(pet: Pet) {
pets.add(pet)
}
fun listPets(): List<Pet> = pets
fun getPet(id: Long): Pet = pets.first { it.id == id }
}Warning:
This repository implementation is a stub!
Many implementation aspects are ignored (e.g., duplicate IDs, persistence) and by using the generated Pet model class in what is usually the domain layer of the application, we couple the domain code to the presentation.
This is often not desirable but ignored for this example focussing on the presentation layer only.
With this repository in place, we can now implement the REST API:
class PetsApiImpl(private val petRepository: PetRepository) : PetsApi { 1
@ServerExceptionMapper 3
public fun mapNoSuchElementException(e: NoSuchElementException) =
Response
.status(Status.NOT_FOUND)
.entity(Error(Status.NOT_FOUND.statusCode, e.message ?: ""))
.build()
override suspend fun createPets(pet: Pet): Response {
petRepository.addPet(pet)
return Response.created(URI.create("/pets/${pet.id}")).build()
}
override suspend fun listPets(limit: Int?): Response {
return Response.ok(petRepository.listPets()).header("x-next", "fake value").build() 2
}
override suspend fun showPetById(petId: Long): Response {
return Response.ok(petRepository.getPet(petId)).build()
}
}- We create an implementation class for the API that implements the generated interface.
- Only when returning
Responseinstances, we are able to modify response header values dynamically such as needed for the exemplary pagination header. - Fulfilling the contract for error responses is usually done by mapping domain exceptions (or due to the stub implementation, general JDK exceptions) to appropriate responses using framework features.
The resulting implementation is pretty concise and easy to read. The generated interface ensures that the implementation provides the required routes with matching input parameters. Iterating the specification is easy and free from loss of manually implemented code. When the OpenAPI specification is modified, the manually implemented endpoints remain untouched and the compiler will spot the most severe discrepancies that need to be addressed. For instance, if a new route is added, we cannot forget to implement it, because otherwise the interface is not implemented correctly. This gives a lot of automated support for ensuring consistent and complete API implementations, complying with the specification.
Summary
With a few plugins and suitable configuration options, an API-first development workflow can easily set up for Quarkus. The resulting projects gives a lot of developer support that prevents an implementation from drifting from the specification. However, the initial setup shown here is not perfect and of course there are more challenges to tackle and costs to consider.
First, the OpenAPI generator is not perfect and some valid specifications result in awkward or unusable generated code. For instance, polymorphism usually results in struggles that need manual intervention. The outcome is usually on the positive side if the specification is part of the service itself and not taken from somewhere else. In this case, generator problems can be worked around by a mix of tweaking generator templates, making use of vendor extensions in the templates, and – in the worst case – by modelling the API specification in a way that works around these issues. If the specification is immutable and not an asset of the project itself, the resulting trade-off decision might go against a use of the OpenAPI generator to avoid these corner cases.
Second, the setup shown here delegates the task of ensuring consistency of implementation and specification to the Kotlin compiler.
However, not all mismatches are detectable here.
For instance, the method signatures of listPets do not enforce the x-next response header.
Therefore, we need further means to gain more confidence in the match between implementation and specification.
One option is described in the next blog post.
Bibliography
- [Vasudevan2006] Vasudevan, Keshav. "Design First or Code First: What’s the Best Approach to API Development?" Swagger.io, February 21, 2017. https://swagger.io/blog/api-design/design-first-or-code-first-api-development/.
- [Bryzek2018] Bryzek, Michael. "Design Microservice Architectures the Right Way." Presented at the QCon New York, New York, NY, USA, September 10, 2018. https://www.youtube.com/watch?v=j6ow-UemzBc.
- [Fowler2003] Fowler, Martin. "Repository." martinfowler.com, March 5, 2003. https://martinfowler.com/eaaCatalog/repository.html.
