PROTOBUF API
API first with Protobuf
Chapter 2: Define the api contract in Protobuf
Motivation
This article demonstrates the One API to rule them all concept by defining a sample API contract using Protobuf. For a deeper understanding of the underlying principles and motivations, refer to my previous article.
Designing API contracts
API design is challenging. Many APIs are accessed indirectly, through automated systems and business process integrations, including asynchronous communication. All these interactions require well-defined contracts to ensure reliable message exchange. It is crucial to avoid basing the API design solely on user interface requirements or the needs of synchronous communication. Therefore, aligning the API design with the underlying business domain is paramount.
The Domain-Driven Design (DDD) approach can be valuable in this context. It involves breaking down the business domain into smaller, more manageable subdomains, each of which can be implemented as a separate microservice. However, this decomposition requires careful consideration to avoid overcomplicating the technical solution. Effective domain decomposition necessitates close collaboration between business and software experts to ensure that the resulting microservices are both aligned with business needs and maintainable.
Domain: E-Commerce
This article explores the development of an API for a simple e-commerce application. I will focus on core concepts and practical implementation, deferring in-depth discussions on design principles like Domain-Driven Design (DDD) – including the use of verbs for domains and the past tense for events – to future articles. My goal is to provide a clear and concise API, both structurally and functionally.
The core functionality of the Kindatta Marketplace centers around the management of products and orders. Sellers list their products on the platform, while buyers purchase these products. The platform handles order fulfillment, payment processing, and customer support for the seller. This C4 model provides a high-level view of the key components and relationships within the Kindatta Marketplace domain.
Offer-Product Domain
Sellers are individuals or companies who offer their products on the Kindatta Marketplace. Sellers maintain information about the products they offer. The product entity, as it relates to the Offer-Product domain, includes attributes such as product name, description, price and availability.
Sellers change the information infrequently and distributed evenly over time. Product information is managed by automated processes designed for high fault tolerance. These processes can effectively handle high loads and compensate for potential timeouts through retry mechanisms.
Order-Product Domain
Buyers are individuals who use the Kindatta Marketplace to browse and purchase desired products. Within the Order-Product domain, the Product entity serves as a reference to the specific items included in buyer orders. Ordering products can result in the creation of multiple orders, for example, when products are purchased from different sellers.
The Order-Product domain must be designed to handle high volumes of buyer interactions, including peak traffic events such as “Black Friday” sales. The system must be highly scalable to accommodate fluctuating demand and ensure a smooth and responsive user experience for all buyers, even during periods of high demand. Effective communication within the domain is critical. Upon order creation, the domain must notify all relevant downstream processes.
Protobuf tooling
While I primarily edit Protobuf files using Visual Studio Code, I also use additional tools to improve my workflow and efficiency.
Protobuf compiler
The Protobuf compiler, also known as protoc, developed by Google, generates source code for various programming languages from Protobuf files. To simplify managing protoc versions, I created a shell script that automates the download and extraction process. Updating protoc is easy — I just change the version number within the script located at tools/protoc/install-protoc.sh. To avoid to check in in large binary files, I add the directory containing the extracted protoc files to my .gitignore file.
Buf CLI
Buf CLI is an addition built on top of protoc, enhancing it with features such as linting for protobuf. Buf Technologies, the creator of Buf CLI, also provides a Visual Studio Code extension. I have evaluated this extension in depth in a separate article, focusing on its linting capabilities.
For this project, I require a slightly modified Buf configuration. The APIs of the two domains are not permitted to share any proto definitions. Therefore, I need to place them in separate modules. Each module will reside under the root folder “proto”, followed by the name of the company “kindatta” and the project “ecommerce”.
version: v2
modules:
- path: proto
includes:
- proto/kindatta/ecommerce/product_offer
- path: proto
includes:
- proto/kindatta/ecommerce/product_order
lint:
use:
- STANDARDDefine the API contract
I will define the API contract iteratively. This means that some aspects of the contract can be scoped out initially for later extension and refinement. This approach allows rapid prototyping and provides a early preview of the application’s future functionality.
Get Nils Diekmann’s stories in your inbox
Join Medium for free to get updates from this writer.
I will begin by defining the Product-Offer domain, starting with the definition of a ‘Product’ entity. Each entity will have an ‘ID’ property of type string, which will be assigned a Universally Unique Identifier (UUID). You can define your own custom data types in Protobuf, a topic I will cover in more detail in a future article. The initial ‘Product’ entity will have only two properties: ‘name’ and ‘description’. This minimal set of properties is enough to build a simple frontend.
package kindatta.ecommerce.product_offer.v1;
message Product {
string id = 1;
string name = 2;
string description = 3;
}At this stage, the specific mechanism for how sellers add their products to the Kindatta Marketplace remains undefined. The ‘Product’ entity acts as a consistent interface within the system, regardless of the chosen seller integration method. I will leave the concrete details of the seller integration process out of scope for now.
To represent sellers, I introduce a simple ‘Seller’ entity with only two properties: ‘ID’ and ‘email’. Protobuf’ allows to add new properties without introducing breaking changes to the API. This minimal definition provides a solid foundation for future enhancements.
package kindatta.ecommerce.product_offer.v1;
message Seller {
string id = 1;
string email = 2;
}The buyer can interact with the application via a web browser or a mobile app. As the buyer requires immediate feedback, synchronous communication is essential. Each product order request includes a unique identifier, the buyer’s ID, and a list of desired products.
package kindatta.ecommerce.product_order.v1;
import "kindatta/ecommerce/product_order/v1/product.proto";
service OrderProductService {
rpc OrderProduct(OrderProductRequest) returns (OrderProductResponse);
}
message OrderProductRequest {
string request_id = 1;
string buyer_id = 2;
repeated kindatta.ecommerce.product_order.v1.Product products = 3;
}
message OrderProductResponse {}User authentication occurs against the application via a token-based mechanism. This token is then exchanged by the API gateway for a unique Buyer ID. The “Order Product” service is then invoked with the Buyer ID, eliminating the need for further authentication within downstream services. This approach simplifies the system architecture and improves overall performance. For the initial application preview, authentication will be simplified by using a fixed Buyer ID, allowing for a faster and more focused initial development phase.
In the Order-Product domain, the ‘Product’ entity differs from its counterpart in the Offer-Product domain. Here, identifying the product solely by its ID is sufficient. To allow for future extensions, the ‘Product’ entity is defined using a message, ensuring compatibility with future requirements.
package kindatta.ecommerce.product_order.v1;
message Product {
string product_id = 1;
}The products in the request are split into multiple orders, one for each respective seller. To avoid making the buyer wait for the entire order splitting process, the request is saved and then processed asynchronously. An event is published to notify interested subscribers of the new request. This asynchronous approach allows for efficient background processing and provides a more responsive user experience.
package kindatta.ecommerce.product_order.v1;
message ProductOrdered {
string request_id = 1;
}The complete API contract is available in my GitHub repository. Please consider starring the repository if you find it helpful.
Summary
In my previous article, I introduced the concept of my One API to rule them all approach. This article applies this concept by defining an API contract for two distinct domains. You might have noticed that these domains currently have no shared Protobuf definitions. This is an intentional design choice. Maintaining domain-specific definitions, even if it means some redundancy, will enhance the flexibility of the API in the long run.
Using the ‘product’ entity as an example, I demonstrate how the same entity can require different properties across various domains. To meet the buyer’s requirement for immediate feedback, I defined a synchronous gRPC call. This call is then followed by an asynchronous event that triggers subsequent downstream processes.
Following Chapters
Now I have everything in place to generate code from this API definition for various programming languages. The following chappers will be written according to user feedback.
- See below for the C# solution
- Clap 20 times 👏 for the Golang solution
- Clap 30 times 👏 for both C# and Golang solutions
- Leave a comment 💬 if you need another language
Thank you for your attention! I appreciate your feedback. ❤
Finished following Chapters
Suggested related articles while you wait
I have a GitHub repository where I experiment with my API concepts and their implementation. Please note that this repository may contain incomplete or inaccurate implementations. Use it with caution.
Part of the series: One API to rule them all
