Enhancing Data Retrieval in Microservices Architecture

In the microservices ecosystem, information is often distributed across multiple systems or microservices. In certain scenarios, applications are required to generate complete response data by combining information from various internal and external data sources. This capability is necessary to serve a set of client systems or adhere to industry-standard data exchange interfaces.

In this article, we will explore the challenge of retrieving information from multiple systems and dynamically generating responses based on the specific requirements of each client. We will begin by breaking down the entire response into smaller sections, with each section representing a logical category derived from either a local database or an external web service. Additionally, we will define a functional interface that takes request and response objects and returns a status object.

Implementing the DataAccessor Interface: To handle data retrieval for each logical section, we introduce the DataAccessor interface, which is a BiFunction taking APIRequest and APIResponse objects and returning a MappedResponse. The MappedResponse object includes a CompletableFuture to support asynchronous operations and can be extended with additional response properties if needed.

public interface DataAccessor 
  extends BiFunction<APIRequest, APIResponse, MappedResponse>{
}        

Defining the Request and Response Objects: To facilitate data retrieval and response generation, we define two objects: APIRequest and APIResponse. The APIRequest object includes properties such as the user, user request details, and a list of data sections required for the response. On the other hand, the APIResponse object represents the complex response that the API will generate.

@Getter
@Setter
public class APIRequest{
    private String user;
    private UserRequest request;
    List<String> dataSections;
}

public class APIResponse{
// This is the complex response the API would generate
}


@Getter
@AllArgsConstructor
@RequiredArgsConstructor
public class MappedResponse{
    private CompletableFuture<Void> completableFuture;
    // Add any other response as needed.
}        

By implementing DataAccessor for each logical sections, we could define small chunks of data loading functions that can operate independantly of each others. As a bonus, these can operate Async, and return CompletableFuture if so.

Mapping Data Accessors: We populate a map called accessors, where each key represents a logical section, and the value is an implementation of the DataAccessor interface. By implementing the DataAccessor interface for each logical section, we define small, independent data loading functions.

private Map<String, DataAccessor> accessors = new HashMap()

// Populate these  map with the required DataAccessor implementations.
public void initHandlers(){
    accessors.put("invoice", (req, res) -> {
        invoiceService.get(req, res);
        return new MappedResponse();
    }
    
    accessors.put("accounts", (req, res) -> {
        CompletableFuture cf = CompletableFuture.runAsync(()-> {
            accountsService.get(req, res);
        });
        return new MappedResponse(cf);
    }
    
    accessors.put("orders", new OrderHandler());
};        

Master Implementation: In the master implementation, we initialize the accessors map by adding the required DataAccessor implementations for each logical section. The populate() method takes an APIRequest and APIResponse as input and iterates through the requested data sections. For each section, it retrieves the corresponding DataAccessor from the accessors map and invokes it to fetch and populate the response data. If the DataAccessor returns a CompletableFuture, it is added to a list for later synchronization. Finally, if there are any asynchronous operations, we wait for all CompletableFuture instances to complete before returning the populated APIResponse.

// populate data based on the list of accessors
public APIResponse populate(APIRequest request, APIResponse response){
    List<CompletableFuture> cfs = new ArrayList<>();

    for(String section:request.getDataSections()){
        DataAccessor accessor = accessors.get(section);
        MappedResponse response = accessor.apply(request, response);
        if(null != response.getCompletableFuture()){
            cfs.add(response.getCompletableFuture());
        }
    }
    
    if(cfs.size() > 0) {
        CompletableFuture[] sdf = cfs.toArray(new CompletableFuture[cfs.size()]);
        CompletableFuture.allOf(sdf).join();
    }

    return result;
};        

Now by specifying the data sections needed as part of the API request object, corresponding functions would be invoked and the data would be populated accordingly.

Conclusion: Retrieving data from multiple microservices and generating dynamic responses requires careful planning and implementation. By breaking down the response into logical sections and implementing a functional interface, organizations can effectively manage data retrieval and response generation. Additionally, leveraging CompletableFuture enables asynchronous operations and enhances performance. By specifying the required data sections in the API request, the corresponding functions are invoked, and the data is populated accordingly, providing tailored responses to client systems.