⚙️ Case Study: Building Efficient Pagination in Spring Boot When 3RD Party Doesn’t Support It
🚀 Introduction
I faced one of those real backend challenges that every Spring Boot developer eventually runs into:
How do you implement pagination when your data comes from an ESB or third-party API that doesn’t support any paging parameters?
My service had to fetch 10 000+ transaction records from the ESB in a single response — and the frontend expected a neat paginated API like: /transactions?page=1&pageSize=100.
To solve this, I explored two possible solutions: 1️⃣ An in-memory, configuration-driven pagination utility. 2️⃣ A Redis-based tokenized caching approach.
Here’s the full breakdown, the trade-offs, and what I finally chose.
🧠 Approach 1 – Configurable In-Memory Pagination
In this approach, the service retrieves the entire dataset from the ESB and performs pagination inside the Spring Boot layer using a generic pagination utility.
I configured default values (page = 1, pageSize = 100) in application.yaml, so even if the client didn’t send them, the backend handled it gracefully.
Once the ESB response arrived, I passed the list to a reusable PaginationService<T> that sliced the list, calculated total pages, and returned metadata like hasNextPage, totalRecords, etc.
✅ Benefits
🟢 Simplicity – Easy to implement, test, and maintain. 🟢 Reusable – Works for any list type (Transaction, Customer, etc.). 🟢 Config-Driven – Defaults stored in YAML, no hardcoding. 🟢 Data Freshness – Every request fetches live, real-time data from ESB. 🟢 Regulatory Safe – No sensitive data stored or cached, ideal for banking or compliance systems.
❌ Drawbacks
🔴 Performance Overhead – Each request re-downloads the full dataset; response time depends entirely on ESB latency. 🔴 Increased Network Load – Every pagination call hits ESB again, multiplying upstream traffic. 🔴 Limited Scalability – Under heavy load or large datasets, this can strain both the ESB and the application server.
This approach is perfect for banking and regulated environments, where data integrity and freshness matter more than performance speed. Every response is guaranteed to reflect the latest state of the system — which is crucial for transaction data, balances, or compliance reporting.
⚙️ Approach 2 – Token-Based Pagination with Redis Caching
The second idea focused on improving speed. Instead of calling ESB for every page, the service would call ESB once, cache the dataset in Redis, and generate a unique token.
Subsequent requests from the frontend would send that token along with page and pageSize, and the backend would serve slices directly from the cached data.
✅ Benefits
🟢 High Performance – After the first call, pages load almost instantly (milliseconds). 🟢 Reduced ESB Load – Only one ESB call per session, protecting upstream systems. 🟢 Smooth User Experience – Ideal for dashboards or reports that require browsing multiple pages quickly. 🟢 Scalable for Read-Heavy Systems – Perfect when many users view similar static data.
❌ Drawbacks
🔴 Stale Data Risk – Cached data may become outdated if ESB updates after caching. 🔴 Memory Usage – Large datasets stored in Redis consume RAM; expensive for high concurrency. 🔴 Complex Infrastructure – Requires cache cluster setup, TTLs, eviction policies, and token management. 🔴 Security & Compliance Concerns – Storing customer data in cache can violate internal or regulatory policies (PCI DSS, GDPR).
This approach shines in analytics or reporting systems where real-time accuracy isn’t critical, but performance is. However, in banking applications, caching entire transaction datasets — even temporarily — introduces compliance and data-staleness risks.
🧩 Performance Discussion
Both methods begin the same way — an ESB call returning thousands of records. The difference lies in what happens afterward.
With the in-memory method, every page call fetches fresh data, keeping results accurate but slower.
With the Redis approach, only the first request is heavy; subsequent ones are lightning-fast but potentially outdated.
In tests:
⏱ In-Memory: 1.5 – 2 seconds per call (consistent freshness).
⚡ Redis: 1.8 s first page, then 50 ms per subsequent page (stale snapshot).
So, it becomes a trade-off between speed vs truth.
🏦 Which Approach Is Better?
After weighing everything, here’s my takeaway from a banking-grade engineering perspective:
✅ In-Memory Pagination Wins Because:
Real-time data is non-negotiable.
No sensitive caching.
Simple to audit and maintain.
While the Redis caching model is attractive for speed, freshness and compliance outweigh performance in financial systems.
If the ESB later adds page or limit parameters, this in-memory pagination structure can evolve seamlessly into native pagination — no redesign needed.
💡 Key Lessons Learned
🟢 Always prioritize data integrity over optimization when dealing with customer or financial data. 🟢 Keep pagination logic generic and reusable — you’ll thank yourself later. 🟢 Externalize defaults (page, pageSize) to configuration files for flexibility. 🟢 Monitor ESB latency; sometimes backend caching isn’t needed if the source is fast enough. 🟢 If performance becomes critical, consider cursor-based pagination using timestamps or IDs — it’s safer than caching full datasets.
Few other approaches are mentioned below:
Streaming + Reactive Approach
Cursor Based Pagination
Asynchronus Pagination
Graphql with Resolver Level Pagination
Will discuss these approaches soon as well.
🏁 Conclusion
Building pagination without ESB support taught me that the “quick” solution isn’t always the right one. In enterprise systems — especially banking — every architectural decision must respect compliance, accuracy, and traceability.
✅ In-Memory Pagination keeps data truthful and code clean. ⚡ Redis Caching offers speed, but at the cost of data reliability.
For me, the clear winner in this context was the generic in-memory pagination approach — simple, compliant, and future-proof.
#SpringBoot #Java21 #Pagination #Microservices #SystemDesign #BackendEngineering #CleanCode #SoftwareArchitecture #BankingTechnology #APIDesign