Pulling the full stack the correct way (part 1 of 2)

Read Part 2 - The “lean manufacturing” concept

PART 1 - The common case

"I have learned the hard way that a website redesign project plan can save you enormous time and money and effort. Your website is outdated if it is more than 4 years old. I encourage people to keep up to date on current trends and graphics, and implement these into your website's design. Do not completely rebrand your company when redesigning your website, remain true to your company's values while maintaining an aesthetically pleasing and efficient site." —Leonard Ang, CEO of iPropertyManagement

You and your team were assigned the redesign of the current website because reasons (the competitor's website looks better or is offering a new functionality, your site is old and/or looks ugly, you are the new manager and want to imprint the new brand vision or identity…). There are tons of guides about how to redesign a website, but the most part are focused on the frontend: prioritize user experience, improve site’s SEO (Search Engine Optimization), improve site navigation, device-specific optimization, CMS needs, and so on. ¿What about the backend APIs?

Every new design starts by the mockups. You work with your (brand-new) UI/UX expert to design the mockups and the new user flows, but you don’t care too much about the backend APIs because they provide the data to the current site so it’s just a matter of reusing them. You’ll look at the APIs later. However, at some point during the re-design process, you ask your UI/UX expert to forget completely about the current site. You want something completely new, with a different approach where the focus is in a different entity. For example, if the old site was focused in the customer, the new one will be focused in the order flow (this decision can affect the data expected from the current APIs as we see later).

When the mockups are finished, you present them to the frontend developers, and they start the implementation. You ask them to use mock data and defer the integration with the backend until you can take care of the backend APIs. ¿Who defines the mock data? Normally, they use the data provided by the designer in the mockups. In the best case, they work with your product owner to define mock data based on the business needs/rules. ¿Does your product owner’s knowledge of the business include the existing APIs and the business rules they apply?

Then, you start looking at the backend APIs that will provide the real data. More sooner than later you'll discover that's too late.

Some common implementation issues

When you start the implementation of the website based solely on the mockups, without clearly defining the source of the data first, you'll be more likely to encounter some common implementation issues. Next, we’ll look at some of them.

Missing input data for existing APIs

There are existing APIs the UI can consume, but these APIs expect data the UI is not able to provide unless you modify the mockups or the user flow.

For example: to see the list of orders, according to the user flow, the user must open the left panel and click on the Orders icon. Then, the UI will make a call to the Orders API to get the list. Since a user can contain multiple accounts, the Orders API requires the customer account number as a parameter, whereas the user flow is not allowing the user to select any customer -¿do you remember you asked your designer to forget completely about the current site?-. You need to modify the current flow and the mockups to add a drop-down list allowing the user to select the customer. This can cause a delay in the frontend implementation, which is already in progress.

Also, to populate the drop-down list, you need to make an extra call to the Customers API to get the list of associated accounts. You need to integrate that call in the frontend.

Complex business logic on the client side

There are existing APIs the UI can consume, but the UI needs to call multiple endpoints to get the data to populate a complex screen like a dashboard or a report. You didn't know the legacy site was applying complicated business logic in between these calls based on the data received from each API.

If you want your new site to be lightweight (for example if you want to support mobile platforms in the near future), you need a new API that will do the dirty job so the UI will just need to make a single call to get the data ready to display.

Internal or sensible data exposed

You don't want your customer facing website allowing a customer to access internal business data, or even other customers' data. Some of your existing APIs were designed to be accessed internally only. Adding multi-tenancy is not an option for several reasons (you don't have ownership of the source code, the API is consumed by multiple teams across the organization so you risk introducing a bug that could affect all the consumers, etc). You need a wrapper that will consume the original API, filter the data by customer accounts, along with any other filtering, sorting and pagination criteria that could be applied, and exclude the sensible data (fields) from the response. The wrapper will be the customer facing API.

Unexpected amount of data

The new site will show a list of orders that will retrieve from an existing API. You played with the API in lower environments (testing, staging) and you found the average amount of orders by customer is around 500-600 orders in the last 30 days. For that reason, you decide that sorting, filtering, and pagination will be performed on the UI side.

After integration and successful testing, you deploy to Pre-Production and you find that some customers have thousands of orders, causing the UI to completely freeze. You need to move the pagination, sorting and filtering to the backend side. ¿Is it possible to modify the existing API? If so, as outlined in the previous issue, ¿are you ok with the risk of introducing a bug that could affect other consumers of the API?

If the answer is No to any of these questions, you may need to create another API that will access the data from the same data source of the original API (database, Elastic, etc), and apply pagination, sorting and filtering. ¿Is it possible to know what business rules are being applied in the original API? (We’ll see more about this, later).

Nor the wrapper or any of the new APIs were anticipated when you started implementation of the UI. ¿Do you have the resources to build them?

Frontend and backend development not sync'd

You've prioritized frontend development based on the user flow. For example, when users navigate to the home page, they are presented with a list of orders. Below the list there is a CRUD and cancelation menu to manage orders. The user can select other entities from the left menu like customers, invoices, etc.

Using mock services that provide mock data, the frontend developer started by the Orders List screen. However, to test the List Orders endpoint, the backend developer needs to create an order first. Therefore, they started by the Create Order endpoint. When this endpoint is ready, frontend developer must wait to integrate it until they finish with the Create Order screen. They cannot integrate the Order list either because this endpoint was not implemented yet. Your full-stack development is not in sync.

You must demo the new site to the management in a regular basis so they can see the progress. For the first demo, you want to show the Orders flow. Since you haven't integrated the backend, you'll have to demo all the flow with mock data. This may not be what the management expects. On next demo, showing the Orders flow again, although fully integrated, could not be seen as real progress. Therefore, you’ll want to show something new, like the Customers flow. Since you have the full-stack development not in sync, you may need to demo another flow using mock data too.

At some point, it could be difficult to justify what the backend is doing, with the risk of affecting your budget.

Delays and budget deviations

You need to implement new APIs that were not anticipated. If you don’t want to penalize the planning, you need to hire additional developers, penalizing the budget. If you don’t want to penalize the budget, or if you don’t have more budget, you need to delay the planning. At the end, you will affect both, the planning, and the budget.

Even with the appropriate budget, it could be difficult to find the resources with the skills and the seniority you need. You may have to borrow or disaffect them from other projects.

New developers will need some time to setup their development environments and, depending on the stage of the project development, to catch up with current developers about the specifics of the project (documentation, undocumented process and conventions, points of contact, etc). They may require permissions to access tools and environments, which could take some time also depending on your internal processes.

Unknown business rules

You know the Orders API provides an endpoint that returns the list of orders by passing the customer account number. You get the list of accounts from the Customer API based on the logged in user identifier, and you populate a drop-down list where the user can select the account they want to see the orders from.

The only documentation you have about the API is the OpenAPI specification, which provides a brief description about the operation and the input and output parameters.

The frontend was implemented with a mock service that provided mock data. Now, you’re integrating the real API. In dev and test environments, testing with a few accounts, it works. Then, you move to staging environment where data is seeded from Production… and it fails with the error “Invalid account type”.

¿What happened? Maybe you get a more descriptive error message indicating why the account type is not valid. In any case, this is something you haven’t anticipated. ¿What are the other invalid types? ¿How do you know what validations you must perform in the UI side before adding each account to the drop-down list? ¿Is it possible to filter the account list when calling the Customer API? If so, ¿what is the filter criteria?

If the validation rules were not documented in a document library or content management system, or they were not updated, and the product owner does not have the answers, you’ll need to contact the API development team. The API developers may need to look at the code to get the validation rules. All this process may require some time, possibly affecting your deadline.

Missing enough and appropriate data

When implementing the UI, the frontend developers used mock data. When implementing the APIs, the backend developers created some data to test in local environment and during the integration phase. This data was enough to validate the integration between the UI and the APIs.

Now, you need to demo the new site to the management for the first time, or a new functionality. At this moment you realize the amount and/or diversity of the data is not enough because no one wants to see two rows in a table of orders or customers. ¿What about pagination, sorting and filtering? ¿What about performance? ¿How do you plan to show the different user flows if you don’t have enough and appropriate data?

Since the current site is only bug fixed, no new features are added regularly, the existing test data can be outdated or not enough.

We already spoken about the disadvantages of using mock data to demo the new site to the management. However, even with mock data it can be impossible to demo all the new functionalities if the mock data consists of the same row, duplicated one hundred times.

The same happens if you need to move to staging or pre-production environments. You need hundreds of orders, with different customers, different products and product types, different locations, and so on, so you can ensure the most part of the validations and business rules have been covered before going to Prod. This is not to be confused with load testing.

Generating the data you need, or seeding data from Prod, may take some time. Since you haven’t anticipated it, you’ll have to decide moving the deadline, or giving the green light to continue without a better validation of the different functionalities.

Summing up

The above list of issues is not meant to be complete. These are just some of the issues you may encounter when you start the implementation of the website based solely on the mockups, without correctly defining the source of the data first.

In the next article, we’ll see how we can avoid these issues.