A Comprehensive Guide to Data Engineering - Part Three: Designing Good Data Architecture (3)

This is the final part of the article discussing designing good data architecture. In this concluding article, we look at the various examples and types of data architecture.

Data Warehouse

A data warehouse is like a big digital storage unit that businesses use to keep and analyze all their data. This isn't something new; it's been around since the late '80s when a guy named Bill Inmon came up with the idea. He said a data warehouse should be a place where data is collected in a certain way to help management make decisions. This idea still holds up today.

In the past, only big companies with lots of money could afford these data warehouses because they were costly and needed a lot of people to run them. But now, thanks to cloud computing, even small companies can use data warehouses without having to buy and manage all the tech themselves.

There are two ways to think about data warehouse architecture:

1. Organizational: This is about how the data warehouse fits into a business's team structures and processes. It's got two key parts: It keeps the heavy data analysis stuff separate from the day-to-day data (like sales transactions) so that the day-to-day stuff doesn't slow down. It brings all the data together in a neat and organized way using a process called ETL, which stands for Extract, Transform, and Load. This means you take data out of where it's coming from, clean it up and organize it, and then put it into the data warehouse ready for analysis.
2. Technical: This reflects the data warehouse's actual tech setup, often using something called MPP, which stands for Massively Parallel Processing. This is a fancy way of saying it can handle a ton of data all at once and do complicated calculations quickly. Cloud data warehouses have gotten even better at this over time because they're set up to handle huge amounts of data more efficiently.

ELT is a newer variation where you pretty much dump the data straight into the warehouse and then sort it out there, using the warehouse's big computing power to handle the heavy lifting.

Cloud data warehouses have changed the game a lot. Amazon Redshift was one of the first to make data warehouses something you could just spin up when you needed it, without all the big upfront costs. Then others like Google BigQuery and Snowflake came along and pushed the idea of splitting up the storage part (where data is kept) from the computing part (where data is processed), which can save a lot of money and offer even more power when you need it.

Because cloud data warehouses are so good now, some folks think we should stop calling them "warehouses" and start thinking of them as a whole new kind of data platform with much more to offer.

Lastly, a data mart is like a small, specialized shop inside a big warehouse, meant for a specific part of the business. It makes things easier for the people in that department to get what they need and can also speed up the process of analyzing data by having everything pre-arranged and summarized to their needs.

So, whether it's the old-school data warehouse or the latest cloud data platform, it's all about having the right tools to store, manage, and analyze data to help businesses make smarter decisions.

Data Lake

A data lake is like a big storage pool where a company can pour all its data, whether it's neatly organized or completely unstructured. It's supposed to be a one-stop shop where you can dive in and find any information you need.

The early version, known as Data Lake 1.0, was cheap for storing huge amounts of data in the cloud, and you could use powerful cloud computing to analyze the data on demand. However, it ended up being a jumbled mess for most companies. The data was often disorganized, hard to manage, and difficult to process. Plus, even though using open-source technology was meant to cut costs, in reality, it was expensive because it required hiring specialists to handle the complex systems.

While some big tech companies managed to make good use of their data lakes, for many others, these data lakes became cluttered and costly, failing to live up to their potential.

Next Generation Data Lakes

After the first version of data lakes (huge pools of raw data) proved to be difficult to manage, companies started improving the idea. For example, Databricks came up with something called a "data lakehouse," which is a mix of a data lake and a data warehouse. It's like having a big storage shed (like a data lake) that's as organized and easy to manage as a house (like a data warehouse). This lakehouse model allows for better control over the data; you can now update or delete data, which wasn't easy to do in the old data lakes.

Now, cloud data warehouses (services that store and manage data on the internet) are becoming more like data lakes. They can handle massive amounts of different types of data and use powerful tools to process it. It's starting to be hard to tell the difference between a data lake and a data warehouse because they're both getting features from the other.

Big cloud service providers like AWS, Azure, Google Cloud, Snowflake, and Databricks are leading the charge. They're creating one-stop shops for data that can handle everything from neat databases to messy, unstructured info. In the future, instead of deciding whether to use a data lake or a data warehouse, data workers will choose from these blended services based on what fits best with their needs and preferences.

Modern Data Stack

Imagine you're building a complicated model aeroplane. In the old days, you might have had to buy a big, expensive kit that had lots of parts you didn't need, and it was all or nothing. That's how old data systems were—big, bulky, and expensive.

Now, think about the "modern data stack." This is like having the freedom to pick and choose the best and easiest-to-use pieces for your model aeroplane from a variety of different kits, all available online, often with clearer instructions and community support. This is what's happening with data systems now. Businesses can use cloud services to pick out just the parts they need—like data storage, ways to move data around, tools to change and manage that data, and programs to make sense of the data through charts and graphs.

The modern data stack is all about mixing and matching these components to build a system that's both flexible and cost-effective. You're not stuck with one big, clunky tool; you can plug in different tools for different jobs, and if something better comes along, you can swap it out without tearing down the whole system.

This approach also encourages a community vibe, where users and creators are constantly talking, sharing ideas, and making improvements, much like hobbyists sharing tips and tricks about their model aeroplanes.

Lambda and Kappa Architecture

Back in the 2010s, companies started to get interested in the idea of "streaming" data. This is like a constant flow of information, kind of like tweets streaming in on Twitter in real time. To manage and analyze this flowing data, they used new tools like Kafka, which is great for handling tons of messages zooming around, and others like Storm and Samza for crunching that data on the fly.

The tricky part was figuring out how to handle two different types of data: the fast-moving "streaming" kind and the big, heavy "batch" kind that's processed in chunks. The Lambda architecture was one answer to that puzzle. Picture it like a factory with two separate assembly lines—one for handling immediate, quick orders (streaming) and another for big, planned-out projects (batch). Both lines work from the same pile of resources (data), but they make different things. A third space, called the "serving" area, would then take the outputs from both lines and mash them up into something useful.

But this setup got pretty complicated. Imagine trying to make sure both assembly lines were using the same materials in the same way—it was a headache and mistakes happened a lot.

That's why the Kappa architecture was suggested. Instead of two lines, there's just one super flexible assembly line that can handle everything—both the real-time orders and the big projects—just by switching modes as needed. It's like having one machine that can make both a fresh cup of coffee and a huge banquet, depending on what recipe you feed it.

However, even though the Kappa architecture sounds cool, it didn't take off as expected. There are a couple of reasons: first, for many companies, streaming data is still like rocket science—intriguing but also hard to do right. Second, while the Kappa approach is awesome for the streaming part when you have a massive history of data to deal with, it's often cheaper and easier to just process it in big batches, like cooking a whole week's meals on a Sunday, instead of making every meal from scratch.

The Internet of Things (IoT)

Think of the Internet of Things as a big team of gadgets that all have the job of gathering and sharing information without needing a person to help them. These gadgets could be anything that can connect to the internet – like fitness trackers, smart fridges, or even tiny sensors that can detect pollution in the air. They don't need someone to type or input data; instead, they automatically pick up information from what's happening around them and send it over the internet.

Why it's a big deal now:

IoT isn't exactly new, but when smartphones became popular, it's like we suddenly had a whole army of these smart gadgets. Now we've got even more stuff, like thermostats that learn your schedule, cars that entertain you, and speakers that can order your groceries.

Understanding IoT helps with big data trends:

Knowing a bit about how IoT works is useful because it's a huge part of how we're starting to handle lots of data. Here's a quick look at the important parts:

Devices:

These are the actual physical things – from simple sensors to complex cameras – that pick up data from their environment. Some can do a bit of data processing on their own before sending it out.

Interfacing with devices:

This is about how we get the data from these gadgets:

• IoT gateway: Think of this like a post office for IoT devices. It's a middleman that helps these gadgets send their data with as little energy as possible.
• Ingestion: This is the process of the IoT system taking in the data, which can be done in various ways, like batch uploading when the device gets internet access.

Storage:

Where and how you keep the data depends on how fast you need to get it back. If you can wait, you can store it in big batches. If you need it fast, you use systems that let you access it almost immediately.

Serving:

This is about what you do with the data. It could be put into a report, used to send alerts for things like fires or break-ins, or even used to adjust devices to make them work better.

IoT is complex:

IoT can get pretty complicated with all the different gadgets and ways to handle the data they collect. It's a whole area of expertise that's growing and changing all the time, and it's worth learning more about if you're into data engineering.

Data Mesh

The "data mesh" is a new way of managing and organizing all the information that companies and organizations collect. It's a reaction to older systems where all the data was kept in one big place, like a warehouse or a lake, and there was a clear split between the data used for everyday business operations and the data used for analysis.

In simple terms, the data mesh is like turning a big, centralized supermarket into a farmer's market with lots of different stalls, each run by different people who know their products. Here's how it breaks down:

1. Decentralization:Instead of keeping all the data in one place that one group controls, each team or department (called a "domain") manages its data. It's like each team has its mini-library of information that it's an expert on.
2. Data as a Product:The idea here is to treat data like a product that needs to be well-made, useful, and easy for others to use, just like a good smartphone or a car. So each team makes sure their data is clean, clear, and ready for others to use without problems.
3. Self-serve Data Infrastructure:This is about giving teams the tools they need to handle their data on their own, without waiting for IT to help them. It's a bit like having a good kitchen in your home, so you can cook what you want, when you want it, without having to go out to a restaurant.
4. Federated Computational Governance:Even though each team controls its data, there's still a set of rules everyone follows to make sure everything works together. This is like a neighbourhood association where everyone agrees on certain standards for their houses, so the neighbourhood works well as a whole.

The data mesh is getting a lot of attention because it's quite a shift from the old ways of doing things, and it's seen as a solution to some big problems that come with those old systems. It's all about giving more responsibility to individual teams, making data more accessible, and ensuring that everyone plays by the same rules for the greater good.

Other Examples

Data architecture is like the plan for a building, but instead of a building, it's for managing all the information a company uses. Just like there are many different styles of buildings – from cottages to skyscrapers – there are many different ways to organize data. Some are like spider webs (data fabric), some are central bus stations (data hubs), and others are like bustling city streets where things are happening all the time (event-driven architecture). Over time, new styles keep popping up as people find better tools or think of smarter ways to put everything together.

If you're someone who works with data, it's like being an architect or a city planner. It's important to keep an eye on the new styles and ideas that are coming out. You should be curious and ready to learn about them, just like you'd check out a new building material or a new kind of city layout.

Don't fall in love with just one way of doing things, because the best plan might change as new materials come out or people's needs change. If you find something that could make things better for your company, take the time to understand it and see if it makes sense to switch things up.

Small changes or big ones to how your company's data is organized can make things run smoother, like updating an old building or even rebuilding it to be more modern. It's all about making sure the company can use its information as effectively as possible.

Conclusion

In conclusion, designing good data architecture is an evolving process, shaped by both the changing tides of technology and the ever-increasing complexity of business needs. From the foundational data warehouses inspired by Bill Inmon to the elastic capabilities of cloud-based solutions like data lakes and lakehouses, the landscape of data storage and analysis has expanded to a myriad of options. The modern data stack has transformed data architecture into a more modular and flexible craft, empowering organizations to pick and choose the solutions that fit their unique challenges.

The shift towards real-time processing through Lambda and Kappa architectures, coupled with the proliferation of IoT devices, has ushered in an era where data is not only vast but also ceaselessly flowing and rich with potential insights. This requires a robust, scalable, and intelligent framework to harness its full potential. Enter the data mesh concept, which decentralizes data ownership and governance, encouraging a culture where data is a product and quality is paramount.

As we navigate this complex terrain, data professionals must remain adaptable, learning and unlearning as they sculpt the architecture that will hold the data-driven insights of tomorrow. The tools and methodologies may change, but the core objective remains steadfast: to build systems that enable businesses to make informed decisions with agility and precision.

Whether you're constructing a data warehouse, diving into a data lake, or weaving a data mesh, the key is to understand the principles and practices that will stand the test of time. By fostering a culture of innovation and maintaining a willingness to adapt, data architects can ensure that their data infrastructure is not only robust for today's needs but also poised for tomorrow's opportunities. As we stand on the cusp of what might be the next revolution in data architecture, it is this foresight and flexibility that will define the success of organizations in a data-centric future.

Resources

1. Fundamentals of Data Engineering by Joe Reis & Matt Housley