Interaction of Database and Application

While working with backend system we often write database queries, such as:

await db.query("SELECT * FROM users");
        

our application smoothly interacts with our DB and wallah!! the results are there. It seems very simple, from multiple layers of abstractions. But under the hood enormous amount of engineering goes, which have been developed by great problem solvers and researches in the span of multiple decades.

lets us dig out to understand, how things are working.

When we talk about Application and database, we are talking about 2 different systems which can be completely different machines or different processes on same machine. In both cases they are isolated from each other by the operating system.

By Isolation I mean here is:

• They do not share memory.
• They do not execute in the same environment.
• They cannot directly access each other’s data.

Now the question comes if they are interacting with each other then where and how they are interacting, obviously things are not happening in the air!!. The answer is: through the network.

What is a network and why it is required?

Assume you and your friend are sitting in 2 sides of the world, you opened you mobile phone and want to send a picture to them, how will u do that? You took picture on your mobile phone and try to send using some social app. You select the picture and click send. But it doesn’t work.

Then you realize that your phone is on flight mode. As you turn off flight mode, hurray it reached to your friends phone. What changed? This is the importance of network. Without it your digital system is an isolated system. Whenever 2 digital systems wants to communicate, they need physical or wireless communication paths. These paths are called networks.

When you send any kind of data through a digital device, it passes through multiple intermediate systems:

Your Phone → Router → ISP → Internet → ISP → Router → Friend’s Phone        

At each step:

• Data is forwarded
• Routed
• Verified
• Delivered

There are multiple structures in which which multiple devices are connected these structures are known as Topologies.

Some of the known topologies are:

1. Bus Topology: All devices share a single communication line. Not reliable If communication line fails everything fails. Each device uses same cable. If one device send others hear because of single cable. It is noisy.
2. Star Topology: All devices are connected to a central node. Widely used If one central node fails entire network dies. Each device has its own cable connected to central node, even if one cable dies others remain intact. In real world there are very less chances that central node dies.
3. Ring Topology: Devices connected in closed loop.
4. Mesh Topology: Devices are interconnected High cost & High reliable, because 2 devices are connected with multiple link. Even if one link fails, traffic reroutes.
5. Tree Topology: Hierarchical structure like a tree. If one branch fails only that branch is effected rest works smoothly.

Reliability order:

Mesh > Tree > Star > Ring > Bus        

The modern internet is built using hybrid of multiple topologies which are interconnected across continents.

If we see from the perspective of Application and Backend:

Application → Network → Database → Network → Application        

Now the question comes, if I sent 10MB of data to a device from my device then will it just reach at once? The answer to this is NO!!.

Data transfer over a network

When a data is transferred over a network, it is not transmitted as one large continuous block. Instead, the data is divided into multiple pieces called Packets.

Each packet contains:

• A portion of the data
• Source address
• Destination address
• Control information

These packets are sent independently.

But why in packets? Why inventors didn’t invent a way to transfer whole data at once?

Some of the problem which may occur when a large data is sent at once:

• To avoid the network line getting blocked by a single data. Decades back we had telephones which used Circuit switching. When you made a call a dedicated line was reserved and nobody else could use it until we hung it up.
• If data lost everything would be lost at once.
• Routers may not be able to forward large data efficiently because router buffer will explode. Routers have limited amount of memory, if huge chunk of data is being sent at once, router must store it which would lead to filling of buffer.
• Recovery becomes impossible in case of failure. Suppose you sent 100MB of data at once, at 99MB it suddenly stopped. Now we have resend everything again.

Now think from the perspective of internet, for 1000s of user we need to have 1000s of dedicated lines for each if all of them send a large data at once.

Therefore instead of sending a big chunk of data at once, it is divided into small packets of data.

But who and how the data packets are divided?

How Data Is Divided Into Packets?

When an application send data over the network, it is not converted to packets directly. Instead, the data passes through multiple layers inside the operating system. Each layer performs a specific responsibility and adds its own information.

Network Stack Inside an Operating system

Modern operating system has a layered stack as follows:

• Application Layer
• Transport Layer
• Network Layer
• Link Layer
• Physical Layer

1. Application Layer

socket.send("SELECT * FROM users");        

The application hand over this data to the operating system. Wait!!! But isn’t the data already in memory which is managed by the OS?? Ya it is. But the data is in Private space i.e. Process memory. The memory belong to the program we run. The OS cant directly use it for networking.

What happens is:

• Your app calls send()
• OS copies data from your process memory
• OS puts it into kernel network buffers
• Now the OS controls it

So “hands data to OS” means:

Your program gives permission to the OS to take over that data for transmission.

2. Transport Layer

The transport layer is implemented inside the OS Kernel. This layer is responsible for dividing large data into smaller pieces. There is a term for this called Segmentation.

There are many protocols at this layer For example TCP, UDP, QUIC, SCTP, DCCP, HTTP/3, WebRTC. Depending on the problem statement these protocols are used. Every protocol has their own way of segmentation. For TCP its TCP segmentation and for UDP its UDP segmentation, each resulting in their own segment.

TCP segmentation

• Data is divided into segments.
• Each segment receives a sequence number.
• Control information is added
• Retransmission logic is prepared

Following things are being added to header by TCP into each segment

Field Purpose Source Port Which app sent Destination Port Which app receives Sequence Number Order of data ACK Number What was received Window Size Flow control Flags (SYN, FIN, etc.) Connection control Checksum Error detection

So each segment looks like:

[TCP Header][Data Chunk]        

TCP keeps a copy of sent segments and follows this retransmission logic which is very important from the perspective of Reliability because packets can be lost, dropped or corrupted.

1. Sender sends segment
2. Receiver sends ACK
3. If ACK not received in time → resend

UDP Segmentation

• Data is packaged into datagrams
• No ordering or reliability information is added
• UDP simply prepares the data for delivery
• UDP Header includes: Source port Destination port Length Checksum

3. Network Layer

Each segment is passed to the network layer, which uses the Internet Protocol.

This layers is responsible for:

• Assigning source and destination IP address.
• Selecting routes across networks.
• Forwarding packets b/w routers.
• The IP layer wraps the transport segment inside an IP header, forming an IP packet.

[IP Header][TCP Header][Data]        

4. Link Layer (Framing)

Now, packet reaches to this layer. This layer handles communication within the local network.

It:

• Add MAC addresses
• Create frames

[Dest MAC][Src MAC][IP Packet][CRC]        

• Performs error detection - Cyclic Redundancy check. Its mathematical fingerprint. Sender computes CRC(data) = 12345, then receiver recomputes, and if found different then packet is dropped.
• The IP packet is wrapped inside a frame which is suitable for Ethernet, Wi-Fi, or other local technologies.

5. Physical Layer (Transmission)

Finally, the frame is converted into physical signals:

• Electrical pulses (copper cables) - uses voltage, If high voltage = 1 else 0
• Light signals (fiber optics) - Uses light, If light on = 1 else 0
• Radio waves (wireless) - Different frequencies represent bits.

These signals travel through physical media to reach the next network device.

[Frame]
  [IP Packet]
    [TCP Segment]
      [Application Data]

        

Complete End to End View

App memory
↓
Kernel buffer
↓
TCP: segment + seq
↓
IP: address
↓
Ethernet: MAC + CRC
↓
NIC: signals
↓
Wire
↓
Routers
↓
Receiver NIC
↓
Reverse process
↓
App
        

But Who Performs All This Work?

Almost all of this processing is handled by:

• The operating system kernel
• The network drivers
• The network interface card (NIC) - takes frame, converts to signals and receives and converts back

Application only calls send() or write(). OS takes care of everything else.

This all from the perspective of sender’s perspective, what about receiver’s perspective? Its all reverse of what happened earlier.

Signals → Frames → Packets → Segments → Data        

Each layer removes its header and verifies correctness before passing the data upward. This process is called Decapsulation.

Now that we understand how data travels across networks, the natural next step is to explore how databases receive, interpret, and execute these requests. This will be covered in the next part of this series.