What This Article Covers
When we think about improving the performance of a computer or server, we often first think of the specifications of individual components such as the CPU, GPU, and memory.
Using a faster CPU, adding a more powerful GPU, and replacing the storage device with a faster one can increase the processing capacity of each component.
In actual infrastructure, multiple components share the work involved in a single task. Data is read from storage, loaded into memory, processed by the CPU or GPU, and then the necessary results are stored again or transmitted over the network.
The actual performance of infrastructure depends on how smoothly this entire flow works together.
The point that limits overall performance is called a bottleneck.
Infrastructure Is a Flow of Multiple Connected Components
Earlier, we looked at digital infrastructure by dividing it into computing, connectivity, storage, deployment and delivery, and the operating environment.
In actual workloads, these elements are connected to one another.
For example, consider an AI server processing data.
- It reads data from storage.
- The CPU loads the data and prepares it in the required form.
- Memory stores the data and delivers it to the CPU or GPU.
- The GPU performs large-scale computations.
- The processed results are stored.
- The necessary data and results are transmitted to other systems over the network.
Throughout this process, each component must continuously pass the data required for the next stage.
Even if a GPU can perform calculations very quickly, it can use that performance only when data is supplied at a sufficient speed. Storage, the CPU, memory, and the network are connected in the same way.
Therefore, when evaluating system performance, we need to look at the overall processing flow along with the maximum specifications of individual components.
What Is a Bottleneck?
The term bottleneck originally refers to the narrow neck of a bottle.
Even when a wide bottle contains a large amount of water, the narrow neck limits how much water can flow out at once.
A similar principle applies to IT infrastructure.
A bottleneck is a point in the overall workflow that limits throughput or creates significant delays.
For example, consider a server with the following characteristics.
- The GPU can process a large amount of data per second.
- The storage device can read a smaller amount of data than that.
- The GPU must work at the speed at which data arrives from storage.
In this case, the storage device’s data delivery speed has a major effect on the overall processing performance.
Even when the GPU has sufficient computing capability, actual throughput is affected by the speed at which data is supplied.
Even a Fast GPU Can End Up Waiting for Data
GPU performance is often emphasized in AI infrastructure.
GPUs play an important role in AI training and inference because they can perform large-scale matrix calculations quickly.
For a GPU to work, the data it needs to process must first be ready.
For example, in a large-scale AI training environment, data is read from storage, the CPU preprocesses it, and memory and the network deliver it to the GPU.
If one part of this process cannot keep up, the amount of time the GPU spends waiting for the next data can increase.
GPU utilization falls, the amount of time spent on actual computation decreases, and it becomes difficult to fully utilize the GPU’s processing capacity.
That is why building AI infrastructure requires looking not only at the number and performance of GPUs but also at the following elements.
- CPU data-processing capacity
- Memory capacity and bandwidth
- Storage read and write performance
- Internal server connection speed
- Network bandwidth between servers
To turn high computing performance into actual processing performance, the system needs a structure that can supply data to computing devices at a sufficient speed.
Bottlenecks Can Also Occur in the CPU, Memory, Storage, and Network
Bottlenecks also occur in general business systems.
CPU Bottlenecks
When the CPU has too many requests to process, CPU utilization can rise and task wait times can increase.
This can occur in environments with heavy CPU workloads, such as web servers, data-processing systems, and application servers.
Memory Bottlenecks
When a program does not have enough available memory, it becomes difficult to keep enough of the required data in memory.
In high-performance CPUs and GPUs, memory bandwidth, which refers to the speed at which data is exchanged with memory, also becomes an important performance factor.
Storage Bottlenecks
In databases and large-scale file-processing systems that frequently read and write data, storage input/output performance can have a major effect on overall performance.
When evaluating storage performance, IOPS (Input/Output Operations Per Second), which refers to the number of input/output operations performed per second, is also used along with transfer speed.
Network Bottlenecks
In systems where servers exchange large amounts of data, network bandwidth and latency are important.
In environments such as cloud systems, distributed databases, and AI clusters, where multiple servers participate in a single task, the connection performance between servers affects overall processing speed along with the performance of the servers themselves.
The Location of a Bottleneck Changes Depending on the Workload
Even on the same computer or server, the most important resource changes depending on the type of work being performed.
For workloads that involve large amounts of computation, CPU or GPU computing performance can become important.
For workloads such as databases that repeatedly read and write data, memory and storage performance can become important.
In distributed systems where multiple servers exchange large amounts of data, the network can become an important factor.
Improving one bottleneck can increase overall throughput and cause another limiting point to appear.
For example, if storage is improved so that data can be supplied more quickly, CPU processing capacity or the network may then have a greater effect on overall performance.
Therefore, a bottleneck is best understood as the point that currently limits overall performance within a particular workload and system configuration.
What Does “Balance” Mean in Infrastructure?
Balance in infrastructure means connecting the processing capacities of each component in a way that fits the actual workflow.
For example, a system that processes large volumes of video files may require high processing capacity from storage and the network.
A system that trains large-scale AI models requires not only GPUs but also high-speed connections between GPUs, memory, and a structure that can supply data efficiently.
In a general enterprise business system, the CPU, memory, storage, and network need to be configured to match the workload so that services can operate reliably.
Good infrastructure is a structure in which each resource is naturally connected according to the throughput required by the actual workload.
Therefore, when evaluating infrastructure performance, it is important to look not only at which component is the fastest but also at where the overall workload is waiting.
Finding a bottleneck means finding the point where that waiting occurs.
Why Do Companies Identify Bottlenecks First?
Expanding infrastructure costs money.
Adding servers, GPUs, storage devices, and network equipment increases not only equipment costs but also electricity, cooling, space, cloud usage fees, and operating costs.
That is why, when performance problems occur, companies identify which resource is currently limiting overall processing.
For example, when a web service becomes slow, they may examine metrics such as:
- CPU utilization
- Memory usage
- Storage input/output volume
- Network usage
- Request processing time
- Number of tasks waiting to be processed
Identifying the actual bottleneck through this data also makes investment priorities clearer.
If a system lacks CPU processing capacity, CPU resources can be increased. If storage input/output is the limiting factor, the storage structure can be improved. If network bandwidth is insufficient, the network configuration can be adjusted.
The key to infrastructure performance management is placing the required resources where they are needed.
DANA NOTES in One Line
Infrastructure performance improves when bottlenecks are reduced across the overall workflow and the processing capacities of each component are connected in a balanced way.
