Boost App Performance: Fight Latency in Distributed Systems

I think it’s obvious that a request to an external network is much slower — and less reliable — than accessing local memory or a local network. The time difference is at least 7 times. That is why your application’s architecture and your code must take this into account.
The only exception is when latency is not a critical quality attribute. This is usually the case when the task is not time-sensitive. For example, when generating analytical reports, a delay of several seconds or even minutes does not affect the service quality.

Clearly Separate External Network Calls

We use a DI container for convenience. The client-side code in the application layer may look completely normal:

// client code after DI container wiring
$order = new Order($id, $userId, $total);
$this->orderRepository->save($order);

❗ However, what happens inside can be very different:
One implementation:

final readonly class OrderRepository
{
    public function __construct(
        private PostgresConnection $connection
    ) {
    }

    public function save(Order $order)
    {
        $this->connection->execute('INSERT INTO orders (id, user_id, total) VALUES (?, ?, ?)', [
            $order->id(),
            $order->userId(),
            $order->total()
        ]);
    }
}

An entirely different implementation:

final readonly class OrderRepository
{
    public function __construct(
        private OrderApiClient $client
    ) {
    }

    public function save(Order $order)
    {
        $this->client->createOrder($order);
    }
}

In this design, the client code does not make it clear that an external network call is being made. Help your future self and your fellow developers by making it obvious what you are working with.

🔥 Introduce naming conventions that clearly indicate external calls
For example, use names like OrderProvider, OrderGateway, or OrderIntegration. These names should be clearly different from names like OrderRepository or OrderStorage.

🤓 Maybe in the future quantum particles will transfer information instantly. But for now, we are strictly limited by the speed of light

Try to retrieve all required data in one request

If you are making an external call, try to return all the needed data in one request. This approach is similar to solving the classic n+1 problem.
For example, avoid this:

$employees = $this->employeeProvider->getList($limit, $offset);
foreach ($employees as $employee) {
    $department = $this->departmentProvider->getById(
        $employee->getDepartmentId(),
    );
    // do something with department
}

Instead, do this:

$employees = $this->employeeProvider->getList($limit, $offset);
$departments = $this->departmentProvider->getAll();
// Prepare a map of department id to department
foreach ($employees as $employee) {
    $department = $departments[$employee->getDepartmentId()];
    // do something with department
}

Caching Is Your Friend, But Use It Carefully

If you frequently request the same data and that data rarely changes, caching can be a great solution. It reduces the call time, speeds up data processing and lowers network load.

❗ Spend time planning your cache invalidation strategy.

When you use the cache-aside pattern (first check the cache — if the data is there, return it. If not, fetch it from the source, store it in the cache, and then return it), keep these points in mind:

• If your cache is “in-process” (meaning your service becomes stateful), it will not scale across multiple processes. In distributed systems, you should use remote caches (like Redis). However, these also work over the network and can add extra latency. Also, consider the time needed for data serialization and deserialization.
• Your cache should not become an additional point of failure. For example, what if Redis goes down?
• Invalidation is about ensuring data consistency. Mistakes in invalidation can lead to frequent cache misses, which will hurt performance.

$employees = $this->cache->get($cacheKey);
if (null === $employees) {
    $employees = $this->employeeProvider->getList($limit, $offset);
    $this->cache->set($cacheKey, $employees);
}
return $employees;

Invert the Data Flow

Instead of calling other services every time you need data, consider using the Publisher-Subscriber pattern to store data locally (including in the cache).

How does it work?

• When data is updated (for example, when an order is created or changed), the responsible service publishes an event to an event bus.
• Other parts of your system that need this data subscribe to these events. When an event arrives, they receive the updated data and store it locally.
• This way, when you need the data later, you do not need to perform a network call. Instead, you use the fast local storage.

Of course, this approach makes your architecture more complex and may not be suitable for every scenario. However, this technique can elegantly solve the problem in your project.

Conclusion

As you design your distributed system, remember that every optimization decision requires careful thought. Clear naming conventions, caching strategies, and data flow inversion can significantly improve performance, but there are always nuances based on your product, industry, team, and technology stack.

Take the time to evaluate your specific needs:

• Product & Domain: The importance of low latency may vary greatly between real-time applications and less time-sensitive reporting systems.
• Team & Culture: Your team’s familiarity with certain patterns and technologies can shape which approaches work best.
• Technology Stack: Different stacks come with their own trade-offs, so choose solutions that align with your existing infrastructure.

Stay flexible, test your assumptions, and continuously iterate. Treat your architecture as a living project that evolves with new insights and requirements. Your commitment to thoughtful, informed decision-making will be the key to building high-performing, reliable systems.