System Design + Concurrency
How would you safely transfer money between accounts?
A safe money transfer system must handle concurrency, consistency, deadlocks, retries, duplicate requests, and distributed system failures while ensuring money is never lost or duplicated.
The Real Problem
This question gets asked a lot. Transferring money sounds simple:
from.balance -= amount;
to.balance += amount;But then we need to run several hundreds or thousands of these transfers concurrently. And some of these may involve the same account. In one transfer, account may be credited to account 100. And in another, money may be debited from the same account. Once multiple threads, retries, failures, or distributed systems are involved, the problem becomes much harder. Financial systems have to guarantee such operations. You don't want extra money debited from your account (or charged to your credit card) even by mistake.
Money must never be lost, duplicated, or partially transferred.
The Concurrency Problem
Imagine two transfers happening simultaneously:
Thread A:
Transfer $100 from Account X to Account Y
Thread B:
Transfer $50 from Account X to Account ZWithout coordination, both threads may read the same balance and overwrite each other's updates.
Unsafe
Safe
Naive Locking Solution
synchronized(fromAccount) {
synchronized(toAccount) {
from.withdraw(amount);
to.deposit(amount);
}
}This protects the critical section, but introduces another major problem:
The Deadlock Problem
Imagine:
Thread A:
locks Account X
waits for Account Y
Thread B:
locks Account Y
waits for Account XBoth threads wait forever.
The Classic Interview Solution
Lock accounts in a consistent order, always locking the smaller account id first.
Account first =
from.id < to.id ? from : to;
Account second =
from.id < to.id ? to : from;
synchronized(first) {
synchronized(second) {
from.withdraw(amount);
to.deposit(amount);
}
}This avoids circular waiting because every thread acquires locks in the same global order.
Why This Still Is Not Enough
The JVM locking solution only solves part of the problem.
Real systems also face:
- duplicate requests
- network retries
- partial failures
- multiple application servers
- database crashes
- distributed transactions
If two different servers process the same transfer request, JVM locks on one server do not protect the other server.
The Idempotency Problem
Imagine a client retries the same transfer request because it timed out waiting for a response.
POST /transfer
Idempotency-Key: abc123The original transfer may actually have succeeded already.
Without idempotency handling, the retry could transfer the money a second time.
Database Transactions
In real financial systems, correctness is usually enforced primarily through database transactions.
BEGIN TRANSACTION
UPDATE accounts
SET balance = balance - 100
WHERE id = 1;
UPDATE accounts
SET balance = balance + 100
WHERE id = 2;
COMMIT;The database can then provide:
- atomicity
- isolation
- rollback on failure
- durability
Important Production Insight
Real financial systems typically rely heavily on transactional databases, idempotency keys, durable ledgers, and carefully designed failure handling.
The Interview-Friendly Explanation
Common Interview Follow-Ups
Why can deadlocks happen?
Two threads may acquire locks in opposite order and wait forever for each other.
How does consistent lock ordering help?
If every thread acquires locks in the same global order, circular waiting cannot occur.
Why are JVM locks insufficient in distributed systems?
Locks only coordinate threads inside the same JVM. Multiple servers may still process the same accounts concurrently.
Why are idempotency keys important?
Retries can accidentally execute the same transfer multiple times. Idempotency keys help detect duplicate requests safely.
Would a database transaction alone solve everything?
Transactions solve many consistency problems, but distributed retries, messaging, and external side effects may still require idempotency and additional coordination.