When we first build microservices on Cloud Run in Google Cloud, everything feels smooth. Each service gets a nice URL, requests flow, and things just work. But then reality kicks in: how do these services actually talk to each other at scale?
This is the story of that journey — discovering one problem at a time, fixing it, and then moving to the next.
Table of contents
Open Table of contents
-
- Step 1: The Obvious Way — Public URLs
- Step 2: Restricting Traffic — Private Access Needed
- Step 3: Introducing the Load Balancer
- Step 4: The DNS Layer
- Step 5: Local Development Realities
- Step 6: One Load Balancer or Two?
- Step 7: Multi‑Region Deployments
- Step 8: The Latency Dilemma
- Step 9: Enter the Global Load Balancer
- Step 10: Internal Multi‑Region Calls
- Step 11: DNS + Multi‑Region Awareness
- Step 12: The Cost Factor
- 🏁 The Destination: A Balanced Architecture
Step 1: The Obvious Way — Public URLs
At the beginning, Service A calls Service B using its .run.app URL. Easy!
But wait…
- All requests go out to the internet (through Google Front Ends).
- Traffic between services in the same project still makes a “round trip.”
- Latency creeps in, and you start paying for egress you shouldn’t need.
👉 First Problem: Microservices inside the same cloud are communicating as if they’re strangers on the internet.
Step 2: Restricting Traffic — Private Access Needed
We think: “Fine, let’s make services private.” Cloud Run allows ingress settings:
- Public (default).
- Internal & Load Balancer only.
Now, only Google Cloud resources can call our service. This improves security. But… where’s the private IP?
- Cloud Run doesn’t have one.
- You can’t just say
10.x.x.x/serviceB. - Everything still routes through GFE.
👉 Second Problem: Without private IPs, we can’t have true internal service-to-service communication.
Step 3: Introducing the Load Balancer
The fix: put Cloud Run behind a Load Balancer.
- An External LB gives a stable public IP + custom domain.
- An Internal HTTPS LB (ILB) gives a private IP in your VPC.
Now, finally, Service A can call Service B through a private entrypoint. Latency drops, egress costs vanish. But then comes a new snag…
👉 Third Problem: Do we really want to hardcode ILB private IPs everywhere? And what about TLS certificates? They expect hostnames, not IPs.
Step 4: The DNS Layer
Enter Cloud DNS private zones.
orders.internal.example.com → ILB private IPpayments.internal.example.com → ILB private IP
Now services call each other by name, not by number. TLS certs validate properly. If the ILB IP changes, DNS handles it transparently.
We solved discovery. But then, developers ask:
👉 Fourth Problem: What happens when we run apps locally on laptops? Private DNS won’t resolve outside the VPC.
Step 5: Local Development Realities
Locally, the .internal.example.com names fail. Why? Because Cloud DNS private zones live only inside the VPC.
Options emerge:
- Simulate mappings with
/etc/hostsordnsmasq. - Use a Cloud VPN so laptops join the VPC and resolve private DNS.
- Or create a dev subdomain (
*.dev.example.com) that resolves differently in dev vs prod.
Now local dev matches production — one hostname everywhere, but with split resolution.
Step 6: One Load Balancer or Two?
At this point, we have:
- Public traffic (from internet clients).
- Private traffic (from microservices inside VPC).
Do we need two load balancers?
- One (Hybrid): An external LB + split-horizon DNS. Inside VPC, it routes internally; outside, it routes publicly. Simpler.
- Two (Separated): One external LB for internet users, one ILB for service-to-service. Cleaner isolation, but more overhead.
- Internal Only: If no public access is needed, rely solely on ILB with private DNS.
The decision depends on your priorities: simplicity vs strict separation.
Step 7: Multi‑Region Deployments
So far, everything is within one region. But what happens when we scale out to multiple regions?
We deploy Cloud Run in:
us-central1for North America.europe-west1for Europe.asia-northeast1for Asia.
Each gets its own URL. Now…
👉 New Problem: How do clients (and other services) know which region to hit?
Step 8: The Latency Dilemma
If a European user calls the US endpoint, latency spikes. If Service A in Europe calls Service B in the US, cross‑continent delays pile up. If one region fails, there’s no automatic failover.
👉 We need smart routing.
Step 9: Enter the Global Load Balancer
Google Cloud’s Global HTTPS Load Balancer solves this:
- One global entrypoint (
api.example.com). - Routes traffic to the nearest healthy region.
- Handles automatic failover.
Now, clients always get low latency and high availability.
👉 External traffic is solved. But internal traffic?
Step 10: Internal Multi‑Region Calls
ILBs are regional.
orders.internal.example.comineurope-west1doesn’t reachus-central1automatically.- Cross‑region calls travel over Google’s backbone, but design is needed for replication and data consistency.
👉 Multi‑region introduces new challenges: state management, routing, and cost.
Step 11: DNS + Multi‑Region Awareness
We can extend DNS with region‑aware records:
orders.eu.internal.example.com → ILB in europe-west1.orders.us.internal.example.com → ILB in us-central1.
Services choose their regional peer. For more automation, add Traffic Director or a service mesh.
Step 12: The Cost Factor
Cross‑region calls aren’t free.
- Inter‑region egress charges apply.
- Every replicated DB write or chatty service call across regions increases both latency and cost.
👉 Efficiency matters: don’t let microservices chatter across continents.
🏁 The Destination: A Balanced Architecture
Our journey uncovered challenges step by step:
- Public URLs → easy but inefficient.
- Ingress restrictions → secure but still public path.
- Load Balancers → private entrypoints.
- Cloud DNS → service discovery.
- Local development → VPN or DNS tricks.
- One vs two LBs → simplicity vs isolation.
- Multi‑region → latency, failover, consistency.
- Global LB → external smart routing.
- DNS + replication → internal multi‑region solutions.
✅ Key Lesson: Cloud Run is regional and public by default. To build resilient microservices, you need Load Balancers, Cloud DNS, and a multi‑region strategy. Each fix solves one problem, but opens the door to the next challenge.