How Magnite uses Amazon VPC Route Server and Border Gateway Protocol (BGP) to build dynamic hybrid-cloud routing

Magnite, the largest independent sell-side advertising company, runs an engineering team that processes more than a trillion ad requests each day across Amazon Web Services (AWS) and its own data centers. In Magnite’s hybrid-cloud environment, network behavior is not background infrastructure. It’s part of the application. The team needs deterministic traffic steering, fast failover, and routing patterns that engineers can automate without building custom control-plane tooling.

Table of Contents

For a long time, AWS networking building blocks—static route tables, managed Network Address Translation (NAT) Gateways, and attachment-based routing constructs—covered most of what was needed. They work well when the network is simple and homogeneous. As Magnite’s network grew in complexity—including participants from outside AWS environments—the team needed dynamic routing capabilities that static route tables can’t easily provide.

Amazon Virtual Private Cloud (Amazon VPC) Route Server fundamentally altered this approach. By bringing BGP (Border Gateway Protocol) into Amazon VPC routing, Amazon VPC Route Server provides an AWS-managed control plane that learns routes from applications and propagates the best paths into route tables. With Amazon VPC Route Server, Magnite can manage routing in the cloud the same way it manages routing on its on-premises network devices, which simplifies operations and broadens the team’s ability to dynamically adjust the routing environment.

This post walks through how Magnite used Amazon VPC Route Server, BGP (Border Gateway Protocol), and Bidirectional Forwarding Detection (BFD) to build two reusable patterns: intelligent outbound routing for a custom NAT layer, and an internal active-active load-balancing pattern for high-performance services. The key takeaway is not the individual applications. It is the design space that opens when dynamic routing is available inside a VPC.

Solution overview

At a high level, both patterns use the same control-plane model. BGP-capable nodes (running on Amazon Elastic Cloud Compute) in each Availability Zone peer with Amazon VPC Route Server endpoints. Those nodes advertise routes that represent the needed behavior: a preferred default route for outbound traffic, or /32 Virtual IP (VIP) routes for an internal service. Amazon VPC Route Server learns those routes, selects the best path, and propagates the result into the route tables selected for management. BFD provides fast liveness detection and simple BGP policy (primarily AS path prepending) allows Magnite to express preference without inventing a parallel orchestration system.

This approach provided a reusable routing building block with several immediate benefits:

Route advertisement and withdrawal based on actual node health
Failover without external services mutating route tables
A cleaner path to hybrid-cloud traffic engineering
One operational model applicable to more than one networking problem

Why dynamic routing mattered

Static routes are simple, which works well until requirements outgrow them. The moment route preference based on node health, failover without control-plane glue code, or different paths for different classes of traffic become necessary, static route tables start forcing the wrong kind of work. Teams end up building liveness systems, route mutators, and orchestration around something the routing protocol already supports.

Amazon VPC Route Server provides a more natural model. BGP-speaking nodes advertise routes, and Amazon VPC Route Server selects the best path and propagates it into the designated route tables. When liveness detection is configured with BFD, failure handling becomes more direct: the peer drops, the route is withdrawn, the best remaining path wins, and new traffic follows it.

That matters because Amazon VPC Route Server is not limited to one narrow routing surface. It can propagate routes into subnet route tables, VPC route tables that are not associated with subnets, and internet gateway route tables. That makes it useful for both east-west and north-south traffic patterns, while keeping the control plane AWS managed.

Use case 1: Intelligent outbound routing with MagNET

One of the first places Magnite applied this model was outbound routing. The team built MagNET, a custom NAT/transit layer, on Amazon EC2 instances running Netgate’s TNSR,software router from Netgate. The goal was to simplify network topology by using technologies already deployed in Magnite’s own data centers on AWS. The fundamental problem was that Magnite needed more control over routing behavior, failure handling, and traffic engineering than static outbound patterns provided.

The MagNET pattern is straightforward:

TNSR instances run in multiple Availability Zones
Each instance peers with its AZ-local Amazon VPC Route Server endpoint over BGP
The instances advertise a default route, 0.0.0.0/0, into the routing domain
Amazon VPC Route Server propagates the preferred path into the route tables used by private subnets
BFD provides fast liveness detection, so failed peers stop attracting new traffic quickly
TNSR instances use BGP to peer with Magnite’s on-premises network devices to learn about available networks and paths

Figure 1. MagNET dynamic outbound routing with Amazon VPC Route Server.

During normal operations traffic on the happy path stays AZ-local — workloads in each Availability Zone route through the local TNSR instance, which advertises the preferred default route (0.0.0.0/0) with a shorter AS path to the AZ-local Amazon VPC Route Server endpoint. During a primary node failure the BGP session drops, the route is withdrawn, and Amazon VPC Route Server automatically propagates the backup route from the partner TNSR in the adjacent Availability Zone allowing traffic to flow cross-AZ without any route table manipulation or human intervention.

BGP attributes express preference with one node advertising the preferred path and a second node advertising a backup. Under normal conditions, traffic follows the primary path. If the preferred node fails or is deliberately drained the BGP session drops, the route is withdrawn, and new flows converge on the surviving path without any route table manipulation and without humans in the loop.

This provided a cleaner foundation for traffic engineering. After a BGP-speaking outbound layer was in place, the team can reason about prefixes, backup paths, and degraded-mode behavior in one consistent way. In the production design lower-priority fallback outbound routing remains available through AWS native constructs so the network still has a valid exit even when both preferred NAT nodes are unavailable.

Efficiency gains were a significant byproduct of this architecture. Magnite realized an 80% reduction in VPC Interface Endpoints and NAT Gateway dependencies within the highest-volume Region. By transitioning from a variable pay-as-you-go structure to a tiered, throughput-based cost model, expenses shifted toward simpler, more predictable AWS networking constructs. While these savings were substantial, the primary advantage is the ability to treat outbound routing as a deterministic system characterized by explicit intent, rapid convergence during failures, and an automation-first operational profile.

Use case 2: Internal active-active load balancing with route-aware VIPs

After Magnite trusted Amazon VPC Route Server as a control plane it became clear that NAT was only one pattern. The more interesting one, at least architecturally, was internal load balancing.

The route server isn’t a general replacement for the AWS family of managed load balancers (Application, Network, and Gateway). It is a specialized pattern for internal, high-performance workloads where Magnite controls the client route tables and wants tighter control over node preference, traffic flows, readiness, and failover behavior.

Magnite has experience running HAProxy at scale on its own servers so it was the natural choice for the software component of the new load balancing stack. The way Magnite historically architected for high availability and load sharing was not going to work on AWS, and it wasn’t until the team gained experience with the Amazon VPC Route Server offering that a new viable model became apparent.

The model works like this:

Load-balancer nodes are assigned Virtual IPs (VIPs) from address space that belongs to the VPC but is intentionally left unallocated to subnets
Each node advertises the VIPs it currently owns to Amazon VPC Route Server over BGP
Amazon VPC Route Server propagates /32 host routes for those VIPs into the route tables used by clients
Nodes advertise both primary and backup VIP ownership using different preferences
DNS publishes the VIP set for the service and AZ-specific records can bias clients toward local nodes when locality is needed

Figure 2. Internal active-active VIP routing with Amazon VPC Route Server.

Each HAProxy node advertises /32 VIP routes to Amazon VPC Route Server over BGP which propagates host routes into the client subnet route tables directing traffic to the specific node currently owning that VIP. When a node fails its routes are withdrawn and Amazon VPC Route Server automatically promotes the backup paths advertised by partner nodes, redistributing traffic automatically—no manual route table changes required.

This pattern is attractive because it separates concerns cleanly. DNS answers the question, “What are the service IPs?” Routing answers, “Which node should receive traffic for this VIP right now?” Because the solution uses a purpose-built controller Magnite can use advanced capabilities such as mixed instance type clusters and fully dynamic frontend and backend configuration changes.

The failure mode is attractive too. Since each node owns primary VIPs and carries backup responsibility for VIPs from a partner node a single-node failure does not produce an all-at-once traffic cliff on the survivors. In Magnite’s design the redistribution event is bounded. Partner nodes absorb roughly a 25% increase in load for the affected VIP set rather than forcing the whole service to rebalance at once.

Figure 3. Cross-VPC Scenario — clients and backends in different VPCs.

This solution also bridges accounts and VPCs efficiently and without additional data-transfer costs. Magnite’s technical history includes several acquisitions and important products operate in their own AWS accounts and within legacy VPCs.

The architecture uses a combination of AWS Resource Access Manager (RAM) sharing and multiple Elastic Network Interfaces (ENIs) homed to different VPCs so traffic flows without additional hops removing even the VPC peering hop when connecting traffic across VPC boundaries.

Operating the pattern at Magnite scale

Magnite’s environment isn’t a single self-contained cloud footprint. Magnite operates a hybrid footprint across AWS Regions and its own data centers and does so for systems that care deeply about availability and latency. That context shaped how Magnite adopted Amazon VPC Route Server.

The team started in Magnite’s lab. BGP behavior, liveness detection, route convergence, and route propagation mechanics were validated before putting the pattern in front of production traffic. After that the rollout proceeded in phases, beginning with narrower scopes where operational behavior can be validated under real load.

Observability was regarded as a mandatory design principle from the project’s inception. Magnite monitors peer state, BFD state, route propagation status, traffic levels, and node health. The team also built the surrounding operational tooling so Site Reliability Engineers (SREs) can refresh nodes, inspect live state, and understand what the control plane is doing without relying on specialized individual knowledge.

Automation was regarded as an equally critical priority. The stack around these patterns includes Terraform for modeling the AWS resources, Packer and Ansible for Amazon Machine Image (AMI) creation, Ansible and NETCONF workflows for day-2 configuration, and OpenTelemetry-based telemetry flowing into managed monitoring systems.

With MagNET, tuning BFD and building the surrounding operational flow got the team to the point where triggered transitions are graceful and hard failures converge on the order of two seconds. That’s fast enough to be viable for production workloads and allows Magnite to collapse on a single networking device for both NAT and Transit Gateway needs that is natively integrated with its on-premises networks.

Lessons learned

A few design lessons stood out.

Plan around Amazon VPC Route Server quotas early. This is a real control plane, but it is not unbounded. The default quotas for route server endpoints, peering sessions, and advertised routes shape the topology sooner than expected.

Keep the policy simple. Magnite experimented with multiple BGP attributes, but AS path prepending proved to be the most intuitive and predictable way to express preference. MED still has its place, but shorter AS paths are easier for humans to reason about during an incident.

Treat IP address management as part of the application design, not an afterthought. The internal load-balancing pattern only works cleanly when VIP space is planned up front. Those addresses are most easily managed when they belong to the VPC, while remaining outside any overlapping subnet allocations.

Keep a recovery path simpler than the primary path. One useful aspect of dynamic routing is that teams can express degraded-mode behavior explicitly. Magnite made use of lower-priority fallback outbound paths so the network always had a valid place to go even when the preferred path was unavailable.

Build the routing pattern and the operating model together. Dynamic routing is powerful, but only when the surrounding system is disciplined. The overall system is complex, so teams must actively manage configuration drift and keep the components aligned.

Conclusion

Amazon VPC Route Server provided a managed routing control plane inside the VPC that Magnite combined with standard BGP behavior, fast liveness detection, and infrastructure automation to create a fault-tolerant, customer-managed routing solution.

At Magnite, that enabled two patterns the team cares about: a smarter outbound routing layer with MagNET, and an internal active-active load-balancing model built around route-aware VIP ownership. The two patterns differ on the surface, but underneath, they rely on the same idea: routing should respond to health and intent directly.

If you’re operating a hybrid environment, running network appliances on AWS, or need more traffic-steering control than static routes provide, Amazon VPC Route Server is worth a serious look. It opens design space that used to be cumbersome to reach inside a VPC, and it does so with constructs that network and platform engineers already know how to reason about.

To learn more about Amazon VPC Route Server, refer to the documentation.

___________________________________________________________

About the authors

Ryan Williams (Guest) is a Fellow in Magnite’s Technical Infrastructure Engineering & Operations team focussed on scaling and improving the hybrid infrastructure that underlies all of Magnite’s applications and especially the network plumbing. Ryan is the primary technical architect for routing traffic amongst cloud based applications across Availability Zones and VPCs and between cloud and on premises applications. Ryan also spends considerable time on ensuring the the tooling to configure, operate, monitor, and manage applications and networks scales and meets the Magnite’s needs as they evolve over time.

Varad Ram is an Enterprise Solutions Architect supporting customers in Advertisement and Marketing vertical at Amazon Web Services. He collaborates closely with customers to design scalable and operationally efficient solutions. Currently, his primary focus is on analytics and helping customers maximize their return on investment in Generative Artificial Intelligence. In his free time, Varad enjoys biking with his children and playing tennis.