In collaboration with MITLaboratory for Information and Decision Systems
A benchmark study in collaboration with MIT finds Routific outperforms four open-source algorithms on route quality. This open benchmark dataset samples real routes and defines a success metric that aligns better with real business value.
How do you assess the quality of a route optimization algorithm? The old way is to look at total travel time. But in the real world, it turns out there's a lot more to a good routing solution — and seasoned delivery planners and drivers can tell a good one at a glance.
Which of these two route plans would you choose?
Which would you dispatch?
Plan B is mathematically shorter. But most experienced dispatchers still choose Plan A. Why?
You could also say that Plan A just looks better — which turns out to be a really good way to judge the quality of a route. In academic-speak: "enhancing visual attractiveness in a routing plan has proven to be an effective way to facilitate practical implementation and positive collaboration."
What makes a route plan visually attractive? Mainly, routes that are compact and don't cross or overlap each other. In other words, they're not spaghetti routes.
Here's an example: a route that diverts one driver off a highway when another driver is already serving the same neighbourhood:
That single stop requires decelerating, exiting onto a minor road, navigating local traffic, hunting for parking to make the drop, then merging back onto the highway. Traditional route optimization solvers don't see the costs here, but they're real — and instantly obvious to experienced drivers.
There's another dimension of route planning that affects driver management, which can be summed up as route fairness: Do all drivers serve a roughly similar number of stops, and are their shift lengths equal?
So this benchmark looks at three high-level measures of route quality:
How often drivers' routes cross into each other's territory
Result 1 ↓Even stop counts and even shift lengths
Result 2 ↓The traditional measure: total driving time across the fleet
Result 3 ↓"Drivers have to get off the road, do two stops, then keep driving — they hate it!"
Joe Wemert, BeCool Couriers
"In real-life operations, the quality of a route is not exclusively defined by its theoretical length, duration, or cost. Experienced delivery drivers have tacit knowledge about the complex operational environment in which they serve customers on a daily basis."
Amazon Last-Mile Routing ChallengeMessy routing solutions have business consequences. When planned routes don't make sense, drivers throw the plans out and drive their own routes. Or they stick to the planned route, but hate it — and say so. Driver happiness is top of mind for most of the businesses we work with. They are ultimately the ones that represent your brand with the quality of their deliveries. Dispatchers are willing to spend hours manually tweaking routes until they make sense in the real world and keep their operations efficient and their drivers happy.
We compared the routing solutions from Routific and three leading open source solvers: VROOM, PyVRP and Google's OR-Tools.
We also tested jsprit, an open-source Java routing library. But it can't run problems above roughly 400 stops, and more than half the problems in this dataset are larger than that.
We tested each solver using 264 real-world delivery scenarios sourced from Routific users – in each case, one day's worth of actual deliveries from a real business (addresses were anonymized). Some scenarios had fewer than 50 stops, but most had more than 200 and 56 had more than 1,000 stops.
We included routing problems with multiple vehicles, time windows, and capacity constraints:
Table 1. Problems by type
| Type | Problems |
|---|---|
| One vehicle (TSP) | 12 |
| Multiple vehicles (VRP) | 46 |
| Capacity limits (CVRP) | 120 |
| Time windows (VRPTW) | 63 |
| Capacity and time windows | 23 |
Table 2. Problems by size
| Stops | Problems |
|---|---|
| Under 50 | 37 |
| 50–200 | 84 |
| Over 200 | 143 |
| of which over 1,000 | 56 |
Spaghetti routes send drivers into each other's territory. For example, two drivers might find themselves serving the same neighbourhood on the same day.
The benchmark identified spaghetti routes by looking at the number of crossings between routes: How often one driver's path crossed another's in the same day's plan, averaged across every problem we tested. A lower score indicates less overlap between drivers.
The result: Routific had a quarter of the route crossings compared to the worst performer.
Table 3. Mean crossings between routes per plan1. Routific is best or tied on 86% of problems.
| Solver | Crossings |
|---|---|
| Routific | 12.5 |
| VROOM | 33.7 |
| PyVRP | 36.0 |
| OR-Tools | 49.0 |
The importance of route quality really starts to show up at problem sizes larger than 500 stops, where Routific's performance vs open source algorithms begins to diverge significantly. It's on these large problems that the operational cost of poor routing is highest.
12.5 vs 49.0Mean crossings between routes: Routific against the worst performer, OR-Tools.
42.5 vs 155.3The same on problems with over 1,000 stops.
Route fairness measures how workloads and work hours are spread across the driver pool. When routes are not well-balanced, some drivers finish early while others run late, creating overtime costs and morale problems.
To measure it, we looked at two things:
On average, Routific's routing solutions had more balanced workloads and more equal shift times.
Table 4. Workload balance and shift spread1
| Solver | Workload imbalance | Shift spread |
|---|---|---|
| Routific | 73.8 stops | 2 h 31 m |
| VROOM | 97.3 stops | 3 h 38 m |
| PyVRP | 98.9 stops | 3 h 42 m |
| OR-Tools | 94.6 stops | 3 h 32 m |
1 hourRoutific's shifts are about an hour more even: the gap between the longest and shortest shift is 2 h 31 m, against 3 h 32 m or more.
All solvers land close together on the traditional metric of travel time. Across 143 problems, Routific's mean total travel time was 19 minutes (1.6%) longer than the fastest solver, VROOM, out of about 19 hours of driving across a whole fleet, and 7 minutes shorter than OR-Tools. That is a small trade-off in mathematical travel time for clean clusters and more balanced routes. Once these routes hit the road, the actual time to complete these routes may be faster in reality (though we have not verified this last statement in this study).
Table 5. Mean total travel time per problem2.
| Solver | Travel time |
|---|---|
| VROOM | 19 h 00 m |
| PyVRP | 19 h 12 m |
| Routific | 19 h 19 m |
| OR-Tools | 19 h 26 m |
(Note: To compare total travel time, we only counted solutions that featured the same number of vehicles. More vehicles automatically means more travel time, so a solution that used a larger fleet wouldn't be a fair comparison.)
19 minRoutific's extra travel time against the fastest solver (VROOM): 1.6% of a 19-hour fleet day. Against OR-Tools, Routific is 7 minutes shorter.
Table 6. Every metric we measured (lower is better). Means across the benchmark; per-instance numbers are in the repository.
| Metric | Routific | VROOM | PyVRP | OR-Tools | Routific best or tied |
|---|---|---|---|---|---|
| Crossings between routes, whole plan1 | 12.5 | 33.7 | 36.0 | 49.0 | 86% |
| Crossings on the worst leg1 | 2.1 | 3.9 | 3.2 | 4.4 | 80% |
| Crossings within a route1 | 84.0 | 86.0 | 85.8 | 86.8 | a tie |
| Idle and wait time1 | 1 h 07 m | 1 h 41 m | 1 h 30 m | 1 h 45 m | 98% |
| Workload imbalance (stops)1 | 73.8 | 97.3 | 98.9 | 94.6 | 59% |
| Longest minus shortest shift1 | 2 h 31 m | 3 h 38 m | 3 h 42 m | 3 h 32 m | 49% |
| Vehicles used1 | 13.9 | 14.2 | 14.3 | 14.4 | 91% |
| Total travel time2 | 19 h 19 m | 19 h 00 m | 19 h 12 m | 19 h 26 m | within 1.6% of the fastest |
| Mean solve time (same 6-vCPU node) | 1 m 48 s | 2 m 48 s | 3 m 37 s | 3 m 36 s | — |
1 Over the 215 instances where all four solvers serve exactly the same stops (crossing and balance metrics over the 177 of those with more than one route). 2 Over the 143 instances where all four also use the same number of vehicles. PyVRP and OR-Tools were given twice Routific's runtime. Definitions: analysis/README.md.
If travel time is the only metric that matters, there's very little real difference between the four route optimizers we tested.
But, as we've heard from our customers time and again, it's not a good enough metric on its own. Route plans that take the same time to drive can differ dramatically on points of quality: More criss-crossing routes and less-balanced shifts translate into real costs.
Spaghetti routes aren't a cosmetic issue. They're evidence that the routing algorithm missed something most drivers and dispatchers already know.
We ran these benchmark tests to bridge a real gap. Academic routing benchmarks like Solomon and CVRPLIB run on synthetically generated datasets and Euclidean distances. An algorithm can set records on those benchmarks and yet be completely unusable on a real delivery day, because the real world has street networks. One-way streets, asymmetric travel times, uneven stop density, vehicle capacity limits and fixed delivery windows all make the problem more difficult. And as we've shown, minimizing travel time alone is not enough.
We're publishing the full dataset of sampled real-world scenarios and the benchmark methodology. Anyone can re-run the open-source solvers, recompute every number, or pit their own solver against the same problems.
Our hope with this benchmark is to guide route optimization research in a more practical direction, to unlock more efficiencies for real-world logistics — and a greener, more efficient planet.