Autonomous Vehicle Engineers Need to Solve the Weather Problem.

Take a minute to imagine some common driving scenarios that humans face while navigating today’s roads. A driver enters a mountain pass when suddenly heavy fog descends and obscures the view of the road and other vehicles. A heavy local downpour washes over the area and blinds a driver, making lane markings hard to discern despite windshield wipers on full tilt. Snow squalls and sandstorms engulf rural roads, blinding the drivers, blanketing the road, hiding ditches, and obscuring markings that delineate the road edge.

These are all common road conditions, but today’s best lidars, cameras, automotive radar, and GPS fail spectacularly in these same conditions. They might be able to perceive objects ahead, but these technologies can’t reliably localize a vehicle on the road or navigate to a destination. When the driver is blinded and when the environment is obscured are two of the most critical times for autonomous vehicles and driver assistance systems to work. As vehicles become more automated and rely less on human intervention, these gaps in performance will prove fatal.

An insurance or safety company or government agency is unlikely to support the mass deployment of autonomous vehicles if they regularly cause significant injuries or fatalities during common weather conditions. This is true not just for passenger vehicles and taxis but also 40-ton 18-wheelers operating on the same roads.

This article will delve into the localization problem and industry’s weather related technological gaps. It then details the new GPR sensor and shows with data and videos that it solves the all-weather localization performance gap.

The weather problem.

Let’s take a quick look at where the current deployments of autonomous vehicles are primarily located:

The location of all autonomous trucking deployments tracked by NHTSA. All deployments are in the desert southwest where there is little challenging weather to deal with. Source: https://www.nhtsa.gov/automated-vehicle-test-tracking-tool

The location of all autonomous car testing on public streets tracked by NHTSA. Note that deployments in Las Vegas do not appear to be called out on this map. A scattering of small tests are occurring in northern climates, but the primary deployments are in the desert southwest where there is little challenging weather to deal with. Source: https://www.nhtsa.gov/automated-vehicle-test-tracking-tool

The primary deployments do not involve operating in challenging weather conditions.

Let’s next review a few permits and manuals to see what the regulators and insurers are allowing currently (this is from digging into available information, not to pick on any companies in particular — every single one of the companies out there has this issue).

Top Level 4 systems

Top Level 2 and 3 systems

It is clear that there is a significant long standing problem with weather and the current sensor suite. Not a single company currently has a solution to this problem.

This problem has now been solved, which I’ll get into in a minute…

GPS, maps, and radar limitations

There are two critical problems to solve to enable accurate autonomous vehicle performance during poor conditions, let’s take them one by one.

1. Not hitting things means being able to detect objects on the road even when the snow, rain, dust, or fog are blocking much of the light going into optical sensors and obscuring landmarks. While lidar and camera can do object detection to some extent, study after study has shown automotive cameras and lidar sensors experience serious degradation in heavy weather conditions. Forward-facing radar, while susceptible to ice buildup and some degradation, can typically detect upcoming objects in these conditions. Fusing sensors, such as radar and cameras, is a method of effectively detecting objects in challenging conditions. While addressable challenges in the perception space do remain, that’s just half the problem.
2. Staying on the road means knowing the vehicle location and the road location at all times. Maps can provide the basis to know the road location. It can tell you when things have changed and enable you to operate even without seeing the edge of the road or even project the lane markings onto the windshield, so you know where it’s safe to drive. It is what will keep you out of a ditch in a snowstorm. However, to use a map for safety in these conditions, it’s essential to know the vehicle position precisely and reliably. Let’s call this the “localization problem”

GPS is insufficiently accurate or reliable (multipath, blockage, jamming) to solve the “localization problem”. As a practical matter, GPS systems also have trouble determining when their outputs are accurate and should be relied upon. From a safety perspective, this typically prevents them from being used as a primary localization system.

Cameras and lidars just don’t have the information to perform safely (too little light, too much variability in the environment, too much obscuration of the maps or markings). They can’t penetrate the snow or ice in most conditions so landmarks are substantially changed or obscured for positioning applications.

Automotive radar is too low resolution currently for accurate positioning, but even newer high resolution 77GHz automotive radar systems degrade when trying to find key landmarks on a map in tough weather conditions such as snow accumulation. They can’t penetrate accumulated snow or impure ice to much depth (~1cm) due to the radar’s high frequency.

The “localization problem” enables staying on the road and is currently not solved by GPS, cameras, lidars or automotive radar. The common issue? These technologies are all looking in the wrong place — the highly dynamic surface environment surrounding them.

A new sensor, Ground Positioning Radar (GPR), solves the “localization problem” using low frequencies. It is insensitive to tough weather conditions and provides centimeter-level positioning accuracy during snow or sandstorms, heavy rain or fog, and while in GPS denied areas.

GPR demonstrating cm-level positioning in heavy snow.

The above video shows accurate GPR positioning in weather conditions that are challenging to lidar and camera based localization approaches. The edge of the road isn’t clear, the lane markings are obscured, the optics need to be continually cleaned, and the trees don’t provide a reliable reference. Let’s dig into the GPR technology a bit to understand why this solves the localization problem completely.

How Ground Positioning Radar solves the “localization problem”

Ground Positioning Radar is uniquely designed using low-frequency radar (~300 MHz) that looks up to 3 meters into the ground to create a character-rich 3D map of the road’s unique subsurface signatures.

Ground Positioning Radar map showing approximately 3 meters of depth into the road subsurface.

A map is collected of the subsurface. Additional layers of information, such as road markings or GPS positions are layered into that map. Vehicles with the GPR sensor integrated can then drive over a mapped area. The sensor then registers its images and calculates the vehicles’ real-time position to less than the size of a post-it note.

Get Byron Stanley’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

GPR also enables operation inside of parking garages where the decks are thin and there is neither GPS nor infrastructure available to operate.

The system runs even in adverse conditions while completely without GPS — below is a cool internal video we made in a parking garage with rain falling from the ceiling above. Note just how closely the autonomous vehicle is retracing its prior tracks.

Autonomous vehicle guided solely by GPR-based positioning demonstrates exact retracing of tire tracks in a multilevel parking garage with heavy active rain falling from the ceiling above

So let’s look at some of the fundamentals as to why GPR works when other sensors fail…

First of all, the GPR sensor uses low-frequency radar waves (~300MHz). This is critical for two reasons:

1. To see the character inside of the ground, you need to first be able to penetrate the surface. Low frequencies penetrate the ground, which higher frequencies just don’t allow.

Model of depth of penetration into dry soil (green) and wet soil (purple) as a function of frequency. Automotive Radar at 77GHz radar penetrates ~1cm into dry soil and less in wet soil. GPR at 300Mhz penetrates ~1 meter into this dry soil and ~0.5 meters in wet soil. [Chart referenced from Recommendation ITU-R P.527–6]

2. Low frequencies are orders of magnitude less sensitive to snow and ice accumulation, as shown in the plot below. This also enables the GPR sensor to operate without cleaning requirements.

Penetration depth by frequency at different densities of dry snow and pure ice. At 77GHz that is used by automotive and HD radars, the depth of penetration is approximately 1cm in typical road snow conditions. While pure ice yields up to 15cm of penetration, salted ice, as typically seen on the roads, yields 2-3cm penetration depth at 77GHz. GPR’s frequency (~300MHz) is just below the frequencies shown on this chart yielding over 100m of penetration depth in all cases. [Chart referenced from Michel Gay et al]

Secondly, the GPR sensor is looking straight into the ground. This is a distinct advantage.

1. GPR data is statistically independent of the dynamic above ground environment, which is powerful. As an example, if a camera and a lidar are mounted on a vehicle and the vehicle is in a lane between two tall trucks, most of the localization landmarks are completely obscured for both optical sensors. With a statistically independent sensor like GPR, the landmarks are still clearly visible in the same situation.
2. When you combine statistically independent sensors together, you get a substantially more robust system. As a simple example, take a set of completely statistically independent sensors that each fail at once every 100 hours. If combined properly, the two independent sensors would both fail at the same time only once every 10,000 hours. GPR is one of a limited set of statistically independent sensors and as a result, will significantly increase reliability and safety of the localization system.

Demonstration of reliable and highly repeatable GPR guided autonomous operation with no GPS, no camera, no lidar, and no automotive radar. The use of the cover emphasizes the independent nature of the GPR sensor.

Lastly, the features are stable and rich. The GPR data comes from changes in soil types, changes in soil densities, rocks, roots, cavities, structure in the road or deck, and a range of other factors associated with the electromagnetic and magnetic properties of the subsurface layers. An example of the data collected is shown below:

A single depth layer of the map is shown for a highway route to illustrate the typical level of character in GPR maps. Each depth layer contains additional unique content.

The uniquely rich character and variation of the subsurface is shown above. This character is derived from changes in the permeability and permittivity of the ground and road infrastructure within a few meters of the GPR sensor. It is this quality along with the stability of the maps that enables GPR localization solutions to converge to centimeter level accuracy.

GPR Performance

Let’s start with the accuracy, precision, and repeatability of GPR systems. We filmed 24 autonomous loops of the GPR guided vehicle driving with its cover on (no camera, no lidar, no radar, no GPS). The blue arrow points to the same visual location on the ground. As a visual demonstration of precision and repeatability, the outer edge of the autonomous vehicle wheel bisects the ~5cm wide patch of grass on the ground every run as shown below.

Repeated tracks using a GPR guided autonomous vehicle (no lidar, no camera, no radar, no GPS). The blue arrow denotes a visual reference that shows that the GPR is repeatedly putting the wheel in exactly the same spot over 24 autonomous runs.

As a demonstration of accuracy, a sampling of datasets collected over five conditions and environments is shown in the table below:

Centimeter level accuracy demonstrated on highway and local roads in dry, rain, and snow conditions. Accuracy was measured based on an OxTS RTK GPS/INS system when GPS coverage was available.

The GPR system shows centimeter level accuracy in both along and cross track localization as referenced by an RTK based GPS/INS system while it was available. A similar example in challenging optical conditions in the rain at night is shown below.

While the datasets and rain and snow videos clearly show solid ~5cm cross-track and 5–10cm along-track performance in all conditions (fog clearly also has no impact on GPR), they are only part of the story. It is important to understand what percentage of measurements are unique in their surrounding areas.

We developed deterministic metrics to isolate and identify uniqueness of the GPR data itself along a run. The combined statistics show the probability that any particular radius will contain a value greater than the central radius. This measures how likely a the GPR system is to converge to an accurate solution if the vehicle knows its position to within a known radius.

Steps to calculate the probability of a convergence for a given radius over a single pass width map.

Below shows the results of a 10km highway route with perfect probability of convergence from within a 10m radius to within a 0.57m safety radius.

10km Highway route demonstrating exactly 100% probability of a good match at all radii within a 10m radius. The probability of a good match to better than a 0.57 meter safety radius is calculated for each radius for each measurement on the route. [safety radius derived from Reid et al.]

While perfect convergence is beneficial, if the solution is heavily reliant on any form of propagation-based technique, it can be susceptible to erroneous localization solutions. The GPR team has been working on isolating the performance of the signal processed data from the localization and propagation-based algorithms. This is important as the signal processed data is the fundamental contributor to robust system operation.

Probability of good match curve for the 10km highway route above, but calculated without propagation based solutions. The GPR team continues to improve upon these results by continuing to improve upon our signal processing chain.

The above plots show the probability of convergence of a solution within a maximum area radius of 0.57m given an input position within a 10m search radius for a single car width map.

While the GPR team continues to improve upon performance, GPR has demonstrated that it can accurately and reliably localize in all weather conditions, including the conditions that existing sensor stacks fail in during true tests of safe operation for autonomous vehicles.

GPR is defining the standard for positioning in assisted and autonomous driving

Vehicles that deploy GPR will be safer than other vehicles on the road due to their ability to localize in all conditions and improved reliability. With this substantial safety improvement, insurance companies and regulators are likely to take a hard look at requiring GPR in every automated or ADAS vehicle. GPR will also unlock geographies that couldn’t allow autonomous vehicles before due to weather limitations and enable autonomous vehicle deployments to finally scale.

GPR is currently partnered with a number of automotive OEMs, Tech companies, and suppliers to integrate its technology into consumer vehicles and commercial trucks. Similar to the automotive radar pathway, mass production material and assembly cost for GPR will be well under $100.

What would consumers, trucking companies, and riders be willing to pay for a vehicle that could confidently and safely drive or assist drivers through even the roughest weather conditions? What about a vehicle that could also detect kids and animals under the car, position electric vehicles on top of wireless chargers even in winter outdoors, and a whole host of other capabilities? I suspect that answer is greater than $100.