Parsing map service URLs with regex

A table of parsed service URLs used by the City of Boulder. Screenshot by author.

If you have ever accessed geospatial data over the internet, you’ll likely recognize URLs like these:

https://hazards.fema.gov/gis/nfhl/rest/services/public/NFHL/MapServer/28

https://hydro.nationalmap.gov/arcgis/rest/services/wbd/MapServer

https://maps.bouldercolorado.gov/arcgis/rest/services/fire/FireResponseTimesOpenData/MapServer/0

All of these are examples of ESRI ArcGIS map services, and they are some of the most prevalent APIs for accessing and rendering geospatial data on the web. And for the eagle eyed among you, they also follow a pretty distinct pattern:

https://<web-domain>/<server>/rest/services/<optional-folder>/<service-name>/ MapServer/<optional-layer>

Patterns? Strings? If you’re thinking, “Regex was made for parsing these URLs…” Welcome, friend. Please take your shoes off at the door.

But first…

Maybe you’re also thinking, “Sure, regex and URLs are a match made in heaven, but why parse these URLs at all?”

Well, I work for the City of Boulder, CO, and we are in the midst of a migration from ArcGIS Online to Portal. In the course of that massive undertaking, it became clear that we needed a way to catalog, sort and filter map services based on characteristics like web domain, server name, and more. Ultimately, this would help GIS staff easily track down web maps and update URLs for map services that were transferred to servers with different names, or replace bad URLs with correct ones.

Get Jesse Nestler’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

If you are curious and want to follow along with map services used in web maps in your own deployment of ArcGIS Online or Portal, download and run this hosted notebook in ArcGIS Pro (while logged in).

The structure of a map service URL

Let’s revisit the structure of these URLs and build the regex expression that matches our needs as we go.

Protocol and web domain

At the beginning of every URL, there is a protocol followed by a domain name:

https://hazards.fema.gov/gis/nfhl/rest/services/public/NFHL/MapServer/28

While we weren’t necessarily interested in whether a given URL was using secure protocol or not, we were very interested in extracting the domain name. The regex we used to match and capture this group was:

https?://([^/]+)/

In essence, this regex says, “The string starts with ‘http’ and might be followed by zero or one ‘s’ characters. Capture all the non-forward-slash characters between that ‘https?://’ pattern and the first ‘/’ in the URL.”

Server

Next, we want to extract the name of the server hosting the map services. For our purposes, we called this everything after the domain but before ‘/rest/services/’:

https://hazards.fema.gov/gis/nfhl/rest/services/public/NFHL/MapServer/28

https?://([^/]+)/(.+)/rest/services/

Now the regex says, “The second capture group is everything between the forward slash that comes after the web domain and the literal string ‘/rest/services/’.”

Optional Folder

Navigate to the root of a map service, and you’ll notice that there are various folders that contain map service endpoints.

The structure of folders on one of the City of Boulder’s server sites. Screenshot by author.

It is not strictly necessary for GIS Administrators to publish services into folders; indeed, when we summarized all the map services being used in our AGO deployment, we realized many were not. So, we also needed a way to extract the name of the folder if it existed in the URL:

https://hazards.fema.gov/gis/nfhl/rest/services/public/NFHL/MapServer/28

https?://([^/]+)/(.+)/rest/services/(?:([a-zA-Z]+)/)?

The details of this capturing group are especially tricky, but in essence, this regex simply says, “there might be a folder name in the URL after ‘/rest/services/’, and if it’s there, capture it; otherwise, it’s okay if the folder is not present.”

Service Name

This one goes without saying. We wanted to be able to search by the name of the service, which always appears after the optional folder:

https://hazards.fema.gov/gis/nfhl/rest/services/public/NFHL/MapServer/28

https?://([^/]+)/(.+)/rest/services/(?:([a-zA-Z]+)/)?(.+)/

Service Type

‘Lo and behold, there’s more than one type of service one can have on an ArcGIS Server, including MapServer, FeatureServer, and ImageServer (to name a few). Luckily, these always take the form of “<alpha-characters>Server” after the service’s name:

https://hazards.fema.gov/gis/nfhl/rest/services/public/NFHL/MapServer/28

https?://([^/]+)/(.+)/rest/services/(?:([a-zA-Z]+)/)?(.+)/([a-zA-Z]+[Ss]erver)

Layer

Lastly, we wanted to optionally extract the layer number if it was present in the extracted URL. This is because when people use services in their web maps, they can either specify an individual layer to pull into the map, or pull in every layer in the map service by omitting a layer number.

https://hazards.fema.gov/gis/nfhl/rest/services/public/NFHL/MapServer/28

https?://([^/]+)/(.+)/rest/services/(?:([a-zA-Z]+)/)?(.+)/([a-zA-Z]+[Ss]erver)/?(\d*)

Implementing the regex

We’ve arrived at the final form of our regex, and boy-howdy it’s a doozy!

https?://([^/]+)/(.+)/rest/services/(?:([a-zA-Z]+)/)?(.+)/([a-zA-Z]+[Ss]erver)/?(\d*)

Rather than iterate through a list of URLs and apply the expression one-by-one, we made use of vectorized operations in pandas, which took less than a second to parse around 10,000 URLs!

import pandas as pd

# Sample DataFrame
data = {'url': [
    'https://hazards.fema.gov/gis/nfhl/rest/services/public/NFHL/MapServer/28',
    'https://hydro.nationalmap.gov/arcgis/rest/services/wbd/MapServer',
    'https://maps.bouldercolorado.gov/arcgis/rest/services/fire/FireResponseTimesOpenData/MapServer/0'
]}

df = pd.DataFrame(data)

# Define the regex pattern
pattern = (
    r'https?://'  # Match 'http://' or 'https://'
    r'([^/]+)/'  # Capturing group 1: Match and capture the 'web_domain'
    r'(.+)/rest/services/'  # Capturing group 2: Match and capture the 'server'
    r'(?:([a-zA-Z]+)/)?'  # Capturing group 3: Match and capture the 'service_folder' (optional)
    r'(.+)/'  # Capturing group 4: Match and capture the 'service_name'
    r'([a-zA-Z]+[Ss]erver)/?'  # Capturing group 5: Match and capture the 'service_type'
    r'(\d*)'  # Capturing group 6: Match and capture the 'service_layer' (optional)
)

# Use str.extract to create new columns based on the pattern
df[['web_domain', 'server', 'service_folder', 'service_name', 'service_type',
    'service_layer']] = df['url'].str.extract(pattern)

# Change missing values to '' in columns that are optional
df.fillna('', inplace=True)

# Display the result
print(df)

There you have it! I hope this helped you understand regex a little more, or inspired you to go forth and parse your own strings!

We ended up enriching this output with information about the URL status and any associated web maps, then input the data to an ESRI dashboard that GIS practitioners within the City could use to track down and easily replace old/broken services.