Local search on Microsoft Local Live

Steve Lombardi from Microsoft emailed me about the local search issues I raised in regard to Google Maps on the iPhone (and other mobile platforms) in my earlier post. In that post I compared the results I got with Google to those I got from MapQuest - I would have tried Microsoft Local Live also, except I had been under the mistaken impression that it didn't work on my Mac ... turns out it doesn't work with Safari (you get a very scaled back site which is pretty useless), but it works fine with FireFox (for 2D stuff - Virtual Earth in 3D, and Photosynth, are actually the things I miss most in Mac land so far!). So I was pleased to discover that I can at least use the 2D stuff still.

Steve had tried the same tests I did on Local Live, and I tried them for myself and it fared well. With luck, this live link will give you more or less the same thing as the screen shot above. I ran several tests in my previous review, all centered on 1792 Wynkoop St, Denver, CO. For full details see the earlier post, but I'll summarize the earlier results here too. These were the test searches and the results I got with Local Live:

• King Soopers (a local supermarket chain): Google returned 4 non-existent results in the top 10; Microsoft and MapQuest both had no errors but some duplication, in the sense of having multiple addresses close by for the same store. M&M both correctly located the closest store but Google didn't (unless you count manually discarding results with incomplete addresses)
• Tattered Cover (a well known Denver bookstore with 3 locations): Google returned 4 non-existent results in a list of 8, while M&M both returned only the 3 correct store locations.
  
• Office Depot and Home Depot seemed to work fine with everyone, with no obvious errors.
  
• A search for grocery yielded 4 out of 10 incorrect results on Google and 10 reasonable looking results on MapQuest (though I didn't verify them all). With Microsoft the top result was incorrect, an incomplete address which just said "Denver, CO", similar to those which caused a lot of the errors with Google. And the second address on the list was interesting - it was Cowboy Lounge, which is a nightclub which used to be Market 41, which appeared on the Google list also, incorrectly categorized as a grocery store (you can understand how the mistake occurred given the original name). Interesting that Microsoft picked up the name change, which Google didn't, but still has the incorrect categorization. However, one good thing with Microsoft is that I was given the option to provide feedback that the result was incorrect, so I did that and asked this it should be removed from the grocery categories (and I provided feedback on the previous incorrect result too). The rest of the list appeared to be legitimate establishments, though the Rocky Mountain Chocolate Factory and Cookies by Design are a broad interpretation of "grocery" :) !! So Microsoft did better than Google but not as well as MapQuest on this one. I will be interested to see if and when my feedback on the incorrect categorization gets through the system, I will check it out every so often to find out!
  

So overall, both Microsoft and MapQuest appear to have much better quality in their points of interest data than Google does (admittedly based on a fairly small sample). As I said previously, as consumers start to use location based services more, data quality will be a really critical factor in end user satisfaction. Your system might have the coolest user interface in the world, but by the second or third time it has taken you to a non-existent location, I can pretty much guarantee that you will switch to something less sexy but with more reliable data. This is a scenario where false positives (returning a location of interest which does not really exist) are in general much worse than false negatives (not returning a real location of interest). If I'm desperate for a beer and you direct me to a bar which is two miles away but ignore one which is one mile away, at least I get my beer pretty promptly (though I would have been happier if I could have walked). But if you take me to several nonexistent locations first and I hunt around at each one for a while to try to find the bar which is supposed to be there, I will be thirsty and grumpy and resolving to find a new LBS to help me next time :).

A couple of other quick observations on the Microsoft Local Live implementation. One is that it lets you place a pin at your original location of interest, and also display pins in different colors showing multiple different query results at the same time - this feature is nicely done and not available in Google either online or mobile. The results include some ads, but these are clearly separate from the result listing. The results come back sorted in order of "relevance", which I think is probably in most cases a euphemism for "more or less by distance, but with scope to move sponsored results further up the list" - which is what Google appears to be doing with its mobile maps as I discussed previously. But with one click I can change this to sort by distance, and on both these lists it shows the distance of each result from my starting point. As a user I am quite happy with this approach - it gives the service provider (Microsoft) the chance to monetize their service with ads and preferred placement, which ultimately is necessary otherwise service providers won't be able to continue to provide their service, but it doesn't hamper my ability to easily get the specific information I want (as Google Maps Mobile does).

Steve tells me that the same local search capabilities are available in Microsoft Live Search for mobile ... but unfortunately there's not much chance of me testing that any time soon, as I am already suffering enough abuse over having both an iPhone and a BlackBerry 8800, so I don't think I can justify adding a Windows-based smartphone to the collection :) !!

Google Maps local search on the iPhone really has some serious flaws

Yesterday I posted my main review of Google Maps on the iPhone, as well as my general impressions of the iPhone. To say some positive things first, I really think that while I've commented on a few areas for improvement, I really think the iPhone is great - it's definitely a jump forward in terms of user interfaces, and I think introduces a lot of ideas in this regard which will become much more widely used. It's just a lot of fun to use. On Google Maps specifically, I talked about a lot of things I liked yesterday, and one other nice feature I have discovered is that the Bookmark page also has a "Recents" tab which saves all your recent searches, routes, etc, and will restore those with all the context - this is especially nice for retrieving routes. Oh, and the built in camera, which I had low expectations of, actually isn't too bad - here are a few sample snapshots in case you're interested.

However, I expressed some reservations about the way that local search was implemented yesterday, and having played around with this some more I really feel that there are some serious issues that need addressing here. There are two main issues: one is the way that the functionality has been implemented, and the other is the quality of the data for points of interest.

Let's take the functionality first. Frankly, Google seems to have lost track of the basic user scenario for local search. I suspect that probably a (conscious or unconscious) contributor to this was the desire to include establishments which paid for placement higher in the list of results, as I commented on yesterday, without it being too obvious that they are doing this. But anyway, the basic scenario for local search is that you identify a location, which might either be where you are now (the most common scenario), or perhaps an address where you plan on being in the future, or where a friend is located. You then search for something close to that place - this might fairly specific, like "tattered cover" (a bookstore I used in my example yesterday) or might be more generic, like "coffee". In either case, knowing how far away each of the search results are from the specified location is a really crucial piece of information in deciding whether that search result is of interest to you. Sorting the results by distance, and knowing that the closest (or the closest 3, or 5, or whatever) results have been shown to you are fundamental requirements. And then if you decide that one of the search results is of interest to you, by far the most common follow up action is that you want to figure out how to navigate from the point you first specified to the search result you have selected.

Most other local search implementations I have used meet these requirements pretty well, but local search on the iPhone fails pretty badly. Using the TeleNav software on my BlackBerry 8800, I choose "Find business" and the first thing it asks me for is the search point, and I have six options for specifying this: current location (which it knows from the GPS, which is of course its big advantage over the iPhone), recent addresses, favorites, key in an address, near airport, or contacts. You then specify the search term, and are immediately given a list of results sorted in order of distance from the search point, and showing how far away each result is. You click on a result and it shows more information including the address, and gives you the option to drive to it, save it, map it, or call it. The one feature which TeleNav does not provide which the iPhone does is the ability to display multiple results on a map at the same time - but it meets the main user scenario much better.

Google Maps online also addresses this user scenario well. You type in the address of the point you are interested in, and hit "Search maps". Then you choose "Find businesses" and you now have two text boxes, one containing the address of the search point, and one for your search term. You hit search and get an ordered list back, by distance, with the distance from the search point displayed for each result. If you click on a search result and ask for directions, the start point is already filled in with the address of the original search point.

But all this goes by the board in Google Maps on the iPhone. You will typically precede a search by centering the screen on a location, most often by typing an address (or retrieving a bookmark). But that point is not explicitly used by the search on the iPhone. Instead, you just get back any ten results which satisfy your search term and which are somewhere within the bounds of the current screen display - except that it's not just any ten, it seems that there is a ranking order which is presumably determined by how much establishments have paid for placement. There is no guarantee that the closest (or closest 3, or 5, etc) are returned. There is no easy way to find out how far away a given search result is from the point you were interested in (the only way I have found is to select a result, display its details, choose "directions to here", and then you have to fill in the start address, which you can do by selecting a bookmark or recent search address, and then calculate a route - and you have to repeat this for every item whose distance you want to know).

I tested a couple of scenarios where there were fewer than ten results in the original map display window, and in this case the map window stays the same and displays pins for the results which are in that area, but you still get a list of ten items (not sure if these are guaranteed to be the ten closest in this situation or not). A bad thing in this scenario is that if you select an item from the list which was not in the original map window, it pans you to that location, but now you have no idea how that relates geographically to your original search point (beyond re-entering the original point and calculating a route).

If there are no search results in the original map window, the map window zooms out so that it fits in all the results (at least in the cases I tried). However, there is no symbol on the map showing where your original point was, so you risk losing your orientation in this situation if it's an area you don't know well, unless you calculate a route from your original point again.

So functionality wise I really think someone lost the plot here. Yes, if you are zoomed in close (i.e. at the default scale which is displayed which you search for an address), the system will find you some results that are "sort of close" to the address you typed, and you get a nice pretty display with animated flying pins. But there appears to be no guarantee that the closest result will definitely be returned (though it probably will in most cases, especially if you are zoomed in close), there's no easy way of determining how far away any given result is (you have to calculate routes individually), there is no way at all of sorting results in order of distance, and in order to calculate a route to a search result you have to re-enter the original search point again (which you can admittedly do via a bookmark or recent result, but this still adds at least two or three clicks which shouldn't be necessary). One other thing I noticed while thinking about distances in this context is that there is no scale bar on the map, so you can't even estimate graphically. I know it's not a big screen, but all in car navigation systems I have used have some nice compact scale bar, so I think this should be added in there too.

So anyway, that's the functionality rant, sorry about that :) !! Now for the data rant!

These issues are exacerbated by the fact that Google's point of interest data really seems to have a lot of issues. I mentioned some examples in yesterday's post, but categories of error that I have found in more than half of my test cases so far include incomplete addresses (either no street number, or no street), establishments which have closed, and incorrect categorization (examples below). I tested Google Maps online and it had the same errors (which is at least consistent - you would hope that they use the same data). For comparison, I went to try MapQuest (online) and it did MUCH better. Here are just a few example searches I tried, centered on 1792 Wynkoop St, Denver, CO:

• King Soopers (a local supermarket chain): in the top 10 on Google, there were three entries with an incomplete address (just Denver, CO), all of which show as being closer than the closest real King Soopers, and one entry which says in Google Maps online that it is an "unverified listing" (at 1150 Stout St) - but on the iPhone this just appears in the list without any indication that it is unverified (this one does not appear in MapQuest and I am pretty sure that it is not a real location). So 4 out of 10 results are wrong (non-existent), therefore you would have a 40% chance of ending up somewhere with no King Soopers. On MapQuest, 10 legitimate addresses are returned, though in some cases they appear to have multiple addresses for the same store (e.g. entry points on two different streets) - but this is not a serious error as you would still find a King Soopers.
  
• Tattered Cover (a well known Denver book store with 3 locations, which I mentioned in my post yesterday). MapQuest returns just 3 results, all correct. Google Maps online returns 8 entries in its initial list, of which two have incomplete addresses (duplicates of entries which do have complete addresses, but they show up at entirely different locations on the map), one is a store which closed a year ago, one says it is unverified and is in Boulder (where there is no Tattered Cover), and one is a duplicate of a correct store. The same 8 entries appear on the iPhone, again with no indication that the Boulder one is unverified - so in this case you have a 50% chance of not showing up at a Tattered Cover.
• Searches for Office Depot and Home Depot were more successful, with no obvious errors - hooray :) !!
• Searching for grocery in Google Maps online returned Market 41, a nightclub which closed a year ago, and four entries with incomplete addresses (two King Soopers which we saw before, and two Safeways). The iPhone returned a different list in this case, but included Market 41, two entries with incomplete addresses (one the same, one different), and Lemon Sisters market, which was a small grocery store but closed several years ago. So a 40% chance of going to the wrong place in this case. The same search on MapQuest did not return Market 41 and had 10 complete addresses, all of which looked legitimate.
  

So anyway, I won't belabor this. Maybe I'm just having really bad luck here. And because I live close to the center of Denver, I am picking up more of these incomplete addresses than some others might. And if you are aware that there is this sort of junk in the Google point of interest database, you can manually ignore results that don't have complete addresses (if you check their details) - but many users will not do that and will trust the pin that they see on the map. Hopefully the overall quality of the Google point of interest data is not as bad as in my random tests, but nevertheless I think that they will see significant pressure to clean this up as people use it a lot more, which should be the case as the iPhone (and other mobile systems) make it much easier for people to use this data on the go.

So anyway, to finish on a not entirely negative note, let's reiterate a few good things. Overall, the iPhone rocks! Google Maps on the iPhone has a lot of great features - the quality of the map display is great, panning and zooming using the touch screen is really fast and intuitive, routing works well and the way that it saves the full state of complex operations including routing and searches in "Recents" is really nice. BUT, the local search really doesn't match up to the high standards that we all expect from Google. Yes, it will give you what you want in many cases, but is missing key functionality like displaying and sorting by distance, common workflows like navigating to the point of interest you want to go are much more complex than they should be, and in my tests I have encountered way too many data errors to be comfortable relying on it. The good news is that fixing the functionality really shouldn't be difficult technically, though I think it will need Google to accept that sponsored results should be displayed separately somehow (as they are with regular Google search). Fixing the data problems may be a bigger effort, but others have done it as the MapQuest comparison shows.

Fixing local search on Google Maps for the iPhone is now top of my wish list, but I also still think that auto-correction of typing and rotation of the map are high priorities, as I mentioned yesterday. That's enough iPhone reviewing for this weekend I think :) !!