2018 House Model Autopsy
Let’s start off with a simple statement about everyone who was predicting the 2018 elections with a quantitative model: We all got lucky. Well, not those who made a Senate model where statewide polling, especially in red states, seems to be off and partisanship didn’t cleave cleanly. But definitely if you were trying to predict the outcome of the House. My model, 538’s, Bitecofer’s, Elliot’s and several others all used different methods and pathways to predict the outcome and as a whole pretty much nailed it. More interestingly, we saw herding between all these models. Considering the different pillars each stood on, I think that makes it harder to declare any one quantitative sweeping theory the winner. For instance, Dr. Bitecofer’s model relied on predicting turnout for a suburban wave but my model is almost singularly focused on predicting ticket splitters, and we both had the same result of confident suburban flips months out. This has made me more confident that 2018 has a huge wealth of knowledge to exploit by looking at what went wrong with my model if the baseline is a good performance.
The rest of this piece is going to break down the aspects of my model that performed the worst in the last cycle and how I’m trying to make corrections for the next cycle. For the things that went well, I’m keeping them, including using the governor’s numbers within reason as a baseline (worked in NY, WV, OK, and others). Additionally, campaign effects such as spending or concerted field efforts really did appear to move the numbers and I don’t believe you can fix the results on the fundamentals alone. This is good because otherwise this whole exercise would be worthless. It is my belief that prediction for the sake of prediction is useless, especially if you’re not paid per click. The point of these models is to guide the actual stakeholders with the means to change a race. Models that ignore things like spending have no real purpose beyond being the topic of conversation in Beltway bars. Decisions campaigns, SuperPACs and donors both big and small make can move the margins and flip seats. Models that remain too fixed and don’t account for this are doomed to fall behind. Or they cause panics on Election Night early on by not accounting for later post polling spends in places like FL-15 or matching IE spending in KY-6 like when 538 believed Rs would hold the House during first reporting and markets moved.
On that note, let me start with a heaping scoopful of humble pie and say, on one of the core tenets I believed in in 2018: I was wrong. No, I’m too vain for that. I was MOSTLY wrong. Like…85%, take it or leave it. But mostly an apology to Dave Wasserman, Kyle Kondik, and other qualitative forecasters.
I spent a lot of time arguing candidate quality didn’t matter and to a point, I still sort of believe that. The way some candidates get labeled as good or bad can be media effect, personal projection, or ignore past performance (In KY-6, good candidate Jim Gray lost to Amy McGrath, who then lost to underacknowledged actual good candidate Andy Barr who usually has one of the highest favorable split tickets). This happens a bunch and really most candidates that overperformed the baseline still did so, even as they lost a district but maybe not so much as they could have.
One of the things that made me discount candidate quality so early was that it was such a nebulous term. It’s hard to pin down what a good or bad candidate was and how to put that in a quantitative model. It wasn’t until running an autopsy after the election that I would find out that there were a bunch of quantifiable candidate features that really did have an impact. And even if they are just proxies, the series of candidate quality factors I’m adding in still depend on who’s name is on the ballot. These new additions for how the candidate themselves change the nature of the race are what make of the bulk of this piece and is a big reversal on the margins from last time around. I was able to eliminate a ton of model error in back testing all elections since 2012 with these new methods and I’m confident it strengthens the 2020 model by a lot. Dismissing these too early or not bothering to even consider some factors such as ideology, age, race, and others hindered my ability to make the most accurate model.
The silver lining is that’s what this whole thing is for. Finding the gap between what should happen as a baseline and what the results are is where the real interesting data is. This is what you can present to the parties, PACs, and engaged voters that can alter the outcome and I hope people take these factors to heart in 2020.
In the meantime, these last several months have been a lot of testing, tweaking, and playing around with the model. I’m excited to be able to go public with an interactive version again in 2020. Because the model now takes candidate quality into account, the actual release probably won’t be until most primaries are done by the early summer and by then hopefully the spending picture will come more into play. I hope all of this is interesting to someone and these initial findings in this write-up are useful. I hope this also helps those rare few (and it’s always a noble gamble to stake it all out on a model you worked on and everyone has something to offer and is always interesting) doing more rigorous analysis or other forecasting modelers looking to potentially add more accuracy to their own models.
So that you don’t have to hunt it all down to use it, all the 2018 House candidate demographic info can be found on this Google sheet: https://docs.google.com/spreadsheets/d/1_-ikIPdNw2-6zFpcEyCbrqYiGnk9zJ-9SkYASmGqjNA/edit?usp=sharing
And 2020 candidate information, which I’ll be updating after each congressional primary occurs, can be found here: https://docs.google.com/spreadsheets/d/1meoPWyqBHIuL7ng_X1SycK8P57oAIpe--BZTTFV2ezs/edit?usp=sharing
Thank you all so much for being along for the ride in 2018, I love doing this fun project and I wouldn’t be able to keep doing this without all of you. So, without further ado…
How Did It Do?
Overall, the model got close when it came to the overall picture. It predicted 42 seats flipping and depending on the count either 39 or 40 that actually flipped. It also caught the large suburban shift only in places like NJ-7, KS-3 or most famously OK-5 before polling or other attention was focused on those. Overall, I had a stretch goal of a median error of under 4% and less than 15 incorrect seats. Unfortunately, I missed those targets though I still believe it performed well, though work is needed. For comparison, I put a bunch of total measures in the table below.
The first is the 538 Fundamental model, which is only the set of fixed factors that 538 looked at, including past margins, incumbency, and others. The next is the 538 Classic model that included polling. Then I used my two 2018 models, not including any of the new factors. The Election Night has my predictions for generic ballot and governor while the Correct 2018 Toplines used the real results as well as late spending that wasn’t disclosed until after the election.
The measures are:
Seats Wrong — Number of Seats where the model believed the other party would win
Overall Median Error — The median absolute error across all competitive seats (minus Louisiana with any jungle primary general election)
Competitive Median Error — Median absolute error for all seats decided by less than 15%
Closer — I put my election night model up against the 538 fundamental model and saw which one had less absolute error for each seat. This is the share of seats that each one of ours was the more accurate
Competitive Seats Closer — Share of competitive seats with less absolute error comparison
The boxplot under shows all the errors plotted out for each model.
My model posted a median error around 5% and a little higher for competitive races. While 538 had more polling for those and so they were more accurate, I ended up getting less wrong in the fundamental model. Part of this is the money impact which would drive things in the right direction, but with the magnitude being off and that is one of the many changes below. In terms of less error, my model was closer in 42.3% of seats overall but only 39.6% of competitive seats. It’s a bit disappointing as I was hoping to get under 4%, but encouraging for the first try and there were only a few seats (like OR-2) that weren’t explained in the post-mortem.
The good news is a lot of this error is self-correcting. The main sources of error come from relying too much on 2014 or 2012 and the model weights by recency and cycle, so before 2014 in a midterm year was given a lot of weight. If the generic ballot polling is as close as it was in 2018, I’d expect the House models to do well with 2 cycles of this new realignment and the battlefield has been settled.
What I’m Not Changing:
Get The Hot Seat’s stories in your inbox
Join Medium for free to get updates from this writer.
There were a few things I settled on for the live version of 2018 that were in a trial run that appeared to come out as they were originally set and don’t need modification. The first is that I’m not setting different bonuses for campaign and PAC spending. This makes sense because campaigns need to hire staff and other expenses even though PACs have a higher rate and the dollar spending across the races didn’t matter where the money came from.
The second thing was one I was very skeptical of adding based on past races but 538 included where instead of spending they looked at % of donors from within a district for enthusiasm. This led to some Dems underperforming the 538 model in lesser known races and I could find no effect for in district or in state money raised, even when trying to break it out into large and small donations, though that had much wider confidence intervals. In 2020, I won’t be adding that feature in though it will be interesting to see what 538 does.
What’s New:
Ideology Bonus
A year ago, I wrote about how it appeared that extreme candidates suffered a split ticket penalty while moderates saw a bonus, but the real measure was how much closer to the center the candidate was against their opponent. Linking this piece, I’m not going to repeat all the details now, but this will be a factor in future races and did a great job of finding misses in tossups. I will also continue to use DIME and I thank Adam Bonica for responding to my emails and pointing me in the right direction on updated numbers.
So, then you can imagine my surprise that when I added the ideology effect to my 2018 back test, the median error margin increased a bit! At the same time, the number of incorrect seats went down. Digging into this, the initial ideology effect helped in tossup seats but increased error in the safer seats. Very quickly, the connection made sense.
I was basing the median voter at 0. So, in a tossup seat where the ideology was broken evenly, the effect would be close. However, parties run a lot of more moderate candidates in safe seats held by the other party that better reflects the district, so a progressive incumbent does not face the same penalty in a Safe Democratic seat as opposed to a tossup race. Because of this, I’ll be adding in the ideology effect but shifting the median point to reflect the partisan lean of the district and demographic makeup. I know that predicting safe seats is not as high of a priority, but this model goes for the lowest median error across the board, and this is a minor enough shift to be useful.
Spending Effects
Before I say what, I got wrong, I would like to note that the candidate spending is also what the model probably nailed the most relative to other quantitative modeling that doesn’t use it. There are several races where the model predicted the races more accurately and the House by giving an electoral boost at the rate of about ~1% for every 400k more spent than the other side. This worked well in predicting the CA sweep in a lot of races that traditional forecasters may have had closer such as CA-25, but also money dumps that came out after polling made it more accurate than models such as 538’s that relied too much on snapshot polling, like in FL-15. However, there are certainly missteps made that introduced error into the model and as one of the few actual resources stakeholders can use to influence the race, it is imperative to cover all the bases.
The first is diminishing returns, pictured in the graph below. It only happened in a few races but there appears to be a linear correlation until you hit a $6 million advantage and eventually tapers off entirely after $8 million and adjusting for that brings a few races closer to the actual outcome. The next error was that I just put in the total topline spending, but a lot of candidacies see a large amount of waste or mismanagement and this also lumped primary and general spending together. Separating out primary spending from spending in the general got closer results in California but also reduced the error in NY-11, a race where the Republican spent a lost to survive a primary challenge and then didn’t spend much after. This still needs to be back tested which is difficult to sort through but based on the early results, I’m expecting that to be added this cycle. I will also be piloting only adding campaign money spent of operating expenditures, which is still an imperfect measure but eliminates spending such as committee transfers that were confusing the model’s predicted margin.
The last effect that marginally reduced error so it will get added in but may be on the chopping block if it’s not accurate this cycle is tweaking the spending for media market complexity. This may also be the only factor that sees PAC and campaign spending undergo a different effect. Complexity here refers to a measure I’ve made that considers how diverse the DMA landscape is. First, I look at how split up the seat is between how many different markets, and then the size of those markets, both measures that can be found here compiled by David Jarman of Daily Kos Elections. This factor will then get a small reduction on the spending effect for more complex districts that fractured or are larger and more expensive markets, and a boost to the impact of money in districts in smaller, cheaper markets, or where ads can reach more people at one. This won’t be given a lot of weight this cycle but is an interesting case that I’ll keep watching and had some success with helping in 2018.
Age Bonus
Age wasn’t even on my mind when I made the model for the midterm, and this was tied for maybe one of the worst oversights of the model that I didn’t even bother to check. The post-mortem showed that looking at the difference between the age of the candidates was the #1 factor that reduced the model’s error and it also reduced error in the other back-tested cycles of 2012–2016. This is especially interesting because I had looked previously at length of incumbency and found no effect and mentally brushed off age as a factor then. Age is also a key factor in how much a candidate fundraises but even when accounting for spending differentials, age has a massive effect and younger candidates see more of a boost. Maybe it’s because they canvass more or are more appealing (one recent study showed a natural election boost for more attractive candidates) but being uncomfortable with rating candidates’ looks and not knowing the exact source, age will be a great stand-in.
Age collecting was and is not easy. It came from a variety of sources (special help to Sandra Fish @Fishnette for starting the work and doing a lot of initial collection and then reaching out and sharing it with me), about half of which I tracked down in website bios or local newspaper questionnaires. The cumulative hours for data collection on age are probably why this is an understudied demographic for matchups, but I hope to have the 2020 candidate ages out publicly for every race again this cycle before the election. In the model I put the age differential on the following regression line to spit out an effect that my model gives. An early look has shown that gender of the candidates does not appear to matter or at least has enough going on under the hood to even out, but this could be updated with a larger sample size in the 2022 version.
Woman Bonus
This was mostly based on seeing a small boost of a couple of points for women candidates in a few previous cycles, though it wasn’t significant at the time. I still added it in hoping that this would pay off, but instead it only introduced more error into the model. Why? Because women did tend to do better, but it was a combination of a few other more predictive factors I talked about in earlier sections: money and ideology. The gender-based ideology factor is detailed above, and this cycle Democratic women raised more money than men and any electoral boost for women disappeared when adjusting for spending. I’ll be eliminating this factor in 2020.
Race Bonus
After seeing this tweet by Vance Ulrich, I decided to look more into whether there was a bump for ethnic candidates in the areas that are the same ethnicity. 2018 testing has shown that it is the case, but it doesn’t show up when two of the same races are facing each other. Because of this, I’m going to be adding a bonus for minority candidate equal to about 3% multiplied by whatever share of the district is the same race.
Further analysis could look at if there is a negative effect for white candidates in ethnic areas or ethnic candidates in white areas but initially, I could find nothing. For racial estimates, I use the Citizen Voting Eligible Population calculated by Daily Kos but may switch this to registered voters by layering registration rate on top of the population. I’m skeptical to do this because Catalist analyses in places like TX-32 seem to show a turnout boost in co-ethnic areas and the original measure will hopefully account for that variability. Race is also a tricky subject, and in 2018 I relied on a survey candidate themselves filled out to determine race. What I noticed is that it’s not always clear, as Hispanic candidates usually put a wide array of options and Republicans would default to White. I’ll put out these demographics publicly (as did Daily Kos here) but I may use some personal discretion, though never outside the bounds of what would be acceptable or even helpful to boosting the model.
Homefield Advantage Bonus
One of the model’s biggest misses was PA-1, where Republican incumbent won a close race against Democratic self-funder Scott Wallace. Adjusting for ideology reduced a large chunk of the error, but it still showed a Democratic win as the topline. So, I dove into the seat. Seeing density was the largest polarizer, I plotted the graph below by town showing shift? between the 2016 and 2018 elections. You can see that density played a huge part but there are several medium or large sized towns that stand out as Republican congressional overperformance. Below these towns are also colored red on the map. These are all clustered in one area, specifically the area of ____, where Rep. Brian Fitzpatrick hails from. Democratic candidate Scott Wallace was a carpetbagger and didn’t have any similar areas to help him.
Probably due to name recognition and familiarity, congressional candidates in back testing analysis did well where they hail from or overperform in any counties or state legislative districts they represented. In 2020, I will be adding a large boost multiplied by the area that a candidate previously represented or have a longer-term hometown with some boost to the surrounding area. I haven’t yet decided whether to add a carpetbagger penalty as that appears in effective ads, but a one-sided boost may be all that’s needed. This is a new factor so I do anticipate some error to stem from it and other issues such as statewide office holders running for Congress (possibly in PA-10 and IN-05 in 2020 for example) may be up to personal discretion. In the meantime, though, this should provide a good fix on the topline for what happens under the hood and captures another candidate characteristic to make the model more comprehensive.
Demographic Changes
Before I was just throwing in every measure from the census and down-weighting to nil any non-predictive ones. This time I’ll be eliminating those entirely but adding in CityLab’s density measures to hopefully combine with other useful factors like race and create a more stable split ticket prediction based on demos. In 2018 it did well, providing an anchor and will play a similar role in 2020, though now adjusted to weight harder on presidential years and recency instead of midterm years. One of the sources of error mostly in safe seats was not giving the Republicans more weight in minority-heavy districts and overestimated Democratic support. The right adjustments will be made based on this for a presidential year.
Polls-Plus Model
Ugh. Ugggggggggggggggghhhhhhhhhhhhhhhhhh. I really didn’t want to do this, and I still believe layering polls on top of a quantitative model is a cheap way out and doesn’t help you look at candidate or campaign effects. I’m also skeptical that polling, especially district-level, is even that useful when it’s only public polling included. I will probably refuse to acknowledge it the entire cycle. However, it can be useful for looking at individual races with scandals that are near impossible to model away the impact of, and can catch macro trends, such as the suburbs maybe shifting a bit more than usual. Because of this I have built out a page that layers district polls on top of the model’s predicted results and changes the outcome. It does this by weighting on recency especially in the seat but also all other seats that fall under the same CityLab density classification because this is where herding and seat look-alikes are the strongest. It will then spit out a different prediction than the model only and comparing the two will be a fun exercise.
Some of these changes introduced some more error in individual seats but overall the median error was down to a little over 3% which gives me a lot of hope for 2020. I didn’t hit exactly the targets I wanted to in 2018 and there was a lot to change and still a lot more to do but I’m encouraged by finding concrete patterns in the errors. I’m hoping the 2020 model performs better and I’ll be releasing that publicly too before this cycle. Because it is more candidate-centric on more factors, this will likely not be released until this summer when more spending and nominees are chosen to get a good picture of how the House looks. Thank you again to everyone and we’ll see which changes works and if there will be any surprise results this time around!
