This model rests on three assumptions:

(1) Objective Voter Position: You can capture most (not all!) of what distinguishes the political values of voters by asking them some questions and using their answers to position them on two axes (Economic Left-Right, Social Liberal-Conservative) relative to the answers of other voters. Their Objective Voter Position is reasonably consistent over time (ask them the same/similar questions a year later, expect to get pretty much the same answers).

(2) Perceived Party Position Consensus: Voters’ perceptions of how Economic Left/Right or Socially Liberal/Conservative a party is are incoherent — but that’s because for most people “the centre” is the patch of earth right under their feet. Think of voter perception as a judgement relative to their Objective Voter Position then voter perceptions do (largely) line up and you can talk about a single Perceived Party Position shared by most of the electorate.

(3) Voters like Parties who they perceive as “Close” to them: This model originally came from a spatial competition model in economics — you’re on a beach, you want an ice cream, which ice-cream stand do you go to? The closest.

From this setup party coalitions “emerge” without having to do anything else the same way the position of ice-cream stands implicitly carves up a beachfront.

That’s it — these three assumptions are pretty well supported (at the “spherical cow” level of detail) and they take you straight from the data you have about voters (e.g. British Election Study) to thinking about Party Value Coalitions as contiguous pieces of territory in Voter Value Space.

This is a very simple model — do we actually see something like this pattern if we look at the last time the two major UK parties were most dominant?

Looking at this stylised chart, pretty well — using only the positions of voters in Voter Value Space you can predict whether someone voted for the Conservatives or Labour with ~80% accuracy.

But what happens if you have more than one party? Consider the UK 2019 Euro Elections with 5 national parties in contention:

The motivation for this article was, in part, follow-up to this (rather long) response to Simon-Wren Lewis’ blog post which contains links to Simon-Wren Lewis’ article where he lays out a similar chart (but with greater ambition since he tries to go beyond ‘Perceived Party Position’ to actual position) and Paula Surridge’s plot of the average OVP for party-euref electorates.

• Can we think about politics from Blair onwards in one chart?

Yes. Paula Surridge has made a lot of very clear charts doing this and I’ve made a whole bunch of charts which are less clear but expose more of the underlying complexity of summarising distributions with just their means.

However, even I think putting two different sets of perceptions into one chart is a bit ambitious (and it’s not like I’m not “ambitious” about how much you can convey with one chart).

Getting perception data on two axes is tough — the British Election Study elicits “leftRight” perceptions from respondents but voters tend to project the two axes down onto that one axis in a complex and tough to reverse manner — as they do for their perceptions of their own position. (There are, I believe, expert surveys that do elicit 2d perception positioning — but I haven’t looked into what relation that has to public opinion).

But what you can do is take BES data, use the standardised battery of questions to position each respondent on the 2d political values (“Political Compass”) chart and then position parties/politicians on the basis of *where the people are who vote for them/like them*.

It’s not quite the same thing as “exactly where is this party/person on these axes I’ve just briefly explained to you?” because you are — after all — invoking the concept on the implicit basis that voting/liking is in some part a function of distance on this chart and that (cf Protagoras) that positioning is relative to the UK voting age population.

• If people are uniformly distributed around this map, then the centre is the place to be in a two party, FPTP system. Parties do not go there because their ideology/principles, maintained mainly by their members, stops them.

I’m a big fan of Robert Saunders’ arguments about the significant (and, arguably, negative) impact of (increasing) party membership veto power — but I think it’s an error to frame members/parliamentary member preference as the *only* thing stopping parties orbiting each other at a radius of .0001 from the centre.

The neatness of the Hotelling-Black-Downs result that politics — and parties — should always converge on the median voter is that it assumes that everyone *must* vote. That is not generally the case (notably, I think the result *does* apply a lot better to Australia where voting is compulsory).

I’ve looked at non-voting in the UK and it tallies *really surprisingly well* to (i) distance from parties (ii) being in the centre/left-auth end (people who are genuinely not motivated by politics/people who might be but are just living in the country for 2 years/people who find liberal parliamentary democracy a turn off).

(I’ll try my hand at making a little interactive d3js 2d Hotelling-Downs simulation tomorrow but when non-voting is an option all voters exert a gravitational force on parties *modified by how responsive their voting x turnout probability is to party positioning* — which means parties won’t converge *that close* to the centre even without 3rd party competition)

• Regrettably, social conservatism has more appeal when times are bad, at least in part because the (incorrect) real wage/public service argument gains traction.

I’m not sure this really stands up — the widespread assumption that the Radical Right (political representation of socially conservative disquiet) do well in rough economic times is something that everyone *who isn’t a scholar of the radical right* seems to take on faith.

There’s definitely evidence that Left-Social Conservatives frame immigration-angst in terms of competition over resources — but “mode of framing” does not imply “objective causal mechanism”.

(There’s a similar issue with assuming net migration rates relates to attitudes about immigration — the last time the UK has an immigration panic *it had negative levels* of net immigration)

This may seem (/be) pedantic, but it leads into my second point that I don’t think “Austerity” necessitates right-wing newspapers to sell to social conservatives — you can go back 20 years and social conservatives have *always* been extremely skeptical of anything that smacks of “Welfareism”.

There *is* something dodgy about Austerity — it’s the fact that it merges and elides the distinction between cutting public spending as a % of GDP (Economic Left-Right concern) and cutting spending on specific minority groups (Social Liberal-Conservative concern).

There is a conspiracy (“”) there — but it’s a conspiracy between the two major political parties to bind the two things tightly together and try to make sure the political debate happens only on that axis because it unites both parties’ coalitions.

Socially conservative newspapers focus on Austerity-as-preventing-people-abusing-the-system, socially liberal newspapers focus on Austerity-as-needless-cruelty-to-minorities, economic-right and economic-left focused on public spending as a % of GDP (notably that 1-d language doesn’t stop 2-d positioning — it just stops people talking clearly about it!).

• Ed Miliband’s defeat in 2015 was narrow but hard for Labour to take. Most of the political commentators (as they always do) said Labour should move to the right

I’m unclear if you mean “Economic Right” or Socially Conservative“Right” — it’s confusing because you’ve elsewhere consistently used Social Conservative/Liberal but it’s weird not to note that all of the big post 2015 reports were very clear that Labour’s problem was with social conservatives. Once you filter out the stopped clocks, I think political commentators broadly agreed with the obvious proposition.

• Brexit shows more than anything how we have to think about perceptions. What made Brexit a narrow winner when its true position suggested an easy loss? In short a brilliant if totally dishonest campaign that painted it as something it is not. Project Fear, with the help of the media, completely nullified the right wing economic dimension of Brexit, and turned it into a plus by talking about more money for the NHS. Staying in the EU was successfully painted as ultra liberal (letting the whole of Turkey come to the UK). As with austerity, the perceived position of parties and policies is what matters when it comes to winning elections and a referendum.

I think it’s wrong to paint *the media* as flattening out the economic value impact on Brexit — I think that comes down to two factors:

(i) Cameron delivering the votes of a chunk of loyalist Soft-Euroskeptic Social-Centre Economic-Right Conservatives (which would go some way to explaining why Con-Remainers are so sticky)
(ii) the very low volume of any Economic Left campaign (“protection of rights” is the major Economic-Left campaign sub-component)

• However crucially he also lost because the UK is not a two party system. The Liberal Democrats, the SNP and the Greens are all perceived to be in liberal/left space, and together with Labour they won more votes than Johnson and the Brexit party put together.

Two issues here —

(i) LD, SNP — and even Greens (really good brand, expect a lot of young liberal Conservatives to have used this as a protest vote)— all have broader reach in some direction than Labour (SNP do well with Left-Authoritarians, LD compete with Conservatives right up to the Right-Liberal corner). If they suddenly disappeared over night, a lot of those voters would not have voted Labour (in fact, quite a few would have voted Conservative).

(ii) I used to think the role of LD/Greens/Labour was obligate competitive but I’ve come to realise that they are actually commensal organisms that enhance each other’s performance when there’s a popular Labour leader and their positions are close-ish. In those circumstances, voters whose priority is ousting the Tories will freely move their vote to whichever party is best placed to do it in their constituency. E.g. 1997, Lib dem vote falls 1.2% … and they go up 25 seats while Labour goes up 132.

The effect is that Labour retains a broader coalition than it could ever hope to assemble or maintain on it’s own. The closest thing the UK had to that in 2019 was the Brexit Party unilaterally standing down in Tory seats (impossible for a real political party).

• Of course it leaves out a lot (the popularity of leaders, which is related to their charisma, the effectiveness of campaigns etc)

Actually, if you position parties/politicians based on *who votes for/likes them* then you capture “charisma”:

(I’m going to stop here and hit publish because I’ve rambled on for too long — but I may come back and edit this down a bit/add more links/think about cutting it up into smaller posts — all suggestions welcomed!)

Visual metaphor: think of each party as holding up an opened “umbrella” — a coloured surface whose height at every point reflects the average abstract appeal (“would you ever vote for?”) for voters at that point of the Political Compass.

Previously, the way I’ve visualised this sort of thing is to have one graph per umbrella side by side or overlaid them on top of each with the colours merging. Due to inferior human cerebral hardware, the results can be hard to read.

So what I’ve done here is to put all the umbrellas on one chart, but now we’re looking down from above and we only see the uppermost surface at any given point.

Upside, pretty chart with nice clean lines. Downside, we’re throwing away a lot of information. For instance, the “umbrellas” for UKIP and TIG were calculated, they’re just completely obscured by higher “umbrellas”.

A more subtle issue is that when you look at the Lib Dem orange swatch of colour you’re not seeing “the areas where people would like to vote Lib Dem” — you’re seeing “the areas where people would like to vote Lib Dem *that are uncontested by any other party*”.

(Light is Very Likely to vote for party, Dark is Very Unlikely)

The Lib Dem vote midpoint — on this chart scale — is actually Socially Liberal:Economic Centre/Economic Centre-Left. But they are contested on the Economic Left by the Greens and Labour — while on the Socially Liberal:Economic Right the Conservatives have actively ceded ground.

And the same is true for the Brexit Party. What we’re seeing is not “where their votes are” but “which territory do they have to themselves”. And that’s partly because we’re not seeing the variation in how enthusiastic voters are about the *best* choice. We can see that in this sensually-terrifying half-Cronenberg half-Doctor Who chart below:

Is the British population evenly distributed across this space? I’ve put considerable time into tinkering with the Political Compass data to make the distribution as flat, even and uniform as possible without significantly deforming it.

The answer is “mostly” — the top 50% of the chart has 50% of the British Election Study sampled people (total N~90,000+) — and so on for any vertical/horizontal slice.

But where the axes interact — which they do less than people reflexively assume and more than is convenient for political scientists — there is structure that can’t be easily massaged out.

And that’s why I’ve added subtle contour lines. Note that this is pretty fine structure — e.g. the bottom right quadrant contains 24.2% of the sample (rather than 25%). So the Lib Dem dominance — viewed in terms of ares — as a fraction of the UK voting population is inflated, but probably only by ~5%.

• On Welfare (specifically cash transfers from the state to individuals) — state cash transfers are just obviously progressive in the sense that more will go to the poor than the rich (yes, there are some universal benefits like state pensions, but all that does is dilute the average “progressiveness” of total state cash transfers)
• On the UK’s relative Economic Position —The UK, while obviously not as far to the economic right as the US, was well to the right of most other European states (let alone states outside of Europe)

So if someone had asked me to place the UK on a scale of what share of Public Social Benefits spending goes to Poorest Fifth/Richest Fifth (relative to other rich OECD club countries) I’d have thought:

• this is a scale that will max out somewhere significantly shy of 1:1
• the UK will end up somewhere on the right hand of the graph

Those intuitions turn out to be super-duper wrong. Even within the OECD — the richest/highest-welfare 20% of the world’s states — the mean position is parity. The UK sits way off on the left in a weird little annex of Anglosphere and Scandinavian countries.

We’re only looking at 36 out of 193 states — I’m going to stick my neck out and predict that this is highly unrepresentative chart because most non-OECD states will sit to the right end of that graph (if anyone has the data for a full breakdown, please poke me!).

I originally posted this chart while people were still focused on Eurozone austerity — which made the relative position of the “PIGS” (Portugal, Italy, Greece and Spain) particularly poignant — especially with Germany sitting almost dead on the OECD mean.

If you’re wondering why — there’s a detailed answer from the OECD with extra breakdown states — and a more accessible write up in the FT here.

The short answer is that all public spending across the world is, to a first approximation, pensions.

And most state spending on pensions is a % of your salary (in extreme cases, specifically your *final salary*!).

In case you’re wondering whether 2011 was a really weird year (say, because of the financial crash), here’s the same chart for 2014:

Your weak inferior brain doesn’t allow you to superimpose and differentiate between two 72 element bar charts? Fine — here’s the changes between 2011 and 2014 — some fairly dramatic (in some cases, suspiciously dramatic!) shifts, but they don’t ultimately change the story.

Are you a very suspicious/obsessive person who is wondering what’s happening with the other fifths?

We learn only two things here:

• line plots are just better than bar charts in every way (the bar is 95% wasted space — I just want to know where the top of the bar is!)
• even before you hit the really disturbing countries where the top fifth are cashing in (you know, disturbingly capitalist countries like *checks graph* France???), when you get past the Anglosphere/Scandinavian countries the Poorish overtake the Poorest

Initial feedback suggested — quite reasonably — that maybe *shares* of public spending is misleading because the % of GDP states spend on Public Social Benefits (cash transfers) varies *a lot* for different reasons, chief among them the fact that for some states occupational pensions are private and for some public.

You can go further than that and assume — as I have in the past — that states buy support/tolerance for redistribution/welfare by giving some beak grease. E.g. Spending *as a % of GDP* on the highest fifth is higher in some states as the price for increasing spending *as a % of GDP* on the lowest fifth.

So I replotted those graph as a % of GDP and the difference is … not much. Just looking at the lowest fifth, France ends up looking more like Germany, but still well behind left-wing utopias … like the UK. Otherwise, this serves to separate out non-European states (Korea, Chile, Mexico, Turkey).

Here’s the notebook that created those charts

Source data directory is here (I had some problems getting some of the data from the website, emailed the OECD and they were quick, nice and helpful)

There is a fairly solid consensus (62/38) that climate change is happening and that it is due to human activity.

If you broaden the category to “the environment”, focus on whether or not enough is being done or cue that there is a trade-off then the results get progressively more even.

Probably some people are already looking at those graphs and wondering whether it is even reasonable to assume that they are capturing the same thing as the climate change skepticism — they really are different questions that touch on quite different debates. So, let’s look at what answers to the climate change question correlate/anti-correlate with across the British Election Study dataset (you do not need to read/understand every variable in the chart below — but if you want a partial explainer, it is covered in my earlier blog post):

• Those two questions — enviroProtection and enviroGrowth — do correlate (+/-) pretty strongly with Climate Change Skepticism
• Surprising no one, climate change skepticism anti-correlates with liking (&thinking they are competent/good at handling the ‘most important issue’) the Green Party/their leaders
• What is surprising is that every other high pos/neg correlation variable relates directly (“which way did you vote in the euref?”) or indirectly to Brexit (“how do you feel about Trump/overseas aid/immigration?” all three correlate heavily with euref vote)
• In case you are wondering how unlikely this is, my processing of the British Election Study dataset creates ~6,800 variables, of which ~5% relate directly/indirectly to Brexit. So, quite unlikely.

So what does the breakdown of Climate Change skepticism against euRef vote look like?

It is a bit less stark if you look at the distribution for each of Leave and Remain rather than the disparity within each climateChange category (a bare majority of Leavers believe climate change is occurring and is caused by humans).

But maybe this is some sort of fluke pattern that only applies to that one question? No.

Okay, so that is kind of weird — but maybe this is all just a confound with Green politics — e.g. Green Party is opposed to Brexit, supportive of immigration and presumably not keen on Trump. Maybe if we control for how people feel about the Green Party the pattern would disappear?

Here is the result of regressing all the variables in the British Election Study dataset (bar the other two variables mentioned above) against Climate Change Skepticism (Believes in Human Created Climate Change ← → Doesn’t Believe in Climate Change).

This is a very, very lazy approach (as above, if you want more details on how that graph was made/how to read it, partial explainer here)— I have not narrowed it down to a single wave so you can see the same question asked in different waves — but you can see how variables that *obviously are* confounds for each other (whether you like the Greens in 2017 as as well as 2016) suppress each other’s impact.

It is notable how much this reads like a list of lazy stereotypes (I stress again, this is an entirely automated process running on 6,800 variables) — Trump support, homophobia, rejection of foreign aid, rejection of experts, support for austerity, support for traditional gender roles, the belief that the UK will not have to concede anything to get full “Single Market” access. And, of course, the Daily Mail readership!

Clearly, culture war stuff (notable that there are party markers in there — but no left-right economic split). The UK is not the US — you can see core Conservative vote bias towards Climate Change Skepticism — and a reflection among some Conservative elites (Matt Ridley/Nigel Lawson) … but I do not think you could argue that this is something created at the party level and disseminated to voters through partisanship.

Beyond noting that this is probably related — like attitudes to austerity, immigration, Trump, gender roles, homosexuality — to psychological dispositions *at least as much* as explicit ideology it is difficult to know what to do with this.

We know only two things for sure about those sorts of sentiments:

• They can shift really quickly
• Nobody knows precisely what it is that changes them

Early comments(well, comment) on this article raised the question of whether all these things were simply “Right” issues. As per previous articles, it’s always ambiguous what people mean by “Right” — here’s the breakdown on the Political Compass (warning this chart is an experimental — as in “not very good yet” — approach to getting several overlapping distributions all on one graph by showing only the upper contours (peaks), the x’s mark the means of each category):

Addendum — at Leo Barasi’s request, I’m adding regression breakdowns for enviroGrowth and enviroProtection (but I’m dropping variables relating to the Green Party):

• Worth stressing that these are quick runs (will schedule more in-depth versions to run overnight) and since I’ve excluded many of the best predictors these models explain only a minority of the variance … but gosh, it’s really stark
• UPDATE: Runs are now slow versions (R²/variance explained ~25–30% — not bad given we’ve stripped out all the Green Party variables)
• To be clear — and to preempt in some small part the scorn of statisticians — the purpose of these regression models is exploratory analysis. There’s a not-wholly-unfair tendency among statisticians to view machine learning as statistics-without-the-professionalism — that’s true in the sense that machine learning models let people imagine they can skip the model generation/analysis steps. But, so long as you view this as an in-exhaustive breakdown of “variables that are implicated”, you’re alright.

• expectAccess/negotiationSpecifics_None = the UK will get full Single Market access post Brexit(/without conceding anything)
• redistCon = where respondents would place the Conservative Party’s preference on redistribution
• lr3 = Ordinary working people do not get their fair share of the nation’s wealth

• nonelecParticipation: 2=Signed a petition, 7=Bought/refused to buy product for ethical reasons
• houseBuild … is missing from the British Election Study questionnaire pdf — correlates strongly positively with councilHouse … which also isn’t in the pdf! Have to ask the BES team politely.

• Gender split is interesting — also explains the pattern of the Authoritarianism scale predicting prioritising economic growth, but one element (al4 = censorship) predicting prioritising the environment because there’s a strong gender split on al4 (there’s an argument for dropping it from the al_scale)
• There’s an (albeit small) income effect for prioritising Economic Growth (profile_gross_household)
• Newpapers readership “None” effect hard to read — do *most* newspapers give you a reason to prioritise Economic Growth or is it something about *not* reading a newspaper? (Note previous blog showed that non-newspaper-readers don’t deviate from general population in values)

Python Notebook (Fairly legible)

Climate Change — and its weird connection to Brexit was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

People responded on twitter by asking (paraphrasing heavily for convenience):

• we’re only seeing the newspaper they read most often (if any) — people often read more than one newspaper — how do second preferences break down?
• the major divide seems to be between Broadsheets and Tabloids — what’s driving that?

A while back, Yougov published a breakdown of the crossover between newspaper websites (%of Daily Visitors to <newspaper X website> who are also Weekly Visitors of <newspaper Y website>). Newspaper website readers aren’t a perfect proxy for newspaper readers … but the data has already been gathered and compiled so it’s not particularly time costly to check out.

I didn’t like Yougov’s chart — I’m not interested in how popular newspaper websites are on average, I want to know which other site’s readers they are *relatively more/less popular with*. So I divided out the average popularity and then turned the results into a network diagram[1]:

The width of the arrows reflects *relative preference* (as mentioned above, controlling for some sites being just more popular with *everyone*). Unsurprisingly, Daily “Independent” website readers tend to be *at least* Weekly “i” readers. The link between the “i” and the “FT” is more surprising.

What does this tell us — other than “don’t use yellow on egg-shell background”? The clustering into Tabloid and Broadsheet “news testicles” is consistent with the Political Compass value distribution overlaps above, even to the point that the Telegraph has the strongest links to the “Tabloid Newsticle”.

But what causes that divide? Ultimate causality is hard so I’m going to recast that as “what predicts whether the newspaper someone reads most often is a Broadsheet or a Tabloid?”.

To answer that, I tossed the entirety of the British Election Study online panel dataset — a dataset with literally hundreds of political, demographic, attitudinal and personal variables — at a machine learning classification algorithm[2] to see which questionnaire variables floated to the top as a predictor of whether people’s preferred newspaper[3] was a Broadsheet ( →) or a Tabloid (←), as categorised by the above diagram. Basically, dynamite fishing.

Here’s the output (via an algorithm that makes machine learning results that bit less opaque[4]):

Choose Your Own Blog Adventure: Do you want to know how to read the opaque graph above (read below)— or do you just want to skip to the breakdown (scroll down)?

Do Want To Read The Graph: Top is most important, bottom is least important[5]. Each blob in each row is a person, coloured by the value of that variable, positioned by the *impact* of that variable on the final prediction in the fitted model (which turned out to be 80–90% accurate —pretty good for real world data).

The variables are (mostly) guessable at first glance, but here is the link to the pdf showing which variables relate to which questions — I’ve automatically modified the variable names for (slightly) enhanced legibility[6]. I’ll take the top variable as an example:

Still at school/20+ is bright red/red (you participated in Higher Education). Bluer than that means you left 19–15. HE participation => more likely to prefer Broadsheet/no HE participation => more likely to prefer Tabloid.[7]

Don’t Want To Read The Graph: Here’s roughly clustered/ordered breakdown of the above variables

• Higher Education →
• Age ←
• Social Conservatism (aka Authoritarianism) — particularly death penalty/tougher sentences ←
• Class self-identification WC ← → MC [8]
• Atheism →
• Political knowledge/attention variables →
• Immigration/black&female equality →
• Live in London/high household income →
• 2005GE Labour voter ←
• Energy Price Cap (text of question not in pdf, but reasonable to assume *for*) ←
• Heard stuff about euRef during 2016 campaign from radio →

Not a lot of surprises here — age, education, social conservatism, immigration/equality sentiment— are all known to correlate heavily (although above we see their separate impacts, ~effectively controlling for each other).

There’s nothing here that automatically tells you which way causation is going (well, I assume picking up a copy of the Guardian doesn’t teleport you to London or reading The Sun cause you travel back in time to vote Labour in 2005).

The “2005GE Labour vote” link to Tabloid readership is probably indirectly relevant for contemporary debates about when Labour started to lose hold of its Socially Conservative vote (more on that in the next blog) — but note that the 2005GE is the *earliest* one in the data (e.g. significant split point may well have been earlier).

Next Up: UK Party Support by Political Compass Position — Where it Does and Does Not Match

Recommendation: If you liked this, you’d probably enjoy Chris Hanretty’s blog *even more*

Code (&Data):

• Yougov Data/Network Diagram (pretty human readable notebook)
• Machine Learning Dataset Dredging Code (a horrible mess of mid-refactored uncommented code — and you’d have to run other notebooks in order to prep the BES data)

[1] I may also have been looking for an excuse to experiment with network diagramming software — in this instance, Gephi

[2] Xgboost classification

[3] Restricted to people who had a preferred paper and whose preferred paper was in that Newsticles diagram — people whose preferred newspaper was regional/None/Other newspapers were dropped from consideration

[4] Scott Lundberg’s SHAP python module

[5] We’re only looking at the top 30 because the further you go down the more you’re probably just looking at statistical noise. This is only really the exploratory phase of a serious analysis

[6] During processing I’ve automatically added text to the end of variables:

• ordinal variables “blah” becomes “blah__bleh” where “bleh” is the top category e.g. “lrUKIPW2__Right” means that a high value means that the respondent thinks UKIP is a right-wing party
• categorical variables “blah” becomes “blah_bleh” which means the variable had 3+ non-ordered categories each of which became separate binary variables e.g. “subjClassW2_W4W7W9_Yes, working class” means that respondents were asked whether they thought of themselves as belonging to any particular class and “Yes, working class” was one of the options, a high value means they chose this option, low value means they didn’t
• variables that were already numbers — like Age/age — were just left as they are (the convention on age going *up* is widely understood)
• WXWY_WZ refers to the waves in which these variables were sampled (i.e. you can probably ignore it)
• The use of underscores before the end of the variable is from the original variables e.g. “profile_past_vote_2005_Labour Party” came from a categorical variable called ““profile_past_vote_2005”

[7] “Can’t remember” I have classified as a “don’t know” response (I have a big list of “weasel answers” that automatically replaces all Don’t Know/Don’t Know-alike answers with the code for “no response”) so it appears as gray (if I didn’t how do you deal with questions where all answers but the DK option are clearly ordered?).

[8] Worth stressing that this is self-identification — if your self-identification contradicts where the National Statistics Socio-economic Classification would place you they won’t come round to your house and beat you with sticks until you correct your ways (of course, once the UK leaves the EU, there will be many new opportunities)

What’s up with the UK Broadsheet/Tabloid split? was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Probably it would involve some sort of monstrously time-consuming text analysis project. But you know what’s a lot easier/faster/freer? Looking at a large sample of the UK population which already contains standardised measures of their political values and splitting it up by their favourite newspaper:

What is this graph? It’s a contour map where the ‘height’ represents the concentration of British (voting-age) people — deeper blue, more people; lighter blue, less people. The red ‘x’ represents the mean position of all the people in that subplot. I’ve added 8 “compass point” guidelines to help the eye. The (N=X ) subtitle beneath each newspaper shows the raw sample size — lower the sample size, less you should read into the distribution.

These axes are the Political Compass. Everyone knows — or thinks they know — what Economic Left/Right mean. But what about Social Liberal/Conservative (aka Libertarian/Authoritarian)? It’s complicated — common sense associations with those terms are neither totally wrong nor totally right[1]. Here are the “standard” political science questions for placing people on these scales[2] (as with all the data here, all courtesy of the British Election Study):

The standard approach to turning this questionnaire data into scales for Economic Left-Right/Social Liberalism-Conservatism … is just to add up the individual scores[3]. That works fine for most political science needs but the resulting distributions are neither nicely Gaussian (bell curves) nor are they well spread which makes visualisation — particularly comparing different distributions — a chore.

What I’ve done[4] is to divide the distribution equally along each axis — e.g. the right-most 1/200th of the chart contains the most Economic Right 1/200th of the sample — *every* vertical and horizontal slice of the chart contains the corresponding 1/200th slice of all participants in the British Election Study[5]. If the two axes were completely independent — and these axes were originally created by political scientists to reflect two independent political factors — the result of charting all participants in one plot would then be beautifully, boringly monochrome because each position on the chart would contain exactly the same number of people:

Clearly that was a technical failure (albeit a big step forward from dense misshapen blob). The problem is that the axes are mostly but not wholly independent. That reflects a mixture of something about the questions[6] and something about the actual distribution of values. The Economic Left *really does* just polarise more along the Social dimension than the Economic Right.

That’s a real-world problem for the people on the Economic Left. But — more importantly — it’s a problem for me making visualisations so I’m going to keep looking into whether I can synthesise better behaved axes by adding/dropping questions from the mix and/or means of directly minimally deforming that distribution (automated panel beating) into something a bit more even[7].

Until then, I’m afraid you *do* have to view all these charts while trying to remember that the baseline distribution has these three clusters — e.g. The Sun readership mean position is bang in the middle of the Economic axis — the Ec. Left-Social Conservative blob is just pronounced because people in that quadrant cluster together in the top left corner. The Mirror *is* significantly to the Economic Left of The Sun … but only because less Economic Right-Social Conservatives/more Economic Left-Social Liberals read it.

Now you know all that, here’s that chart again — as an animated slide show broken down for specific “waves” (dates that the polling was carried out). As far as I can tell, no really meaningful trends — but a really cheap, easy way to get a sense of how stable this method is (kinda)/what to look for:

So, about that opening-question/flimsy-bait-and-switch for a methodology dominated blog article?

You can use this approach to define “Right Wing Press” as Economic Right (by readership). But that puts The Sun/Star as neither definitively RWP (or not-RWP).

Or you can use it to define “Right Wing Press” as Socially Conservative (by readership) — but that means The Mirror becomes RWP, the Telegraph fall into ‘neither’ like The Sun/Star above and The Times/Financial Times conclusively become not-RWP.

I propose a new, objective categorisation:

• Left-bloid: Daily Mirror/Record
• Mid-bloid: The Sun/Star
• Right-bloid: The Express/Daily Mail
• Left-sheet: The Guardian/Independent
• Right-sheet: The Times/Financial Times
• Right-broadbloid: The Telegraph

Clearly, the naming scheme is unquestionably perfect, but what about the methodology? Don’t people pick newspapers for reasons other than their personal politics (e.g. TV pages)? What about people who stick with a newspaper when it shifts its positions? Yeah, well … let’s see you do better!

Next Up: What’s up with the Broadsheet/Tabloid split?

Recommendation: If you liked this, you’d probably enjoy Paula Surridge’s blog *even more*

Code: Pan-Dataset Values notebook (warning: this notebook is currently just a vast uncommented mishmash of hacks, dead-ends and random approaches — will try to refactor something cleaner — also I haven’t integrated the R code that does the multiple imputation yet)

Data: pan_dataset_allr_values.csv (includes both uniform and Gaussian-ish distributed versions and the id column from for all 14 waves

[1] There’s a great deal to be said for using academic labels that are obviously labels rather than descriptions (or descriptions that are so overtly out of date — which side of the French King would you sit? — that they’ve become labels).

[2] There are variants, but they don’t vary “a lot” and they “tend” to give the same answers.

[3] “Strongly Disagree” ->0 … “Strongly Agree” -> 4, DKs get replaced by the mean for whole sample, al_scale=al1+al2+al3+al4+al5, lr_scale=20-(lr1+lr2+lr3+lr4+lr5)

[4] Data processing: replace DKs with nans, run Hmisc (R package) multiple imputation on all BES al/lr variables across all waves (one at a time), then run sparsePCA (python module Sklearn) to spit out two orthogonal-ish components made from blending all those variables together. Then I box-cox (python Scipy stats) it to get something Gaussian/use pandas qcut (python module) to transform into a discrete uniform distribution. Visualisation is Seaborn (python module built on Matlplotlib) using FacetGrid and kde plots.

[5] Health Warning: I didn’t apply the British Election Study’s demographic weighting partly because “profile_newspaper” is a variable across all the waves and I know they’re going to bring out some code to generate a proper cross-wave weighting. But mostly because it’s a hassle to build those charts with it due to the python module I’m using. It’s doable, but it’s the last item on the to-do list and, from past experience, I know it doesn’t solve problems of lumpy distributions/make a huge visual difference.

[6] lr1 — redistribution — correlates quite heavily with Education/Social Liberalism — that’s why most Political Compass style analyses give the impression that there *are no* Economic Right-Social Liberals because their support for redistribution auto-converts them to Economic Centre-Social Liberals. As HE provision has expanded, it looks like this has become more or a problem for this sort of analysis — so there’s a weird-but-reasonable case for dropping “support for redistribution” from the definition of Economic Left!

[7] Please contact me if you know of a nice, formal way to do this — for some reason, I just keep failing to find the right google keywords — ideally with a functional python module!

What is the Right-Wing Press? was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

I browse a number of different forums where Brexit is discussed and the field for the most irritating argument is, as you might imagine, pretty competitive.

But this argument is the paragon of efficient obnoxiousness.

It’s not merely vice-signalling a la “You’re worried your insulin supplies? Well — your insulin, your problem!”.

It’s worse than that because at least the vice-signaller straight-up acknowledges that you (*albeit you alone*) have a legitimate concern.

Instead, it has a sort of cringing can’t-quite-meet-your-eye please-drop-the-subject quality to it that is particularly grating.

And it’s capped by an irritatingly fake matey chuckle as if they feel that they’ve effectively wished you good luck when all they’ve done is wished away the burden of empathy.

What’s really happened is that I (or someone else) has said something that makes them uncomfortable — e.g. I’m worried that Brexit means I won’t be able to live and work where the people I love are — and they’ve responded by trying to change the subject with all the usual be-a-good-fellow-can’t-you-see-I’m-uncomfortable-ha-ha signalling.

Without fail, it makes me want to scream “if you’re *sure*, how about you *insure* me against the risk?” and whip out some the legal documentation for them to sign over some slice of their savings/housing property in the event that their “surety” turns out to be in error — lets put a £-value on that surety!

But surely this, like most human failings, is found on all sides of the debate? No, not really — at least, nowhere near equally. Remainers are, if anything, more willing than Leavers to engage with the content of Leaver worries (not always admirably, but generally avidly). Where there’s confusion, it’s more often because there are *too many* responses to the content (e.g. EU Army — do you want to argue that it’s unlikely any time soon or that it’s not at all clear why it would be a bad thing).

But maybe this is just because there’s no other response people can make that allows them to stick to their beliefs? No - some Leavers do have the sense to realise that the correct response is “I hope that doesn’t happen. I realise there’s a real risk (and, in any case, the burden of uncertainty) — ultimately, I think it’s worth it — but I’m not going to diminish the possible/actual costs”. Obviously, it’s emotionally tiring. But, you know what — you’re not legally obliged to respond to posts on the internet/appear on Question Time if you’re past the point where you can engage.

This post is an attempt to put down some preliminary observations in a semi-coherent lump rather than spraying them randomly over twitter threads. I say preliminary observations because none of this fully digested and the graphs are all exploratory constructs (rather than the things you build *after* you’ve decided what the narrative is you are trying to convey).

Is there such a thing as “UK Immigration Sentiment”? It’s perfectly conceivable that the variation in immigration sentiment variables could break into two dimensions (same way variation in political sentiments does) or more.

Turns out, that doesn’t happen — there is just one significant(!) axis of “UK Immigration Sentiment”. Below is the output from a specific version of Principal Components Analysis (Factor Analysis) — it’s the laziest/dumbest form of data analysis there is (bar staring at raw correlations — see later)[1].

If a person thinks immigration is *bad for the economy*, they probably also think *it’s bad for culture* and *places a burden on the welfare state* and *think Brexit negotiations should prioritise controlling immigration over preserving the Single Market*.

For the rest of the article, I’ll stick with individual variables (because doing this on a per-wave basis would be a hassle — problem for Part II) — but just bear in mind that the immigration-sentiment specific variables do appear to be mostly all sampling the same thing — hence a somewhat cavalier attitude in which ones I do/don’t include in any graph.

Has it changed over time? Yes, quite significantly. You can see significant shifts before/after UKIP’s 2014 vote (around their EU election victory; later W3->W4) and during/after the 2016 EU referendum (later W8->W9). You can also see what *might* be a falling back in the last wave.

Can we really just take these shifts in the mean at face value? Being paranoid, I assumed not and built a huge series of extremely over-engineered Sankey diagrams to visualise the flow of people who were in every wave for a specific variable, coloured by their initial answer.

Take home point — yes, you more or less can take the means at face value — the shifts in the means don’t rely on people of just one opinion shifting — a similarish (eyeball precision!) chunk of people in every group shifted whenever there was a shift.[2]

Okay — but *why* did those people change their opinions after the 2016 EU referendum result? Good question. I thought I’d keep things simple by looking at *just the people with the most immigration-negative position during the 2016 EU referendum campaign* and create a variable called ChangedMind (=1 if their answer changed after the 2016 EU referendum result, =0 if it stayed the same).

I then looked at every other variable in the British Election Study dataset[3] and then looked at the top (20 — not 30 as table states!) each most negatively and positively correlating variables. I did this for each of the immigration variables with decent coverage for wave 8 and 9/10 (1000+ in each category of ChangedMind/didn’t ChangeMind).

The upshot was … well, more interesting for what it *didn’t* show. One of the first proposed explanations of the thawing of anti-immigration sentiment is that people changed their mind *because they believed immigration would be more controlled/levels would be lower*. Actually, immigration-most-negative people were *less likely* to change their mind the *more completely* they believed post-Brexit immigration would be controlled.

The overall pattern is that the positive correlates are almost entirely “being more positive about immigration”, negative correlates are almost entirely “being more authoritarian/anti-immigration/pro-Brexit”.

When I say “authoritarian” I don’t just mean immigration/Brexit related variables — or standard authoritarian questions like stance on death penalty — I mean questions like whether you think “Attempts to give equal opportunities to ethnic minorities” have “Gone much too far”.

Conclusion: At this coarse level of “analysis”, what distinguishes people who changed their mind from those who didn’t … is that they were just a bit less generally socially conservative/just a bit less committed to their anti-immigration sentiments.

That doesn’t answer the question of *why* they shifted — but it does underline that it’s hard to separate possible causal factors from the Hard/Softcore Anti-Immigrant position (hardcore position: the level of immigration is getting a lot higher, but Brexit will give us complete control, softcore position: level of immigration is getting a bit higher/staying the same, but Brexit will give us a lot of/some control)

Notes on dataset/how to read the variables:

The British Election Study dataset I’m looking at is high quality, freely available, online panels ~30,000 people polled per wave with reasonable levels of retention between waves. Each of the thirteen waves (W1…W13) was carried out at a specific time over the period Feb 2014 — June 2017 (not evenly spread — they focus on … elections — and not every question is asked in each wave).

The variables I refer to relate to questions in this questionnaire. Where variables are ordered (“How do you feel about immigration — would you like it: lower/same/higher”) I’ve added the topmost (non “Don’t Know”) answer category after the variable with two underscores (“feelImmigration__higher”). Where the variables are not ordered (“What’s your favour animal? Cat/Dog/Pigeon”) I just create 0/1 variables (“favouriteAnimal_Cat, favouriteAnimal_Dog, favouriteAnimal_Pigeon”). That makes analysis easier and in *most* cases makes variables human readable without having to look up the full question wording (but you can in that questionnaire links above).

Footnotes:

[1] There’s a lot of interesting structure — but it’s all second-order party-political stuff, next three components are basically BlameLabour Yes/No, BlameConservatives Yes/No, UKIP Yay/Boo

[2] Ipsos Mori saw the same thing in their own dataset, but find people who are pro-immigration more stable — not clear if that’s a confound with nowhere to go from “most favourable” when there’s a shift in favour of immigration, but I would hazard they accounted for that: https://www.ipsos.com/ipsos-mori/en-uk/shifting-ground-attitudes-towards-immigration-and-brexit

[3] Some winnowing on coverage, dropping/mean-replacing DKs, linear-ish variables turned into numbers, categorical variables dummied). I left directly related variables in there as a sanity check (/reasonable baseline /laziness).