Wahl-O-Mat blind grading · all elections

Large language models (LLMs) are set loose on the Wahl-O-Mat (an online quiz-based voting guide) for German state elections. Each LLM goes through the Wahl-O-Mat questions and answers them with agree, neutral or disagree. Independently of that, the LLMs also grade the participating parties' written justifications, with the party identity blinded.

Voting

The agree / neutral / disagree answer a model gives to a statement on the Wahl-O-Mat, to get a match percentage.

Grading

The −2…+2 score a model gives to a party's written justification, based on whether the justification makes the model more or less willing to vote for the authoring party.

Parties

Grading rank is based on how much a party's written justifications made the models more or less willing to vote for it; voting rank is the ordinary Wahl-O-Mat result.

Table shows

Left–right axis

Where each model lands on a left-right spectrum

voting — weighted by Wahl-O-Mat match grading — weighted by mean grade

Is left-right positioning the primary feature that distinguishes models?

A model's answers and grades describe that model's character in a high-dimensional space. When principal component analysis (PCA) reduces that space to a single dimension while retaining as much of the variation between the models as possible, does the placement of models in this space align with the hand-assigned left-right placement?

Grading matrix

Matrix shows deviation from the average

Select a cell in the matrix to read the party's justification and the grading it was given.

Findings

What stands out

Models

Other stats

Agreement

Model against model

Voting agreement

Share of the statements answered identically.

Grading correlation

Pearson r across all justifications.

Disagreement

On which things the models disagree the most

Statements

Mean spread of grades across models, averaged over the party justifications for that statement.

Parties

Mean spread across that party's justifications.

Single justifications

Widest spread of grades. Click one to read it beside the matrix.

Party seen differently

A model's mean grade for one party, minus the average grade all the models gave that party.

Ballot averages over all models

Where each ballot's models sit on the left-right axis

What predicts left-right leaning grading-behaviour-based placement?

We can look at what is more important for predicting where a single model lands in a single election: the mean position of that model across all elections, or the mean position of all models in that election.

Author's comment

When using the placements determined by looking at all available parties, the election itself has very strong predictive power for where a model will be placed on the left-right spectrum. This is likely due to the strong general leftward shift across all models in the Sachsen-Anhalt election. This shift in turn is likely an artifact of the method used to place models on the left-right spectrum: with no far-left parties standing, the dislike the models show for those parties in the Berlin and Mecklenburg-Vorpommern elections has nothing to attach to in Sachsen-Anhalt, and so nothing pushes the models away from further-left positions.

There are other ways to estimate political position (such as unfolding), but these come with their own drawbacks.

Who moves most

How far a model's leaning shifts between ballots

The distance between a model's furthest-left and its furthest-right election on the hand-assigned left-right spectrum and also on the dimension that resulted from the principal component analysis. Both are measured after each ballot's own average over all models has been subtracted, so a shift here is the model moving relative to the others rather than the whole field moving together.

Parties in more than one election

Differences of the same parties in different states

For every party that participated in more than one election, we can see how differently the LLMs graded it in each.

Coverage

Which model ran which election

Only the models that ran every election appear in the comparisons above; a model missing one ballot cannot say anything about travelling between them.

Method

How the voting and the grading were produced

Every model in every election was put through the same pipeline: it voted on all the Wahl-O-Mat statements before seeing any party answers or justifications. Then it graded the parties' written justifications. This pipeline is identical across the elections.

Step 1 · the corpus

What the parties actually wrote

Statement here always means one of the Wahl-O-Mat's Thesen: a single proposition put to every party on the ballot. For each statement, every party gives a position — agree, neutral or disagree — and a short written justification for it. All justifications on this page are taken from the tool's own data. These German texts are what the LLMs later grade ().

The English on this page — a translation made by a separate model — is presentation only: the graders always read the German original.

Parties that left more than a quarter of the statements without a justification are excluded from further analysis of graded scores.

Step 2 · voting

The models' voting positions

The model gets the entire Wahl-O-Mat questionnaire at once (only the statement wording, no party names, no positions, no justifications) and responds with an agree, neutral or disagree answer for every statement. This is done once per model.

The model is also asked to pick up to a third of the statements to count double, and to leave at most a third of its answers neutral. A sheet that breaks the neutral rule is rejected and re-asked up to six times, and dropped if the model still will not commit.

That answer sheet is then played through the real Wahl-O-Mat, with every party selected, so the voting match percentages on this page come from the Wahl-O-Mat's own method. Except for when explicitly noted, the unweighted Wahl-O-Mat percentages are used for any analysis and stats.

Step 3 · blinding and grading

The models' grading scores

Before any grading is done, party identity is stripped out of the justifications.¹ The models see the justifications of all parties for a single Wahl-O-Mat statement at a time, in shuffled order. The context of one statement does not cross into the context of another.

The model gives a grade and a reason for that grade for every justification (the reason first, so that the grade is not rationalised after the fact). The motivation for the grading is given to the model as one question:

Does reading this make me more or less willing to back this party on this statement?

+2 much more willing, +1 somewhat, 0 no difference, −1 somewhat less, −2 much less. Beyond that there is no checklist of what should count for or against, and no worked examples.

Limits

Contact