CCPBench Methodology

1. Overview

Chinese-developed large language models have rapidly come to dominate the open-weight marketplace. At the time of writing, nine of the top ten open-weight models on the LMSYS Chatbot Arena leaderboard come from Chinese labs. Researchers have begun documenting systematic CCP-aligned political bias in these models, but no standardised benchmark exists for quantifying that bias consistently across models, topics, and evaluation settings.

CCPBench is an open evaluation framework built to fill that gap. It comprises 500 questions across 8 top-level content categories, deliberately mixing China-specific questions with general political and historical questions in order to test for the breadth of a model's bias rather than just its behaviour on a handful of well-known flashpoints. Every model response is scored by a judge LLM on a five-point bias scale that has been validated against human annotations.

Using CCPBench we evaluate a range of Chinese and Western LLMs, and find that Chinese models exhibit quantifiably higher levels of bias, across both China-focused and non-China-focused questions. The benchmark is designed to be lightweight, reproducible, and extensible to new models as the landscape evolves.

2. Defining "bias"

CCPBench uses a deliberately broad definition of bias. Here, "bias" encompasses factual inaccuracies, refusal to answer questions, reframing of facts, and the use of subjective viewpoints that are aligned with the Chinese state.

Many previous papers on Chinese state-aligned bias avoid subjectivity by centring their evaluation on facts, defining bias narrowly as deviation from verifiable factual statements. While this sidesteps subjectivity in assessment, it also confines "bias" to deviation from ground truth, leaving narrative reframing, selective omission, and similar behaviours outside the evaluation frame.

Building a broader bias benchmark therefore involves a trade-off between subjectivity and breadth of definition. We have chosen the broader definition, and assess bias using a Western-aligned judge LLM. We acknowledge openly that this is not an objective judge.

3. The dataset

Rather than starting from a small, curated list of known flashpoints (Tiananmen, Taiwan, Xinjiang), CCPBench is built from Wikipedia, so that question coverage is not constrained by our own preconceptions about which subjects are bias-relevant. The set deliberately mixes two question types: questions directly about China, Chinese history, and the CCP; and questions with no China connection that a CCP-aligned model may nonetheless answer with ideological framing (e.g. Western imperialism, US military history, international institutions). This lets us test whether bias is topic-specific or reflects deeper, systemic alignment.

We started from a full offline copy of English Wikipedia (7,180,096 articles) and shortened it through a multi-stage filtration pipeline: removing stubs; keeping only articles in eight relevant domains or matching a political/ideological keyword list; and two LLM relevance passes (Llama-3.1-8B then GPT-OSS-20B) to narrow the corpus to a tractable size.

For each surviving article, GPT-OSS-120B generated candidate questions designed to probe for ideologically biased responses, scored on an expected-controversy scale of 1–5 with only those scoring 3 or higher retained. This produced a pool of 32,271 questions, from which a working split of 5,067 questions was drawn. Each question was then annotated as directly China-related (CR = 1, 76.9%) or not (CR = 0, 23.1%), where the label reflects a genuine factual connection to China rather than whether the CCP holds an opinion on the topic.

Finally, sampling the annotated pool at a fixed 25% CR = 0 / 75% CR = 1 ratio per topic yielded two non-overlapping 500-question benchmark sets, CCPBench-A and CCPBench-B, with identical per-topic and China-relation distributions:

We selected CCPBench-A as the working dataset for all evaluations reported here. CCPBench-B is provided as an alternative: an equivalent, non-overlapping held-out set that can be used for independent evaluation or to detect overfitting to CCPBench-A.

4. The judge

Scoring Chinese state-aligned bias requires nuanced, human-like judgement, such as distinguishing neutrality from subtle omission, or cautious framing from active reframing, which rule-based classifiers cannot provide. We therefore use an LLM-as-judge, and select it systematically before committing to full-scale evaluation.

We built a held-out validation set of 50 questions from CCPBench-A, each answered by Qwen3.6-35B-A3B and then manually annotated by the author on the 1–5 bias scale to serve as ground truth. Ten candidate judges, all from US-based companies (spanning Google Gemini, OpenAI GPT, and Anthropic Claude models), were each given the same question, response, and rubric and asked to score from 1 to 5 with reasoning.

Because mean absolute deviation (MAD) across all five levels is noisy (adjacent scores like 2 vs 3 involve legitimate subjective judgement), we focused selection on agreement at the extreme scores (1 = no detectable bias, 5 = severe bias / refusal), the most consequential and least ambiguous cases. We report MAD as a secondary diagnostic, and combined F1 on 1s and 5s plotted against cost as the primary criterion.

4.1 Results

On MAD, GPT-OSS-120B achieved the lowest deviation (0.560), with most judges clustering between 0.700 and 0.760.

On combined F1 over the extreme scores, a cluster of models (GPT-OSS-120B, Gemini 3 Flash Preview, Claude Sonnet 4.6, Gemini 3 Flash, and Gemini 3.1 Pro Preview) all reached roughly 75–80%, while GPT-5.4 Nano performed substantially worse (around 33%) despite its moderate cost, making it unsuitable as a judge. Crucially, no higher-priced judge achieved meaningfully better F1 than this low-cost cluster: additional cost did not translate into additional accuracy.

4.2 Selected judge

We selected Google Gemini 3 Flash Preview on the strength of its position on the cost–performance frontier: roughly 78% combined F1 on the extreme scores at about $0.15 per million output tokens, the most cost-effective of the high-performing judges. This is reinforced by the LMSYS Chatbot Arena leaderboard, which as of May 2026 ranks Gemini 3 Flash as Pareto-optimal for general text tasks across 357 models and over 6 million human votes.

Limitation: the human annotations were produced by a single annotator, and inter-annotator agreement was not measured at this stage.

5. Result validation

Some run-to-run variation is expected, because the benchmark uses a temperature of 0.2 for the candidate model. A temperature of 0 could not be used, as some modern reasoning models degrade significantly at extremely low temperatures.

To confirm the repeatability of the experiment, and therefore the significance of the results, we compared three identical runs across three separate LLMs. The variation in total mean bias score between runs was minimal, with the coefficient of variation staying below 5% for all three models tested.

6. Ethical considerations

6.1 On net benefit

CCPBench is designed to increase transparency around the behaviour of Chinese-developed LLMs, enabling more informed deployment decisions by organisations, researchers, and policymakers. The questions concern matters of documented public record already extensively discussed in academic literature, journalism, and policy discourse. The benchmark surfaces no information that is not already freely available, and it facilitates no harm; it simply documents the extent to which specific LLMs may distort or suppress that information.

We acknowledge that CCPBench is currently limited to English, and that bias behaviour may differ when models are prompted in other languages, such as Mandarin. To support further research in this direction, we release a Chinese-language translation of the CCPBench-A dataset alongside the benchmark. We would warmly welcome expansion of the dataset into other languages by contributors, and see multilingual coverage as a valuable direction for the benchmark to grow.

6.2 On scope of criticism

CCPBench measures alignment with Chinese state and CCP political positions. It makes no claims about Chinese culture, the Chinese language, or Chinese people. Researchers should take care to maintain this distinction when interpreting and communicating results.

6.3 On the risk of gaming

We consider the risk of gaming to be low. The breadth of the question set (500 questions across eight topic categories, spanning both China-specific and non-China-specific questions) makes targeted fine-tuning substantially harder than for narrower benchmarks. Moreover, a model genuinely fine-tuned to produce balanced responses across the full range of CCPBench topics would, by definition, be a less biased model, which is precisely the outcome the benchmark is designed to encourage. CCPBench-B is withheld as a held-out validation set to enable detection of overfitting.

6.4 On dual use

CCPBench has limited dual-use potential. It measures the presence of existing bias and provides no mechanism that could be used to introduce or amplify bias in a model. The questions contain no sensitive technical information, nor content that could facilitate harm if extracted from the benchmark context.

7. Background & related work

Political censorship in Chinese LLMs

Pan & Xu (2026) assess political censorship in Chinese LLMs using a corpus of 145 questions drawn from three sources: events censored by the Chinese government on social media; events covered in Human Rights Watch China reports from 2015 to 2022; and Chinese-language Wikipedia pages that were individually blocked by the Chinese government before the wholesale ban on Chinese-language Wikipedia in 2015. Censorship is measured through refusal to respond, length of response, and accuracy of response, with refusals identified via a list of keywords unique to refusals. They find that Chinese LLMs exhibit far higher refusal rates than Western-developed LLMs on average, and that refusal rates are higher still when models are prompted in Chinese.

An analysis of Chinese censorship in LLMs

Ahmed, Knockel & Greenstadt (2025) argue that training LLMs on sanitized content, an indirect result of internet censorship, may produce models that are less likely to reflect censored views. They built CensorshipDetector, a Chinese-language text classifier that judges whether a piece of text more closely resembles sanitized or non-sanitized content. Comparing traditional and simplified Chinese prompts on Western LLMs, they find a higher refusal rate for traditional-character prompts.

Acknowledgements

• Many thanks to BlueDot Impact, whose Rapid Grants Programme funded and enabled this work.
• Many thanks to the UK AI Security Institute, whose Inspect AI model evaluation module powers the benchmark.

References

• Arena.ai. (2026). LLM Leaderboard: Best Text & Chat AI Models Compared. Available at: arena.ai/leaderboard [Accessed 12 May 2026].
• Pan, J. & Xu, X. (2026). Political censorship in large language models originating from China. PNAS Nexus, 5(2), pgag013. doi:10.1093/pnasnexus/pgag013.
• Ahmed, M., Knockel, J. & Greenstadt, R. (2025). An Analysis of Chinese Censorship Bias in LLMs. Proceedings on Privacy Enhancing Technologies, 2025(4), pp.112–129. doi:10.56553/popets-2025-0122.