Evaluating Chatbot Responses to Sensitive Political Questions: A Science Protocol

Digital opportunity (v4): Protocol for Systematic Comparison of Chatbot Responses to Contested Facts Target users: Researchers, educators, journalists, and civil society groups concerned with information accuracy and narrative framing in AI chatbot outputs, especially in regions with contested histories. How it would work: Develop a standardized checklist and workflow for submitting identical sensitive political or historical questions to multiple chatbots. Collect, anonymize, and categorize the responses by framing, factual accuracy, and narrative style. Output a comparative table and summary report highlighting differences and potential biases. First version: A paper-based or spreadsheet checklist with 10-20 pre-selected sensitive questions, a template for recording chatbot responses, and a summary table. Operated by a small team (2-3 people) over one week, using publicly accessible chatbots. No personal data collected; all responses anonymized. Alternatives to investigate: Existing fact-checking workflows, academic protocols for media analysis, and any published guidelines for evaluating AI-generated content. Compare with manual search engine queries and traditional reference sources. Difference to test: Whether a standardized protocol enables more transparent, reproducible, and actionable comparison of chatbot outputs than ad hoc or anecdotal approaches. Test if the protocol reveals systematic framing differences. Source evidence of a capability or need: They also interpret, prioritise and frame information in ways shaped by their training data and the technological environments in which they operate. This was one of the central Possible uses and limits: Growing reliance on chatbots as reference tools creates a need for systematic evaluation of their outputs, especially in contested or sensitive contexts. Actual adoption of such protocols is not reported. Potential users and optional payers: Unverified analysis, not established facts; confirm before acting. No established paying customers; this is a non-commercial research and education tool. Community benefit is the primary goal. Competitors and alternatives to check: Unverified analysis, not established facts; confirm before acting. Manual fact-checking, academic media analysis protocols, and any published AI content evaluation guidelines. No specific competing product identified in the excerpt. Optional legal business model: Proposed hypothesis; revenue may never occur. Monetization is not required for this project. No commercial hypothesis established; community or academic use only. Skills and time to test: Critical reading, basic spreadsheet or checklist skills, and familiarity with political/historical context. Estimated time: 1 week for a small pilot. Skills not independently verified. Geographic availability and limits: Unverified analysis, not established facts; confirm before acting. Source names the Western Balkans and international chatbots; actual coverage and access depend on local internet and platform availability. Availability is unknown until checked. Legal and platform constraints to verify: Unverified analysis, not established facts; confirm before acting. Must verify local privacy, intellectual property, and platform terms for chatbot use and publication of outputs. No legal advice given; compliance required. Current prices and platform terms: Unverified analysis, not established facts; confirm before acting. Actual prices and terms for chatbot access are unknown/not checked. Some chatbots may have usage limits or require payment; local verification needed. Age, payments and eligibility checks: MLAIEL requires age 16 or older; external eligibility and payments are unverified. MLAIEL minimum user age is 16. External chatbot age limits, KYC, payment, and country eligibility are unverified and must be checked before use. Smallest low-cost validation test: Run the protocol with 3 chatbots and 5 questions using a spreadsheet, record outputs, and review categorization agreement between two team members. No funding or special access required. Human responsibilities: Selecting questions, submitting them to chatbots, recording and categorizing responses, and compiling the summary report. Requires critical reading and basic spreadsheet skills. AI contribution and limits: No autonomous AI work required; chatbots are used as black-box sources. Optional: use AI for text clustering or sentiment analysis, but manual review is primary for transparency. Data and access needed: Access to public chatbot interfaces; no login or personal data required. All chatbot outputs anonymized. Consent not needed for public chatbot use, but check each platform's terms for research use. Source-supported setting and capacity: They also interpret, prioritise and frame information in ways shaped by their training data and the technological environments in which they operate. This was one of the central questions guiding our research. What our research revealed In 2025, I led a research team at Action for Democratic Society/Hibrid.info that examined how three widely used AI chatbots responded to sensitive political and historical questions: ChatGPT (launched in the United States), DeepSeek Chat (launched in China), and Alice (launched in Russia). Ordered steps and owners: 1. Select 10-20 sensitive questions. 2. Submit each to 3 chatbots. 3. Record and anonymize responses. 4. Categorize framing and accuracy. 5. Compile comparative table and summary report. 6. Review and revise protocol as needed. Minimum resources and dependencies: Publicly accessible chatbot interfaces, spreadsheet or paper checklist, basic internet access. No proprietary tools required. Confirm chatbot terms of use for research. Costs and affordability unknowns: Actual costs unknown; requires local internet access and time. No reported funding or price for protocol use. Local quotes needed for any paid chatbot access. How a newcomer can help: 2-3 team members: question selection, data collection, categorization, and reporting. Supervised beginner can assist with data entry or anonymization under guidance. Pilot and success criteria: Proposed pilot target: 10 questions, 3 chatbots, 2 reviewers, 1 week. Baseline: manual comparison without protocol. Success: at least 80% agreement between reviewers on categorization of framing and accuracy. Measured by inter-rater reliability and completeness of summary table. Risks and safeguards: Chatbot access may be restricted or outputs may change over time. Legal risks if platform terms prohibit publication of outputs. Potential for controversy if findings are politicized. Limited generalizability beyond selected questions and chatbots. When to stop or change direction: If chatbots block access, outputs are too inconsistent, or legal constraints prevent publication, the project should pause or adapt. If reviewers cannot reach agreement, protocol may need revision.