AI & Intelligent Systems Evaluation Portfolio.
Evaluating AI-Generated Responses: Accuracy, Relevance, Completeness and Reliability
Artificial Intelligence is increasingly being used to generate information, answer questions, support research and assist decision-making. However, an AI-generated response is not automatically accurate simply because it sounds convincing. The ability to evaluate AI-generated information critically is therefore an important skill in the development and improvement of reliable AI systems.
In this practical project, I examine how two AI systems respond to the same question and evaluate their outputs using a structured assessment framework.
Project Objective: The objective is to assess the quality of AI-generated responses based on five criteria:
- Accuracy
- Relevance
- Completeness
- Clarity
- Instruction following
Methodology: Two AI systems were given an identical question under the same conditions. Their responses were reviewed against a predefined evaluation rubric. Each criterion received a score from 1 to 5. The responses were then be compared to identify strengths, weaknesses and opportunities for improvement.
1: Poor performance; major weaknesses.
3: Satisfactory performance; some weaknesses.
5: Excellent performance; meets the criterion very well.
Evaluation Question: Online Learning vs Traditional Classroom Learning
The following prompt was submitted to both ChatGPT and Google Gemini:
“Explain the differences between online learning and traditional classroom learning”.
1. Define online learning and traditional classroom learning.
2. Explain at least four advantages of each learning method.
3. Discuss the disadvantages of both methods.
4. Compare their flexibility, cost, student interaction and learning experience.
5. Explain which type of learner may benefit most from each method.
6. Provide a balanced conclusion.
“Present your answer in a clear, structured and easy to understand format. Avoid unsupported claims and exaggerated statements”.
AI systems evaluated: ChatGPT and Google Gemini.
Assessment approach: Both responses were assessed using the same five evaluation criteria and a scoring scale of 1–5.
Results and Comparative Analysis
Both ChatGPT and Google Gemini produced structured responses to the same evaluation question. Each system addressed the requested definitions, advantages, disadvantages, comparison factors, learner profiles and conclusion.
The evaluation produced the following results:
Google Gemini achieved a slightly higher initial score, primarily due to its concise presentation and clarity. ChatGPT also performed strongly, providing more detailed explanations and examples. However, these results represent an initial assessment of the generated responses and would not be interpreted as a final judgement of factual reliability because a well-structured and convincing AI-generated response is not necessarily an entirely accurate one.
Although both systems produced relevant and comprehensive answers, their factual claims require independent verification against reliable sources.
I therefore conducted a further fact-checking exercise to examine the accuracy and reliability of some selected statements from both ChatGPT and Gemini;
- Online learning can Reduce Education Costs
- Online learning offers greater flexibility
- Traditional classroom learning provides more direct student interaction.
The findings may influence the final accuracy scores and overall assessment
Fact-Checking Claim 1: Online Learning Can Reduce Education Costs
Finding: Partially supported.
A randomized controlled trial by Maloney et al. (2012) compared web-based and face-to-face education for health professionals.
The researchers found that both delivery methods produced comparable outcomes in participation, satisfaction, knowledge acquisition and changes in professional practice.
From the education provider’s perspective, web-based education was more financially efficient because it required fewer enrolments to reach its break-even point. However, cost-effectiveness analyses from the health service and participant perspectives favored face-to-face education in some circumstances.
This shows that the financial advantage of online learning depends on the perspective being considered and the costs included in the analysis.
Evaluation conclusion: The claim that online learning can reduce education costs is partially supported. However, online learning is not automatically cheaper in every situation. Its cost-effectiveness depends on the course design, delivery model and the costs incurred by providers and learners.
Evidence source: Maloney, S., Haas, R., Keating, J. L., Molloy, E., Jolly, B., Sims, J., Morgan, P., & Haines, T. (2012). Breakeven, Cost Benefit, Cost Effectiveness, and Willingness to Pay for Web-Based Versus Face-to-Face Education Delivery for Health Professionals. Journal of Medical Internet Research, 14(2), e47.Breakeven, cost benefit, cost effectiveness, and willingness to pay for web-based versus face-to-face education delivery for health professionals — PubMed
Fact-Checking Claim 2: Online Learning Offers Greater Flexibility
Finding: Supported, with qualifications.
Get Aishat Oluwaseun Idowu’s stories in your inbox
Join Medium for free to get updates from this writer.
The research reviewed identifies flexibility and convenience as important advantages of online learning. According to Lee (2024), online learning provides students with access to different learning resources and opportunities for self-directed learning.
In comparison, traditional classroom learning generally involves fixed schedules and physical attendance requirements, which may limit students’ flexibility.
The article also recognizes that online learning has limitations, including reduced face-to-face interaction, possible feelings of isolation and limited opportunities for certain hands-on activities.
Evaluation conclusion: The claim that online learning offers greater flexibility is supported by the research reviewed. However, the extent of flexibility depends on the course structure, particularly whether learning activities are self-paced or require attendance at scheduled online sessions.
Evidence source: Lee, C. M. (2024). Online Learning versus Face to Face Learning toward Students: Which can be an effective way of Learning Methodology to our current Educational System? Technological University of the Philippines. Christopher M. Lee — Online Learning versus Face to Face Learning
Fact-Checking Claim 3: Traditional Classroom Learning Provides More Direct Student Interaction
Finding: Supported, with qualifications.
A study by Yin and Shi (2022), involving 3,999 undergraduate students from a research university in Northern China, examined the relationship between face-to-face and online interactions and students’ learning experiences.
The researchers found that face-to-face synchronous interaction was generally more strongly associated with positive learning experiences, including teacher academic assistance, student cooperation, sense of belonging and overall satisfaction.
However, the study also identified benefits of online asynchronous interaction. These included supporting student autonomy and contributing to the completion of in-depth thesis writing.
The researchers therefore recommended using online asynchronous technologies to complement regular face-to-face interaction through a blended learning approach.
Evaluation conclusion: The claim that traditional classroom learning provides more direct student interaction is supported by the research. However, online learning also provides opportunities for meaningful interaction and independent learning. The findings should not be interpreted as evidence that face-to-face interaction is always superior to every form of online interaction.
Evidence source: Yin, H., & Shi, L. (2022). Which Type of Interpersonal Interaction Better Facilitates College Student Learning and Development in China: Face-to-Face or Online? ECNU Review of Education, 5(1), 9–36.
https://doi.org/10.1177/20965311211010818
Final Evaluation Results
Following the independent fact-checking of three selected claims, the initial evaluation scores were reviewed. “The fact-checking of the three selected claims did not provide sufficient grounds to change the initial accuracy scores. However, this exercise was limited to those selected claims and does not establish the accuracy of every statement in either response.”. Both systems therefore retained their original scores.
The final results remained unchanged following the selected-claim review. Google Gemini maintained a slightly higher score, primarily due to its clarity and concise presentation. It is important to note that only three selected claims were independently fact-checked. Therefore, these results should not be interpreted as confirmation that every statement in either response is accurate.
Final Comparative Analysis: ChatGPT vs Google Gemini
Following the initial evaluation and independent fact-checking of three selected claims, both ChatGPT and Google Gemini demonstrated strong performance in responding to the evaluation question.
However, some differences were observed in their response styles and presentation.
1. Accuracy and Reliability: Both systems provided generally reasonable explanations of online and traditional classroom learning. The three selected claims were broadly supported by the research reviewed, although the evidence highlighted important qualifications. For example, online education may reduce certain costs, but it is not necessarily more cost-effective in every situation. Similarly, although face-to-face learning offers opportunities for direct interaction, online learning can also support meaningful communication and independent learning.
The fact-checking exercise demonstrated that AI-generated information should be evaluated against independent evidence rather than accepted solely because it appears convincing.
2. Relevance and Completeness: Both systems addressed all the major requirements of the evaluation question. They defined the two learning methods, explained their advantages and disadvantages, compared their characteristics, identified suitable learner profiles and provided balanced conclusions.
Neither system omitted a major section requested in the prompt.
3. Clarity and Presentation: ChatGPT provided a more detailed response, with additional explanations, examples and supporting points. However, some information was repeated across different sections.
Google Gemini presented its response more concisely, using clear headings and a straightforward comparison table. This made its answer easier to scan and understand quickly.
4. Instruction Following: Both systems followed the instructions and maintained a clear structure. They also avoided making absolute claims that one learning method was universally better than the other.
Overall Comparative Finding
Google Gemini achieved a slightly higher overall score of 24/25 (96%), compared with ChatGPT’s 23/25 (92%).
The difference was primarily attributable to clarity and presentation rather than a significant difference in factual accuracy.
ChatGPT demonstrated strength in providing detailed explanations, while Google Gemini demonstrated strength in concise and organized communication.
The results suggest that both systems can produce relevant, comprehensive and well-structured responses. However, their outputs still require critical evaluation and independent verification.
Limitations of the Evaluation
This evaluation was based on one prompt submitted to two AI systems. The findings therefore represent their performance in this particular test and should not be interpreted as a general ranking of ChatGPT and Google Gemini.
The scoring process involved evaluator judgement, and only three selected claims were independently fact-checked. A broader evaluation using multiple prompts, additional sources and more extensive fact-checking would be needed to draw stronger conclusions about the overall reliability of either system.
Conclusion
This practical evaluation compared ChatGPT and Google Gemini using an identical prompt and a structured assessment framework consisting of five criteria: accuracy, relevance, completeness, clarity and instruction following.
Both systems demonstrated strong performance, producing relevant, comprehensive and well-organized responses to the evaluation question.
Google Gemini achieved a slightly higher overall score of 24/25 (96%), while ChatGPT scored 23/25 (92%). The difference was mainly observed in the clarity and presentation of their responses.
However, the independent fact-checking exercise highlighted an important consideration: a well-written AI response is not necessarily entirely accurate or complete. Reviewing selected claims against published research helped identify the context, limitations and qualifications behind the information generated by both systems.
The findings also demonstrated that neither online learning nor traditional classroom learning is universally superior. Their suitability depends on factors such as cost, flexibility, student interaction, course structure and individual learning needs.
Overall, this project reinforced the importance of critical thinking, evidence-based verification and structured assessment when evaluating AI-generated information.
As AI systems continue to develop, the ability to assess their outputs objectively will remain an important skill in improving the quality, reliability and usefulness of artificial intelligence.
Key takeaway: The quality of an AI-generated response should not be judged by how convincing or well-written it appears, but by how accurately, completely and reliably it addresses the question.
