TLDR: A study evaluated six LLMs (ChatGPT 4, DeepSeek, Gemini, Claude 3.5, Llama 3, Mistral AI) on their career guidance for entry-level computing roles across ten African countries. It found that while LLMs consistently recommended core technical skills, they often lacked contextual awareness, making assumptions about infrastructure and overlooking local factors. Open-source models like Llama and DeepSeek generally outperformed proprietary ones in contextual relevance and balancing technical and professional skills, suggesting a need for decolonial approaches to AI in African education that prioritize local needs and open-source solutions.
In today’s rapidly evolving job market, large language models (LLMs) are increasingly becoming a go-to resource for students seeking career guidance. But how well do these AI tools truly understand the unique needs and contexts of students in different parts of the world, especially in Africa?
A recent study titled Evaluating LLMs for Career Guidance: Comparative Analysis of Computing Competency Recommendations Across Ten African Countries delves into this very question. Authored by Precious Eze, Stephanie Lunn, and Bruk Berhane, the research offers a comprehensive look at how six prominent LLMs—ChatGPT 4, DeepSeek, Gemini, Claude 3.5, Llama 3, and Mistral AI—provide career advice for entry-level computing roles across ten diverse African countries.
Understanding the Study’s Approach
The researchers aimed to uncover similarities and differences in LLM recommendations, focusing on both technical and non-technical computing competencies. They also evaluated how well these models adapted their advice to specific country contexts. The study used the globally recognized Computing Curricula 2020 framework as an analytical lens, alongside theoretical perspectives like Digital Colonialism Theory and Ubuntu Philosophy, to interpret the findings.
Ten African countries were selected for the study: Egypt, South Africa, Tunisia, Morocco, Nigeria, Senegal, Kenya, Benin, Ghana, and Zambia. For each country, a standardized prompt was given to each of the six LLMs, asking about necessary skills and preparation for an entry-level computing job. A total of 60 responses were collected and analyzed.
Key Findings: What LLMs Recommend
The study found that LLMs consistently highlighted core technical skills. Programming, particularly Python, and areas like AI/Machine Learning/Natural Language Processing (ML/NLP) were almost universally mentioned across all responses. Cloud computing also featured prominently. On the non-technical side, adaptability, lifelong learning, teamwork, and communication were frequently recommended.
However, there were notable gaps. Competencies like cybersecurity, data analytics, and project management appeared less frequently. More critically, the coverage of ethics and responsible AI use was inconsistent, often being implicitly mentioned through terms like ‘privacy’ or ‘regulatory compliance’ rather than explicitly stated.
The Challenge of Contextual Awareness
One of the most significant findings was the limited contextual awareness of most LLMs. Many responses assumed universal access to advanced technological infrastructure, such as major cloud services like AWS or Azure, without acknowledging the varying realities of access and affordability in different African countries. This ‘contextual blindness’ often led to generic, Western-centric advice that overlooked local tech industries, specific language requirements, national policies, or unique educational programs.
For instance, while ChatGPT-4 might recommend gaining experience with global cloud services for Nigeria, an open-source model like Llama recognized Kenya’s unique fintech landscape by mentioning the importance of understanding M-Pesa and mobile money technologies. This highlights a crucial difference: proprietary models often provided standardized advice, while some open-source models demonstrated greater sensitivity to local conditions.
Open-Source vs. Proprietary Performance
Surprisingly, open-source models generally outperformed their proprietary counterparts in the study’s weighted evaluation. Llama achieved the highest composite score, followed closely by DeepSeek. These open-source models showed a better balance between technical and professional skills and demonstrated stronger contextual awareness. This challenges the common assumption that proprietary AI tools are always superior, especially in resource-constrained environments where cost-effective open-source alternatives could be more beneficial.
However, not all open-source models were equally context-aware; Mistral, for example, provided entirely generic responses. This suggests that factors beyond just being ‘open-source,’ such as training data composition and model architecture, play a vital role in contextual responsiveness.
Also Read:
- Evaluating AI Ethics Courses: A Multi-Perspective Approach with AI Assistance
- Navigating Systematic Literature Reviews: How Prompting Strategies Shape LLM Performance in Screening
Implications for African Computing Education
The research underscores the need for a ‘decolonial approach’ to AI in education. This means moving away from simply adopting external AI tools that may carry Western biases and infrastructure assumptions. Instead, African educational institutions should critically evaluate AI tools for their contextual relevance and alignment with local values.
The study suggests that hybrid approaches, combining the technical strengths of LLMs with the invaluable contextual knowledge of local human experts, could be most effective. Furthermore, the strong performance of open-source models points to opportunities for African institutions to actively participate in AI tool development, customizing them to reflect local languages, cultural practices, and specific industry needs. This could foster technological self-determination and educational sovereignty across the continent, ensuring that AI truly empowers African youth for their unique career pathways.
