Evaluation Process: Quantifying the Potential Impact of GenAI on Job Competencies

================================================================

A recent study, conducted using data from US job postings between August 1st, 2023, and July 31st, 2024, has shed light on the potential impact of Generative AI (GenAI) technology on various occupations. The research, which assessed over 2,800 job skills, ranging from account management to wound care, was based on a prompt created by OpenAI and compared the performance of the GPT-4o model across different AI models.

The study found that only four skills were given the highest rating for providing theoretical knowledge: generative AI, grammar, text classification, and writing skills. Interestingly, data entry was identified as "very likely" to be replaced by GenAI in one of the 15 runs, but not in the consolidated final rating.

The majority of skills (53.8%) require essential or high physical execution and are therefore very unlikely to be replaced by GenAI. Conversely, only 2.8% of skills require no or little physical execution and are likely to be replaced. The study also noted that 80 skills were rated as "likely" to be replaced, all having good problem-solving abilities and either good or proficient theoretical knowledge.

The ratings were used to assess the likelihood that GenAI could replace a human in performing any of the assessed skills. Each skill was evaluated across three main areas: the ability of GenAI to provide theoretical knowledge, solve problems, and determine the importance of physical presence. The ratings were scored on a 5-point scale, with a rating of 1 indicating very limited or no ability, and a 5 indicating a strong ability.

To increase the reliability and robustness of the results, each skill was assessed 15 times. The justification for the ratings was always selected from the first run within the group of ratings that share the mode. The range of differences was minimal for most skills, with 99.3% having a difference of 1 for theoretical knowledge and problem-solving.

The study's findings were validated by human researchers, and the prompt was adjusted if the results did not meet expectations. The model's performance was compared across different models, with GPT-4o generating more reliable results. The mode was calculated as the most frequently occurring rating across 15 runs for each dimension. The final prompt was structured in seven sections and followed common prompt engineering strategies.

In conclusion, while GenAI shows promise in replacing certain tasks, the majority of jobs requiring physical execution are unlikely to be replaced. However, skills such as data entry, problem-solving, and theoretical knowledge, particularly in areas like writing and text classification, could potentially be automated. The study underscores the need for continuous monitoring and adaptation as AI technology continues to evolve.

Evaluation Process: Quantifying the Potential Impact of GenAI on Job Competencies