A Performance Study of LLM-Generated Code on Leetcode

Coignion, Tristan; Quinton, Clément; Rouvoy, Romain

Document type :

Autre communication scientifique (congrès sans actes - poster - séminaire...): Communication dans un congrès avec actes

DOI :

10.1145/3661167.3661221

Title :

A Performance Study of LLM-Generated Code on Leetcode

Author(s) :

Coignion, Tristan [Auteur]
Self-adaptation for distributed services and large software systems [SPIRALS]
Quinton, Clément [Auteur]

Self-adaptation for distributed services and large software systems [SPIRALS]
Rouvoy, Romain [Auteur]

Self-adaptation for distributed services and large software systems [SPIRALS]

Conference title :

EASE'24 - 28th International Conference on Evaluation and Assessment in Software Engineering

City :

Salerno

Country :

Italie

Start date of the conference :

2024-06-18

English keyword(s) :

LLM
Large Language Model
Leetcode
Performance
Python

HAL domain(s) :

Informatique [cs]/Génie logiciel [cs.SE]
Informatique [cs]

English abstract : [en]

This study evaluates the efficiency of code generation by Large Language Models (LLMs) and measures their performance against human-crafted solutions using a dataset from Leetcode. We compare 18 LLMs, considering factors ...
Show more >This study evaluates the efficiency of code generation by Large Language Models (LLMs) and measures their performance against human-crafted solutions using a dataset from Leetcode. We compare 18 LLMs, considering factors such as model temperature and success rate, and their impact on code performance. This research introduces a novel method for measuring and comparing the speed of LLM-generated code, revealing that LLMs produce code with comparable performance, irrespective of the adopted LLM. We also find that LLMs are capable of generating code that is, on average, more efficient than the code written by humans. The paper further discusses the use of Leetcode as a benchmarking dataset, the limita- tions imposed by potential data contamination, and the platform’s measurement reliability. We believe that our findings contribute to a better understanding of LLM capabilities in code generation and set the stage for future optimizations in the field.Show less >

Language :

Anglais

Peer reviewed article :

Oui

Audience :

Internationale

Popular science :

Non

ANR Project :

Comprendre, Améliorer, Réduire les impacts Environnementaux du Cloud computing
Service de recommandation pour des logiciels en ligne plus durables
Configurations d'environnements fog large-échelle

Collections :

Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189

Source :

Harvested from HAL

Files

document
Open access
Access the document

Tristan___LLM_Code_Performance_Study.pdf
Open access
Access the document

2407.21579
Open access
Access the document

document
Open access
Access the document

Tristan___LLM_Code_Performance_Study.pdf
Open access
Access the document

A Performance Study of LLM-Generated Code ... BibTeX CSV Excel RIS

Files

A Performance Study of LLM-Generated Code ...

BibTeX

CSV

Excel

RIS