Comparative Analysis of LLM-Based Conversational Agents as an Assistive Technology for Web Interaction

This study investigates whether LLM-based Conversational Agents (LLMCAs) could be employed as assistive technologies, helping anyone navigate a website by explaining its content and interactive features, and suggesting actions to accomplish specific goals. This would make the web more accessible especially to seniors, blind users and people with mild cognitive impairment. Specifically, the study examines whether such tools can analyze the source code of a webpage to identify its structure and interactive elements, understand which interactive components are required to support users in achieving their goals, and act effectively as assistive technologies. In addition, the work compares leading tools to assess differences in their effectiveness and behavior in supporting accessible web interaction. The work presents two empirical studies in which four popular LLMCA tools are evaluated on a set of public websites, in order to assess their ability to understand web page structures and interactive components and assist users in their tasks. The evaluation combines qualitative and quantitative measures, focusing on accuracy, completeness, and practical usefulness of the responses. Results show a good degree of pertinent and good-quality answers, even on websites with a poorly accessible structure. The user-friendly way to interact with a LLMCA makes it much more comfortable to use for the average user. However, the study also highlights that some LLMCAs appear more ’skilled’ in accessibility tasks than others. LLMCAs show promising potential not only to complement existing assistive technologies but also to support or replace the human assistance that is often required for many users to effectively access the web.