Web Scraping e extração de dados em estruturas HTML com visualização em interface web: um estudo de caso

Abstract

The virtual item market in electronic games has grown significantly, particularly in titles such as Counter-Strike, where skins can reach high commercial value. Given the constant price fluctuations across platforms, access to real-time data is essential to support purchasing decisions. This study presents a case study and the implementation of an automated system for the real-time collection and centralization of Counter-Strike skin data from five specialized platforms. Data extraction was conducted using the Selenium library, combined with WebDriver and undetected_chromedriver, to bypass anti-bot mechanisms. Information such as item name, price, finish type, wear level (float), and image was extracted based on specific HTML tag selectors such as <div>, <span>, and <img>, and subsequently processed using the pandas library for standardization and analysis. Data visualization was enabled through an interactive interface developed with Dash Bootstrap, allowing for user-friendly navigation and analysis. The application was further optimized using multithreading to enhance scraping efficiency, particularly in environments with multiple sources. Additionally, a dynamic scraping mode was implemented for users with prior knowledge of HTML structure, allowing manual customization of HTML selectors, page scroll type (paginated, infinite, or alternating), and target URL. This feature broadens the tool’s applicability by enabling integration with new websites following the same technical principles used in the main study. Among the key challenges addressed were asynchronous JavaScript content loading and automation detection mechanisms, which were overcome through forced rendering and stealth navigation techniques. The proposed solution aims to facilitate user access and analysis of data, promoting greater efficiency in interpreting item values and characteristics.