Efficient Data Matching
In 2022, Analytics Engines initiated a project for a software company based in the United States. This client provides services aimed at helping large enterprises gain insights into their utilization of hardware, software, SaaS, and cloud assets, which allows them to optimize their IT expenditures and achieve maximum return on investment. In light of an ambitious modernization plan, the client sought a development partner to create internal microservices that would replace some repetitive, manual tasks within their current workflow. The client suggested that a Machine Learning-based solution could address this challenge. However, drawing from past experience, our data scientists concluded that an algorithmic, rules-based method could achieve comparable, if not superior, results. Our team also felt that a rules-based method could resolve many challenges linked to Machine Learning, such as higher costs, more complex deployment, and the necessity for ongoing maintenance. We ran an experiment using a sample dataset provided by the client, where we tasked both the Machine Learning model and the rules-based algorithm to identify and classify software version numbers in the dataset. The results revealed that the rules-based approach achieved a success rate exceeding 99%, while the Machine Learning method only reached 70%. Based on these findings, we developed a solution that employed a rules-based strategy to integrate data cleansing with automatic evidence matching. This algorithm is scalable and retains the high accuracy of the manual matching process, successfully automating the identification and classification of software versions with an accuracy rate above 99%. Consequently, the solution has automated a considerable portion of the project and significantly diminished the need for manual processing.