DISCOVERING DORMANT, DUPLICATE AND LOWVALUE ENTERPRISE DATA TO DRIVE AUTOMATED DATA LIFECYCLE AND STORAGE OPTIMIZATION - (DATA WASTE - SENTINEL)

Abstract

Organisations store far more data than they actively use. File shares, cloud buckets, databases, and backup systems accumulate copies of documents, abandoned project folders, old logs, and tables that no report reads any more. This unused data consumes storage, increases backup time and cost, complicates compliance with retention and privacy rules, and makes it harder for employees to find the information they actually need. This paper presents Data Waste Sentinel, a data engineering and analytics system that discovers dormant, duplicate, and low-value enterprise data and drives automated lifecycle actions to optimise storage. The system collects metadata rather than content from storage platforms. Crawlers and connectors gather file system metadata, object storage inventories, database catalogue statistics, and access logs, including sizes, owners, creation and modification dates, last access times, and query history. A metadata pipeline normalises these records into a common catalogue, computes content hashes for files where permitted, and builds a lineage graph that shows which tables and files feed reports, applications, and downstream jobs. All of this is stored in a metadata warehouse that is updated incrementally every day. Three analysis modules operate on the catalogue. A dormancy analyser measures how long each asset has gone without being read or written and compares this with the typical access rhythm of similar assets. A duplicate detector finds exact copies through hashing and near-duplicates through MinHash similarity on file content signatures and table schemas. A value scoring model, based on gradient boosting, combines access frequency, number of distinct users, lineage connections, business ownership, and retention requirements into a score that estimates the importance of each asset. A policy engine then maps each asset to an action such as keep, move to cold storage, archive, deduplicate, or recommend deletion, subject to owner approval.