Adaptive Storage Tiering in Data Engineering Systems for Mixed Analytical and Operational Workloads

Abstract

Mixed analytical and operational workloads create significant storage management challenges in modern data engineering systems because the same data environment must support low-latency transactions, large analytical scans, streaming ingestion, historical retention, and dashboard refreshes. Static storage placement often leads to inefficient use of high-performance storage, delayed access to hot data, and unnecessary cost for inactive historical datasets. This article presents an adaptive storage tiering framework that classifies data objects according to access frequency, latency sensitivity, update intensity, analytical scan demand, and retention requirements. The proposed framework assigns data to performance, balanced, capacity, or archive tiers using workload profiling, latency-cost scoring, migration thresholds, and feedback-based placement correction. The results indicate that adaptive tiering improves query latency reduction, hot-data placement accuracy, and storage cost efficiency across repeated tiering cycles. The framework also improves workload stability and placement efficiency while controlling migration overhead across different storage tiering policies. Overall, the article demonstrates that adaptive storage tiering can serve as an intelligent data engineering control layer for balancing performance, cost, and operational reliability in mixed workload environments.