arXiv:2409.07796v3 Announce Type: replace-cross Abstract: Resource-constrained IoT devices increasingly rely on deep learning models, however, these models experience significant accuracy drops due to domain shifts when encountering variations in lighting, weather, and seasonal conditions. While cloud-based retraining can address this issue, many IoT deployments operate with limited connectivity and energy constraints, making traditional fine-tuning approaches impractical. We explore this challenge through the lens of wildlife ecology, where camera traps must maintain accurate species classification across changing seasons, weather, and habitats without reliable connectivity. We introduce WildFit, an autonomous in-situ adaptation framework that leverages the key insight that background scenes change more frequently than the visual characteristics of monitored species. WildFit combines background-aware synthesis to generate training samples on-device with drift-aware fine-tuning that triggers model updates only when necessary to conserve resources. Our background-aware synthesis surpasses efficient baselines by 7.3\% and diffusion models by 3.0\% while being orders of magnitude faster, our drift-aware fine-tuning achieves Pareto optimality with 50\% fewer updates and 1.5\% higher accuracy, and the end-to-end system outperforms domain adaptation approaches by 20--35%\% while consuming only 11.2 Wh over 37 days -- enabling battery-powered deployment.