Real-time frame-level fish feeding behavior detection using a lightweight CNN-TCN architecture

Aquaculture, particularly fish farming, is a rapidly growing sector critical for global food security, where efficient feeding management is essential for economic and environmental sustainability. Real-time monitoring of fish feeding behavior can enable intelligent feeding control, thereby reducing waste and optimize growth. Here we proposed frame-level models to make real-time fish feeding behavior detection by combining a Convolutional Neural Network (CNN) with a Temporal Convolutional Network (TCN) from video data. The CNN extracts spatial features from each video frame, while the TCN aggregates temporal dependencies through stacked dilated causal residual blocks, allowing reliable recognition of feeding behavior from frame sequences over time. The models were evaluated against baseline CNN models and CNN/MobileNet + GRU models using grayscale and/or optical flow frames as inputs. Results show that the gray CNN-TCN model achieved the best performance, with accuracy, precision, recall, F1 score, and ROC-AUC all exceeding 0.99, while maintaining real-time inference capability, reaching 1,029 frames per second and a mean latency of 7.77 ms and only moderate computational overhead. Moreover, the input of gray-scale frames alone is sufficient to capture fish feeding behavior, indicating that the commonly used optical flow may be a redundant input for the CNN-TCNs. To align real-time model training and inference, we proposed a three-phase feeding schema comprising the before-feeding phase (B), the immediate after-feeding phase (A), and the subsequent inference phase (S). The B-A-S schema allowed automatic sample labeling and provided a foundation for real-time and intelligent feeder control in recirculating aquaculture systems. The findings indicate that the proposed gray CNN-TCN model, combined with the B-A-S feeding schema, offers an effective and computationally efficient approach for fine-grained fish feeding behavior detection, supporting real-time model training and inference, and thereby enhancing the toolkit available for intelligent feeding control in aquaculture systems.