Labeled Multi-Bernoulli Sensor Management Strategy Based on Twin-Delayed Deep Deterministic Policy Gradient Learning Mechanism
-
摘要: 针对复杂不确定性环境下的多目标跟踪问题，提出了一种基于双延迟深度确定性策略梯度(Twin-Delayed Deep Deterministic Policy Gradient algorithm, TD3)学习机制的标签多伯努利滤波器(Labeled Multi-Bernoulli, LMB)传感器管理方法。首先，联合深度强化学习和有限集统计(Finite Set Statistics, FISST)理论，基于LMB多目标运动感知结果构建信念空间，将传感器管理任务转化为该空间内的连续动作决策问题，使智能体能够根据当前多目标信念状态选择移动传感器的下一步观测动作。其次，为精准量化传感器配置对感知效能的增量贡献，引入柯西-施瓦茨(Cauchy-Schwarz, CS)散度动态评估候选策略驱动下的多目标概率密度信息增益，并据此设计信息驱动型奖励函数。通过TD3网络的双评论家架构、目标策略平滑与延迟更新机制，智能体能够在连续动作空间内学习观测配置，实现对传感器管理的闭环决策。实验结果表明，该方法能够改善LMB滤波器的状态估计性能和传感器轨迹连续性。Abstract:
Objective Multi-target tracking requires sensor management to adapt the observation process to clutter, missed detections, target-number variations, and changes in target motion. Conventional methods often search over a finite set of sensor actions, resulting in increasing computational cost and limited control resolution. Moreover, rewards formed by combining several single-target quantities may not adequately represent the joint multi-target posterior. A continuous-action sensor management method integrating the Twin-Delayed Deep Deterministic Policy Gradient algorithm with the Labeled Multi-Bernoulli filter is therefore developed to optimize mobile-sensor heading according to the multi-target belief state. Methods The LMB posterior, including target existence probabilities and state densities, is used to construct the belief state. At each filtering step, the mobile sensor selects a continuous heading angle that affects the sensor-target geometry, detection probability, and LMB update. Predicted target states and candidate headings are used to generate ideal measurements and obtain pseudo-updated LMB densities. The Cauchy-Schwarz divergence between the predicted and pseudo-updated densities is used to construct the information-gain reward. TD3 employs two critics, target policy smoothing, and delayed actor updates to reduce value-estimation bias. Random control, policy-gradient control, an information-driven discrete method, and DDPG-based continuous control are used for comparison. Results and Discussions DDPG-LMB and TD3-LMB produce more continuous steering changes than the discrete-action methods ( Fig. 2 ). TD3-LMB obtains the highest or near-highest detection probabilities for most targets (Fig. 3 ) and gives relatively large Cauchy-Schwarz divergence values during most time steps, while random control remains relatively low (Fig. 4 ). TD3-LMB also achieves the lowest overall OSPA in the tested scenario, with DDPG-LMB generally outperforming the discrete baselines (Fig. 5 ). These results show that continuous heading control improves observation quality and overall LMB tracking performance.Conclusions A TD3-based continuous-action sensor management framework for the LMB filter is presented. Candidate heading actions are evaluated through pseudo-updated LMB densities and Cauchy-Schwarz divergence, linking action selection directly to the joint multi-target posterior. The simulation results show more continuous sensor motion, higher detection probabilities for most targets, larger information gain, and lower OSPA in the tested scenario. Future work will consider higher-dimensional actions and cooperative multi-sensor management. -
[1] LIU Zhihao, SHANG Yuanyuan, LI Timing, et al. Robust multi-drone multi-target tracking to resolve target occlusion: A benchmark[J]. IEEE Transactions on Multimedia, 2023, 25: 1462–1476. doi: 10.1109/TMM.2023.3234822. [2] HU Juan, ZUO Lei, VARSHNEY P K, et al. Resource allocation for distributed multitarget tracking in radar networks with missing data[J]. IEEE Transactions on Signal Processing, 2024, 72: 718–734. doi: 10.1109/TSP.2024.3352915. [3] COX P B and VAN ROSSUM W L. Split-aperture phased array radar resource management for tracking tasks[J]. IEEE Transactions on Aerospace and Electronic Systems, 2025, 61(3): 6476–6486. doi: 10.1109/TAES.2025.3531843. [4] HAYKIN S. Cognitive radar: A way of the future[J]. IEEE Signal Processing Magazine, 2006, 23(1): 30–40. doi: 10.1109/MSP.2006.1593335. [5] ZHANG Peng, YAN Junkun, PU Wenqiang, et al. Multi-dimensional resource management scheme for multiple target tracking under dynamic electromagnetic environment[J]. IEEE Transactions on Signal Processing, 2024, 72: 2377–2393. doi: 10.1109/TSP.2024.3390119. [6] CHARLISH A, HOFFMANN F, DEGEN C, et al. The development from adaptive to cognitive radar resource management[J]. IEEE Aerospace and Electronic Systems Magazine, 2020, 35(6): 8–19. doi: 10.1109/MAES.2019.2957847. [7] KURNIAWATI H. Partially observable Markov decision processes and robotics[J]. Annual Review of Control, Robotics, and Autonomous Systems, 2022, 5(1): 253–277. doi: 10.1146/annurev-control-042920-092451. [8] MAHLER R. Objective functions for Bayesian control-theoretic sensor management, II: MНC-like approximation[M]. BUTENKO S, MURPHEY R, and PARDALOS P M. Recent Developments in Cooperative Control and Optimization. Boston: Springer, 2004: 273–316. doi: 10.1007/978-1-4613-0219-3_16. [9] GOSTAR A K, HOSEINNEZHAD R, BAB-HADIASHAR A, et al. OSPA-based sensor control[C]. 2015 International Conference on Control, Automation and Information Sciences (ICCAIS), Changshu, China, 2015: 214–218. doi: 10.1109/ICCAIS.2015.7338664. [10] JONES G, GARCÍA-FERNÁNDEZ Á F, and WONG P W. GOSPA-driven Gaussian Bernoulli sensor management[C]. 2023 26th International Conference on Information Fusion (FUSION), Charleston, USA, 2023: 1–8. doi: 10.23919/FUSION52260.2023.10224220. [11] GOSTAR A K, HOSEINNEZHAD R, and BAB-HADIASHAR A. Multi-Bernoulli sensor control for multi-target tracking[C]. 2013 IEEE 8th International Conference on Intelligent Sensors, Sensor Networks and Information Processing, Melbourne, Australia, 2013: 312–317. doi: 10.1109/ISSNIP.2013.6529808. [12] RISTIC B, VO B N, and CLARK D. A note on the reward function for PHD filters with sensor control[J]. IEEE Transactions on Aerospace and Electronic Systems, 2011, 47(2): 1521–1529. doi: 10.1109/TAES.2011.5751278. [13] GOSTAR A K, HOSEINNEZHAD R, RATHNAYAKE T, et al. Constrained sensor control for labeled multi-Bernoulli filter using Cauchy-Schwarz divergence[J]. IEEE Signal Processing Letters, 2017, 24(9): 1313–1317. doi: 10.1109/LSP.2017.2723924. [14] HOFFMANN F, CHARLISH A, RITCHIE M, et al. Sensor path planning using reinforcement learning[C]. 2020 IEEE 23rd International Conference on Information Fusion (FUSION), Rustenburg, South Africa, 2020: 1–8. doi: 10.23919/FUSION45008.2020.9190242. [15] SHI Yuchun, JIU Bo, YAN Junkun, et al. Data-driven simultaneous multibeam power allocation: When multiple targets tracking meets deep reinforcement learning[J]. IEEE Systems Journal, 2021, 15(1): 1264–1274. doi: 10.1109/JSYST.2020.2984774. [16] LU Ziyang and GURSOY M C. Resource allocation for multi-target radar tracking via constrained deep reinforcement learning[J]. IEEE Transactions on Cognitive Communications and Networking, 2023, 9(6): 1677–1690. doi: 10.1109/TCCN.2023.3304634. [17] OBOREH-SNAPPS O, SHE Buxin, FAHAD S, et al. Virtual synchronous generator control using twin delayed deep deterministic policy gradient method[J]. IEEE Transactions on Energy Conversion, 2024, 39(1): 214–228. doi: 10.1109/TEC.2023.3309955. -
下载:
图(5)
计量
- 文章访问数: 14
- HTML全文浏览量: 5
- PDF下载量: 2
- 被引次数: 0
