Fixed #34903, Refs #34825 -- Made workers initialization respect empty set of used connections.
Thanks to David Smith for the investigation & patch.
Regression in 2128a73713735fb794ca6565fd5d7792293f5cfa.
Follow up to a5905b164dbf52e59fa646af9c3d523c0804d86a.
Co-authored-by: David Sanders <shang.xiao.sanders@gmail.com>