The Yang Family wasn't afraid of Guard Li, because they knew his Identity.
"If he really makes a move... it will definitely draw the attention of higher-ups, and then he might not be able to cover it up himself."
"He's even less willing than we are to let this blow up."
"Good thing we didn't let his people steal the grain, or we'd really have failed at the last step."
Zhong Yi's wave of snatching grain on the city wall had already screwed everyone once; they would absolutely never again let some unrelated person touch their grain.
Especially since along the way, they'd kept stockpiling until they had this amount now, just enough to submit to the court. If some was missing... there was really nowhere they could go to make it up.
"Speaking of... now that this Guard Li has left, where exactly are we supposed to deliver our grain? We don't have any guidance..."
