He's angry at the others (his colleagues?) for goofing off during guard duty, eating sweets (?) like little girls and having fun, all of which is very unbefitting of a serious samurai.
The last paragraph is him reiterating that the schedule of the trip has already been decided (so they should stop asking the same questions).