I must confess I don’t really know where this problem comes from or why rmdir /var/cache/pacman/pkg/download-* works when ls -l /var/cache/pacman/pkg/download-* shows nothing 
This is all a mystery to me.
But maybe you could try @cscs 's Maclean script, that does wonders.