Do I need the Astral Combat skill to battle entities in astral space, even when I am only astrally perceiving them and not projecting ? I have an adept with the Astral Perception power, but I don't know whether I can use my Combat skill (+ weapon focus), or have to use Willpower + Astral Combat.

When you're attacking astral entities, it's Willpower + Astral Combat (+ Weapon Focus, if any).