Alright so I assume usually the way it works is that one test is made for everybody who could see the caster. But I've got a couple questions.

1. Can you "try again" to pierce the illusion through force of will (or intuition). This would be for times like when an invisible mage steps through mud leaving a trail or something else that tells everyone there's an invisible mage here, you just can't see them.

If so how would it work? I'd say it'd take a complex action and you can only try as many times as your relevant stat.

2. If someone does detect an invisible mage do they realize they're seeing someone who is trying to be invisible. (like they pop out of nowhere or retain some "I'm invisible" aura.)

1. You can't try again unless the GM for some reason says so. Seeing the mages steps in the mud would be different from seeing through the illusion. You would still be affected by the illusion but you would realize that something is making footprints in mud. Also realize this would most likely be a Perception test with a threshold of 2 or 1 if the GM felt it is easy to see.
2. If you resist the illusion it is fully resisted. You do not realize the person is trying to hide, you just see them. As far as control thoughts, people do not realize they are under the affect of the spell. If they did realize it, it would say so like it does for mind probe. Even with mind probe people still won't realize who is doing it unless they make a noticing magic test.

One last thing. In a recent FAQ it was stated people always get counterspelling dice for Detection and Illusion spells. This is due to LOS issues with the spells. I personally do not agree with it and when I GM my first Shadowrun game, I will not be implementing this rule. I will find another way to make it easier to have counterspelling dice for Illusion and Detection spells. I haven't come up with a way I like yet.

EDIT:
For number 1 it would actually be vs. the mage's infiltration, unless for some reason the GM was doing a test to spot the footprints only. Usually this will always be a perception vs. infiltration opposed test.
EDIT AGAIN:
Possible scenario for the mud footprints:
The mage is walking with Imp. Invisibility on and he only slightly fails his infiltration test, or gets a glitch on the infiltration test. The GM decides mud footprints show up. A perception test with a threshold of 2 must be made to see the mud footprints.

Hmmm I may have to disagree with my previous post about mental manipulations going unnoticed. Even though the book never outright says it is noticeable like it does for mind probe. Which, one would think would set a precedent for things like this. It says they get force combat turns to shake off the mental manipulation. Wouldn't this then require them to be aware they are being manipulated and actively resisting the effects of the spell. Why wouldn't they then say the same thing they did for mind probe where it states they are aware of the spell but not necessarily know who it is? I seem to want to agree with djinni because it makes sense. On the otherhand, why aren't the spell description consistent with the precedence set in other spell descriptions. For the ease of understanding and clarity one would think they would have done this. It is also very possible every force combat turns requires the spellcaster to regain control relative to the force of the spell. This would then require another resistance test. So then maybe they wouldn't notice it. Hard to tell.

Control emotions sounds like the target isn't aware at all and just has a really drawn out emotion. Control thoughts you are again giving the target orders that they are forced to comply with. Wouldn't they then realize they no longer have control of their body? Or, is it some very strong force of will which they feel has to be done? Then there is influence which is implanting an idea into the targets head. It sounds like they take it to be their very own idea and that it makes sense to them. They will carry it out as if it were the greatest idea in the world they had come up with. If they don't make the willpower test they will even argue they are right.

Since under the rules of mental manipulation it doesn't state the target specifically knows of the mental manipulation I think I would have to say no. My only reason for this is that I think if it was the case that they noticed mental manipulations it would say so. I could definitely see it going djinni's way too in an update to the FAQ for certain mental manipulation spells, but not all of them.

EDIT:
I just read the SR3 spell descriptions and they seem to point even more towards djinni's opinion in regards to Control Actions and Control Thoughts. This is something that should really be cleared up in the FAQ or errata.

I'll definitly agree that it should have been spelled out sooner.

For purposes of gameplay I'd be inclined to have spells that directly control make the target aware and the emotional ones not make them aware.

Another question. How does it work for things in contact with the mage or thing the mage touches going invisible?

For an specific example what if you thought an invisible stealthed mage was behind you so you grabbed some kids silly string and hosed the mage down. Would the silly string vanish or give away the mage.

Does it matter if all the string is just on their shirt or touching the ground?

As long as you are still under the effects of the illusion(you failed the resistance test) you will still see the mage as being invisible. The silly string will just coat the mage. Maybe instead of a -6 for a blind fire test it would be a -2. Not to sure how to handle that one. The players still would not be able to see the mage normally if they failed to resist the illusion spell.