The flashing LED is because your system is in "sleep" mode. By default windows Vista puts your computer in sleep. What this means is it doesn't shutdown the power completely it leaves a little going through certain computer components (more specifically the ram) to speed up the boot time when the power button is pressed again.
In XP a the default shutdown scheme is a complete shutdown where power is turned off after the shutdown process is done. If you are using XP you are putting your computer to sleep when shutting down.
As for the boot problem. Newer motherboards have an LED on the actual board to indicate that it is receiving power. If it is of either the power supply switch (at the back of the case) if off or the motherboard connection is loose and the shaking makes it fall into place. (Shaking is usually not necessary and indicates a problem...)
There are a few possibilities I can deduct from your description:
- the connection from your power supply to your motherboard isn't properly seated and is causing this
- the power supply switch (not the case switch) is being turned off and results in the computer taking a few seconds before the internal LED become solid
- ... the dreaded sign that the power supply is failing and is on it's last legs...
My suggestion is to check all connections and if that doesn't work borrow a power supply from a friend and swap them to see if it boots properly.
(If a friends with a spare power supply isn't an option a local PC repair shop should be able to help you out... but probably for a fee...)