If the computer turns and works (other than the noise) then the wiring is correct. The ATX plug should be keyed so it is almost impossible to hook up incorrectly, and the wiring is for the most part standardized. However, there could be a problem if it is an older computer and your replacement power supply is from a Dell. Dell ATX power supplies, mostly from the PII to PIII era were not wired to the standard ATX pinouts. So, if you put a non-Dell power supply in an older Dell, or if you put an older Dell power supply in a non-Dell PC of any age, your wiring will not be correct. Usually incorrect wiring means either it fries or just doesn't start at all.
Specialized PC's or high end servers may also use non standard power connectors.
If you paid less than $40.00 for the power supply, the problem is most likely just a bad / poorly made power supply. Most of the budget power supplies are extremely cheaply made. They will sometimes have a lot of noise/whine/hum, and often fail quickly. In a lot of cases you will be able to tell the quality of the power supply just by the weight. The cheap ones are usually as light as can be. A good power supply will have some heft to it, as it will have large heatsinks and heavy duty components. If you have a Radio Shack nearby you could pick up a Thermaltake 300 to 450 watt power supply and try it and see if the problem goes away. They have a pretty flexible return policy. If you have a high quality power supply, it could be that you just got a bad one. Try another to confirm. If you try a Thermaltake and it still whines (and never made noise before this problem came up) then it could be that other components in the PC are near failure--possible as a result of your previously failing power supply. My experience though has been that unless the power supply popped (as in literally blew out) the motherboard and other internals aren't usually affected.
I am assuming here that the computer is starting up. If not, make sure that that all the necessary power leads are hooked up. Newer motherboards may have have a 20 or 24 pin main ATX power connector. They may also have a secondary 4-pin 12v connector to power the CPU. If you are mixing a really old or new revision power supply with an opposite age motherboard there may be a problem of not having the appropriate connections available.
Finally, if you are under-powered, you could also experience problems. What is the wattage of your new power supply and what type of computer/CPU? If you are running a 200 watt power supply to power a modern motherboard/CPU you may not be able to supply the power needed. This could result in a stressed power supply, which may make noise, or just power offs or system instability.
I very much recommend spending the extra dollars for a well made power supply. I have replaced many of the $20.00 units. Even if they say they are 350 or 450+ watts, they don't hold up. Often the computer will start random power offs and then the supply will fail completely. A good supply will usually come with a 5 year warranty and will be generally problem free.
Here is a link to a wiring diagram for the ATX plug--just to be complete.