Protect Mode with RCA's
It sounds like you have some DC voltage on the RCA input.
You need to figure out if the problem is coming from the head unit or if it is in the amplifier.
If you have a voltage meter, you can use it to find out very quickly.
Set the meter for DC and for a max level of more than 12 volts. Most meters I use have a 30 volt level. That would be a good setting.
Take the black lead and make sure it is grounded to the chassis of the car or to the ground of the power terminal, be careful and make sure it does not touch anything else.
With the RCA cable disconnected put the red lead on the inner part of the RCA cable and see if there is any DC voltage and then check the ground or outer part of the cable. Do the same for the other channel as well.
It should be very close to zero volts. A couple of millivolts is fine to. Any voltage here will cause the amplifier to go into protect mode, which is what is happening to you. You will then need to have the head unit serviced.
If you have a Pioneer head unit, I have seen many of them with an open ground fuse for the RCA jack. That fuse is a surface mounted fuse and is very small and hard to locate on the main board of the radio. If you do have a Pioneer head unit the voltage may seem OK, but you will have an open ground, but that normally does not put the amp into protect. It makes the audio sound very weak and you would have a loud hum. I can help you with that if it turns out to be the problem.
If the voltage on both the RCA leads is OK, then you need to check the amplifier.
You can put the meter lead on the ground of the RCA jack on the amplifier, if you have anything abnormal there, you have a problem in the amplifier. Be careful checking this because you can easily touch the lead to the chassis of the amplifier at the same time as the RCA ground. Most of the time, if yu have a problem in the amplifier like the one you are having, this check of the RCA ground will result in some DC voltage on it. If it still checks OK, then put the lead into the RCA ground and check the positive part of it. You may need to move the lead around a little to get it to touch the metal inside, it is on the bottom part of the hole inside the jack.
I think you will find that you have some voltage on the RCA inputs of the amplifier. To fix this properly you will need to take it in to be serviced.
Sometimes you can ground the RCA jack ground somehow with a jumper wire and then turn the system on with the RCA cable plugged in and it will seem normal. But you do not want to do that as a repair, only to see if the amp will play. If it does play, you know that the problem is in the pre-amp of the amplifier. That would be the only reason to connect an external ground to the RCA jack. You still have a problem in the amplifier. If it does not play normal, you still have a problem in the amplifier if there was some DC on the RCA jack.
Sorry this answer is so long, but the problem you are having will require service of one of your units, you just need to know which one it is. If your inputs get grounded, you will not have the amp go into protect, you just won't hear anything.
Let me know if you need more help and if this was helpful to you a good rating is always appreciated.