This is all programmed in the phone system settings. Your phone can be used on different Avaya phone systems (AT&T, Lucent is Avaya now) and you can also have different voice mail systems. They all program differently.
Sounds like your phone port is not programmed to use voice mail. You can have your phone port programmed to do many different things if nobody answers. Mostly you have the call transfered to another station after so many rings. That other station (extension) could be the operator, a calling group, a voice mail box, etc... Some systems allow you to change the number of rings on a per-phone basis, others only have one no-answer timer for all calls.
You can get all the manuals from www.avaya.com
for free but you need to know which system and version you have first. Programming can get very complicated and you may find it easier to hire a local phone installation company that is familiar with your system to do the changes for you.