Which model Canon All-in-one do you have? What OS is on your computer? Wired or wireless connection - if wired USB, then try a new cable. Scanners often fail when the printer works due to a lack of bi-directional communication with the computer.
Did you install the Canon drivers (or full software) if available? These are available here: http://www.usa.canon.com/cusa/support/consumer
for supported OS versions for each model. Click on Printers and Multi-function then the product family and then the model. Make sure to select x64 if you have a 64-bit Windows OS. Not all Canon all-in-ones have support for scanning under the Mac OS.
For USB printers, if you plugged in the data cable before installing the Canon drivers, you usually get a generic driver that only runs the printer. If you have done this, then uninstall the driver and remove the data cable before installing the Canon software.
For networked printers, sometimes you need to connect the printer via USB and install additional setup files to get the scanner to work. (Not all scanners work over a network; with a wireless network, signal loss can be a problem.) Again it depends on the model - check the documentation.
Next are you trying to use the buttons on the all-in-one or launching the scans from the computer? If you are using the buttons on the scanner, you need the full software (MP Navigator for Windows) installed and a monitor program running on your machine. Otherwise, you will be limited to launching a scan from the computer. (Please note: you need to set up the destinations for the scan buttons in the software first; so scan from the computer at first to see what the best default settings are for most of your documents.)
I hope this helps.