Custom Controller:

• It is an Apex class that implements all of the logic for a page without leveraging a standard controller.
• You can use only one Apex class.
• You can Use custom controllers when you want your Visualforce page to run entirely in system mode, which does not enforce the permissions and field-level security of the current user.

Extension:

• A controller extension is any Apex class containing a constructor that takes a single argument of type ApexPages.
• It provides additional functionality to a controller – either a standard controller or a custom controller.
• You can use multiple Apex classes separated by comma.
• Use controller extensions when:You want to leverage the built-in functionality of a standard controller but override one or more actions, such as edit, view, save, or delete.You want to add new actions.You want to build a Visualforce page that respects user permissions. Although a controller extension class executes in system mode, if a controller extension extends a standard controller, the logic from the standard controller does not execute in system mode. Instead, it executes in user mode, in which permissions, field-level security, and sharing rules of the current user apply.