Custom software is generally software (or pieces of software) that are not generally available to the public either for sale or for free download. This software is generally for a specific purpose.
Two examples would be:
A application written inside a company for their mainframe or AS/400 that is written by RPG developers employed by the company to enter accounting information.
A web site developed for a large online auction site (ebay or amazon) that was developed for the sole use of this company, which could have been developed inside the company or for the company by another company.