This refers to the technique of deciding whether an application is malicious or not, according to what it does. If an application does something that falls outside the range of ‘acceptable’ actions, its operation is restricted. For example, trying to write to certain parts of the system registry, or writing to pre-defined folders, may be defined as a threat. The action can be blocked, or the user notified about the attempted action. This fairly simple approach can be further refined. It's possible, for example, to restrict the access of one application (let's say allowing a web browser read-only access to limited portions of the system registry) while giving unrestricted access to other programs that do not use the Internet.
An alternative behavioral method is to 'wrap' a downloaded application and restrict its action on the local system. Here the application is run in a protective 'sandbox' [sometimes called a ‘playground’, or ‘secure cache’] to limit its actions according to a pre-defined policy. The activity performed by the program is checked against a set of rules. Depending on the policy, the program’s actions may be considered a violation of the policy, in which case the rogue action is blocked.