The key difference between 'forgetting a member' and 'deleting a member' is that 'forgetting' allows all of the reports and statistics to remain in place, while at the same time removing personally identifiable information about the user. By 'deleting' a member's account, you are wiping the system of all of their information. 

Delete member is meant to be used sparingly and primarily for accounts created in error or during testing. Delete member removes the user record from the database as if it never existed, then removes that users' orders, their logs, their access rights. If you look in the database, there is a virtual hole where that user used to be. Additionally, when you delete a member from MemberMouse, any content in WordPress associated with their account will also be deleted, unless you have your WordPress install configured not to function this way.

Forget member is a more refined tool. It will randomize or remove personally identifiable user data while keeping the data structure intact, so that reporting, order and subscription data aren't affected. Additionally, it is possible to configure some of the settings to not be included in the Forget Member process, though all data types are enabled by default. The username, password, and IP addresses are always included in the Forget Member process. While the first and last name, email address, activity log, custom fields, and billing & shipping addresses can be excluded from the Forget Member process if you are required by law to keep those data types. WordPress content remains intact and if deletion is desired, this would have to be accomplished through WordPress directly.