The threat here is humorous, like the scare line at the end of a ghost story. The first line of this verse is an impossibility: we cannot give back to the Divine all that we owe, no matter how much we are tortured. This is why the second half is important because it limits the "torture" to the time when we do not let go of others. Note that neither "debts" nor "mistakes" are mentioned. Instead, holding something in our hearts against someone is equated to slavery.

Notice also that this mistake is personal, even though Jesus is addressing a crowd. This verse is an odd combination of second-person plural forms and third-person singular forms. It is a second-person plural statement with a third-person singular aside, "each person to that brother of his" in the third-person singular. The end is a relatively rare example of Jesus using the plural form of the Greek word for "hearts," probably because it follows this "each person" phrase. Most of the plural hearts we see in English are actually a singular Greek form. See this article.

The negative here is indicates a desire or opinion, "wanting to forgive," not something that doesn't happen.