For years Citigroup Inc. has been an anomaly among big banks, relying on an obscure piece of software it helped develop to manage loan payments. Just as the bank tried to replace it, things went terribly wrong.

The saga began emerging in court this week as the firm blamed human errors for mistakenly sending $900 million to a fleet of hedge funds reluctant to return it.