To remove rvm and its traces, you can use the rvm command itself:
rvm implode
This command will remove all traces of rvm from your system.