Unless you're running out of disk space, there's usually no problem having the mods installed. They aren't used unless you start them. But if you want to remove them, you can go to C:\Users\USER\AppData\Local\Driftmoon and delete the whole directory. (replace USER with your username)
I am planning on including a button to delete mods in the game, just haven't found a good place for it yet.