You can’t really modify your modem as all it’s doing is sending the data you are requesting. If there is a way it doesn’t seem logical as it would have to somehow encode the data and then decode it again which would slow your shit down quite a bit.
There are various ways you can go into a SSH session and tunnel all your traffic through SSL so it looks like it all came from a https:// address, but what a pain in the ass. That works well for browsing restricted websites but not so good for file sharing.
If you are downloading files like torrents, i assume most modern programs have built-in protocol encryption. I use utorrent and you can enable or even force the encryption which makes it so your ISP can’t see it as being bittorrent traffic. They will however see you flooding the shit out of a port and might assume it to be bittorrent but it will not be able to be shaped. I know azureus has the same thing.
Otherwise you can browse the web through the TOR network which won’t show up as standard packets and you can set a lot of things to use port 80 so they look like standard web traffic. Between doing these things I have never heard a peep from my ISP’s.