You've got the right idea. This table should be useful.
According to RFC 4594, the appropriate DSCP marking for "low-priority data" is CS1. This corresponds to tos=0x20 for rtorrent.
Assuming you're got Tomato's QOS configured and running, then to match that on mark, add
iptables -t mangle -I QOSO 3 -m dscp --dscp-class cs1 -j CONNMARK --set-return 0x105/0xff
to the custom Firewall script section under Admin.
- "-I QOSO 3" inserts this at the top of the QOS classifications. Change the number to something higher if you want certain other QOS rules to take precedence.
- "xt_dscp" is if you're running K26; change to "ipt_dscp" for K24.
- The final digit of "0x105" is what sets the classification to be used inside Tomato's QOS. 1 = Highest, 2 = High, …, 5 = Lowest, 6 = Class A, etc. My example sets it to "Lowest", but you might have your QOS configured differently.