I did a search in the forum and found another guy with the same problem with his RT-N16.
My case is, I flashed my WL-520GU with tomato-NDUSB-1.28.8754-Ext. everything looks fine. then I install transmission following the instructions. when editing the setting.json file, I got the error "vi: memory exhausted" and the putty window frozen. On the other tries, I tried open vi without open any files, it works. I also try the same thing in my RT-N16, it also work fine. My guess is for WL-520GU, the remaining memory is really limited after I flash the version tomato-NDUSB-1.28.8754-Ext. But why the swap space in the hard disk isn't used (I used the tutorial and attach an external HDD to 520GU's usb. I format the HDD with two partitions, one linux swap one ext3)? Will this memory issue have some impact if i want to install other stuff like plowshare? It also bother me since vi is the only one I know to edit the file under linux. If it didn't work, I have to transfer the file back and forth (download the file, do the edit and transfer back).
Sorry if this question is too basic. I really try to search similar issue but found nothing. Thanks!