TomatoUSB

Router: E3000
Build: tomato-E3000USB-1.28.9054MIPSR2-beta-Ext

I've been experimenting with the dnsmasq adding more upstream DNS servers and using the "all-servers" directive in my configuration. Specifically, I added the following in the dnsmasq custom configuration box:

server=8.8.8.8
server=8.8.4.4
server=208.67.222.222
all-servers

My ISP will provide 3 DNS servers on DHCP, so this will result with a total of 6 DNS server for dnsmasq to use. In addition, the "all-servers" option will force dnsmasq to send queries to all 6 and use the result from the first server that replies.

I started noticing that PCs in LAN, making DNS resolution against dnsmasq on the router will intermittently get no IP address on query results. This was not an issue with any of the upstream DNS servers because queries done directly against them had no issues.

I further installed tcpdump in a thumb drive and ran some packet capture. Dumps from vlan2 (WAN) showed that the router received 6 DNS replies on every single request made from a LAN PC. However the corresponding dumps on vlan1 (LAN) showed that dnsmasq will occasionally send a blank DNS response back internally.

This symptom seems to occur only with "all-servers" option and if dnsmasq has more than 4 upstream DNS servers to query for results.

Why are you using "all-servers" at all? That's a mis-use. Unless I'm missing something.
Dnsmasq does a very good job all on its own. Just put your server lines in the config and let it work its magic.

My ISP's DNS has been flaky and their tech has refused to acknowledge the problem. Occasionally, one of their DNS servers may timeout, or refuse a recursive lookup (clearly logged by dnsmasq).

However I'm not ready to give up using ISP's DNS, thanks to wide use of Akamai and Limelight content delivery network. If I use my ISP's DNS, I will get one of their servers in-ISP or in-country, which is very fast for downloads and videos. If I used Google or OpenDNS, I will usually get an out-of-country server.

So the reason for the "mis-use" of "all-servers" is to have my router use the results from my ISP (usually fastest) and if they die, my router will pickup the results from either Google or OpenDNS. This configuration works perfectly well, if I had only 1 server= line in the configuration, resulting with just 4 upstream DNS for dnsmasq. The problem happens when there are 5 or more servers (3 from DHCP, 2 from server= lines)

In a perfect world, I'll be happy with the "default" settings. Unfortunately I'm on contract with an ISP running a bad DNS and (lack of) DNS query results will break the Internet experience.

Dnsmasq periodically issues a query to all the servers to re-select the "best" one, regardless of which one it had previously chosen as best. "Best" = first one to reply. "Periodically" = pretty danged often. IIRC every ~50 queries and every few dozen seconds. Or, of course, if the currently selected server times out.

All-servers just makes it do that EVERY time. A total waste of resources. You can watch what's happening by doing "killall -USR1 dnsmasq", and then look at syslog. It will show the overall stats and the query counts for each server. I have a cron job that does this every hour.

Here's a great utility to show you which public servers are best from your location:
http://code.google.com/p/namebench/

[[edit]] Here's what I get on my main router for the last two hours:
cache size 1500, 0/1512 cache insertions re-used unexpired cache entries.
queries forwarded 588, queries answered locally 89
server 8.8.8.8#53: queries sent 171, retried or failed 0
server 156.154.71.1#53: queries sent 417, retried or failed 2
server 208.67.222.222#53: queries sent 168, retried or failed 0
server 74.82.42.42#53: queries sent 190, retried or failed 0
server 192.168.1.254#53: queries sent 314, retried or failed 1

my requirement is to use ISP provided DNS if available but fall back to Google/OpenDNS when ISP DNS is down. it is important for me because services like netflix use the closest CDN to my home to stream video. with OpenDNS, i am not sure if that can be accomplished. so, was wondering, if OpenDNS/GoogleDNS can be configured in such a fashion where they come into play only when my ISP provided(DHCP) DNS servers are not functioning.please let me know how to go about including the same in tomato in a clean way.

@E2000user, in GUI Basic>Network>LAN add the non-ISP DNS to Static DNS and then in Advanced>DHCP / DNS check Use received DNS with user-entered DNS.

i have tried that in the past and here is what i noticed with the suggested settings.
when you add anything to the staticDNS, then it gets added to the DNS servers from ISP(static DNS shows up first in the list when you do Status->Overview). Notice here though that the ordering cannot be enforced(in fact i have noway to tell dnsmasq to use my ISP DNS and staticDNS in that order and NOT based on optimization that dnsmasq might make based on what is the fastest DNS server to use.(i also tried adding the OpenDNS in custom config and strict-order clause.now, when i try adding some junk.com in my browser it takes me to openDNS search page as opposed to my ISP search page)

No, it doesn't work that way.

E2000user, you don't need to do what you are asking how to do.

Dnsmasq is very smart and will "do the right thing" automatically. So automatically that you would have to grossly misconfigure it to force it do what you ask.

Just add "server=8.8.8.8" to the Dnsmasq Custom configuration box in Advanced>DHCP / DNS. That's all you need to do.

Here's the key item: "dnsmasq will send queries to any of the upstream servers it knows about and tries to favour servers that are known to be up"
and
"The reply from the server which answers first will be [used]."
Also, if it has more than one server, it will occasionally send a query to all of them. It also does this at startup and if the "favoured" server doesn't respond. Whichever server responds first, that's the one that is marked as favoured.

You really, really can't control how the dns servers will go about doing their thing. For example, with my old DSL modem (R.I.P.) the servers set up via DHCP were 209.68.51.4 and 174.137.125.73 (which are att.dns1.com & att.dns2.com). My new DSL modem says the server is 192.168.1.254. Which is itself!
Guess what! My modem has dnsmasq (or the equivalent) running inside itself. I could configure my router's dnsmasq all day long, but the modem is going to place dns queries to whomever it wants.
In fact, just about any major open dns server is not really one computer in one location. There are lots of servers with the ip address of 8.8.8.8 scattered all over the country & the world. Your queries to 8.8.8.8 get automatically & invisibly directed to the preferred computer, which generally means the one that is closest to you.

hi ray123, thanks. understand the theory. just hope in practice too it gets me back a CDN nearest to me every single time. also used namebench based on your suggestion above. pretty nifty.helped me put it in fastest "secondary" DNS.(just a fallback when my ISP DNS stops working)

To see what's going on, do "killall -USR1 dnsmasq" and then "tail /var/log/messages"
It will show you all the servers it has queried, and the counts.

just hope in practice too it gets me back a CDN nearest to me every single time.

It does a damn good job. I looked at the code and was impressed at what it does. Very elegant. And regardless of how fast or slow the favored server is, every X seconds or Z queries, it sends a query to *all* the servers it knows about. And always, whichever one responds first, that's the one that gets designated the new favored one. So "nearest" is deemed to be "closest in time" — which is as good as you can get in network space.