Category: Networking

Your computer and mine talking to each other

Categories
Networking Open Source Web Development

Yahoo Traffic Server Open Sourced

Way back in 2002 Yahoo acquired Inktomi who was largely know for their search products. Their software powered some early search engines like HotBot in the pre-Google days. One of their lesser known products was something called Traffic Server. Even if it was lesser known it was still used by ISP’s including AOL, who in those days was big. Their business disappeared with the great bubble and they were acquired by Yahoo, who was using Traffic Server themselves ever since.

Fast forward to 2009. Yahoo is now in the process of opening up Traffic Server as an Apache project. It’s already in incubator. Yahoo says it’s capable of 30,000 requests per server. Noteworthy is that this runs on generic hardware.

These days most websites use either Squid, Nginx, Pound or Varinish on the open source side. On the proprietary side there’s Citrix NetScaler, Foundry (now Brocade) ServerIron, Zeus ZXTM or F5’s Big-IP. The proprietary side can be either expensive software running on generic hardware or an appliance (which is generally a Intel based server with a custom modified Linux install for low maintenance and top performance).

At this point it’s apparently not 64-bit and doesn’t have native IPv6 support. However it appears to be usable and likely competitive with some of the other stuff out there already. Yahoo has been using it all along, and I hear they are pretty popular (problems aside).

It should be noted that commercial CDN’s aren’t really an alternative for reverse proxy or load balancer since they still require a robust and redundant origin. If anything they will reduce your requirements, not eliminate them.

Given everyone’s interest in scaling computing quickly and cheaply this is pretty noteworthy open source event. It tends to be an afterthought but these applications can be critical. Squid handles 78% of Wikipedia’s requests. Given all their traffic, you can see how it matters.

It will be interesting to see if a community builds around Traffic Server and if it sees adoption.

Categories
Networking Open Source Software

Remove Spikes From RRDtool Graphs

I use RRDtool to make graphs on various things I monitor like server stats, network stats and it does a relatively good job. My one (big) complaint is that when you restart you occasionally see these gigantic spikes that completely mess up the data. I’ve even seen spikes larger than what the system can technically handle.

Nobody mentioned there’s a removespikes.pl script (download) that will remove these outliers from your rrds. I put together a quick shell script to make it quick for when I need to run it again:

!/bin/sh
 
for i in /path/to/graphs/rrd/*;
do
        perl removespikes.pl $i;
done;

If you have a ton of graphs a quick shell script to iterate through the directly may be quicker. If you only have a handful like me, no big deal.

Keep the script around for the next time you have spikes to deal with.

Categories
Networking

802.11n Finalized

802.11n, something I was starting to think would never get beyond draft is now approved. Having suffered through “compliant” 802.11b devices I long ago decided wireless networking is fussy enough to warrant stricter standards. As a result I stuck to Wi-Fi Alliance certified 802.11g devices, and the results have been awesome. I’m still of the opinion that the difference between “compliant” and “certified” is gigantic. Certified 802.11n devices should start to appear in the next few months.

Looks like the goals for any 802.11n upgrade are MIMO (obviously) and preferably dual-band (2.4GHz and 5GHz). I can’t see why I would want to do anything otherwise.

Considering most ISP’s don’t yet provide the downstream or upstream bandwidth necessary to take saturate a good 802.11g network, I’m not sure it’s really necessary to upgrade just yet. Thanks to a solid signal I can sustain up to about 19 Mbps over 802.11g even with WPA2 overhead and slight signal degradation. Under 1ms pings as well. ISP currently offers up to 16 Mbps, 12 Mbps plans for mortals. Rarely is that performance actually seen thanks to “the Internets being a series of tubes”. At least for today upgrading would only improve local network performance, not Internet performance. Most traffic is going outside the network anyway. 802.11n would bring capacity up to 130 Mbps, but since the uplink is still 12 Mbps, that really provides no real performance boost.

For anyone who would argue the faster CPU’s on the newer access points would improve performance, I’ve found that my current AP rarely sees more than a 2% load, with rare spikes up to about 40% capacity.

Of course hardware providers, and retail outlets will continue to tell people that downloading will be 6X faster1, but logic and common sense proves otherwise. It’s the equivalent of a Bugatti Veyron stuck behind a funeral procession.

That of course also assumes all devices are connecting via 802.11n. If you have an 802.11g and 802.11n devices connecting over 2.4 GHz, you’re going to be in mixed mode and slow down while 802.11g devices send/receive anyway. As far as I know there’s no way around that.

Then there’s the issue of all the pre-N adapters sold in laptops over the past few years and their compatibility, which is generally pretty good, but not perfect when mixing vendors.

So despite the marketing getting even stronger, I don’t see how it would be really beneficial to upgrade just yet. The actual performance increase for most activity will be virtually non-existent until ISP’s get faster. I’d rather wait until the hardware matures and prices drop more.

1. up to 6X faster, actual results may vary.

Categories
Google Internet Networking

Measurement Lab

Google today unwrapped Measurement Lab (M-Lab) which can measure connection speed, run diagnostics and check if your ISP is blocking or throttling things via it’s blog.

In general it’s a good idea, but it’s nothing new. Sites like dslreports.com and SpeedTest.net have been measuring speed and providing diagnostics for years. The BitTorrent test however isn’t replicated by many yet.

One thing that isn’t addressed is how they will detect if an ISP is adjusting their routing to handle M-Lab servers specially. What stops an ISP from not throttling from one of Google’s “36 servers in 12 locations in the U.S. and Europe” but throttling all other data? Perhaps Vint Cerf and friends have a plan in mind but it seems to me this could be a cat and mouse game.

Categories
Apple Networking Open Source

Compiling RRDtool 1.3.x On Mac OS X

I’ve recently been trying to upgrade RRDtool to 1.3.x as I’ve previously been using 1.2.x. New in 1.3 is moving to Cairo graphics, which is pretty cool and provides much better anti-aliased text. MMAP IO should also help since I run it on an old box. It’s a worthwhile upgrade.

Continue reading “Compiling RRDtool 1.3.x On Mac OS X”
Categories
Networking

Network Perils

It’s been a week of networking pain. For the past few weeks Comcast has been using a low DHCP lease time. 30 minutes to be exact. This is typical of when they are doing network upgrades/repairs and is what a normal network administrator does. It’s similar to lowering the TTL for DNS.

Renewing the DHCP lease is normally a pretty transparent process, but this time around it’s been causing network connections to drop. While this process is relatively quick, it still resulted in a brief network outage that would kill connections. Occasionally it created a spike when things came back online, which made it feel even longer due to the resulting lag.

No configurations have changed in a long time other than a firmware upgrade a few months ago. Strange I thought. Why is it insisting on on loosing the IP and rediscovering, rather than just renewing? I let it go for a few days hoping Comcast’s low DHCP lease time would be temporary. After 2 weeks I decided to dig deeper.

After a few emails with Motorola’s tech support (Motorola bought Netopia in 2007) they came to the conclusion that the renew-lease ACK packet wasn’t reaching the router. They suspected the firewall wasn’t allowing it to pass, as a result it was dropping the IP, and requesting it again from the DHCP server. They suggested opening up UDP/67 and UDP/68 on the firewall. This seems to resolve the problem. I’m still seeing the lease drop at about 1:00 AM for the past 2 nights, but that’s really a low priority issue and may indeed be on Comcast’s end. It’s possible the router was renewing the hard way every 24-72 hours for a few months, but I know Comcast’s DHCP lease time has been lowered before and the router didn’t exhibit this behavior. Perhaps the firmware upgrade changed the firewalls behavior? I don’t recall that in the docs. Regardless, it’s fixed.

Now today, the UPS for the router, modem and file server’s battery died. Yet another pain. I was able to swap the battery with a similar model UPS from another computer for now. I can deal with that other computer later.

Now maybe I can take my networking hat off for a little while.

Categories
Google Networking Spam

Google Mail Fail

Found an interesting header when doing some tests with mail filtering:

Received: from qb-out-1314.google.com ([172.21.30.5])
        by mx.google.com with ESMTP id k29si2692710qba.7.2008.09.06.14.48.05;
        Sat, 06 Sep 2008 14:48:06 -0700 (PDT)
Received-SPF: softfail (google.com: domain of transitioning user@example.com does not designate 172.21.30.5 as permitted sender) client-ip=172.21.30.5;
Authentication-Results: mx.google.com; spf=softfail (google.com: domain of transitioning user@example.com does not designate 172.21.30.5 as permitted sender) smtp.mail=user@domain.tld
Received: by qb-out-1314.google.com with SMTP id d5so1543676qbd.6
        for <destination@example.com>; Sat, 06 Sep 2008 14:48:04 -0700 (PDT)

See the problem? Look closely. In particular look at this line:

Received-SPF: softfail (google.com: domain of transitioning user@example.com does not designate 172.21.30.5 as permitted sender) client-ip=172.21.30.5;

Look at that IP. RFC 1918 states the “20-bit block” (172.16/12) is for private internets. Google is softfailing emails because it’s sent through it’s own mail servers. Google’s own SPF record looks like this:

;; QUESTION SECTION:
;_spf.google.com.               IN      TXT

;; ANSWER SECTION:
_spf.google.com.        292     IN      TXT     "v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ?all"

I really don’t understand why Google is doing this. They should have their SPF checker whitelisting mail sent from their own servers. SPF is intended to verify the sender. When sent locally it’s pointless and can only be harmful. They can still do other spam checks.

From what I can tell, this seems to happening about 50% of the time, meaning this is something deployed on some but not all Google clusters.

Categories
Networking Politics

Nobody Is Using IPv6

Arbor Networks found that almost nobody is using IPv6 (a peak of 0.012% to be exact). Not exactly shocking.

This is due to a chicken or the egg problem:

  • ISP’s don’t give out IPv6 addresses because the majority of their customers can’t handle it. Modern operating systems support IPv6, but these days most people use broadband routers, which only support IPv4. As a result most can’t use IPv6.
  • Hardware vendors that make routers and switches often don’t support IPv6 to keep costs low, and performance high. It’s not needed since most ISP’s don’t support it anyway, and that doesn’t look like it’s about to change anytime soon.

There are however a few ISP’s that have experimented with IPv6.

Of course another issue is that most websites don’t use IPv6, but I think that’s the easiest to fix. Since most servers are hosted in data centers with expensive routers that could be upgraded. Nobody bothers because it’s not much more than a novelty. Servers themselves use modern operating systems that can easily support IPv6.

So what will change this? A massive government push. Something along the lines of Digital TV transition. It would need to do the following:

  • Specify a date after which all hardware and software sold must be IPv6 compatible.
  • Specify a date after which all ISP’s with more than X customers, or a certain bandwidth level must support IPv6.

Unlike the Digital TV transition, there’s no real push to kill IPv4, so it wouldn’t be so bad if it died a natural death like Gopher and just became antiquated and disappeared. DTV is different since the space freed up can be auctioned for large sums of money, which is the real incentive for the switch anyway.

Will it happen? I doubt it.

The Olympics is available via IPv6 (more info here). China has a better IPv6 plan since their growing population means they see the need for more IP space. Not to mention the US has a much higher allocation than China.

The DoD as well as the US Government in general has been moving to IPv6, but they have yet to make any real push for the private sector.

Until the US Government realizes a push is necessary it’s not going to happen. To bad. I’d love to point a domain name at a toaster. I’d love even more to get rid of NATs, since they are a nightmare for software to work with.

Categories
Networking

WMM Slowdown

I turned on Wireless Multimedia (WMM) support the other day on my wireless network, figuring QoS for a wireless network would pretty much be a slam dunk. For those who don’t know, the four access categories it uses are:

  • voice
  • video
  • best effort
  • background

I was surprised to find, at least with the Netopia box that this actually resulted in a significant slowdown in http traffic, even when there was no other services being used. To put some numbers out there, we’re talking 10000 kbps with it enabled vs. 17400 kbps when disabled (these aren’t scientific, they are just bandwidth tests). I think the performance hit negated any real benefit, at least in this case. The box doesn’t handle much VoIP, so it really doesn’t do much. Video is more about raw bandwidth these days than latency thanks to CDN’s becoming more common and reducing the bulk of the latency issue. Also interesting is that the CPU hit seems pretty minimal. Daily average increased from 2% to about 4%, it’s double but really nothing serious. With it enabled it never spiked past 50%, and that was only one time.

So after a few days testing, WWM is turned off. Seems QoS at least in this case doesn’t pay. I can’t complain, wireless performance (20Mbps+) and signal strength are fantastic (when the microwave isn’t on) for an 802.11g network. Despite that, there’s always the desire to find ways to make it even better. Next step would be 802.11n, but I have a thing against uncertified gear. Once it’s standardized, I’d strongly consider it, especially if I can find a device that supports Linux firmware.

Experiment complete.

Categories
Networking

Goodbye MRTG

I’ve finally got just about all the network graphs I maintain using RRDtool rather than MRTG. I started doing this since MRTG isn’t good for graphing things with more than 2 sets of numbers. I started doing this because I wanted to track cable modem data. I figured while I’m at it, I should move other stuff too.

So much easier to read, and so much faster since your not spitting out graphics files every time the system polls. Not to mention the quality of the graphing image is much nicer. These are shrunk a little and they still look great.

Traffic

Ping

MRTG is still running, but I think I’ll shut it down as soon as I can verify things are working the way I want.