According to the W3C Systeam’s blog, there’s a lot of poorly designed software out there. It’s pretty rare that something has a legitimate need to pull down a DTD in order to work. They should never be requesting it on a very frequent basis. It’s a very cachable asset. The post includes some pretty impressive stats too:
..up to 130 million requests per day, with periods of sustained bandwidth usage of 350Mbps, for resources that haven’t changed in years.
They also make a few requests which really all developers should follow. Here’s my summary:
- Cache as much as possible, to minimize your impact on others (not to mention improve your performance).
- Respect caching headers
- Don’t fetch what you don’t need
- Identify yourself. Don’t use a generic UA.
- Try not to suck.