Recently, I've noticed that in ubuntu (6.06) dapper server with apache-2.0.55, apache by default listens to IPv6, thus was causing slow response times. The response times was much improved by having apache listen to IPv4 instead.
Edit /etc/apache2/ports.conf and specify an IPv4 address on all Listen directives:
Listen 0.0.0.0:80
Listen 0.0.0.0:443