I have a shedload of OTA HDTV video that I want to compress to make room for more recordings. Can anyone recommend some good settings for Handbrake/x264 that result in a nice balance of quality, space, and encoding time?

What's the hardware on your system?

The thing that annoys me the most is artifacting.

Also, was the video captured in digital format and saved as-is, or was it transcoded when you received it? If it was captured digitally, then you still have the MPEG-compressed audio stream, so re-encoding the audio stream in an MPEG format (such as mp3), even at a lower bitrate, will introduce less transcoding error than if you use an otherwise superior audio codec like ogg vorbis. Just make sure you're keeping the same sample rate as in the original data stream.

There's no principled way of choosing the bitrate, but there are many intelligent things you can do with the encoding options. H.264 does basically the same thing that MPEG2 does to the video, but then it does a lot more beyond that, so you're unlikely to get artifacts if you're not changing the framerate or resolution. The same principle applies to the audio, which is why I suggested keeping it in MP3 format, rather than transcoding. You might even want to keep the audio stream as-is without recompressing at all, depending on how you feel about the audio quality.