What file compression application is the most space-efficient?

"the program must run on OS X

I'd say your best bet would be any one of the 7-Zip derivatives for Mac OS X.

7-Zip is free open source software and the compression ratio is marginally better than Rar.

StuffIt also does compress very efficient, however, the price of $80 is rather steep, for that money you'll get a 1 TB hard disk. :)

What file compression application is the most space-efficient?

Here's an extensive test comparing about 250 different archiving programs (not all of them available for Mac OS X though) in various scenarios:

Lossless data compression software benchmarks / comparisons"

A friend found that if you were trying to compress large numbers of small files then a huge reduction in the size of the final compressed file 50%? could be achieved if all the small files were concatenated first. He even wrote a utility to concatenate/de-concatenate for this purpose. Unfortunately his software is no longer available. But there may be modern software that could do this.