config virtual directory to be served by an isapi extension I've built

"The URL you use should evaluate to a physical ISAPI DLL on the machine on all IIS versions before 6. Since IIS6, configuring ISAPI is done as an application : more orderly but also more complicated.

I'll need to know which IIS version you're using in order to give a better comment. However, there are also some very good tutorials and books available for ISAPI. A full treaty of the question would be rather long.

EDIT

A typical installation for IIS 5.1 on XP is:

Enable scripts

Open Control Panel / Add or Remove Programs / Add remove Windows components
Click on Internet Information Services / Details
Click on World Wide Service / Details
Check Scripts virtual directory
OK all the way thru.

Ensure Permissions

The above created the directory C:\Inetpub\Scripts. Now:

Copy your dll to this directory (I normally link it directly into there).
Open IIS Management, drill into the Scripts directory, right-click and then Properties, and ensure that Execute permissions are ""Scripts and Executables"".
In Windows Explorer ensure that in the Scripts directory the IIS user account has read and execute permissions (IUSR_).

Usage

Use the isapi extension by a URL of the format:
http://localhost/scripts/yourdll.dll?parameters

Some sources you might like to have a look at:

How to debug ISAPI DLLs in IIS 4.0, IIS 5.0, IIS 5.1, and IIS 6.0
Appendix 7: Debugging web application"