Your comment only outlines how to restrict access. That part is easy. The question was "how do you send the person to your custom login page when they click on that document link?"
Do you know how we go about that?
You can configure a front end login page url following these steps: docs.sitefinity.com/frontend-login-page
If the permissions for the documents are setup properly, Sitefinity will automatically redirect the user to the page configured as the frontend login page with a ReturnUrl.