Well, the records for the currently logged in users are stored in the sf_lic_user_activity table. You can delete the record for this user and try again.
This looks like an issue with the code of your custom provider. Basically for each membership providers there are some parameters like applicationName, etc. applicationName is actually importand and you should add it. Also in Administration -> Settings -> Advanced -> Security there are some settings like BackendUsersSessionTimeoutDescription - please check what are their values.