Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After certificate renewal RRAS doesn't accept new connections until after a second manual service restart. #552

Closed
dobrosavljevic opened this issue Mar 23, 2021 · 6 comments

Comments

@dobrosavljevic
Copy link

We support a few Windows servers for small business and generally we implement Certify for certificate renewal on servers to support RRAS and SSTP specifically.

Generally these are single certificate environments that normally don't even require a certificate deployment task into RRAS. RRAS is generally configured to simply use the Default certificate that's tied to the default IIS site.

The RRAS service does require a restart after a certificate has been renewed and we've attempted to use the Deployment Tasks for this effort, however we still seem to be running into connectivity problems after a certificate was renewed and even though Certify shows as successfully having run the restart service task after it has renewed a certificate, we still need to go in and manually restart it again to get the server to accept new incoming connections properly.

We can either use the deployment task within Certify (by manually clicking the run task button) or go into the Services console to manually restart the service but generally that second manual restart makes the VPN connections work again after a certificate was renewed.

We've also tried using the Deploy to RRAS specific task during certificate renewal and that hasn't really made a difference either, usually we need to still go in after a certificate has been renewed to manually restart the service.

I would be happy to provide any additional data necessary to try and figure out why these second restarts are necessary. I would love to not to have to manually restart these services to make them work properly.

@dobrosavljevic
Copy link
Author

I should add that I know I can build my own script for these restarts and while I haven't tried that yet, I am not sure if that would make a difference, I'd be happy to try it if solving this isn't exactly a priority as I understand this is possibly an edge use case.

All I am saying is that we have run into a possible bug and I'd be happy to do the leg work to try and determine why this is happening in case others are running into the same problem as us.

@webprofusion-chrisc
Copy link
Contributor

Hi Igor, so are you using the built in deployment task to apply the cert or are you using your own script? This sounds similar to the issue here : #519 - as mentioned in that thread I think if the problem persists it would be worth asking Microsoft what the exact sequence of service dependencies is so that the correct things can be restarted (instead of restarting the whole server).

We do provide some example deployment tasks but for advanced usage your own PowerShell script is a good idea.

Here is the exact code for our basic Remote Access script. If it can be adapted to handle more situations/configurations that's great but you may also find it easier to make your own custom version:

param($result, [switch] $restartServices = $false)

Import-Module RemoteAccess

# get cert object to apply by thumbprint, assumes My cert store, TODO: could check Web Hosting if get null result
$Cert = Get-ChildItem -Path Cert:\LocalMachine\My -Recurse | Where-Object {$_.thumbprint -eq $result.ManagedItem.CertificateThumbprintHash} | Select-Object -f 1

if ( $restartServices -eq $true) {
	Stop-Service RemoteAccess
}

Set-RemoteAccess -SslCertificate $Cert

if ( $restartServices -eq $true) {
	Start-Service RemoteAccess
}

Here is also the source for our RDP Gateway script, which includes a restart of SSTP

# This is an example script and it will be overwritten when the next update is installed. 
# To use this script copy it to another location and modify as required

# Enable certificate for RDP Gateway
# For more script info see https://docs.certifytheweb.com/docs/script-hooks.html

param($result, [switch] $restartServices = $false)

Import-Module RemoteDesktopServices

# Apply certificate
Set-Item -Path RDS:\GatewayServer\SSLCertificate\Thumbprint -Value $result.ManagedItem.CertificateThumbprintHash -ErrorAction Stop

# Optionally restart TSGateway and related service

if ($restartServices -eq $true)
{

	Restart-Service IAS -Force -ErrorAction Stop
	Restart-Service TSGateway -Force -ErrorAction Stop
	Restart-Service SSTPSvc -Force -ErrorAction Stop
	Write-Host "Services Restarted."
}

@dobrosavljevic
Copy link
Author

Thanks for a quick response Chris.

So to clarify, for RRAS to work in our deployment scenarios it's not necessary to deploy the new certificate to RRAS. RRAS is set to use the "Default" certificate, meaning whatever is bound to the default site in IIS (which Certify already does wonderfully when renewing a certificate), all that's required is the restart of the RRAS service (and only the RRAS service) for it to accept new connections using that new certificate.

However we have also configured the Deploy Cert to RRAS task before as well with the same result. Since that didn't seem to make a difference we reverted back to simply leaving RRAS configured to use the Default certificate and us restarting the service manually.

So what happens often is after a renewal we get reports from end users that they are unable to connect using the VPN connections that we have configured for them. All that we then have to do is connect to the server in question and either run the "Restart service" deployment task within Certify manually (by hitting the play button on it) or opening up the services console and restarting the RRAS service manually.

I am not sure if the timing on when deployment tasks are run matters here (we usually don't hear about a problem at least a few hours after a cert was renewed) but the only reason I can imagine why us connecting after the renewal has completed and manually restarting the service works is because Certify automatically runs the Deployment Restart Service Task too soon or before something else has completed during the renewal process.

@webprofusion-chrisc
Copy link
Contributor

Hmm, so the deployment to IIS definitely happens before we attempt any of the deployment tasks. You can optionally add a Wait task to introduce a delay. Another idea might be to restart the service, perform a wait then restart the service again which sounds a bit hacky - which is why I think it might work :)

I'm not a windows administrator by trade so I don't know the subtleties of most of these services (with the exception of IIS which I know very well), so I'm relying on users with more relevant experience such as yourself to flag up issues like this. Together I'm sure we can get to the bottom of the issue.

@dobrosavljevic
Copy link
Author

I would be happy to do the legwork and help hunt this down because otherwise Certify has been invaluable in some of these deployments and I would love to be a part of making it slightly better.

Help me understand the tasks. If we have a task (like restarting a service) that runs before a certificate is renewed, does Certify wait until the service is properly fully started before it proceeds with the renewal or does it just send off the commands and renew the certificate in parallel?

And the last question I have is, the "Max. Wait Time (secs)" setting, is that to tell Certify how long to wait after renewal before kicking off the task or how long it waits before starting the service after it shuts it down? I am assuming it's the former but I wanted to confirm.

Bare with me on this as it takes a bit to replicate these issues as they only happen on renewal and some of these installations I've pushed the renewal out to every 60 days to allow for less frequent interruptions in service. I'll keep this thread updated as I try different configurations settings out.

@webprofusion-chrisc
Copy link
Contributor

webprofusion-chrisc commented Mar 25, 2021

Thanks! So you should definitely only be restarting a service after the certificate has renewed, not before.

We have two phases of task runs:

  • Pre-request (happens before renewal, generally used for things like temp firewall changes etc)
  • Deployment Tasks (previously called to Post-Request scripting) - these happen after the certificate request has run and after the cert has been stored in the local certificate store and applied to IIS (our default deployment mode).

Tasks by default trigger on success (so if the renewal fails the task shouldn't run) but you can also trigger tasks on error, both or manually (also via the command line). Manual tasks/command line tasks are useful for deployments that need to happen during a maintenance window or via a periodic manual process.

For the Stop, Start or Restart a Service task, the wait time is the maximum time we wait for the operation to complete, thereafter we continue as normal. We don't cancel the operation however so if a service is taking a long time to start then we assume it will eventually start.

As always I would encourage experimenting on a test system before settling on your production strategy. I know from the other thread that the exact nature of the remote access services restarts can be troublesome.

Here is the source code for that task: https://github.com/webprofusion/certify-plugins/blob/master/src/DeploymentTasks/Core/Providers/ServiceManager.cs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants