This post has been republished via RSS; it originally appeared at: IIS Support Blog articles.
We've encountered a few support cases where customers, usually after some time, have an ASP.NET Core application hosted in IIS in-process that starts returning this "HTTP 503 Server has been shutdown" status. In all these cases a restart of the application was needed to temporarily resolve the issue until it happens again, typically a few clicks around the application later or within 10-15 minutes. This post details the most likely cause of that problem and how to fix it.
As of the time of this writing I have come across 5 cases where this 503 issue was occurring. In 100% of those cases, the problem was redundant code. There were a few different ways of triggering the 503s, but the root cause was always the same.
What to look for?
In your application, look for any of the below methods being called multiple times and/or look for more than one of them being called (i.e. your app has a call to both #1 and #2):
- ConfigureWebHostDefaults, and/or
- WebHost.CreateDefaultBuilder and/or
- WebApplication.CreateBuilder, and/or
- explicit calls to UseIIS
- (potentially others I missed, but do similar things)
In 4 of the 5 support cases I've seen, it was only during app startup (in Program.cs and/or Startup.cs) where multiple of the above were being called multiple times. In one case the customer was calling them throughout their code during page execution for various reasons. In all cases, the problem was the same.
There is only meant to be one call of one of those methods, not multiple calls to one or any combination of them.
Thus, the first thing you should do if you see 503 Server has been shutdown, is to analyze your application code to ensure there is only one call above. Regardless of whatever happens that seems to "trigger" the 503s to start, checking for the above should be the first thing done.
Once the above has been rectified, redeploy and retest the application. There's a very good chance the issue will be resolved. If not then double- and triple-check your codebase to ensure a redundant call was not made elsewhere. Of course a support case can also be opened to get help with the investigation.
If you want more information on this problem, then feel free to keep reading into where that specific 503 originates and why it happens.
Deep Dive (optional read)
As of this writing, this 503 originates from the same place in all current versions of ASP.NET Core (6, 7, and 8 Previews).
It's worth noting that this 503 itself is sent from ASP.NET Core Module (ANCM;AspNetCoreModuleV2), which is installed into IIS as part of the .NET Core Web Hosting Bundle. That also means it only happens when running on IIS. It also only occurs when using the in-process hosting model.
Here is the link to the point where the 503 status code is set in 6.0.16 (all other links to code below will be for 6.0.16 as well, unless specified otherwise):
22 static REQUEST_NOTIFICATION_STATUS ServerShutdownMessage(IHttpContext * pContext)
23 {
24 pContext->GetResponse()->SetStatus(503, "Server has been shutdown", 0, HRESULT_FROM_WIN32(ERROR_SHUTDOWN_IN_PROGRESS));
25 return RQ_NOTIFICATION_FINISH_REQUEST;
26 }
Tracing calls backwards leads us here:
85 REQUEST_NOTIFICATION_STATUS IN_PROCESS_HANDLER::ServerShutdownMessage() const
86 {
87 ::RaiseEvent<ANCMEvents::ANCM_INPROC_REQUEST_SHUTDOWN>(m_pW3Context, nullptr);
88 return ShuttingDownHandler::ServerShutdownMessage(m_pW3Context);
89 }
That has multiple callers in the same code file:
44 else if (m_pApplication->QueryBlockCallbacksIntoManaged())
45 {
46 return ServerShutdownMessage();
47 }
and
69 if (m_pApplication->QueryBlockCallbacksIntoManaged())
70 {
71 // this can potentially happen in ungraceful shutdown.
72 // Or something really wrong happening with async completions
73 // At this point, managed is in a shutting down state and we cannot send a request to it.
74 return ServerShutdownMessage();
75 }
The QueryBlockCallbacksIntoManaged() call is defined here:
103 bool
104 QueryBlockCallbacksIntoManaged() const
105 {
106 return m_blockManagedCallbacks;
107 }
This is only set to TRUE in one place, in the same file:
72 void
73 StopCallsIntoManaged()
74 {
75 m_blockManagedCallbacks = true;
76 }
And there is only one place where this StopCallsIntoManaged is invoked:
521 EXTERN_C __MIDL_DECLSPEC_DLLEXPORT
522 HRESULT
523 http_stop_calls_into_managed(_In_ IN_PROCESS_APPLICATION* pInProcessApplication)
524 {
525 if (pInProcessApplication == NULL)
526 {
527 return E_INVALIDARG;
528 }
529
530 pInProcessApplication->StopCallsIntoManaged();
531 return S_OK;
532 }
This is a function used as a managed export, which means it is invoked from managed code.
The managed import of this function is here:
https://github.com/dotnet/aspnetcore/blob/v6.0.16/src/Servers/IIS/IIS/src/NativeMethods.cs#L71
70 [DllImport(AspNetCoreModuleDll)]
71 private static extern int http_stop_calls_into_managed(NativeSafeHandle pInProcessApplication);
And it's invoked by the managed method in the same file:
194 public static void HttpStopCallsIntoManaged(NativeSafeHandle pInProcessApplication)
195 {
196 Validate(http_stop_calls_into_managed(pInProcessApplication));
197 }
Which leads us to its sole caller here (on IISNativeApplication - this is the important class to remember):
29 public void StopCallsIntoManaged()
30 {
31 lock (_sync)
32 {
33 if (!_nativeApplication.IsInvalid)
34 {
35 NativeMethods.HttpStopCallsIntoManaged(_nativeApplication);
36 }
37 }
38 }
There are only 3 references to this IISNativeApplication.StopCallsIntoManaged() method. One of them is the IISNativeApplication finalizer (same code file as above):
71 ~IISNativeApplication()
72 {
73 // If this finalize is invoked, try our best to block all calls into managed.
74 StopCallsIntoManaged();
75 }
The other two invocations are in IISHttpServer.OnRequestsDrained(..) and IISHttpServer.Dispose() -- to date I have never seen these be called outside of a legitimate application shutdown nor causing the problems observed here. The problem in the cases I've seen has always been due to the Finalizer invocation above, so I'll focus on that.
So why would this IISNativeApplication be finalized? Let's go the other direction and see where that class is instantiated. There is only one place I found as of writing where this is done - in the WebHostBuilderIISExtensions.UseIIS(..) method here:
39 return hostBuilder.ConfigureServices(
40 services =>
41 {
42 services.AddSingleton(new IISNativeApplication(new NativeSafeHandle(iisConfigData.pNativeApplication)));
...
It's added as a singleton into the app's Dependency Injection container. This is critical as we'll see later.
The key idea here is this is inside the aforementioned UseIIS(..) method, which is public. If I recall correctly, older versions of ASP.NET Core had project templates where much of the setup-related stuff, including this call, was just part of the template code in Program.cs or Startup.cs as it had not been refactored into the convenience methods we use today. In current versions of ASP.NET Core, this UseIIS() method is invoked in the internal method Microsoft.AspNetCore.WebHost.ConfigureWebDefaults(..) which is invoked in the CreateDefaultBuilder method on the same code page, as well as Microsoft.Extensions.Hosting.GenericHostBuilderExtensions.ConfigureWebHostDefaults(..).
You might recognize these as various calls made in different ASP.NET Core templates over the years. Today, if using VS2022 and creating an ASP.NET Core 6 WebAPI with top-level statements, the very first line of Program.cs is this:
var builder = WebApplication.CreateBuilder(args);
All this does is invoke the WebApplicationBuilder constructor:
https://github.com/dotnet/aspnetcore/blob/v6.0.16/src/DefaultBuilder/src/WebApplication.cs#L106
101 /// <summary>
102 /// Initializes a new instance of the <see cref="WebApplicationBuilder"/> class with preconfigured defaults.
103 /// </summary>
104 /// <param name="args">Command line arguments</param>
105 /// <returns>The <see cref="WebApplicationBuilder"/>.</returns>
106 public static WebApplicationBuilder CreateBuilder(string[] args) =>
107 new(new() { Args = args });
This constructor calls ConfigureWebHostDefaults as noted above. So this means that, in the end, the WebApplication.CreateBuilder() call ultimately leads to a new IISNativeApplication instance being added to the DI container as singleton as shown earlier.
What happens when other code runs that leads to another new IISNativeApplication being instantiated and added as a singleton to the DI container? What appears to happen is the original singleton is ejected by the DI container and is left for finalization (i.e. the DI container does not dispose of it and just replaces the original singleton with the new one). When Garbage Collection runs at some point in the future and the finalizer for IISNativeApplication is invoked, the StopCallsIntoManaged() method as shown earlier above will trigger ANCM to start sending back 503s even though the application is technically still running.