Race condition in ControllerActionInvoker causes NullRef

[oofpez]

An unhandled exception occurs very rarely in this WebApp hosted in Kestrel.
I have attached a section of the stacktrace - anything in between is just rethrowing of the exception.

Please let me know if this is an issue inside Mvc or if I should be handling it elsewhere.

 Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware
	An unhandled exception has occurred: Object reference not set to an instance of an object.
	System.NullReferenceException: Object reference not set to an instance of an object.
    at lambda_method(Closure , Object , Object[] )    
	at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__27.MoveNext() 
	--- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    
	at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
	at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__25.MoveNext() 
	--- End of stack trace from previous location where exception was thrown ---    
	at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)    
	at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)    
	at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextResourceFilter>d__22.MoveNext() 
	--- End of stack trace from previous location where exception was thrown ---    
	at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ResourceExecutedContext context)    
	at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)    
	at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__20.MoveNext() 
	--- End of stack trace from previous location where exception was thrown ---    
	at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    
	at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
	at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext() 
	--- End of stack trace from previous location where exception was thrown ---   

...more stuff...

 --- End of stack trace from previous location where exception was thrown ---    
	at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)    
	at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    
	at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6.MoveNext()Null'

[Eilon]

@oofpez do you have anymore info on how to reproduce this?

@rynowak any idea what this could be?

[rynowak]

That's this code https://github.com/aspnet/Mvc/blob/rel/1.1.2/src/Microsoft.AspNetCore.Mvc.Core/Internal/ControllerActionInvoker.cs#L1233

What comes to mind is if there is some kind of crazy unanticipated method signature.

Sporadic says that either there's one rarely used method that's causing issues or it's a race condition

[rynowak]

Actually, I spaced on this part lambda_method(Closure , Object , Object[] ) - that's the generated expression thunk we use to call the controller method.

[oofpez]

I can't reproduce it, I only got this error occasionally when running parallel integration tests on some compression middleware. It might be some kind of race condition?

The error is rethrown at await _next(context) where _next is the RequestDelegate.

            using (var buffer = new MemoryStream())
            {
                var body = context.Response.Body;
                context.Response.Body = buffer;

                try
                {
                    await _next(context);

                    using (var compressed = new MemoryStream())
                    {
                        using (var gzip = new GZipStream(compressed, CompressionLevel.Optimal, leaveOpen: true))
                        {
                            buffer.Seek(0, SeekOrigin.Begin);
                            await buffer.CopyToAsync(gzip);
                        }

                        context.Response.Headers.Add(ContentEncodingName, new[] { GzipEncodingType });

                        if (context.Response.Headers[ContentLengthName].Count > 0)
                        {
                            context.Response.Headers[ContentLengthName] = compressed.Length.ToString();
                        }

                        compressed.Seek(0, SeekOrigin.Begin);
                        await compressed.CopyToAsync(body);
                    }
                }
                finally
                {
                    context.Response.Body = body;
                }
            }

[fracrtio]

Getting same exception:

66.249.64.154  GET /file/163423/ - An unhandled exception has occurred: Object reference not set to an instance of an object. System.NullReferenceException: Object reference not set to an instance of an object.
   at lambda_method(Closure , Object , Object[] )
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeActionMethodAsync>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextActionFilterAsync>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.<InvokeAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Site.Web.Security.CookieAuthenticationMiddleware.<Invoke>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Site.Web.Security.ApiKeyAuthenticationMiddleware.<Invoke>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6.MoveNext() 

Occurs randomly when executing random action. One of such actions looks like this:

[ HttpGet ]
[ Route("file/{file_id:int:required}") ]
public async Task<IActionResult> Default(int file_id, int snapshot_id = 0, string note = null) {
    ...
    return View("Default", model);
}

Maybe at ControllerActionInvoker.InvokeActionMethodAsync controller is null or became null somehow before calling

result = _executor.Execute(controller, orderedArguments);

or

result = await _executor.ExecuteAsync(controller, orderedArguments);

BTW using autofac as IServiceProvider but is seems no matter because controller is being created by DefaultControllerActivator

[rynowak]

I suspect that the issue here is a race condition in https://github.com/aspnet/Mvc/blob/rel/1.1.2/src/Microsoft.AspNetCore.Mvc.Core/Internal/ObjectMethodExecutor.cs#L219

It's possible to observe partially-initialized data when a thread has initialized _parameterDefaultValues to a non-null value.

[rynowak]

Keeping this bug to track the 2.0.0 issue, will add separate issues to port the fix to 1.0.X and 1.1.X