Question : Application pool suffered a fatal communication error with WWW PS

Urgent issue in production environment on webserver.
Server: Win2k3 Web Edt. SP2, IIS6, ASP.NET
Problem: Web applications restart/recycle often.

Below logs are from two seperate crashes. The information in the logs are the same for eveything crash except for the processid's which obviously changes.

From eventlog under system:
EventID 1011:
A process serving application pool 'DefaultAppPool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '5800'. The data field contains the error number.

Note Process ID here refers to W3WP.exe

Running MS Debug diaognastics tools gives out the following information leading up to a crash:

From DbgSVC log file:
18.06.2010 10:10:34] New process found:        Process Name - w3wp.exe             Process ID - 3796   Process Identity - NT AUTHORITY\NETWORK SERVICE             Web application pool name - DefaultAppPool
[18.06.2010 10:10:34] Attach Debugger:          Process Name - w3wp.exe             Process ID - 3796   Control Script - C:\Program Files\DebugDiag\Scripts\CrashRule_WebAppPool_DefaultAppPool.vbs
[18.06.2010 10:10:37] Process Exited:           Process Name - w3wp.exe             Process ID - 5800   Web application pool name – DefaultAppPool

From w3wp log file:
[18.06.2010 12:34:37] First chance exception - 0xe0434f4d caused by thread with system id 5736
Script Error
Error Code - 0x80004005
Error Source [Unavailable]
Error Description [Could not obtain System ID for this thread]
Line 116, Column 2

Script Error
Error Code - 0x80004005
Error Source [Unavailable]
Error Description [Could not obtain System ID for this thread]
Line 116, Column 2

[18.06.2010 12:34:37] First chance exception - 0xe0434f4d caused by thread with system id 304
[18.06.2010 12:34:37] First chance exception - 0xe0434f4d caused by thread with system id 304
[18.06.2010 12:34:37] First chance exception - 0xe0434f4d caused by thread with system id 304
[18.06.2010 12:34:37] First chance exception - 0xe0434f4d caused by thread with system id 304
[18.06.2010 12:34:37] First chance exception - 0xe0434f4d caused by thread with system id 304
[18.06.2010 12:34:37] First chance exception - 0xe0434f4d caused by thread with system id 304
[18.06.2010 12:34:37] First chance exception - 0xe0434f4d caused by thread with system id 304
[18.06.2010 12:34:37] First chance exception - 0xe0434f4d caused by thread with system id 304
[18.06.2010 12:34:37] First chance exception - 0xe0434f4d caused by thread with system id 304
[18.06.2010 12:34:37] First chance exception - 0xe0434f4d caused by thread with system id 304
[18.06.2010 12:34:38] First chance exception - 0xc0000005 caused by thread with system id 1520
[18.06.2010 12:35:40] Second chance exception - 0xc0000005 caused by thread with system id 1520

[18.06.2010 10:09:31] First chance exception - 0xc0000005 caused by thread with system id 5112
[18.06.2010 10:10:33] Second chance exception - 0xc0000005 caused by thread with system id 5112
[18.06.2010 10:10:33] Thread exited. Exiting thread system id - 3792. Exit code - 0xffffffff
[18.06.2010 10:10:33] Process exited. Exit code - 0xffffffff

From the report generated by the Debug diagnostics tools:
Report for w3wp__PID__3796__Date__06_18_2010__Time_10_26_34AM__343__Second_Chance_Exception_C0000005.dmp
Type of Analysis Performed        Crash Analysis
Machine Name        NOTSHARED
Operating System        Windows Server 2003 Service Pack 2
Number Of Processors        2
Process ID        3796
Process Image        c:\WINDOWS\system32\inetsrv\w3wp.exe
System Up-Time        12:40:41
Process Up-Time        00:16:02
Thread 17 - System ID 2860
Entry point        mscorwks!SVR::gc_heap::gc_thread_stub
Create time        18.06.2010 10:10:39
Time spent in user mode        0 Days 0:0:1.687
Time spent in kernel mode        0 Days 0:0:0.15

Function          Arg 1          Arg 2          Arg 3        Source
mscorwks!SVR::gc_heap::mark_object_simple1+1d9          16813aac          00000001          0015e750       
mscorwks!SVR::gc_heap::mark_object_simple+25a          16813a20          00000001          0224fd8c       
mscorwks!SVR::GCHeap::Promote+115          14817cc8          16813a20          00000000       
mscorwks!WKS::CFinalize::GcScanRoots+42          79fd00d5          00000001          00000000       
mscorwks!SVR::CFinalize::ScanForFinalization+10f          79fd00d5          164b87b8          00000000       
mscorwks!SVR::gc_heap::mark_phase+1fd          00000002          00000000          0015e750       
mscorwks!SVR::gc_heap::gc1+46          00000000          ffffffff          0015e750       
mscorwks!SVR::gc_heap::garbage_collect+246          00000000          00000000          0015e750       
mscorwks!SVR::gc_heap::gc_thread_function+6a          00000000          7c8276dd          0224fef8       
mscorwks!SVR::gc_heap::gc_thread_stub+92          0015e750          00000000          00000000       
kernel32!BaseThreadStart+34          79f23b1c          0015e750          00000000       


MSCORWKS!SVR::GC_HEAP::MARK_OBJECT_SIMPLE1+1D9In w3wp__PID__3796__Date__06_18_2010__Time_10_26_34AM__343__Second_Chance_Exception_C0000005.dmp the assembly instruction at mscorwks!SVR::gc_heap::mark_object_simple1+1d9 in C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000080 on thread 17
Module Information
Image Name:      C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll        Symbol Type:       PDB
Base address:      0x79e70000        Time Stamp:       Tue Oct 20 08:10:14 2009
Checksum:      0x0059659f        Comments:       Flavor=Retail
COM DLL:      False        Company Name:       Microsoft Corporation
ISAPIExtension:      False        File Description:       Microsoft .NET Runtime Common Language Runtime - WorkStation
ISAPIFilter:      False        File Version:       2.0.50727.3607 (GDR.050727-3600)
Managed DLL:      True        Internal Name:       mscorwks.dll
VB DLL:      False        Legal Copyright:       © Microsoft Corporation. All rights reserved.
Loaded Image Name:       mscorwks.dll        Legal Trademarks:       
Mapped Image Name:               Original filename:       mscorwks.dll
Module name:       mscorwks        Private Build:       
Single Threaded:       False        Product Name:       Microsoft® .NET Framework
Module Size:       5.56 MBytes        Product Version:       2.0.50727.3607
Symbol File Name:       c:\symcache\mscorwks.pdb\8FCB438097C9463E95E6907ECDE0C06A2\mscorwks.pdb        Special Build:       &


All help greatly appreaciated as this is a production server.

Answer : Application pool suffered a fatal communication error with WWW PS

I think you will struggle to get anyone to look at/understand the DebugDiag it really is a MS product support case. It looks likely that it is acode issue with your asp.net code.

To troubleshoot this without full understanding of teh DEbugDiag results have a look at what is happening at the time.

At teh mimum seperate off your app into different app pools. So you can find out what is occuring.

Look at the last entries in the IIS logs that can give you some clues about what pages that may cause teh error.

I find procmon a useful initial debugging tool also. Once you are able to recreate the error then this will be even more useful. But use it to compare working and failed states.