WinDbg.info - Forum

Thread: Crash location !

patra04 — Fri, 13 Aug 2010 21:09:14 +0200

Hello Folks,

I have a crash file which gives me the following information.

================================================================================
An Exception Occurred:

Reason:
some.exe caused an EXCEPTION_ACCESS_VIOLATION in module some.dll at 001B:012C300B

Registers:
EAX=00910000 EBX=003425A0 ECX=012C7090 EDX=7C82860C ESI=012C7090
EDI=00000000 EBP=05CBF3C8 ESP=05CBF3B8 EIP=012C300B FLG=00010206
CS =001B DS =0023 SS =0023
ES =0023 FS =003B GS =0000

Call Stack:
001B:012C300B (0x00910048 0x009AE718 0x009AE6E0 0x009AE6EC) some.dll
001B:01A11027 (0x00910048 0x00000000 0x00000000 0x042CF0D8) one.dll
001B:0120834E (0x00910048 0x00941D58 0x00000000 0x00000000) two.dll
001B:0111DD9E (0x00000002 0x05CBFE34 0x05CBFA38 0x00000000) three.dll
001B:01113AC7 (0x05CBFA38 0x05CBFE34 0x05CBF9A4 0x00000001) four.dll
001B:011188BF (0x05CBFA38 0x00CBFE34 0x00000000 0x05CBFEC4) five.dll
001B:01104222 (0x05CBFA38 0x05CBFE34 0x00000000 0x05CBFEC4) six.dll
001B:0110B409 (0x074CA008 0x05CBFA38 0x05CBFE34 0x00000000) seven.dll

================================================================================

I started winDBG and attached it to the instance of the faulty process on my local machine. I have all the correct PDBs and symbol path is set correctly.

The above crash file also stated that some.dll was loaded at 012C0000.

In my running instance I can see that some.dll is loaded at 017a0000.

Can I use the following command in my winDBG session to locate the crash location?

0:007> ln 017a300B

012C300B - 0x012C0000 = 300B = crash location?
For me it should it 017a0000 + 300B = 017a300B?

Or do I have to use

0:007> ln 017a200B

012C300B - 0x012C0000 -1000 = 200B?

Or am I wrong in my approach itself?

Thanks for your help and support.

Regards,
Rahul

Thread: Minidump error

toty — Wed, 11 Aug 2010 17:44:31 +0200

Hi, kindly help. Got 3 minidump files w/ after 3 consecutive restart of the machine by itself.

Debugging Details:
------------------

Could not read faulting driver name

READ_ADDRESS: fffffff0

FAULTING_IP:
nt!ObReferenceObjectByPointer+f
e086c6a1 394608 cmp dword ptr [esi+8],eax

MM_INTERNAL_CODE: 0

CUSTOMER_CRASH_COUNT: 3

DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP

BUGCHECK_STR: 0x50

PROCESS_NAME: System

CURRENT_IRQL: 1

TRAP_FRAME: eeb9dc30 -- (.trap 0xffffffffeeb9dc30)
ErrCode = 00000000
eax=00000000 ebx=00000000 ecx=000007a4 edx=eeb9dd88 esi=ffffffe8 edi=00000000
eip=e086c6a1 esp=eeb9dca4 ebp=eeb9dca8 iopl=0 nv up ei ng nz na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010286
nt!ObReferenceObjectByPointer+0xf:
e086c6a1 394608 cmp dword ptr [esi+8],eax ds:0023:fffffff0=????????
Resetting default scope

LAST_CONTROL_TRANSFER: from e085eced to e0827c63

STACK_TEXT:
eeb9dba0 e085eced 00000050 fffffff0 00000000 nt!KeBugCheckEx+0x1b
eeb9dc18 e088c798 00000000 fffffff0 00000000 nt!MmAccessFault+0xb25
eeb9dc18 e086c6a1 00000000 fffffff0 00000000 nt!KiTrap0E+0xdc
eeb9dca8 e0933d73 00000000 00000000 00000000 nt!ObReferenceObjectByPointer+0xf
eeb9dd64 f3a552b3 00000000 00000000 00000000 nt!ObOpenObjectByPointer+0x27
WARNING: Stack unwind information not available. Following frames may be wrong.
eeb9dd90 f3a553fb 00001fa4 00000ece 00006944 hknlgi+0x2b3
eeb9ddac e0949b7c 00000000 00000000 00000000 hknlgi+0x3fb
eeb9dddc e088e062 f3a552d0 00000000 00000000 nt!PspSystemThreadStartup+0x2e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

STACK_COMMAND: kb

FOLLOWUP_IP:
hknlgi+2b3
f3a552b3 ?? ???

SYMBOL_STACK_INDEX: 5

SYMBOL_NAME: hknlgi+2b3

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: hknlgi

IMAGE_NAME: hknlgi.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 43aee2d0

FAILURE_BUCKET_ID: 0x50_hknlgi+2b3

BUCKET_ID: 0x50_hknlgi+2b3

Followup: MachineOwner
---------

Thread: function plus offset question - reply by: chupper

chupper — Tue, 03 Aug 2010 00:04:29 +0200

Great! Thanks for the help. I have so much trouble sorting through these details in the normal help file / chm.

Thanks again!

Thread: Developing WinDBG extensions in Visual Studio

huku — Mon, 02 Aug 2010 19:07:16 +0200

Hello everyone,

I would like to share with everyone the following link which I found via google.

www.saygoodnight.com/?p=48

So... if you are trying to code a WinDBG extension in VS, don't forget to add __declspec(dllexport) in all function definitions that should be visible to WinDBG :)

Thread: ASP hang

Keith — Wed, 21 Jul 2010 15:07:48 +0200

Hi

I'd be very grateful for any help with interpreting the following analysis from WinDbg. The problem is an ASP application hanging about once a day. The only solution is to manually recycle the application pool. The server is Windows 2003 and IIS6.

Many thanks in advance.

Here's the output:

0:000:x86> !analyze -v
*** WARNING: symbols timestamp is wrong 0x499007dc 0x49900d5a for ntdll.dll
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************

*** ERROR: Symbol file could not be found. Defaulted to export symbols for hotlinkprot.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for ISAPI_Rewrite.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for mtbnotif.dll -
*** WARNING: symbols timestamp is wrong 0x474b83d7 0x474b8467 for asp.dll
*** WARNING: symbols timestamp is wrong 0x45d6cc1d 0x45d70ad8 for comsvcs.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for fcgiext.dll -

FAULTING_IP:
+5befd80
00000000 ?? ???

EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000000000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD: 00000000000068c8

DEFAULT_BUCKET_ID: ZEROED_STACK

PROCESS_NAME: w3wp.exe

OVERLAPPED_MODULE: Address regions for 'activeds' and '輨' overlap

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

MOD_LIST: <ANALYSIS/>

APPLICATION_VERIFIER_FLAGS: 0

PRIMARY_PROBLEM_CLASS: ZEROED_STACK

BUGCHECK_STR: APPLICATION_FAULT_ZEROED_STACK

LAST_CONTROL_TRANSFER: from 000000007d4d8c9e to 000000007d61c846

STACK_TEXT:
0014fc0c 7d4d8c9e 00000190 00000000 00000000 ntdll_7d600000!ZwWaitForSingleObject+0x15
0014fc7c 7d4d8c0d 00000190 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
0014fc90 5a364692 00000190 ffffffff 00000000 kernel32!WaitForSingleObject+0x12
0014fca0 5a366f27 003354c8 5a3af3dd 00000000 w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
0014fca8 5a3af3dd 00000000 64711dcf 00000000 w3dt!UlAtqStartListen+0x2d
0014fcb8 5a3bc315 01001418 010013e4 010012d0 w3core!W3_SERVER::StartListen+0xbd
0014ff0c 0100187c 00000007 00333a30 00000000 w3core!UlW3Start+0x26e
0014ff44 01001a27 00000007 00333a30 003348c8 w3wp!wmain+0x22a
0014ffc0 7d4e7d42 00000000 00000000 fffdf000 w3wp!wmainCRTStartup+0x12f
0014fff0 00000000 010018f8 00000000 000000c8 kernel32!BaseProcessStart+0x28

STACK_COMMAND: ~0s; .ecxr ; kb

FOLLOWUP_IP:
ntdll_7d600000!ZwWaitForSingleObject+15
7d61c846 c20c00 ret 0Ch

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: ntdll!ZwWaitForSingleObject+15

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: ntdll_7d600000

IMAGE_NAME: ntdll.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 499007dc

FAILURE_BUCKET_ID: ZEROED_STACK_80000003_ntdll.dll!ZwWaitForSingleObject

BUCKET_ID: X64_APPLICATION_FAULT_ZEROED_STACK_ntdll!ZwWaitForSingleObject+15

WATSON_IBUCKET: 395708677

WATSON_IBUCKETTABLE: 1

WATSON_STAGEONE_URL: watson.microsoft.com/StageOne/w3wp_exe/6...00000.htm?Retriage=1

Followup: MachineOwner
---------

0:000:x86> ~*k

. 0 Id: d04c.68c8 Suspend: 0 Teb: fffdb000 Unfrozen
ChildEBP RetAddr
0014fc0c 7d4d8c9e ntdll_7d600000!ZwWaitForSingleObject+0x15
0014fc7c 7d4d8c0d kernel32!WaitForSingleObjectEx+0xac
0014fc90 5a364692 kernel32!WaitForSingleObject+0x12
0014fca0 5a366f27 w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
0014fca8 5a3af3dd w3dt!UlAtqStartListen+0x2d
0014fcb8 5a3bc315 w3core!W3_SERVER::StartListen+0xbd
0014ff0c 0100187c w3core!UlW3Start+0x26e
0014ff44 01001a27 w3wp!wmain+0x22a
0014ffc0 7d4e7d42 w3wp!wmainCRTStartup+0x12f
0014fff0 00000000 kernel32!BaseProcessStart+0x28

1 Id: d04c.1003c Suspend: 0 Teb: fffd8000 Unfrozen
ChildEBP RetAddr
0088fea4 7d63f521 ntdll_7d600000!ZwWaitForMultipleObjects+0x15
0088ff48 7d63f9a8 ntdll_7d600000!EtwpWaitForMultipleObjectsEx+0xf7
0088ffb8 7d4dfe37 ntdll_7d600000!EtwpEventPump+0x27f
0088ffec 00000000 kernel32!BaseThreadStart+0x34

2 Id: d04c.16cc Suspend: 0 Teb: fffd5000 Unfrozen
ChildEBP RetAddr
0104ffa0 7d634d89 ntdll_7d600000!NtDelayExecution+0x15
0104ffb8 7d4dfe37 ntdll_7d600000!RtlpTimerThread+0x47
0104ffec 00000000 kernel32!BaseThreadStart+0x34

3 Id: d04c.3c24 Suspend: 0 Teb: fffad000 Unfrozen
ChildEBP RetAddr
010cff74 7d62e159 ntdll_7d600000!ZwRemoveIoCompletion+0x15
010cffb8 7d4dfe37 ntdll_7d600000!RtlpWorkerThread+0x3d
010cffec 00000000 kernel32!BaseThreadStart+0x34

4 Id: d04c.11e0c Suspend: 0 Teb: fffaa000 Unfrozen
ChildEBP RetAddr
0116fd1c 7da3da80 ntdll_7d600000!ZwReplyWaitReceivePortEx+0x12
0116ff84 7da45eac rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
0116ff8c 7da45dd0 rpcrt4!RecvLotsaCallsWrapper+0xd
0116ffac 7da45e94 rpcrt4!BaseCachedThreadRoutine+0x9d
0116ffb8 7d4dfe37 rpcrt4!ThreadStartRoutine+0x1b
0116ffec 00000000 kernel32!BaseThreadStart+0x34

5 Id: d04c.4ae0 Suspend: 0 Teb: fffa7000 Unfrozen
ChildEBP RetAddr
011eff0c 7d4d0ec9 ntdll_7d600000!NtDelayExecution+0x15
011eff74 7d4d14ef kernel32!SleepEx+0x68
011eff84 776bbb0f kernel32!Sleep+0xf
011eff90 776bbab4 ole32!CROIDTable::WorkerThreadLoop+0x14
011effac 776b1704 ole32!CRpcThread::WorkerLoop+0x26
011effb8 7d4dfe37 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x20
011effec 00000000 kernel32!BaseThreadStart+0x34

6 Id: d04c.f2e8 Suspend: 0 Teb: fffa4000 Unfrozen
ChildEBP RetAddr
013ef458 7d628698 ntdll_7d600000!ZwWaitForSingleObject+0x15
013ef494 7d628596 ntdll_7d600000!RtlpWaitOnCriticalSection+0x1a3
013ef4b4 67a86482 ntdll_7d600000!RtlEnterCriticalSection+0xa8
WARNING: Stack unwind information not available. Following frames may be wrong.
013ef594 7c4239cb hotlinkprot!TerminateFilter+0x5462
013ef5e4 fa375121 msvcp80!std::basic_string<char,std::char_traits<char>,std::allocator<char> >::assign+0x6a
00000000 00000000 0xfa375121

7 Id: d04c.4ce8 Suspend: 0 Teb: fffa1000 Unfrozen
ChildEBP RetAddr
0146f458 7d628698 ntdll_7d600000!ZwWaitForSingleObject+0x15
0146f494 7d628596 ntdll_7d600000!RtlpWaitOnCriticalSection+0x1a3
0146f4b4 67a86482 ntdll_7d600000!RtlEnterCriticalSection+0xa8
WARNING: Stack unwind information not available. Following frames may be wrong.
0146f594 7c4239cb hotlinkprot!TerminateFilter+0x5462
0146f5e4 fa4f5121 msvcp80!std::basic_string<char,std::char_traits<char>,std::allocator<char> >::assign+0x6a
00000000 00000000 0xfa4f5121

8 Id: d04c.2720 Suspend: 0 Teb: fff9e000 Unfrozen
ChildEBP RetAddr
014ef458 7d628698 ntdll_7d600000!ZwWaitForSingleObject+0x15
014ef494 7d628596 ntdll_7d600000!RtlpWaitOnCriticalSection+0x1a3
014ef4b4 67a86482 ntdll_7d600000!RtlEnterCriticalSection+0xa8
WARNING: Stack unwind information not available. Following frames may be wrong.
014ef594 7c4239cb hotlinkprot!TerminateFilter+0x5462
014ef5e4 fa475121 msvcp80!std::basic_string<char,std::char_traits<char>,std::allocator<char> >::assign+0x6a
00000000 00000000 0xfa475121

9 Id: d04c.553c Suspend: 0 Teb: fff9b000 Unfrozen
ChildEBP RetAddr
0156f458 7d628698 ntdll_7d600000!ZwWaitForSingleObject+0x15
0156f494 7d628596 ntdll_7d600000!RtlpWaitOnCriticalSection+0x1a3
0156f4b4 67a86482 ntdll_7d600000!RtlEnterCriticalSection+0xa8
WARNING: Stack unwind information not available. Following frames may be wrong.
0156f594 7c4239cb hotlinkprot!TerminateFilter+0x5462
0156f5e4 fa5f5121 msvcp80!std::basic_string<char,std::char_traits<char>,std::allocator<char> >::assign+0x6a
00000000 00000000 0xfa5f5121

10 Id: d04c.c510 Suspend: 0 Teb: fff98000 Unfrozen
ChildEBP Re

Thread: Bugcheck Analysis

Edward — Sat, 05 Jun 2010 11:38:11 +0200

Microsoft (R) Windows Debugger Version 6.9.0003.113 X86
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Windows\Minidump\060510-19390-01 begin_of_the_skype_highlighting 060510-19390-01 end_of_the_skype_highlighting.dmp]
Mini Kernel Dump File: Only registers and stack trace are available

Symbol search path is: SRV*c:\windows\symbols*msdl.microsoft.com/download/symbols
Executable search path is:
Windows 7 Kernel Version 7600 MP (2 procs) Free x86 compatible
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 7600.16539.x86fre.win7_gdr.100226-1909
Kernel base = 0x8280d000 PsLoadedModuleList = 0x82955810
Debug session time: Sat Jun 5 10:17:58.751 2010 (GMT+2)
System Uptime: 0 days 0:10:32.533
Loading Kernel Symbols
.........................................................................................Missing image name, possible paged-out or corrupt data.
...............................................................
Loading User Symbols
Loading unloaded module list
.....
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 10000050, {fd7f0000, 1, 8284be85, 0}

Could not read faulting driver name
Missing image name, possible paged-out or corrupt data.
Missing image name, possible paged-out or corrupt data.
Probably caused by : win32k.sys ( win32k!PALLOCMEM+29 )

Followup: MachineOwner
---------

Thread: Unable to load image ntoskrnl.exe - reply by: Chris Cates

Chris Cates — Tue, 01 Jun 2010 21:49:16 +0200

There does NOT appear to be a descrepancy for the 'R2' variant. We were able to sucessfully download symbols from Microsoft Symbols Server on server A and import/utilize them on server B.

The symbols folder name was 'ntkrpamp.pdb'. There appears to be a dependency on hardware configuration outside of two servers having matching kernels. I found this interesting, but very logical.

Thanks again for your help and letting me bounce this off of here.

Kind Regards,
Chris

Thread: Memory Searching issue (command s)

Brett K — Mon, 31 May 2010 00:43:10 +0200

Hello! When I try to use the 's' command to search memory, it was coming up with no results. Upon dumping the memory region where I knew the byte-sequence I was looking for is located, the memory there was showing up as '???' a bunch of question marks. So I used the .pagein extension to page in that memory, and then it showed up. At this point, if I used the 's' command again, it would now find the byte-sequence. The problem I am having is, the images I debug are large (often 5-10mb), and they seem to get paged out all the time, making searching impossible. If I didn't know the exact address of where to find my byte-sequence, I wouldn't know which of the several thousand pages I'd need to .pagein to find it. So my question is, is there a way to prevent memory from being paged out, or a way to .pagein an entire executable image?

Thread: Memory Access errors in the Kernel - reply by: Brett K

Brett K — Fri, 28 May 2010 22:09:14 +0200

Thanks again for you help, I fully understand it now!

Thread: Wrong display of function Names by WinDbg - reply by: Robert Kuster

Robert Kuster — Fri, 28 May 2010 16:43:43 +0200

JD, welcome.

Whenever you see a large offset like this mydll!wrongfuncanme+0x33360 it is always a sign of missing symbols.

Now strncpy is part of the CRT library and thus resides in MSVCR80.DLL or one of its siblings (usually there comes a new CRT version with every Visual Studio release). Because MSVCR80.DLL is a Microsoft DLL you can actually get its public PDB symbols from the Microsoft server. More precisely WinDbg will fetch the correct symbols automatically for you from the server, if you set it up correctly. You might read WinDbg. From A to Z! - "Symbols in WinDbg" at slide 24. Or simply check out the .symfix+ command...

If you aren't able to get the right PDB files any serious debugger will read at least the export symbols (functions) of your modules. Example: If you open MSVCR80.DLL in Dependency Walker you see all its exported functions. This is the same information that is read and used by IDA or by WinDbg if they fail to get the PDBs. For example in WinDbg:

0:000> ld MSVCR80
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.4053_x-ww_e6967989\MSVCR80.dll

Bottom line: WinDbg should show you and will show you the same symbol information as IDA does.

I hope this helps,
Robert

Thread: CrashMe Application - reply by: Robert Kuster

Robert Kuster — Sun, 04 Apr 2010 23:53:23 +0200

Sudhir,

welcome and thanks for your input. In fact an article about setting up WinDbg and analyze CrashMe and its crash dumps is the very next thing I will do over the next weeks.

I hope to see you around,
Robert

Thread: Determing cause of access denied - USN Journal - reply by: Robert Kuster

Robert Kuster — Sun, 04 Apr 2010 23:48:07 +0200

Hi Will.

Some solutions to this issue have been provided here:
www.mydefrag.com/forum/index.php?topic=1131.0

To peek under the hood:

Behind the scenes "fsutil usn deletejournal /D C:" translates into a call too fsutil!DeleteUsnJournal. Its pseudo-code looks like this:

Code:

fsutil!DeleteUsnJournal() 

{ 

   fsutil!IsVolumeLocalNTFS(..) 

   hVolume = kernel32!CreateFileW(sVolume);   // sVolume: "\\.\c:", OR "\\.\d:", OR.. 

   kernel32!DeviceIoControl(hVolume, FSCTL_DELETE_USN_JOURNAL, ..) 

   fsutil!DisplayError(..) 

}

In short you can attach WinDbg to fsutil.exe, set a breakpoint on DeleteUsnJournal, and debug through it. It can actually only fail on two places:

1) as it tries to open a handle to the volume
2) in the call too DeviceIoControl( FSCTL_DELETE_USN_JOURNAL )

After you find the failed function you could execute a !gle (get last error) in WinDbg. With some luck you will get a hint about what failed. With less luck you might end up debugging FSCTL_DELETE_USN_JOURNAL throughout kernel mode. The stack in the kernel looks something like this:

1: kd> kb
ChildEBP RetAddr Args to Child
acb11adc b9dbdca6 88928a38 88666008 acb11b20 Ntfs!NtfsDeleteUsnJournal
acb11af0 b9da7adc 88928a38 88666008 acb11b20 Ntfs!NtfsUserFsRequest+0x325
acb11b04 b9da7a36 88928a38 88666008 8a6c2020 Ntfs!NtfsCommonFileSystemControl+0x44
acb11b78 804ef19f 8a6c2020 88666008 806e6428 Ntfs!NtfsFsdFileSystemControl+0x116
acb11b88 80658128 8a7294d8 8a729590 88666008 nt!IopfCallDriver+0x31
acb11bac b9e26ee5 8a6a4040 00000000 acb11bf0 nt!IovCallDriver+0xa0
acb11bbc 804ef19f 8a729590 88666008 806e6428 sr!SrFsControl+0x121
acb11bcc 80658128 889f04c0 88666008 00000000 nt!IopfCallDriver+0x31
acb11bf0 b9e43837 88666000 889f04c0 8a786040 nt!IovCallDriver+0xa0
acb11c1c 804ef19f 889f04c0 88666008 806e6428 fltmgr!FltpFsControl+0xd7
acb11c2c 80658128 8a70e130 806e6410 88666008 nt!IopfCallDriver+0x31
acb11c50 8057f982 886661bc 8a70e130 88666008 nt!IovCallDriver+0xa0
acb11c64 805807f7 889f04c0 88666008 8a70e130 nt!IopSynchronousServiceTail+0x70
acb11d00 805792a8 00000788 00000000 00000000 nt!IopXxxControlFile+0x5c5
acb11d34 8054163c 00000788 00000000 00000000 nt!NtFsControlFile+0x2a
acb11d34 7c90e514 00000788 00000000 00000000 nt!KiFastCallEntry+0xfc
0007faf8 7c90d3aa 7c80174d 00000788 00000000 ntdll!KiFastSystemCallRet
0007fbf8 7c9100b8 00090330 0007fcd0 7c910041 ntdll!ZwFsControlFile+0xc

1: kd> !irp 88666008
Irp is active with 10 stacks 10 is current (= 0x886661bc)
No Mdl: System buffer=887e9400: Thread 88600020: Irp stack trace.
cmd flg cl Device File Completion-Context
>[ d, 0] 4 0 8a6c2020 8a70e130 00000000-00000000
\FileSystem\Ntfs
...

A good point to start you investigation is nt!NtFsControlFile as obviously something between it and Ntfs!NtfsDeleteUsnJournal fails.
The following breakpoint will stop the kernel for any FSCTL_DELETE_USN_JOURNAL request:
bu nt!NtFsControlFile ".if( poi(@esp + 18) == 0x000900f4 ) { .echo ***** FSCTL_DELETE_USN_JOURNAL *****; } .else { g;};"

Explanation:
> the sixth parameter is the dwIoControlCode which was passed to DeviceIoControlCode
> FSCTL_DELETE_USN_JOURNAL == 0x000900f4

You can also check out these articles:
Keeping an Eye on Your NTFS Drives: the Windows 2000 Change Journal Explained
Keeping an Eye on Your NTFS Drives, Part II: Building a Change Journal Application

I hope this helps,
Robert

Thread: See in Memory Descriptor List whats on - reply by: Robert Kuster

Robert Kuster — Sat, 03 Apr 2010 15:17:47 +0200

Steffen welcome.

It is easy to find out which driver is leaking or consuming a lot of memory.

Theory
In order to use Pool tags one generally has to enable them with GFlags -> System Registry -> "Enable pool tagging". Luckily on Windows Server 2003 pool tagging is always enabled. From the documentation: "Pool tagging is permanently enabled on Windows Server 2003 and later versions of Windows, including Windows Vista. On these systems, the Enable pool tagging check box on the Global Flags dialog box is dimmed and commands to enable or disable pool tagging fail.". Also alongside your WinDbg installation you should find ..\Debugging Tools for Windows (x86)\triage\ pooltag.txt which lists all tags used by kernel mode components and drivers. Here is what it says about the Mdl tag:

<PoolTag> - <binary-name> - <Description>
Mdl - <unknown> - Io, Mdls

MDLs are described here: What Is Really in That MDL? Further you should use Driver Verifier to track Pool Usage and configure it as follows:

"Create custom settings" -> Next
"Selet individual settings from a full list" -> Next
Select "Pool tracking" -> Next
"Automatically select all drivers installed on this computer" -> Finish, Restart computer

Start Driver Verifier Manager again an select "Display information about currently verified drivers" -> Next (3x). Now you will see the pool usage of each driver:

In WinDbg
After enabling driver verifier you can get even more information from WinDbg. Attach WinDbg as a kernel debugger to the target machine and use the following commands:

0: kd> !verifier 1

Driver Verification List

Entry State NonPagedPool PagedPool Module

8a7e6ee8 Loaded 00000000 00000000 kdcom.dll
8a7e6e68 Loaded 00000000 00000000 BOOTVID.dll
8a7e6df0 Loaded 00023708 00003760 ACPI.sys
8a7e6d70 Loaded 00000000 00000000 WMILIB.SYS
8a7e6480 Loaded 00003710 0001b310 pci.sys
....

; to see all individual allocations of each driver
0: kd> !verifier 0x3

Driver Verification List

Entry State NonPagedPool PagedPool Module

8a7e6df0 Loaded 00023708 00003760 ACPI.sys

Current Pool Allocations 00000059 00000023
Current Pool Bytes 00023708 00003760
Peak Pool Allocations 000000d3 0000002d
Peak Pool Bytes 00024b88 00003be8

PoolAddress SizeInBytes Tag CallersAddress
e10115a8 0x00000080 AcpM b9fa1c47
8a783148 0x00000018 AcpS b9f7fabb
8a7bd148 0x00000018 AcpS b9f7fae7
8a786a10 0x00000030 AcpS b9f836a6
8a7c0130 0x00000030 AcpR b9f877c9
....

I hope this helps,
Robert

Thread: Common WinDbg Commands (Thematically Grouped) - reply by: Robert Kuster

Robert Kuster — Fri, 02 Apr 2010 21:15:09 +0200

Kam,

In general you don't need symbols to know the entry point of a PE image. The entry point is conveniently stored to the PE header and can be read from it. Further the OS loader loads an image into memory (be it an EXE, DLL, or kernel mode driver) and calls its entry point thereafter. In other words by the time DriverEntry is called the driver will always be loaded. If all you need is break into WinDbg after a driver is loaded but before its entry point is called the situation is simple. And if you want to break even before the image is loaded into memory the situation is still simple enough. I described both scenarios bellow.

1) TODOs - break after driver load & before its entry point is called

1) Break into WinDbg -> Debug (menu) -> Event Filters
2) In the "Event Filters" select the "Load module" event
> select Execution -> Enabled
> click Arguments and enter the name of the driver in question (77fba431 in our case)

With this settings WinDbg will brake after loading any driver with the name 77fba431 and before calling its entry point. Here is what happens:

0: kd> .lastevent
Last event: Load module 77fba431.sys at ba644000
debugger time: Wed Mar 31 21:12:56.937 2010 (GMT+2)

0: kd> lm vm 77fba431
start end module name
ba644000 ba645e00 77fba431 (deferred)
Image path: 77fba431.sys
Image name: 77fba431.sys
Timestamp: Tue Mar 30 20:10:51 2010 (4BB23EAB)
CheckSum: 0000AD61
ImageSize: 00001E00
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

;get ImageEntry (== DriverEntry)
0: kd> ? $iment( ba644000)
Evaluate expression: -1167828646 = ba64595a

;now that we have the DriverEntry address we can conveniently set a breakpoint on it
0: kd> bp ba64595a
*** ERROR: Module load completed but symbols could not be loaded for 77fba431.sys

0: kd> bl
0 e ba64595a 0001 (0001) 77fba431+0x195a

0: kd> g
Breakpoint 0 hit
77fba431+0x195a:
ba64595a 8bff mov edi,edi

2) TODOs - break before our driver is loaded into memory (XP SP 3)

Behind the scenes a driver is loaded by nt!IopLoadDriver. Its pseudo-code looks like this:

Code:

nt!IopLoadDriver( hRegKeyOfDriver, ...)

{

   // get full path of our driver

   UNICODE_STRING driverPath = call nt!IopBuildFullDriverPath(...);



   // load driver into memory

   call nt!MmLoadSystemImage( driverPath , ...)       



   // retrieve drivers entry point and call it

   call nt!RtlImageNtHeader

   call nt!IopPrepareDriverLoading   

   call    dword ptr [edi+2Ch]       // call DriverEntry

}

With this on mind we can use the following conditional breakpoint. It will cause WinDbg to break if the driver being loaded contains the name "77fba431" and continue execution in any other cases:

bu nt!IopLoadDriver ".block {as /c HandleOutput !handle poi(@esp+4)}; .block {.if ( $spat( \"${HandleOutput}\", \"*77fba431*\" ) ) { .echo ***** Loading our driver *****; ad *; } .else { ad *; g;}};"

Explanation:

!handle poi(@esp+4) ...... get handle information and registry path of hRegKeyOfDriver; store it into HandleOutput (string alias)
$spat( "${HandleOutput}", "*77fba431*" ) .... is 77fba431 found in HandleOutput? If yes break. If not go (g)

I peeked into nt!IopLoadDriver on Windows XP SP3. It might bee slightly different on Windows Vista or Windows 7 installations. Nevertheless I think you got the idea and will be able to change the breakpoint if necessary. You could set a similar conditional breakpoint on nt!MmLoadSystemImage. Note that its first parameter is the path of the driver:

0: kd> kb
ChildEBP RetAddr Args to Child
ba507c74 8058107b ba507cf8 00000000 00000000 nt!MmLoadSystemImage
ba507d54 80581487 80000748 00000001 00000000 nt!IopLoadDriver+0x371
ba507d7c 80538789 80000748 00000000 8a8e8640 nt!IopLoadUnloadDriver+0x45
ba507dac 805cff72 ae22acb0 00000000 00000000 nt!ExpWorkerThread+0xef
ba507ddc 8054611e 8053869a 00000001 00000000 nt!PspSystemThreadStartup+0x34
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

0: kd> dS ba507cf8
e10e69b8 "\??\C:\CpDrv\77fba431.sys"

I hope this helps,
Robert

Thread: Using WinDbg to examine ASP.NET applications - reply by: Will Steele

Will Steele — Thu, 18 Feb 2010 16:43:00 +0100

Awesome. Thanks for the pointers and the link to the PDF. I had not even seen that one yet. Looking forward to working with it. Again, much appreciated.

Thread: Debugging minGW/GCC built DLL in Visual Studio? - reply by: Edward

Edward — Wed, 17 Feb 2010 19:01:43 +0100

thank for the pointers.
'looks like PDB absorbed in other IDE that already absorbs PECOFF/STABS and PECOFF/DWARF is a more realistic path. (GCC 4 is pushing out PECOFF/DWARF to complicate issues)....

Thread: Articles/tutorials - reply by: Robert Kuster

Robert Kuster — Wed, 17 Feb 2010 15:48:00 +0100

Hi Raminder,

welcome and thanks for your suggestion. To be honest, I'm not yet sure how much I would like to open the article part to the public. It certainly would have advantages, because simply more stuff would be on-line. On the other side I would lose control about the quality of what is on-line. Nevertheless, it is very likely that I will turn at least a part of the article section public so anyone could make contributions there. In the meantime just let me know if you have something really interesting to write on and we will figure out something.

Kind regards,
Robert

Thread: Other forums - reply by: Robert Kuster

Robert Kuster — Sun, 17 Jan 2010 18:32:03 +0100

Hi Will,

From my point of view your questions perfectly fit to the forum; that is what it actually is about. Help people with WinDbg, regardless to what their initial skills might be. Hm, if you nevertheless want other places there are several. You might check Syinternals or CodeProject's debug section; the Forum at Open System Resources (OSR), or the handy blog from Dmitry called Dumpanalysis; then there is Debuginfo and so on. But after all you could also take a day off, read WinDbg. From A to Z!, and play around with CrashMe along the way. In fact WinDbg. From A to Z! answered virtually all of your questions so far. ;)

Hope to see you around, Robert

Thread: Can handled exceptions be seen with WinDbg - reply by: Robert Kuster

Robert Kuster — Sun, 17 Jan 2010 18:07:40 +0100

Hi Will,

here I go again. Sorry for the little delay this time.
Can handled exceptions be seen with WinDbg
Short answer: Nope. A handled/dismissed exception isn't an exception anymore.

Long answer: It depends. The first thing one should know about exceptions is that on Windows (either the environment is Win32, MFC, .NET, or the kernel) exceptions are handled by the OS. To us exceptions are made available through language extensions, for example through the try & except constructs in C++ or C#. Thus it is the OS or more precisely its exception dispatcher that takes care and dispatches exceptions. From the perspective of the exception dispatcher there are two different situations to cover:

A debugger is attached to the application in question
No debugger is attached to the application

In case there is a debugger the exception dispatcher does the following:

Attempts to notify the process's debugger (say, WinDbg). This is known as a first-chance exception.
If WinDbg handles the exception (gH - go, exception handled) that's it. In this case the exception dispatcher won't notify anyone else about the exception because it has been dismissed.
If WinDbg didn't handle the exception, the OS then tries to locate a frame-base exception filter (a C++ or C# catch statement, for instance).
Again, if the exception filter handles the exception that's it. In this case the exception dispatcher won't notify anyone else about the exception.
If along the way nobody handled the exception, the exception dispatcher will notify the associated debugger again. This is known as a second-chance or last-chance exception.
Again, if the associated debugger handles the exception that's it. Your application will continue to run fine as if nothing has happened.
But if the associated debugger doesn't handle even the second chance exception (for instance, in WinDbg: gN - go not handled command) the process will terminate.

The bottom line:

if a debugger is attached, this debugger will always be notified about any exception in the first place (first-chance exception)
once an exception is handled there is no way to display it in a debugger thereafter; no debugger will be notified about handled exceptions

Finally please check WinDbg. From A to Z! once again.

Refer to slides 17-20 about the exception dispatcher and exceptions in general.
Refer to slides 85-86 about debugging exceptions with WinDbg.
Refer to slides 89-92 about event filters in WinDbg. In particular you should enable CLR exceptions and CLR notification exceptions for managed applications as otherwise WinDbg will ignore them altogether.

Said all that there is one more tool to be aware off called Application Verifier. Application Verifier is a runtime verification tool that monitors an application's interaction with the Windows OS. Among other things it profiles and tracks all (handled an unhandled) exceptions that occur within a process. Again, check out WinDbg. From A to Z! and refer to slides 97-102. While Application Verifier will display handled exceptions to you, note that it is just a tracking/logging tool of events that occurred in the past. By no means it will halt a debugger on handled exceptions since this isn't how exceptions were meant to work in the first place.

Last but not least: You might also try out the sos!DumpAllExceptions command.

I hope this helps, RK :)

Thread: How do source code files help? - reply by: Robert Kuster

Robert Kuster — Sun, 17 Jan 2010 17:54:49 +0100

Hi Will,

we are somewhat out of luck here, I'm afraid. By default WinDbg doesn't support debugging of managed code natively - at least not in its public releases. See explanation here: No CLR support in the latest debugger release . The only WinDbg release really supporting it was version 6.7.5.0 which was mistakenly made public. Maybe you can still find it somewhere. If so, you'll be able to see call stacks with functions names, source code information, and so on in almost the same way as you see it for C++ applications.

To keep it short: You can use WinDbg to obtain some extra information for managed applications via its SOS or SOSEX extension. But in order too turn it into a really useful debugger for manged applications stick to WinDbg version 6.7.5.0

I hope this helps, RK