WinDbg.info - Forum

Thread: Random Thread getting stuck

Ben — Wed, 28 Dec 2011 10:59:00 +0100

I've spent the last couple of weeks investigating a very very difficult bug.

Firstly some background... our application prcoesses 600-1000 pairs of audio streams from the network in realtime and uses one thread per pair of audio streams (2 streams because its stereo). This means our app can use upto about 1006 threads (There are a few threads in additon to the audio receiving threads).

Each Audio thread is temporary and may only last a couple of minutes, so threads are being created and exited frequently.

The problem is that after a couple of hours, one of main threads (Not one of the 1000 or so Audio Receiving threads) just stops. It doesn't exit but its not running either!

Using WinDbg I can see the following call stack for the stuck thread:

ntdll!NtWaitForMultipleObjects+0x15 (FPO: [5,0,0])
KERNELBASE!WaitForMultipleObjectsEx+0x100 (FPO: [SEH])
kernel32!WaitForMultipleObjectsExImplementation+0xe0 (FPO: [5,8,4])
kernel32!WaitForMultipleObjects+0x18 (FPO: [4,0,0])
WARNING: Stack unwind information not available. Following frames may be wrong.
boost_thread_vc90_mt_1_47!boost::this_thread::interruptible_wait+0x199
boost_thread_vc90_mt_1_47!boost::thread::get_thread_info+0x144
boost_thread_vc90_mt_1_47!boost::thread::join+0x6c

Firstly, why is some of the stack unwind info unavailable? If I break the process before the problem occurs then the threads call stack looks perfectly normal and shows all my apps methods as you would expect. Could the stack have become corrupt?

Secondly, why doesn't the WaitForMultipleObjects ever return? One of the objects I'm waiting on is a WaitableTimer that fires every 10 seconds, so at the very least I should be seeing that (But I'm not).

I beieve that if one of my wait objects is invalid then WaitForMultipleObjects should return WAIT_FAIL, and not just hang. Plus I'm confident that none of the objects are invalid as I have been successfully calling WaitForMultipleObjects using these objects for well over an hour before the problem occurs, and no other threads close the handles that I'm waiting on.

Anyway, here's the weirdest bit...

If I add any diagnostics to this thread to try and see what's going on then the next time I run the code it may be another thread that freezes, not this one! Basically changing the code changes which thread freezes!

Does this just look like a stack corruption problem (If so then is there any way to get WinDbg to detect what is corrupting the stack?).

Thanks
Ben

Thread: WinDbg showing Wrong Info

Ben — Wed, 28 Dec 2011 10:10:31 +0100

I've got a program that I'm trying to debug but when I attach WinDbg to the process and look at the call stack for the thread that I'm interested in I sometimes see that the 'this' pointer is NULL, or that some of the member data in my class is NULL... after a lot of investigation I concoluded that this cannot be possible, so I got my code to print out the value of the 'this' pointer once a second. Sure enough the printed 'this' pointer is perfectly valid yet WinDbg still says its NULL.

Any idea why WinDbg could be getting it wrong?

The PDB file definitely matches the EXE (i.e. I don't have an old PDB against a nwer version of the EXE).

Also, I'm using the old RaiseException trick to name my threads in the debugger, yet sometimes I only see some of my threads with names in WinDbg even though I know I named them all. Sometimes none of them have names. Could this be because I'm naming my threads before attaching WinDbg to the process? i.e. does the debugger have to be running at the time I call RaiseException?

Thanks for your help
Ben

Thread: StackOverFlowException in .Net

Daniele Foschi — Wed, 07 Dec 2011 14:57:35 +0100

Hi,
I've a question about finding StackOverFlowException in .net programs.
I have a simple program test that generates a StackOverFlowException when I click a button:

private void button1_Click(object sender, EventArgs e)
{
MyMethod();
}

private void MyMethod()
{
MyMethod();
}

Windbg+sos loads symbols correctly, but with analyze command I've not find references to MyMethod that caused StackOverflow; instead if I build in debug mode I can see full stack with recursive call of MyMethod.
What can I do to see full stack in release mode?
Can anyone help me?
Thanks!

Below WinDBG output in release and debug mode (extract from !analyze -v command)

*** WinDBG RELEASE MODE: ***

[...]
MANAGED_STACK:
(TransitionMU)
0019F158 664D8CEE System_Windows_Forms_ni!System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)+0x24e
0019F1F4 664D8957 System_Windows_Forms_ni!System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)+0x177
0019F248 664D87A1 System_Windows_Forms_ni!System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)+0x61
0019F278 66495911 System_Windows_Forms_ni!System.Windows.Forms.Application.Run(System.Windows.Forms.Form)+0x31
0019F28C 0061009A ProvaCrash!ProvaCrash.Program.Main()+0x2a
(TransitionUM)

[...]
MANAGED_OBJECT_NAME: System.StackOverflowException
[...]

STACK_TEXT:
WARNING: Frame IP not in any known module. Following frames may be wrong.
0019ee20 6648f55a 018ed980 019036dc 0019ee4c 0x40c0a2
0019ee30 66a26f64 00000619 000000e4 00100000 System_Windows_Forms_ni+0x1bf55a
0019ee4c 669f7773 018ed980 00000005 00050033 System_Windows_Forms_ni+0x756f64
0019eed0 66d2a202 00000001 00100000 e9f17a94 System_Windows_Forms_ni+0x727773
0019ef30 66d289d1 018ed980 00000000 00000000 System_Windows_Forms_ni+0xa5a202
0019ef74 664c25b0 0019efa8 0019ef88 664c86a0 System_Windows_Forms_ni+0xa589d1
0019ef80 664c86a0 0019ef9c 664c8621 00000000 System_Windows_Forms_ni+0x1f25b0
0019ef88 664c8621 00000000 018eda50 0019efcc System_Windows_Forms_ni+0x1f86a0
0019ef9c 664c84fa 018eda50 00081054 00000202 System_Windows_Forms_ni+0x1f8621
0019f000 76b1c4e7 00081054 00000202 00000000 System_Windows_Forms_ni+0x1f84fa
0019f02c 76b1c5e7 005d0cb2 00081054 00000202 user32!InternalCallWinProc+0x23
0019f0a4 76b1cc19 00273244 005d0cb2 00081054 user32!UserCallWinProcCheckWow+0x14b
0019f104 76b1cc70 005d0cb2 00000000 0019f130 user32!DispatchMessageWorker+0x35e
0019f114 012b110e 0019f1a0 e9f17a94 00000000 user32!DispatchMessageW+0xf
0019f130 664d8cee 018f4dac 00000001 018ea380 0x12b110e
[.....]
0019fe34 76f337c8 6fd54ddb 7ffd5000 00000000 ntdll!__RtlUserThreadStart+0x70
0019fe4c 00000000 6fd54ddb 7ffd5000 00000000 ntdll!_RtlUserThreadStart+0x1b

*** WinDBG DEBUG MODE: ***

[...a lot of rows of MyMethod...]
0030ef70 001f0506 ProvaCrash.Form1.MyMethod()
0030ef7c 001f04cb ProvaCrash.Form1.button1_Click(System.Object, System.EventArgs)
0030ef90 66494170 System.Windows.Forms.Control.OnClick(System.EventArgs)
0030efa8 6648f55a System.Windows.Forms.Button.OnClick(System.EventArgs)
0030efb8 66a26f64 System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventArgs)
0030efd4 669f7773 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32)
0030f060 66d2a202 System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
0030f064 66d289d1 [InlinedCallFrame: 0030f064]
0030f0fc 664c25b0 System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)
0030f108 664c86a0 System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
0030f110 664c8621 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
0030f124 664c84fa System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
0030f2c8 008709e4 [NDirectMethodFrameStandalone: 0030f2c8] System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)
0030f2d8 664d8cee System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)
0030f374 664d8957 System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
0030f3c8 664d87a1 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
0030f3f8 66495911 System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
0030f40c 001f00ae ProvaCrash.Program.Main()
0030f638 69d51b4c [GCFrame: 0030f638]

Thread: ntdll.dll symbols are missing? - reply by: noname

noname — Wed, 23 Nov 2011 10:25:27 +0100

Oh... I've already solved the problem with windbg - I just downloaded local symbols and then gave them to symstore. It's okay now, but !peb doesn't work, neither dt _PEB or dt nt!_PEB, however lml shows that ntdll.pdb has been loaded. I debug XP SP3 so the symbols do fit the system.

kd> !peb 7ffdb000
PEB at 7ffdb000
error 1 InitTypeRead( nt!_PEB at 7ffdb000)...

However this value is the right one - I've taken it from Peb field of !process 0 0 output.

kd> dt _PEB 7ffdb000
ntdll!_PEB
+0x000 InheritedAddressSpace : ??
+0x001 ReadImageFileExecOptions : ??
+0x002 BeingDebugged : ??
+0x003 SpareBool : ??
+0x004 Mutant : ????
+0x008 ImageBaseAddress : ????
+0x00c Ldr : ????
+0x010 ProcessParameters : ????
+0x014 SubSystemData : ????
+0x018 ProcessHeap : ????
//and so on

I'm really at a loss;

Thread: Video or Sound from 'A to Z' presentation? - reply by: Robert Kuster

Robert Kuster — Thu, 03 Nov 2011 11:57:22 +0100

Austin, welcome.

Hm, that's an interesting suggestion indeed. I'll definitely think about it, though I first plan to add more written content to the site. Among others a step-by-step tutorial for the first steps with WinDbg that you will probably also find interesting and useful. But as said your idea is really interesting ...so let's see.

Stay tuned,
Robert

Thread: drwtsn32 - reply by: Robert Kuster

Robert Kuster — Sun, 09 Oct 2011 23:37:58 +0200

Welcome Sybaris.

The official Dr. Watson documentation states that:

If Dr. Watson is activated on Windows XP or later, a message box will appear. This message box gives you the option of sending an error report to Microsoft. If you choose Don't Send, a dump file will be created and stored on your hard disk. If you choose Send Error Report, a dump file will be created and stored on your hard disk, and will also be transmitted to Microsoft over the Internet.

However, if you have manually set the \\HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\AeDebug\Auto value to zero, the Don't Send button will simply terminate the application. The Send Error Report button will create a dump file and send it to Microsoft but will not store it on your hard disk. A third button, Debug, will create a dump file and store it on your hard disk.

Thus, in theory the generated dump should be preserved except if you change the AeDebug\Auto registry value to 0. For one reason or another this never worked on my XP machines. My suggestion is that you simply use ntsd.exe or WinDbg.exe in order to generate the crash dumps - if it is your own application and test environment, it doesn't make sense to send the crashes over to MS just to retrieve them later anyway. To achieve this add the following string value for AeDebug\Debugger in the registry:

"C:\WINDOWS\system32\ntsd.exe" -p %ld -e %ld -g -c ".dump /u d:\mydumps\crash.dmp; q"

Now ntsd is launched as the default postmortem debugger. It creates a crash dump with a unique name (/u) and exits right thereafter.

I hope this helps,
Robert

Thread: Crashdump bitness - reply by: Robert Kuster

Robert Kuster — Sun, 09 Oct 2011 15:26:17 +0200

Welcome Nachiket.

You can actually open a 32-bit dump with both a 32- or 64-bit WinDbg. At the same time you can open a 64-bit dump with both variants of WinDbg too. So, how to find out the bitness of a dump in question? When I open a dump on my machine, I get a text that look like this:

32-bit dump; 32- or 64-bit WinDbg: Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64
64-bit dump; 32- or 64-bit WinDbg: Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x86 compatible

Alternatively you can use another handy trick: Check out what kind of registers were stored in the dump. For example, RAX will exist only in 64-bit dumps.

32-bit dump; 32- or 64-bit WinDbg: r rax -> Bad register error in 'r rax'
64-bit dump; 32- or 64-bit WinDbg: r rax -> returns the register value

I hope this helps,
Robert

Thread: kernel32 symbol in live kernel debug - reply by: Robert Kuster

Robert Kuster — Sun, 09 Oct 2011 14:47:32 +0200

Welcome Thongchai.

The kernel on 2000, XP, Vista, or Windows 7 never loads user32.dll or kernel32.dll. Both are user mode DLLs and thus get loaded by user-mode applications (generally speaking any Win32 process should load kernel32.dll; applications that have a GUI also load user32.dll).

Symbols are loaded into a debugger mainly for two purposes:

to map raw addresses in the executable to source-code lines
to analyze internal layout and data of applications

What you are asking about somehow violates this basic principle and is simply not needed to debug an application or the kernel. You can still check out the Symbol Options for WinDbg or start your investigation by examining the ld or !lmi commands.

I hope this helps,
Robert

Thread: Creating a process server

Pankaj — Fri, 23 Sep 2011 04:45:34 +0200

Hi,

I'm trying to create a process server.
To do this I create a client object and call the StartProcessServer() function.

Code:



if ( (status = DebugCreate(__uuidof(IDebugClient), (void**)&client)) != S_OK )

{

    fprintf(stderr, "DebugCreate() failed, 0x%X\n", status);

    return;

}



if ( (hr = client->StartProcessServer(DEBUG_CLASS_USER_WINDOWS, opt, NULL)) != S_OK )

{

    fprintf(stderr, "Error: StartProcessServer() failed: %X\n", hr);

    return;

}

But the StartProcessServer() always fails with return value 0x8007053D. :(
Why does it fail? Is there anything I missed?

Thanks,
Pankaj

Thread: Finding undocumented swtiches - reply by: Will Steele

Will Steele — Mon, 12 Sep 2011 00:44:58 +0200

Thanks for that tip. I am looking more for what switches are embedded with the image itself. It seems that in C/C++ the command line switches correspond to switch statements within the code. I just have found found what code section I need to check out in the PE or in something like Windbg or Ollydbg to see the switches used to compare against when arguments are passed to the .exe.

Thread: sort lm n t by date/time - reply by: karl

karl — Sun, 11 Sep 2011 16:15:19 +0200

Robert,
I apologize for not getting back. I did some reading and wrote a PowerShell script to sort the list and perform a couple of other clean up operations.

Thanks,
karl

Thread: How to find parameter values to a method?

ponsakthi — Thu, 01 Sep 2011 20:53:31 +0200

Hi all,

I faced an .net exception in my application in the customer machine.
Without closing the exception I attached Windbg and I saw the clrstack.

The method in which exception has occurred accepts a string as an parameter.

But I am not able to identify the value of paramater in !clrstack -p as well.It shows as no data.

Please help me out.

Thread: Parsing SYSTEM_PROCESS_INFORMATION!

Cirta — Thu, 25 Aug 2011 13:03:45 +0200

Hi guys!

I'am trying to print processes list usig the SYSTEM_PROCESS_INFORMATION structures contained in SYSTEM_INFORMATION_CLASS (this list is used by task manager to get processe list) just by using Windbg!

Any idea how to do that? Is it possible ?

Thanks !!

Thread: Dll export table - Exported functions list and @

Blade0x1 — Thu, 25 Aug 2011 13:00:45 +0200

Hello

Can we extract exported functions list from a dll and their asociated addresses using windbg?

Can we determine what dlls's functions are used by an application? for exemple: myapp.exe is just calling MessageBoxA from user32 dll.

Merci !

Thread: CrashMe Application - reply by: Kim Leeper

Kim Leeper — Thu, 18 Aug 2011 12:28:41 +0200

My development machine is Win2000sp4. My development environment is VC6sp6. Does anyone have a version of CrashMe for VC6?

Your article "WinDbg From A to Z" is very inspiring!

Thread: Symbol not found - reply by: Robert Kuster

Robert Kuster — Tue, 19 Jul 2011 23:35:23 +0200

Mmostafaxx,

sooorry for the late reply. If it is not completely to late: Check out slides 14 and 24 at WinDbg. From A to Z! and try to set up _NT_SYMBOL_PATH so that it really points to the correct folder. Alternatively you could try to debug our CrashMe application first - the symbols there should be loaded just fine. Then you can move back to your application and pinpoint the problem there.

Also check:
- that there is indeed a PDB file that is generated alongside your exe (both should have same timestamps)
- your compiler settings - debug information: /Zi (Program Database) should be set, not /ZI (Program Database for Edit & Continue)
- use the !sym noisy WinDbg command (debugger displays info about its search for symbols), followed by ld *
- check out other symbol-related commands at Common WinDbg Commands (Thematically Grouped)

I hope this helps,
Robert

Thread: Remote debugging of CrashMe with ntsd -d - reply by: Robert Kuster

Robert Kuster — Tue, 19 Jul 2011 22:26:23 +0200

Guillaume, welcome.

My experience is that it is often not worth to debug user mode applications from a kernel mode debugger. True, the official docus propose to debug Winlogon just as you did. But hey, Winlogon is almost an ordinary user mode application and with a few simple tricks a user mode debugger will do it just fine.

First note that it is wise to debug Winlogon on a remote machine, because it is considered to be part of the OS. If Winlogon crashes or the debugger screws it up the whole system is taken down. Remote debugging is shortly described in WinDbg. From A to Z! - "Remote Debugging with WinDbg" at slide 87. Basically you have to copy dbgsrv.exe, dbgeng.dll and dbghelp.dll to the remote machine, run dbgsrv.exe on a given port, and connect to that port with WinDbg. The additional trick here is that dbgsrv.exe should run as a service so one can connect to it even before any user logs on. There are two wonderful applications, namely Srvany.exe and Instrsrv.exe, that help you to achieve just that. Just follow the steps described here: How To Create a User-Defined Service. Once you set everything up you should see something like this in the registry of your target machine:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DbgService]..
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DbgService\Parameters]
"Application"="C:\\dbgsrv\\dbgsrv.exe -t tcp:port=1222"

When you restart that machine dbgsrv.exe will be up and running waiting for WinDbg connections. Then attaching to Winlogon will be just one more click away..
Bottom line: I would only use the officially proposed solution with a kernel debugger if debugging a user mode application early in the boot process. In all other scenarios the above solution should yield more satisfactory results.

I hope this helps,
Robert

Thread: Brand New to Windbg - Need some basic answers

Jim Flanagan — Mon, 27 Jun 2011 15:57:33 +0200

Hi.. Just joined and am very new to Windbg..
I have a simple need and that is to monitor the calls
made from NTapm.sys into my system bios. I own
a logic analyzer that runs off of Windows 2000 and
was made back in '96 or so. My system uses the old
APM instead of the ACPI method of power management.
One thing that doesn't work correctly is automatic
shutdown of the analyzer hardware whent the 'Shutdown'
command is issued in Win 2000. This has bugged me for
some time. I'm sure that my bios has some incompatibility
with the NTapm.sys/HAL.dll calls and want to see if I can
track it down.

Is this doable by using the Windbg debugger? I simply want to
set breakpoints at the calls into the bios to trace the
register settings, etc. Any help that could be offered would
be appreciated.
thanks
Jim

Thread: Other forums - reply by: Tom Adams

Tom Adams — Sun, 26 Jun 2011 21:58:40 +0200

Awesome answer, very nice.

americancritic

Thread: WinDbg. From A to Z! - reply by: Tom Adams

Tom Adams — Sun, 26 Jun 2011 06:10:54 +0200

Windbg A-Z is amazing, I have never seen anything that can even come close to this. Thank You so very much.