Tài liệu Memory Dump Analysis Anthology- P8 ppt

WinDbg Tips and Tricks 211
0: kd> ub b8d1a068-2
olddriver!TraceRoutine+0xc1
b8d1a051 mov esp,ebp
b8d1a053 pop ebp
b8d1a054 ret
b8d1a055 cmp edi,8
b8d1a058 jbe olddriver!TraceRoutine+0x157 (b8d1a0e7)
b8d1a05e push 206b6444h
b8d1a063 push edx
b8d1a064 push 0
0: kd> .formats 206b6444
Evaluate expression:
Hex: 206b6444
Decimal: 543908932
Octal: 04032662104
Binary: 00100000 01101011 01100100 01000100
Chars: kdD
Time: Sat Mar 28 05:48:52 1987
Float: low 1.99384e-019 high 0
Double: 2.68727e-315

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
212 PART 2: Professional Crash Dump Analysis
OLD DUMPS, NEW EXTENSIONS
Sometimes we can use old Windows 2000 WinDbg extensions to extract informa-
tion from Windows 2003 and XP crash dumps when their native extensions fail. We can
also do the other way around to extract information from old Windows 2000 crash
dumps using WinDbg extensions written for Windows XP and later. Here is an example.
WinDbg !stacks command shows the following not really helpful output from Windows
2000 complete memory dump:

2: kd> !stacks
Proc.Thread Thread Ticks ThreadState Blocker
[System]
8.000004 89df8220 0000000 BLOCKED nt!KiSwapThread+0x1b1
8.00000c 89dc1860 0003734 BLOCKED nt!KiSwapThread+0x1b1
8.000010 89dc15e0 0003734 BLOCKED nt!KiSwapThread+0x1b1
8.000014 89dc1360 00003b4 BLOCKED nt!KiSwapThread+0x1b1
8.000018 89dc10e0 0003734 BLOCKED nt!KiSwapThread+0x1b1
8.00001c 89dc0020 0000381 BLOCKED nt!KiSwapThread+0x1b1
8.000020 89dc0da0 00066f6 BLOCKED nt!KiSwapThread+0x1b1
8.000024 89dc0b20 00025b4 BLOCKED nt!KiSwapThread+0x1b1
8.000028 89dc08a0 00025b4 BLOCKED nt!KiSwapThread+0x1b1
8.00002c 89dc0620 0003734 BLOCKED nt!KiSwapThread+0x1b1
8.000030 89dc03a0 0003734 BLOCKED nt!KiSwapThread+0x1b1
8.000034 89dbf020 00025b4 BLOCKED nt!KiSwapThread+0x1b1
8.000038 89dbfda0 00025b4 BLOCKED nt!KiSwapThread+0x1b1
8.00003c 89dbfb20 00007b4 BLOCKED nt!KiSwapThread+0x1b1
8.000040 89dbf8a0 00007b4 BLOCKED nt!KiSwapThread+0x1b1
8.000044 89dbf620 0000074 BLOCKED nt!KiSwapThread+0x1b1
8.000048 89dbf3a0 00007b4 BLOCKED nt!KiSwapThread+0x1b1
...
...
...
This command belongs to several WinDbg extension DLLs (from WinDbg help):
Windows NT 4.0 Unavailable
Windows 2000 Kdextx86.dll
Windows XP and later Kdexts.dll
and we can try newer kdexts.dll with better results:
2: kd> !winxp\kdexts.stacks
Proc.Thread .Thread Ticks ThreadState Blocker

[89df84a0 System]
8.0000c8 89db77c0 0000000 Blocked nt!MiRemoveUnusedSegments+0xf4
8.0000f0 89c8a020 0019607 Blocked cpqasm2+0x1ef0
8.000108 89881900 0000085 Blocked CPQCISSE+0x3ae8
8.000110 8982cda0 000000a Blocked cpqasm2+0x2a523
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
WinDbg Tips and Tricks 213
8.00013c 8974a9a0 00007d7
Blocked rdbss!RxSetMinirdrCancelRoutine+0x3d
8.000148 89747b20 000010a Blocked rdbss!RxIsOkToPurgeFcb+0x3f
8.00014c 89758a80 0019493
Blocked nt!NtNotifyChangeMultipleKeys+0x434
8.0002dc 89620680 000000e Blocked cpqasm2+0x5523
8.0002e0 89620400 00000d2 Blocked cpqasm2+0x584d
8.0004ac 895ae9c0 000955b Blocked srv!SrvOemStringTo8dot3+0xb7
8.0004c0 8937b4e0 0018fea Blocked srv!SrvOemStringTo8dot3+0xb7
8.0004a0 895b09e0 0018fe9 Blocked srv!SrvOemStringTo8dot3+0xb7
8.0004cc 893784e0 0018fe8 Blocked srv!SrvOemStringTo8dot3+0xb7
8.0004d0 893774e0 000955b Blocked srv!SrvOemStringTo8dot3+0xb7
8.0004d4 893764e0 0018fe8 Blocked srv!SrvOemStringTo8dot3+0xb7
8.003d68 87abb580 00000b7
Blocked rdbss!RxSearchForCollapsibleOpen+0x17c
8.002b94 88e4f180 00000b9
Blocked rdbss!RxSearchForCollapsibleOpen+0x17c
[89736940 smss.exe]
[896d3b20 csrss.exe]
178.000180 896c8020 0000012 Blocked ntdll!NtReplyWaitReceivePort+0xb
178.00018c 896c5320 0000012 Blocked ntdll!NtReplyWaitReceivePort+0xb
178.001260 88fbcb20 0000060 Blocked ntdll!NtReplyWaitReceivePort+0xb
178.001268 88fbbda0 0000060 Blocked ntdll!NtReplyWaitReceivePort+0xb

[896c8740 WINLOGON.EXE]
174.00019c 896b7740 0000299 Blocked ntdll!ZwDelayExecution+0xb
174.0001a0 896b6020 00015dd Blocked ntdll!NtRemoveIoCompletion+0xb
174.000f08 8913eda0 00000b0
Blocked ntdll!ZwWaitForMultipleObjects+0xb
174.000f0c 8901b020 00000b0 Blocked ntdll!ZwWaitForSingleObject+0xb

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
214 PART 2: Professional Crash Dump Analysis
OBJECT NAMES AND WAITING THREADS
Sometimes we have threads waiting for synchronization objects like events and it
is good to know their names or vice versa because it might give some clues to whether
the particular thread and object are relevant for the problem. For example, we have a
thread from !process 0 ff WinDbg command applied to a complete memory dump:
THREAD 86047968 Cid 01e8.04d4 Teb: 7ffaa000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Non-Alertable
8604b750 NotificationEvent
86013070 NotificationEvent
Not impersonating
DeviceMap e1007d00
Owning Process 86014ba0 Image: winlogon.exe
Wait Start TickCount 997 Ticks: 788709 (0:03:25:23.578)
Context Switch Count 1
UserTime 00:00:00.000
KernelTime 00:00:00.000
Win32 Start Address USERENV!NotificationThread (0×76929dd9)
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init f5d48000 Current f5d47914 Base f5d48000 Limit f5d45000 Call 0
Priority 10 BasePriority 10 PriorityDecrement 0
Kernel stack not resident.

ChildEBP RetAddr
f5d4792c 8082ffb7 nt!KiSwapContext+0×25
f5d47944 808282b0 nt!KiSwapThread+0×83
f5d47978 80930d34 nt!KeWaitForMultipleObjects+0×320
f5d47bf4 80930e96 nt!ObpWaitForMultipleObjects+0×202
f5d47d48 80883908 nt!NtWaitForMultipleObjects+0xc8
f5d47d48 7c8285ec nt!KiFastCallEntry+0xf8
00f1fec0 7c827cfb ntdll!KiFastSystemCallRet
00f1fec4 77e6202c ntdll!NtWaitForMultipleObjects+0xc
00f1ff6c 77e62fbe kernel32!WaitForMultipleObjectsEx+0×11a
00f1ff88 76929e35 kernel32!WaitForMultipleObjects+0×18
00f1ffb8 77e64829 USERENV!NotificationThread+0×5f
00f1ffec 00000000 kernel32!BaseThreadStart+0×34
or we switched to winlogon.exe process and we are inspecting this thread:
kd> .process 86014ba0
Implicit process is now 86014ba0
kd> .reload /user
Loading User Symbols
kd> .thread 86047968
Implicit thread is now 86047968
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
WinDbg Tips and Tricks 215
kd> kv
*** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr Args to Child
f5d4792c 8082ffb7 86047968 ffdff120 00002700 nt!KiSwapContext+0x25
f5d47944 808282b0 86047968 00000002 00000000 nt!KiSwapThread+0x83
f5d47978 80930d34 00000002 f5d47aac 00000001
nt!KeWaitForMultipleObjects+0×320
f5d47bf4 80930e96 00000002 f5d47c1c 00000001

nt!ObpWaitForMultipleObjects+0×202
f5d47d48 80883908 00000002 00f1ff10 00000001
nt!NtWaitForMultipleObjects+0xc8
f5d47d48 7c8285ec 00000002 00f1ff10 00000001 nt!KiFastCallEntry+0xf8
00f1fec0 7c827cfb 77e6202c 00000002 00f1ff10 ntdll!KiFastSystemCallRet
00f1fec4 77e6202c 00000002 00f1ff10 00000001
ntdll!NtWaitForMultipleObjects+0xc
00f1ff6c 77e62fbe 00000002 769cd34c 00000000
kernel32!WaitForMultipleObjectsEx+0×11a
00f1ff88 76929e35 00000002 769cd34c 00000000
kernel32!WaitForMultipleObjects+0×18
00f1ffb8 77e64829 00000000 00000000 00000000
USERENV!NotificationThread+0×5f
00f1ffec 00000000 76929dd9 00000000 00000000 kernel32!BaseThreadStart+0×34
kd> dd f5d47aac l2
f5d47aac 8604b750 86013070
WinDbg !object command shows names for named synchronization objects:
kd> !object 8604b750
Object: 8604b750 Type: (86598990) Event
ObjectHeader: 8604b738 (old version)
HandleCount: 1 PointerCount: 2
kd> !object 86013070
Object: 86013070 Type: (86598990) Event
ObjectHeader: 86013058 (old version)
HandleCount: 10 PointerCount: 18
Directory Object: e19b61c0 Name: userenv: Machine Group Policy has
been applied
We see that one object is named and related to group policies. The same tech-
nique can be applied in reverse. For example, we want to find which thread is waiting
for 85efb848 event:

kd> !object \BaseNamedObjects
Object: e19b61c0 Type: (865cab50) Directory
ObjectHeader: e19b61a8 (old version)
HandleCount: 75 PointerCount: 259
Directory Object: e10012c8 Name: BaseNamedObjects
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
216 PART 2: Professional Crash Dump Analysis
Hash Address Type Name
---- ------- ---- ----

...
...
...
861697f0 Event COM+ Tracker Push Event
85f6fbb0 Event WMI_ProcessIdleTasksComplete
85efb848 Event VMwareToolsServiceEvent
…
…
…
Looking at threads from !process 0 ff command we find that VMwareService.exe
uses it:
THREAD 8633bd40 Cid 0664.0680 Teb: 7ffde000 Win32Thread: 00000000 WAIT:
(Unknown) UserMode Alertable
85efb848 SynchronizationEvent
8633bdb8 NotificationTimer
Not impersonating
DeviceMap e1007d00
Owning Process 862fa938 Image: VMwareService.exe
Wait Start TickCount 789703 Ticks: 3 (0:00:00:00.046)
Context Switch Count 120485

UserTime 00:00:00.093
KernelTime 00:00:00.062
Win32 Start Address ADVAPI32!ScSvcctrlThreadA (0×77f65e70)
Start Address kernel32!BaseThreadStartThunk (0×77e617ec)
Stack Init f5cc8000 Current f5cc7914 Base f5cc8000 Limit f5cc5000 Call 0
Priority 15 BasePriority 15 PriorityDecrement 0
ChildEBP RetAddr
f5cc792c 8082ffb7 nt!KiSwapContext+0×25
f5cc7944 808282b0 nt!KiSwapThread+0×83
f5cc7978 80930d34 nt!KeWaitForMultipleObjects+0×320
f5cc7bf4 80930e96 nt!ObpWaitForMultipleObjects+0×202
f5cc7d48 80883908 nt!NtWaitForMultipleObjects+0xc8
f5cc7d48 7c8285ec nt!KiFastCallEntry+0xf8
00a5fe4c 7c827cfb ntdll!KiFastSystemCallRet
00a5fe50 77e6202c ntdll!NtWaitForMultipleObjects+0xc
00a5fef8 0040158e kernel32!WaitForMultipleObjectsEx+0×11a
WARNING: Stack unwind information not available. Following frames may be
wrong.
00a5ff18 00402390 VMwareService+0×158e
00a5ff84 00402f5a VMwareService+0×2390
00a5ffa4 77f65e91 VMwareService+0×2f5a
00a5ffb8 77e64829 ADVAPI32!ScSvcctrlThreadW+0×21
00a5ffec 00000000 kernel32!BaseThreadStart+0×34
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
WinDbg Tips and Tricks 217
!object command is equivalent to WinObj tool ( />us/sysinternals/bb896657.aspx) and allows inspecting Windows Object Manager names-
pace that existed at the time when a memory dump was saved. Here is the root direc-
tory from my x64 Vista workstation:
lkd> !object \
Object: fffff880000056c0 Type: (fffffa800183fde0) Directory

ObjectHeader: fffff88000005690 (old version)
HandleCount: 0 PointerCount: 50
Directory Object: 00000000 Name: \
Hash Address Type Name
---- ------- ---- ----
01 fffff88000005510 Directory ObjectTypes
03 fffffa80047574e0 Event NETLOGON_SERVICE_STARTED
05 fffff8800156fb00 SymbolicLink SystemRoot
06 fffff880018bfeb0 Directory Sessions
07 fffffa800448eb90 ALPC Port MmcssApiPort
08 fffff8800000a060 Directory ArcName
09 fffff88000081e10 Directory NLS
fffffa80047523c0 ALPC Port XactSrvLpcPort
10 fffffa8004504e60 ALPC Port ThemeApiPort
fffff880018efce0 Directory Windows
fffff88000007bd0 Directory GLOBAL??
fffffa8004199de0 Event LanmanServerAnnounceEvent
fffffa80043027d0 Event DSYSDBG.Debug.Trace.Memory.2a4
11 fffff8800189feb0 Directory RPC Control
13 fffffa8003ed6490 Event EFSInitEvent
14 fffffa8002746bd0 Device clfs
fffff88000fb6b10 -
15 fffffa8003dd5060 ALPC Port SeRmCommandPort
fffffa80040c7210 Event CsrSbSyncEvent
16 fffff880000052e0 SymbolicLink DosDevices
fffffa8004626c70 Device Cdfs
17 fffff8800471c210 Directory KnownDlls32
fffffa8004770490 ALPC Port AELPort
fffffa8004342680 Event EFSSrvInitEvent
18 fffff8800000a2b0 Key \REGISTRY

fffffa8004851900 ALPC Port WindowsErrorReportingServicePort
19 fffff88004732380 Directory BaseNamedObjects
21 fffff88000072d00 Directory UMDFCommunicationPorts
fffffa8004182120 ALPC Port SmSsWinStationApiPort
fffffa8003ddbe60 Event UniqueInteractiveSessionIdEvent
22 fffff88000875a00 Directory KnownDlls
fffffa8003ece330 Device FatCdrom
fffffa8003a16720 Device Fat
23 fffff88000005120 Directory KernelObjects
fffff88000081ab0 Directory FileSystem
fffffa8002a5f620 Device Ntfs
26 fffff88000007300 Directory Callback
fffffa80042e14c0 ALPC Port SeLsaCommandPort
28 fffff880000095f0 Directory Security
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
218 PART 2: Professional Crash Dump Analysis
29 fffffa8004574e60 ALPC Port UxSmsApiPort
30 fffff88000013060 Directory Device
fffffa8004342700 Event EFSSmbInitEvent
32 fffffa8004342260 ALPC Port LsaAuthenticationPort
34 fffffa8003dd7e60 ALPC Port SmApiPort
fffff88004bf5080 Section LsaPerformance
fffffa8003f65160 Event UniqueSessionIdEvent
36 fffff88000081c60 Directory Driver
fffffa8004308c00 Event SAM_SERVICE_STARTED
We can inspect any directory or object, for example:
lkd> !object \FileSystem
Object: fffff88000081ab0 Type: (fffffa800183fde0) Directory
ObjectHeader: fffff88000081a80 (old version)
HandleCount: 0 PointerCount: 31

Directory Object: fffff880000056c0 Name: FileSystem
Hash Address Type Name
---- ------- ---- ----
02 Unable to read directory entry at fffff88004d46ca0
03 fffffa80041a9bc0 Driver mrxsmb20
04 fffffa8004371450 Driver luafv
11 fffffa8003e3b530 Driver rdbss
fffffa8003c6e470 Device CdfsRecognizer
12 fffffa800261c300 Device UdfsDiskRecognizer
fffffa8003c6e680 Driver Fs_Rec
13 fffffa8002626e70 Driver Msfs
15 fffffa8003edc7e0 Driver DfsC
16 fffffa8004640e70 Driver cdfs
17 fffffa800410ed90 Driver srvnet
19 fffffa80046f9420 Driver srv
fffffa800468cc90 Driver MRxDAV
fffff88000072eb0 Directory Filters
21 fffffa80046be400 Driver bowser
fffffa8001c92c40 Driver FltMgr
22 fffffa800261cc40 Device FatCdRomRecognizer
23 fffffa8002756e70 Driver Ntfs
24 fffffa8003dc0530 Driver Npfs
fffffa80027abd20 Driver Mup
fffffa80018476a0 Driver RAW
27 fffffa8003f04270 Driver fastfat
28 fffffa8002745060 Driver FileInfo
31 fffffa800261ce50 Device FatDiskRecognizer
33 fffffa80046c4650 Driver srv2
fffffa8003eaf470 Driver NetBIOS
fffffa800261ca30 Device ExFatRecognizer

34 fffffa8003ce3610 Driver SRTSP
35 fffffa800261c060 Device UdfsCdRomRecognizer

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
WinDbg Tips and Tricks 219
MEMORY DUMPS FROM VIRTUAL IMAGES
Although I haven’t found the way to distinguish the process dump taken from a
physical machine versus virtualized machine there is a way to see it from kernel and
complete memory dumps if VMware Tools are installed inside the guest Windows OS:
kd> !vm
...
...
...
1098 VMwareUser.exe 350 ( 1400 Kb)
...
14e4 VMwareTray.exe 317 ( 1268 Kb)
...
0664 VMwareService.e 190 ( 760 Kb)
...
...
...
In case of a kernel minidump we can check for VMware drivers (as we can ob-
viously do with kernel and complete memory dumps):
kd> lmt m vm*
start end module name
bf9e6000 bf9faa80 vmx_fb Tue Oct 04 08:13:32 2005
f6e8b000 f6e8ed80 vmx_svga Tue Oct 04 08:13:02 2005
f77e7000 f77ede80 vmxnet Sat Apr 22 23:13:11 2006
f7997000 f7998200 vmmouse Tue Aug 02 20:07:49 2005
f79c9000 f79ca5c0 vmmemctl Thu Jul 26 21:50:03 2007

If VMware Tools are not installed we can check machine id:
kd> !sysinfo machineid
Machine ID Information [From Smbios 2.31, DMIVersion 0, Size=1642]
BiosVendor = Phoenix Technologies LTD
BiosVersion = 6.00
BiosReleaseDate = 04/17/2006
SystemManufacturer = VMware, Inc.
SystemProductName = VMware Virtual Platform
SystemVersion = None
BaseBoardManufacturer = Intel Corporation
BaseBoardProduct = 440BX Desktop Reference Platform
BaseBoardVersion = None

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
220 PART 2: Professional Crash Dump Analysis
FILTERING PROCESSES
When I analyze memory dumps coming from Microsoft or Citrix terminal service
environments I frequently need to find a process hosting terminal service. In Windows
2000 it was the separate process termsrv.exe and now it is termsrv.dll which can be
loaded into any of several instances of svchost.exe. The simplest way to narrow down
that svchost.exe process if we have a complete memory dump is to use the module op-
tion of WinDbg !process command:
!process /m termsrv.dll 0
!process /m wsxica.dll 0
!process /m ctxrdpwsx.dll 0
Note: this option works only with W2K3, XP and later OS
Also to list all processes with user space stacks having the same image name we
can use the following command:
!process 0 ff msiexec.exe
or

!process 0 ff svchost.exe
Note: this command works with W2K too as well as session option (/s)





Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
WinDbg Scripts 221
WINDBG SCRIPTS
FIRST ENCOUNTERS
Sometimes instead of writing a debugging extension it is much faster to write a
script. After spending some time I wrote the final version of my first script (based on
WinDbg help sample) which can enumerate processes in a complete memory dump and
output their command line.
I saved the script below in a text file and used the following command to run it
from WinDbg command prompt: $$><script.txt
$$ WinDbg script to get process command line for all processes in complete
memory dump
r $t0 = nt!PsActiveProcessHead
.for (r $t1 = poi(@$t0); (@$t1 != 0) & (@$t1 != @$t0);
r $t1 = poi(@$t1))
{
r? $t2 = #CONTAINING_RECORD(@$t1,
nt!_EPROCESS, ActiveProcessLinks);
.process @$t2
.if (@$peb != 0)
{
.catch
{

r $t3 = @@c++(@$peb->ProcessParameters)
r? $t4 =
@@c++(&((_RTL_USER_PROCESS_PARAMETERS *)
@$t3)->CommandLine)
.printf "_EPROCESS: %N Command Line: %msu\n",
@$t2, @$t4
}
}
}

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
222 PART 2: Professional Crash Dump Analysis
YET ANOTHER WINDBG SCRIPT
One day I got a Windows 2000 server crash dump with 30 IE processes running
and I wanted to find the only one waiting for a specific function. I knew there was one
and I wrote the following script to list all processes and their stacks (of course, I already
opened a log in WinDbg to save that huge amount of output):
$$
$$ List user processes and stacks
$$
r $t0 = nt!PsActiveProcessHead
.for (r $t1 = poi(@$t0); (@$t1 != 0) & (@$t1 != @$t0); r $t1 = poi(@$t1))
{
r? $t2 = #CONTAINING_RECORD(@$t1, nt!_EPROCESS, ActiveProcessLinks);
.process @$t2
.reload
!process @$t2
}
In memory dumps coming from XP/W2K3 and higher systems you can get all of
this plus PEB and module information for all processes by using !process 0 ff WinDbg

command. The command and flags sets process context for every process and reloads
user symbols accordingly.
Another alternative would be to use the following command instead of the script:
!for_each_process ".process /r /p @#Process; !process @#Process"


Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.