Tài liệu Memory Dump Analysis Anthology- P6 docx

Bugchecks Depicted 151
kd> kL
Child-SP RetAddr Call Site
fffffadf`dfcf19b8 fffffadf`dfee38c4 nt!KeBugCheck
fffffadf`dfcf19c0 fffff800`012ce9cf userdump!UdIoctl+0x104
fffffadf`dfcf1a70 fffff800`012df026 nt!IopXxxControlFile+0xa5a
fffffadf`dfcf1b90 fffff800`010410fd nt!NtDeviceIoControlFile+0x56
fffffadf`dfcf1c00 00000000`77ef0a5a nt!KiSystemServiceCopyEnd+0x3
00000000`01eadd58 00000001`0000a755 ntdll!NtDeviceIoControlFile+0xa
00000000`01eadd60 00000000`77ef30a5
userdump_100000000!UdServiceWorkerAPC+0x1005
00000000`01eaf970 00000000`77ef0a2a ntdll!KiUserApcDispatcher+0x15
00000000`01eafe68 00000001`00007fe2 ntdll!NtWaitForSingleObject+0xa
00000000`01eafe70 00000001`00008a39
userdump_100000000!UdServiceWorker+0xb2
00000000`01eaff20 000007ff`7fee4db6
userdump_100000000!UdServiceStart+0x139
00000000`01eaff50 00000000`77d6b6da ADVAPI32!ScSvcctrlThreadW+0x25
00000000`01eaff80 00000000`00000000 kernel32!BaseThreadStart+0x3a
This might be useful if we want to see kernel data that happened to be at the
exception time. In this case we can avoid requesting complete memory dump of physi-
cal memory and ask for kernel memory dump only together with a user dump.
Note: do not set this option if you are unsure. It can have your production servers
bluescreen in the case of false positive dumps.

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
152 PART 2: Professional Crash Dump Analysis
CF
Bugcheck CF name is the second longest one:
TERMINAL_SERVER_DRIVER_MADE_INCORRECT_MEMORY_REFERENCE (cf)
Arguments:

Arg1: a020b1d4, memory referenced
Arg2: 00000000, value 0 = read operation, 1 = write operation
Arg3: a020b1d4, If non-zero, the instruction address which referenced the
bad memory
address.
Arg4: 00000000, Mm internal code.
A driver has been incorrectly ported to Terminal Server. It is
referencing session space addresses from the system process
context. Probably from queueing an item to a system worker thread. The
broken driver's name is displayed on the screen.
Although bugcheck explanation mentions only system process context it can also
happen in an arbitrary process context. Recall that kernel space address mapping is
usually considered as persistent where virtual-to-physical mapping doesn’t change be-
tween switching threads that belong to different processes. However there is so called
session space in multi-user terminal services environments where different users can
use different display drivers, for example:
MS RDP users - RDPDD.DLL
Citrix ICA users - VDTW30.DLL
Vista users - TSDDD.DLL
Console user - Some H/W related video driver like ATI or NVIDIA
These drivers are not committed at the same time persistently since OS boot al-
though their module addresses might remain fixed. Therefore when a new user session
is created the appropriate display driver corresponding to terminal services protocol is
loaded and mapped to the so called session space starting from A0000000 (x86) or
FFFFF90000000000 (x64) after win32k.sys address range (on first usage) and then
committed to physical memory by proper PTE entries in process page tables. During
thread switch, if the new process context belongs to a different session with a different
display driver, the current display driver is decommitted by clearing its PTEs and the new
driver is committed by setting its proper PTE entries.
Therefore in the system process context like worker threads virtual addresses

corresponding to display driver code and data might be unknown. This can also hap-
pen in an arbitrary process context if we access the code that belongs to a display driver
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
Bugchecks Depicted 153
that doesn’t correspond to the current session protocol. This can be illustrated with the
following example where TSDD can be either RDP or ICA display driver.
In the list of loaded modules we can see that ATI and TSDD drivers are loaded:
0: kd> lm
start end module name
...
...
...
77d30000 77d9f000 RPCRT4 (deferred)
77e10000 77e6f000 USER32 (deferred)
77f40000 77f7c000 GDI32 (deferred)
77f80000 77ffc000 ntdll (pdb symbols)
78000000 78045000 MSVCRT (deferred)
7c2d0000 7c335000 ADVAPI32 (deferred)
7c340000 7c34f000 Secur32 (deferred)
7c570000 7c624000 KERNEL32 (deferred)
7cc30000 7cc70000 winsrv (deferred)
80062000 80076f80 hal (deferred)
80400000 805a2940 nt (pdb symbols)
a0000000 a0190ce0 win32k (pdb symbols)
a0191000 a01e8000 ati2drad (deferred)
a01f0000 a0296000 tsdd (no symbols)
b4a60000 b4a72320 naveng (deferred)
b4a73000 b4b44c40 navex15 (deferred)
…
…

…
The bugcheck happens in 3rd-partyApp process context running inside
some terminal session:
PROCESS_NAME: 3rd-partyApp.exe
TRAP_FRAME: b475f84c -- (.trap 0xffffffffb475f84c)
ErrCode = 00000000
eax=a020b1d4 ebx=00000000 ecx=04e0443b edx=ffffffff esi=a21b6778
edi=a201b018
eip=a020b1d4 esp=b475f8c0 ebp=b475f900 iopl=3 nv up ei pl zr na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00013246
TSDD+0×1b1d4:
a020b1d4 ?? ???

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
154 PART 2: Professional Crash Dump Analysis
Examining display driver virtual address shows that it is unknown (PTE is NULL):
0: kd> !pte a020b1d4
A020B1D4 - PDE at C0300A00 PTE at C028082C
contains 14AB6863 contains 00000000
pfn 14ab6 –DA–KWV not valid
ATI display driver address is unknown too:
0: kd> !pte a0191000
A0191000 - PDE at C0300A00 PTE at C0280644
contains 3E301863 contains 00000000
pfn 3e301 –DA–KWV not valid
Let’s switch to another terminal session :
PROCESS 87540a60 SessionId: 45 Cid: 3954 Peb: 7ffdf000 ParentCid:
0164
DirBase: 2473d000 ObjectTable: 889b2c48 TableSize: 182.
Image: csrss.exe

0: kd> .process /r /p 87540a60
Implicit process is now 87540a60
Loading User Symbols
................
Now TSDD display driver address is valid:
0: kd> !pte a020b1d4
A020B1D4 - PDE at C0300A00 PTE at C028082C
contains 5D2C2863 contains 20985863
pfn 5d2c2 –DA–KWV pfn 20985 –DA–KWV
but ATI driver address is not. It is unknown and this is expected because no real
display hardware is used:
0: kd> !pte a0191000
A0191000 - PDE at C0300A00 PTE at C0280644
contains 5D2C2863 contains 00000000
pfn 5d2c2 –DA–KWV not valid

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
Bugchecks Depicted 155
Let’s switch to session 0 where the display is “physical”:
PROCESS 8898a5e0 SessionId: 0 Cid: 0180 Peb: 7ffdf000 ParentCid:
0164
DirBase: 14c58000 ObjectTable: 8898b948 TableSize: 1322.
Image: csrss.exe
0: kd> .process /r /p 8898a5e0
Implicit process is now 8898a5e0
Loading User Symbols
..............
TSDD driver address is unknown and this is expected too because we no longer
use terminal services protocol:
0: kd> !pte a020b1d4

A020B1D4 - PDE at C0300A00 PTE at C028082C
contains 14AB6863 contains 00000000
pfn 14ab6 –DA–KWV not valid
However ATI display driver addresses are not unknown (not NULL) and their 2 se-
lected pages are either in transition or in a page file:
0: kd> !pte a0191000
A0191000 - PDE at C0300A00 PTE at C0280644
contains 14AB6863 contains 156DD882
pfn 14ab6 –DA–KWV not valid
Transition: 156dd
Protect: 4
0: kd> !pte a0198000
A0198000 - PDE at C0300A00 PTE at C0280660
contains 14AB6863 contains 000B9060
pfn 14ab6 –DA–KWV not valid
PageFile 0
Offset b9
Protect: 3



Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
156 PART 2: Professional Crash Dump Analysis

Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
Manual Stack Trace Reconstruction 157
MANUAL STACK TRACE RECONSTRUCTION
This is a small case study to complement Incorrect Stack Trace pattern (page
288) and show how to reconstruct stack trace manually based on an example
with complete source code.

For it I created a small working multithreaded program:
#include "stdafx.h"
#include <stdio.h>
#include <process.h>
typedef void (*REQ_JUMP)();
typedef void (*REQ_RETURN)();
const char str[] = "\0\0\0\0\0\0\0";
bool loop = true;
void return_func()
{
puts("Return Func");
loop = false;
_endthread();
}
void jump_func()
{
puts("Jump Func");
}
void internal_func_2(void *param_jump,void *param_return)
{
REQ_JUMP f_jmp = (REQ_JUMP)param_jump;
REQ_RETURN f_ret = (REQ_RETURN)param_return;
puts("Internal Func 2");
// Uncomment memcpy to crash the program
// Overwrite f_jmp and f_ret with NULL
// memcpy(&f_ret, str, sizeof(str));
__asm
{
push f_ret;
mov eax, f_jmp

mov ebp, 0 // use ebp as a general purpose register
jmp eax
}
}
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
158 PART 2: Professional Crash Dump Analysis
void internal_func_1(void *param)
{
puts("Internal Func 1");
internal_func_2(param, &return_func);
}
void thread_request(void *param)
{
puts("Request");
internal_func_1(param);
}
int _tmain(int argc, _TCHAR* argv[])
{
_beginthread(thread_request, 0, (void *)jump_func);
while (loop);
return 0;
}
I had to disable optimizations in Visual C++ compiler otherwise most of the code
would have been eliminated because the program is very small and easy for code opti-
mizer. If we run the program it displays the following output:
Request
Internal Func 1
Internal Func 2
Jump Func
Return Func

internal_func_2 gets two parameters: the function address to jump and the func-
tion address to call upon the return. The latter sets loop variable to false in order to
break infinite main thread loop and calls _endthread. Why we need this complexity in
the small sample? Because I wanted to simulate FPO optimization in an inner function
call and also gain control over a return address. This is why I set EBP to zero before
jumping and pushed the custom return address which I can change any time. If I used
the call instruction then the processor would have determined the return address as the
next instruction address.
The code also copies two internal_func_2 parameters into local variables f_jmp
and f_ret because the commented memcpy call is crafted to overwrite them with zeroes
and do not touch the saved EBP, return address and function arguments. This is all to
make the stack trace incorrect but at the same time to make manual stack reconstruc-
tion as easy as possible in this example.
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
Manual Stack Trace Reconstruction 159
Let’s suppose that memcpy call is a bug that overwrites local variables. Then we
obviously have a crash because EAX is zero and jump to zero address will cause access
violation. EBP is also 0 because we assigned 0 to it explicitly. Let’s pretend that we
wanted to pass some constant via EBP and it is zero.
What we have now:
EBP is 0
EIP is 0
Return address is 0
When we load a crash dump WinDbg is utterly confused because it has no clue
on how to reconstruct the stack trace:
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(bd0.ec8): Access violation – code c0000005 (first/second chance not
available)
eax=00000000 ebx=00595620 ecx=00000002 edx=00000000 esi=00000000

edi=00000000
eip=00000000 esp=0069ff54 ebp=00000000 iopl=0 nv up ei pl nz ac po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010212
00000000 ?? ???
0:001> kv
ChildEBP RetAddr Args to Child
WARNING: Frame IP not in any known module. Following frames may be wrong.
0069ff50 00000000 00000000 00000000 0069ff70 0×0
Fortunately ESP is not zero so we can look at raw stack:
0:001> dds esp
0069ff54 00000000
0069ff58 00000000
0069ff5c 00000000
0069ff60 0069ff70
0069ff64 0040187f WrongIP!internal_func_1+0x1f
0069ff68 00401830 WrongIP!jump_func
0069ff6c 00401840 WrongIP!return_func
0069ff70 0069ff7c
0069ff74 0040189c WrongIP!thread_request+0xc
0069ff78 00401830 WrongIP!jump_func
0069ff7c 0069ffb4
0069ff80 78132848 msvcr80!_endthread+0x4b
0069ff84 00401830 WrongIP!jump_func
0069ff88 aa75565b
0069ff8c 00000000
0069ff90 00000000
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
160 PART 2: Professional Crash Dump Analysis
0069ff94 00595620
0069ff98 c0000005

0069ff9c 0069ff88
0069ffa0 0069fb34
0069ffa4 0069ffdc
0069ffa8 78138cd9 msvcr80!_except_handler4
0069ffac d207e277
0069ffb0 00000000
0069ffb4 0069ffec
0069ffb8 781328c8 msvcr80!_endthread+0xcb
0069ffbc 7d4dfe21 kernel32!BaseThreadStart+0x34
0069ffc0 00595620
0069ffc4 00000000
0069ffc8 00000000
0069ffcc 00595620
0069ffd0 c0000005
Here we can start searching for the following pairs:
EBP: PreviousEBP
Function return address
…
…
…
PreviousEBP: PrePreviousEBP
Function return address
…
…
…
for example:
0:001> dds esp
0069ff54 00000000
0069ff58 00000000
0069ff5c 00000000

0069ff60 0069ff70
0069ff64 0040187f WrongIP!internal_func_1+0×1f
0069ff68 00401830 WrongIP!jump_func
0069ff6c 00401840 WrongIP!return_func
0069ff70 0069ff7c
0069ff74 0040189c WrongIP!thread_request+0xc
0069ff78 00401830 WrongIP!jump_func
0069ff7c 0069ffb4
This is based on the fact that a function call saves its return address and the stan-
dard function prolog saves the previous EBP value and sets ESP to point to it.
Push ebp
mov ebp, esp
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
Manual Stack Trace Reconstruction 161
Therefore our stack looks like this:
0:001> dds esp
0069ff54 00000000
0069ff58 00000000
0069ff5c 00000000
0069ff60 0069ff70
0069ff64 0040187f WrongIP!internal_func_1+0×1f
0069ff68 00401830 WrongIP!jump_func
0069ff6c 00401840 WrongIP!return_func
0069ff70 0069ff7c
0069ff74 0040189c WrongIP!thread_request+0xc
0069ff78 00401830 WrongIP!jump_func
0069ff7c 0069ffb4
0069ff80 78132848 msvcr80!_endthread+0×4b
0069ff84 00401830 WrongIP!jump_func
0069ff88 aa75565b

0069ff8c 00000000
0069ff90 00000000
0069ff94 00595620
0069ff98 c0000005
0069ff9c 0069ff88
0069ffa0 0069fb34
0069ffa4 0069ffdc
0069ffa8 78138cd9 msvcr80!_except_handler4
0069ffac d207e277
0069ffb0 00000000
0069ffb4 0069ffec
0069ffb8 781328c8 msvcr80!_endthread+0xcb
0069ffbc 7d4dfe21 kernel32!BaseThreadStart+0×34
0069ffc0 00595620
0069ffc4 00000000
0069ffc8 00000000
0069ffcc 00595620
0069ffd0 c0000005
We also double check return addresses to see if they are valid code indeed. The
best way is to disassemble them backwards. This should show call instructions resulted
in saved return addresses:
0:001> ub WrongIP!internal_func_1+0x1f
WrongIP!internal_func_1+0x1:
00401871 mov ebp,esp
00401873 push offset WrongIP!GS_ExceptionPointers+0x38 (00402124)
00401878 call dword ptr [WrongIP!_imp__puts (004020ac)]
0040187e add esp,4
00401881 push offset WrongIP!return_func (00401850)
00401886 mov eax,dword ptr [ebp+8]
00401889 push eax

0040188a call WrongIP!internal_func_2 (004017e0)
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.
162 PART 2: Professional Crash Dump Analysis
0:001> ub WrongIP!thread_request+0xc
WrongIP!internal_func_1+0x2d:
0040189d int 3
0040189e int 3
0040189f int 3
WrongIP!thread_request:
004018a0 push ebp
004018a1 mov ebp,esp
004018a3 mov eax,dword ptr [ebp+8]
004018a6 push eax
004018a7 call WrongIP!internal_func_1 (00401870)
0:001> ub msvcr80!_endthread+0x4b
msvcr80!_endthread+0x2f:
7813282c pop esi
7813282d push 0Ch
7813282f push offset msvcr80!__rtc_tzz+0x64 (781b4b98)
78132834 call msvcr80!_SEH_prolog4 (78138c80)
78132839 call msvcr80!_getptd (78132e29)
7813283e and dword ptr [ebp-4],0
78132842 push dword ptr [eax+58h]
78132845 call dword ptr [eax+54h]
0:001> ub msvcr80!_endthread+0xcb
msvcr80!_endthread+0xaf:
781328ac mov edx,dword ptr [ecx+58h]
781328af mov dword ptr [eax+58h],edx
781328b2 mov edx,dword ptr [ecx+4]
781328b5 push ecx

781328b6 mov dword ptr [eax+4],edx
781328b9 call msvcr80!_freefls (78132e41)
781328be call msvcr80!_initp_misc_winxfltr (781493c1)
781328c3 call msvcr80!_endthread+0×30 (7813282d)
0:001> ub BaseThreadStart+0x34
kernel32!BaseThreadStart+0x10:
7d4dfdfd mov eax,dword ptr fs:[00000018h]
7d4dfe03 cmp dword ptr [eax+10h],1E00h
7d4dfe0a jne kernel32!BaseThreadStart+0x2e (7d4dfe1b)
7d4dfe0c cmp byte ptr [kernel32!BaseRunningInServerProcess
(7d560008)],0
7d4dfe13 jne kernel32!BaseThreadStart+0x2e (7d4dfe1b)
7d4dfe15 call dword ptr [kernel32!_imp__CsrNewThread (7d4d0310)]
7d4dfe1b push dword ptr [ebp+0Ch]
7d4dfe1e call dword ptr [ebp+8]
Now we can use extended version of k command and supply custom EBP, ESP
and EIP values. We set EBP to the first found address of EBP:PreviousEBP pair and set
EIP to 0:
Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark.