Tài liệu Real World Analysis docx

1
1
Network Traffic Analysis
Using tcpdump
Judy Novak
Judy Novak

Real World Analysis
All material Copyright  Novak, 2000, 2001. All rights reserved.
2
2
Real World Examples
•
Introduction to tcpdump
•
Writing tcpdump Filters
•
Examination of Datagram Fields
•
Beginning Analysis
•
Real World Examples
•
Step by Step Analysis
This page intentionally left blank.
3
3
Objectives
•
Learn advanced analysis through examining real
world examples

The final challenge in intrusion detection was to synthesize confirmed anomalies to further
understand the exploit or trends. In this section, we demonstrate, using examples, how we delve into
the analysis process and come up with a more complete explanation of the activity.
4
4
You’ve Had a Compromise!!!!
•
Notification from parent CERT organization of
compromised host
•
The notifier only knew:
•
IP numbers of the hostile host and
compromised host
•
Day and approximate time of incident
•
What tools have we learned about that might help
us validate and investigate this incident?
For starters, we need to examine tcpdump output for the time of the reported event to determine out
what happened.
From there, we can use the new scripts we created if more detail is required.
So, tcpdump records were dumped for the site:
tcpdump –r tcpdumpfile ‘hostile-IP and compromised-IP’
where hostile-IP and compromised-IP were the actual IP’s.
5
5
What a “Normal” TCP Connection
Session Looks Like
•

Before looking at output from the tcpdump output from the break-in,
let’s examine typical output from a telnet session
boulder.myplace.com.38060 > aspen.myplace.com.telnet: S 3774957990:
3774957990(0) win 8760 <mss 1460> (DF)
aspen.myplace.com.telnet > boulder.myplace.com.38060: S 2009600000:
2009600000(0) ack 3774957991 win 1024 <mss 1460>
boulder.myplace.com.38060 > aspen.myplace.com.telnet:. ack 1 win 8760 (DF)
aspen.myplace.com.telnet > boulder.myplace.com.38060: F 4289:4289(0) ack 92
win 1024
boulder.myplace.com.38060 > aspen.myplace.com.telnet: .ack 4290 win 8760 (DF)
boulder.myplace.com.38060 > aspen.myplace.com.telnet: F 92:92(0) ack 4290
win 8760(DF)
aspen.myplace.com.telnet > boulder.myplace.com.38060: .ack 93 win 1024
boulder.myplace.com.38060 > aspen.myplace.com.telnet: P 1:28(27) ack 1 win 8760
(DF)
aspen.myplace.com.telnet > boulder.myplace.com.38060: P 1:14(13) ack 1 win 1024
aspen.myplace.com.telnet > boulder.myplace.com.38060: P 14:23(9) ack 28 win 1024
A TCP session where connectivity has been established and data has been exchanged has the pattern
of the above records.
Namely:
• The connection is established by:
• The client (boulder) sending a SYN to the server (aspen) to the desired port
• The server (aspen) sending a SYN/ACK back to the client (boulder)
acknowledging the request
• The client (boulder) sending an ACK to the server (aspen) to complete the three-way
handshake
• Data is exchanged:
• The PUSH flag is set for data transmission
• Actual data bytes are sent (this can be seen before the ACK on records with the PUSH set,
you see values in parentheses, i.e. ( 27), (13), (9))

• The connection is stopped:
• Gracefully with the two-way FIN exchange
• Aborted with a RESET
Note: timestamps have been removed from the tcpdump records to make the records more readable.
6
6
tcpdump Output from the
Break-in
whatsup.net.24997 > dns.myplace.com.sunrpc: S 2368718861:2368718861(0)
win 512 <mss 1460>
whatsup.net.25002 > dns.myplace.com.139: S 4067302570:4067302570(0)
win 512 <mss 1460>
whatsup.net.25075 > dns.myplace.com.ftp: S 1368714289:1368714289(0)
win 512 <mss 1460>
dns.myplace.com.ftp > whatsup.net.25075: R 0:0(0) ack 1368714290 win 0 (DF)
whatsup.net.25177 > dns.myplace.com.1114: S 3231175487:3231175487(0)
win 512 <mss 1460>
whatsup.net.25189 > dns.myplace.com.tcpmux: S 368146356:368146356(0)
win 512 <mss 1460>
whatsup.net.25118 > dns.myplace.com.22: S 2035824356:2035824356(0)
win 512 <mss 1460>
dns.myplace.com.1114 > whatsup.net.25177: R 0:0(0) ack 3231175488
win0 (DF)
dns.myplace.com.22 > whatsup.net.25118: R 0:0(0) ack 2035824357 win 0 (DF)
dns.myplace.com is the alleged compromised host and whatsup.net is the hostile IP.
What’s wrong with the break-in theory? Well, no connections were established, no data exchanged, and
no connections were stopped after being established.
This looks like a typical scan from whatsup.net. Occasionally, you’ll see dns.myplace.com respond
with a RESET/ACK meaning that the port that is being scanned is inactive. Other than that, there is no
evidence of any two-way conversation.

If no RESET/ACK is seen that means that the traffic never actually made it to the inactive port to be
rejected. In other words it was blocked by an exterior router. The sensor for this site resides outside the
filtering router.
This, of course, assumes that there is no backdoor entry to the network/host that the sensor does not see.
It turns out that this incident was a communication problem among the CERT organizations. One
reported it as a scan and somehow several phone calls later, it got reported as a break-in. Because we
had the tcpdump output, we were confident that a break-in did not occur.
Had we not had tcpdump or some other software that had an audit trail of activity, we would not have
any concrete evidence to refute the break-in claim. You should have an IDS or software that is capable
of maintaining a history of activity such as tcpdump.
7
7
Scan of High-Numbered Port
•
Hostile Activity:
•
Scan to TCP port 32775
•
Site router blocked much of activity
•
Internal sensor detected initial SYN activity to
many hosts
Examining tcpdump output for a given hour, a scan appeared of high-numbered port 32775. The
sensor that picked this up was inside the packet-filtering device signaling that some traffic got
through to internal hosts. At this point, all we could see was that there were initial SYN connections
attempted. We would need to further investigate to discover if there were any responses to these
connections.
8
8
Network View of Activity

Hostile host
1.1.1.1
Filtering Router
sensor
Semi-protected
subnet
TCP port 32775 scan
What we see above is a site that has a filtering router that blocks some traffic and allows in others.
Specifically, they allow in some high-numbered ports for certain subnets to accommodate users or
system administrators who need to use outside services such as active FTP that might try to open up
these ports. The sensor collecting the traffic is inside the packet filtering router so anything we see
has gotten through the packet filtering device. A scan from hostile host 1.1.1.1 was received directed
to destination port 32775.
9
9
Port Scan
16:46:42.690168 scanner.net.4838 > 192.168.5.118.32775: S
2943632224:2943632224(0) win 32120 (DF)
16:46:43.270573 scanner.net.2077 > 192.168.5.120.32775: S
2940858778:2940858778(0) win 32120 (DF)
16:46:43.270737 scanner.net.2080 > 192.168.5.121.32775: S
2944647564:2944647564(0) win 32120 (DF)
16:46:43.272415 scanner.net.2083 > 192.168.5.124.32775: S
2938904083:2938904083(0) win 32120 (DF)
16:46:43.272906 scanner.net.2086 > 192.168.5.125.32775: S
2941596782:2941596782(0) win 32120 (DF)
16:46:43.275637 scanner.net.2091 > 192.168.5.128.32775: S
2950442877:2950442877(0) win 32120 (DF)
16:46:43.275813 scanner.net.2092 > 192.168.5.129.32775: S
2952745733:2952745733(0) win 32120 (DF)

16:46:43.276139 scanner.net.2097 > 192.168.5.132.32775: S
2950485912:2950485912(0) win 32120 (DF)
This is the type of traffic that appeared on the sensor. Many connections were allowed through the
filtering router destined for TCP port 32775. It can be surmised that the scanner actually probed the
entire class B 192.168 subnet, but since our vantage point is from the inside, we can only guess this.
We do know that all subnets and hosts that had access rules that allowed access to ports greater than
1023 were scanned. That is why it is suspected that the entire class B network was probed. Either
that or the scanner did some prior reconnaissance and knew that there were some subnets that
allowed traffic through to high-numbered ports.
TCP port 32775 doesn’t have a well-known service associated with it, but it is in the range of ports
considered to be Remote Procedure Call (RPC) services. This range is 32770 – 34000 both TCP and
UDP. So, we begin to surmise that the prober is looking for some sort of RPC service being offered
– either one that we know is vulnerable or an entirely new exploit that has yet to be reported.
10
10
Conventional inetd Services
telnet server x
listens on
port 23
telnet server q
listens on
port 23
telnet server z
listens on
port 23
A moment of digression to explain the difference in the way conventional internet services started by
the Unix inetd daemon are identified versus the way RPC services must be identified. A server can
start any given internet service on any port, but convention dictates that if we want to offer a
universal service, we must listen on a standard well-known port. For telnet, this is port 23. Almost
all servers offering the telnet service will listen on port 23 so the clients will know where to find the

telnet port.
11
11
RPC Services
Runs ToolTalk
Listens on
port 32775
Runs ToolTalk
Listens on
port 32773
Runs ToolTalk
Listens on
port 32774
Host A
Host B
Host C
RPC services, on the other hand, can listen on different ports for the same service. Depending on the
operating system and the number of RPC services that are run on a given host, the port can vary. As
we see above, we have three different hosts that listen for the RPC service ToolTalk on different
ports in the RPC port range. What this means for someone malicious is if she/he wants to exploit an
exploit in a specific RPC service, she/he has to first determine what port that service runs on.
12
12
Portmapper
•
Registers RPC services run by a host
•
Can query portmapper using command:
•
rpcinfo –p hostname/IP

•
Uses TCP port 111 to query
•
Prudent to block this port on packet-filtering device
•
For each RPC service running, query returns:
•
RPC service number
•
Port number
•
Service name
The Unix portmapper is the program that manages RPC services on a host. It registers each RPC
service as it starts and unregisters it when it stops. It can be queried to list all the RPC services a
host is running. So, a malicious user trying to exploit a vulnerable RPC service is most likely to
attempt to query the portmapper of a host as reconnaissance to discover the port at which the service
is listening. After gleaning that information, the attack usually takes place.
The problem with this approach is that most sites except those that are totally security unaware will
block TCP port 111 which is the port at which the portmapper listens for queries. If this is done, the
only other recourse for the hacker is to scan the ports in the RPC range to see if hosts are listening.
13
13
Sample rpcinfo Output
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100232 10 tcp 32775 sadmind
100011 1 udp 32773 rquotad
100002 2 udp 32772 rusersd
100024 1 udp 32774 status

100024 1 tcp 32774 status
program number
universal
Port number
variable
Since the port at which an RPC service listens may vary, we need a universal indicator to identify a
given RPC service for all hosts running that service. This is known as the program number. In the
above output, we see that the RPC service sadmind used for Solaris remote administration has a
program number of 100232. It listens on port 32775 on this host. Any sadmind service running on
any host will have a program number of 100232. This is the number we must key on to identify
services.
How do you determine associated RPC services and program numbers? Most hosts have a file
/etc/rpc that lists the possible RPC services (much like /etc/services for the inetd services). Yet, this
is often not complete for all services that might be run.
14
14
Responding Hosts
scanner.net IP = 1.1.1.1
Filter: ‘host 1.1.1.1 and tcp[13] = 0x12’
16:46:57.221083 host1.net.32775
> scanner.net.3149: S
4235251605:4235251605(0) ack 2956170941 win 9576 <nop,nop,timestamp
836920240 262467851,mss[|tcp]> (DF)
16:46:57.235307 host20.net.32775 > scanner.net.3147: S
511195160:511195160(0) ack 2953574510 win 9660 <mss 1380> (DF)
16:46:57.237201 host55.net.32775 > scanner.net.3143: S
1103845880:1103845880(0) ack 2965548842 win 9660 <mss 1380> (DF)
16:46:58.225076 host12.net.32775 > scanner.net.3398: S
351313966:351313966(0) ack 2959016215 win 9576 <nop,nop,timestamp
30242489 262467951,mss[|tcp]> (DF)

16:47:01.242766 host4.net.32775 > scanner.net.2490: S 70452835:70452835(0)
ack 2961275373 win 9576 <nop,nop,timestamp 72210076
262468250,mss[|tcp]> (DF)
Back to the original scan! If the portmapper port TCP port 111 is blocked by the packet filtering
device, it might still be possible to scan RPC ports (32770 – 34000) directly. This won’t tell you
what RPC service is running, it will just tell you that something is listening in this range of ports and
it is most likely some RPC server. Additional scanning can be done to determine exactly what RPC
service is running on a responding port.
Since the scan made it through the packet filtering device, the concern then became – did any host
respond to the 32775 probe. If none responded, we could just dismiss this as our good fortune and a
relatively harmless scan. But if any answered, we have more investigation to do.
Unfortunately, many hosts responded with a SYN/ACK indicating that they listened with some RPC
service on port 32775. We don’t know exactly what service each host listened for at this point, we
only know that something was listening. The next step after seeing this was to look at the hosts that
responded and see if they pushed (most often the P flag will be set) data from the probed host to the
scanner or vice versa. Fortunately, this did not occur. This indicates that the scanner most likely
was doing reconnaissance only at this phase. Exchanged data would elevate the level of concern
because this means the scanned host either divulged some information or suffered some indignity at
the hands of the prober.
15
15
What RPC Services Were
Running on Port 32775?
rpcinfo -p hostname | grep "tcp 32775"
+ rpcinfo -p host1.net | grep "tcp 32775"
100083
1 tcp 32775
+ rpcinfo -p host20.net| grep "tcp 32775"
100083
1 tcp 32775

+ rpcinfo -p host4.net| grep "tcp 32775"
100083
1 tcp 32775
+ rpcinfo -p host12.net| grep "tcp 32775"
100007 3 tcp 32775 ypbind
100007 2 tcp 32775 ypbind
For each of the responding hosts, we want to determine the exact RPC service running on port
32775. This can be done by issuing the command rpcinfo –p hostname and then examining the
output for the string “tcp 32775”. The scan looked at TCP port 32775. It is also possible to have an
RPC service running on udp 32775 so that is why we make the distinction. As we see above, we
have three hosts running a service that has no description, but has the universal program number of
100083. This is what we must use to search for the service running on this host. The responding
hosts ran many different services – all of those had to be identified for known exploits.
A best guess is that the scanner is looking for a service that has been successfully exploited recently.
At the time of this writing (7-2000), the most popular exploited RPC service has been ToolTalk. See
SANS write-up entitled “Consensus List of Top Ten Internet Threats” found at
www.sans.org/topten.htm. RPC services (specifically citing ToolTalk) are the third listed threat.
16
16
CERT Advisory
•
CERT Advisory CA-98.11 discusses the ToolTalk
vulnerability
•
States RPC program number for ToolTalk database
service is 100083
•
This is a buffer overflow exploit that can allow root
access
•

Very dangerous, very widespread
The CERT advisory can be found at:

Here is an excerpt from this advisory:
“The ToolTalk service allows independently developed applications to communicate with each other
by exchanging ToolTalk messages. Using ToolTalk, applications can create open protocols which
allow different programs to be interchanged, and new programs to be plugged into the system with
minimal reconfiguration.
The ToolTalk database server (rpc.ttdbserverd) is an ONC RPC service which manages objects
needed for the operation of the ToolTalk service. ToolTalk-enabled processes communicate with
each other using RPC calls to this program, which runs on each ToolTalk-enabled host. This
program is a standard component of the ToolTalk system, which ships as a standard component of
many commercial Unix operating systems. The ToolTalk database server runs as root.
Due to an implementation fault in rpc.ttdbserverd, it is possible for a malicious remote client to
formulate an RPC message that will cause the server to overflow an automatic variable on the stack.
By overwriting activation records stored on the stack, it is possible to force a transfer of control into
arbitrary instructions provided by the attacker in the RPC message, and thus gain total control of the
server process.”
17
17
Recommended Immediate
Action
32775
We cannot be absolutely sure that the scanner was trolling for reconnaissance about hosts that might
be running ToolTalk. But, we see enough evidence on this network that if the scanner returns to do
some damage, it is very possible that we have unpatched versions of ToolTalk that might be
susceptible to this exploit. There could be other RPC services running on other hosts that have
exploits associated with them such as sadmind.
The immediate reaction was to block this port. Granted, this is a stop-gap solution for this particular
event. The hacker can return and probe other RPC ports and probably find other ports that listen and

serve up ToolTalk. A more all-encompassing solution would be to block inbound all the RPC ports
32770 – 34000. The conservative approach until more investigation is done is simply to block this
specific port.
18
18
Destination Port 31789
•
Scan detected to UDP port 31789
12:13:10.570000 1.2.3.4.31790 > 192.168.21.63.31789: udp 1
12:13:10.570000 1.2.3.4.31790 > 192.168.21.90.31789: udp 1
12:13:10.570000 1.2.3.4.31790 > 192.168.21.109.31789: udp 1
12:13:11.020000 1.2.3.4.31790 > 192.168.21.117.31789: udp 1
12:13:11.030000 1.2.3.4.31790 > 192.168.21.118.31789: udp 1
12:13:11.040000 1.2.3.4.31790 > 192.168.21.119.31789: udp 1
12:13:11.040000 1.2.3.4.31790 > 192.168.21.120.31789: udp 1
12:13:11.040000 1.2.3.4.31790 > 192.168.21.121.31789: udp 1
12:13:11.060000 1.2.3.4.31790 > 192.168.21.122.31789: udp 1
12:13:11.070000 1.2.3.4.31790 > 192.168.21.123.31789: udp 1
This anomalous traffic was noticed because the source IP appeared in Shadow’s hourly tally count
and the records were dumped using tcpdump.
19
19
Port 31789 Data
•
Curious only 1 byte of UDP data sent
•
Why not send more, or
•
Why send any at all?
•

What is this curious one byte?
•
Examine records in hex and using tcpshow
•
What scripts can we use to check out this one
byte?
It was fairly unusual to see UDP traffic that purported to be sending only one byte of UDP data.
This traffic was examined further to gain some insight into what that one byte of data might be.
A fair question might be “what made the analyst see or examine this traffic?”. Basically, curiosity
was the motivation. This means that the analyst had the “luxury” of being curious. The point is if
analysts are overloaded, they will not have the time or mindset to be curious. If analysts operate in a
chaotic environment where phones are continually ringing, deadlines are bearing down on them,
they’ve got multiple sites, and a lot of traffic to evaluate, they may not have an atmosphere that
nurtures curiosity.
20
20
Examining Port 31789
Records in Hex
•
Execute following command:
tcpdump –r tcpdumpfile –x ‘dst port 31789’
12:13:10.570000 1.2.3.4.31790 > 192.168.21.63.31789: udp 1
4500 001d a932 0000 7311 10cf 0102 0304
c0a4 153f 7c2e 7c2d 0009 38b1 413b 0000
0001 0000 0000 0000 0356 4752 0341
12:13:10.570000 1.2.3.4.31790 > 192.168.21.90.31789: udp 1
4500 001d c432 0000 7311 f5b3 0102 0304
c0a4 155a 7c2e 7c2d 0009 3896 41cc 8000
0001 0000 0002 0002 0236 3003 3132
This will dump tcpdump data in hexadecimal that contains a particular pattern, in this case

destination port 31789.
By dumping the traffic in hexadecimal format, we can see all the fields that were collected with
tcpdump. Remember that only the default number of bytes captured based on the default snapshot
length will be displayed. You may see length fields that indicate more data than is actually captured
simply because you may not be capturing the entire datagram.
21
21
Examining The Hex Output
More Closely
4500 001d a932 0000 7311 10cf 0102 0304
c0a4 153f 7c2e 7c2d 0009 38b1 413b 0000
0001 0000 0000 0000 0356 4752 0341
Do you see only one byte of UDP data?
IP Header
UDP Header UDP Data
This is the first record dumped in hex. You’ll notice 18 bytes of UDP data.
The IP header encompasses the first 20 bytes of data. The UDP header follows with 8 bytes of data.
The UDP data follows these headers. One data byte consists of two hexadecimal characters such as
the 41.
22
22
Examining Port 31789
Records With tcpshow
•
Generate tcpshow output:
tcpdump –r tcpdumpfile –enx ‘dst port 31789’ | tcpshow
-nolink
In an attempt to discover anything more about the traffic bound for port 31789, we use tcpshow to
interpret the output for us. We can dump the tcpdump data using tcpshow to interpret the output. In
this case, we search for the pattern of destination port 31789.

23
23
Examining the tcpshow Output
Packet 1
IP Header
Version: 4
Header Length: 20 bytes
Service Type: 0x00
Datagram Length: 29
bytes
Identification: 0xA932
Flags: MF=off, DF=off
Fragment Offset: 0
TTL: 115
Encapsulated Protocol: UDP
Header Checksum: 0x10CF
Source IP Address: 1.2.3.4
Destination IP Address: 192.169.21.63
The above tcpshow interpretation of the IP header shows nothing unusual. Note the 29 byte datagram
length.
Packet 1
IP Header
Version: 4
Header Length: 20 bytes
Service Type: 0x00
Datagram Length: 29 bytes
Identification: 0xA932
Flags: MF=off, DF=off
Fragment Offset: 0
TTL: 115

Encapsulated Protocol: UDP
Header Checksum: 0x10CF
Source IP Address: 1.2.3.4
Destination IP Address: 192.169.21.63
UDP Header
Source Port: 31790 (<unknown>)
Destination Port: 31789 (<unknown>)
Datagram Length: 9 bytes (Header=8, Data=1)
Checksum: 0x38B1
UDP Data
A
-----------------------------------------------------------------
Packet 2
24
24
Examining the tcpshow Output
UDP Header
Source Port: 31790 (<unknown>)
Destination Port: 31789 (<unknown>)
Datagram Length: 9
bytes (Header=8, Data=1)
Checksum: 0x38B1
UDP Data
A
-----------------------------------------------------------------
Packet 2
Look at the UDP datagram length derived from the UDP header length field. It assumes that since
that UDP header says it is 9 bytes long (8 bytes for header and 1 for data) that there is only one byte
of data to interpret - so it decrypts the first byte to A.
You’ll also see a “Packet 2” . Chances are that this is the additional data bytes that the tcpshow

command doesn’t know how to interpret.
Sometimes, as you can see, the hex output is more accurate because it reveals the actual bytes.
tcpshow doesn’t always interpret unconventionally crafted traffic correctly.
What this mutant packet shows is that the UDP header was probably crafted. Again, the reason is
unknown; it is worthwhile noting that this probably is not for benevolent purposes. It could be an
attempted denial of service against a particular operating system that will not be able to handle this
packet. Or, it can be an attempt to send a covert message – one that is not to be discovered by
anyone other than the intended recipient.
25
25
Examining The Length Values
4500 001d a932 0000 7311 10cf 0102 0304
IP HEADER
c0a4 153f 7c2e 7c2d 0009 38b1 413b 0000
UDP HEADER UDP DATA
0001 0000 0000 0000 0356 4752 0341
IP datagram total length UDP length
The IP datagram total length agrees with the crafted UDP header
because it is 29 bytes (20 IP header + 8 UDP header + 1 UDP data).
The UDP length has been crafted to be 9 bytes (8 UDP header + 1
UDP data).
This is the hex record that was dumped before. The focus now becomes the fields that contain length
values.
It turns out that the minimum size of an IP datagram over Ethernet is 46 bytes which is the actual
length of this IP datagram. However, a normal datagram that had one byte of UDP data would be 29
bytes (20 IP header + 8 UDP header + 1 UDP data) and would require 17 bytes of padding. The
padding would be zeroes. For instance, in the above packet, we should see:
4500 001d a932 0000 7311 10cf 0102 0304
c0a4 153f 7c2e 7c2d 0009 38b1 4100 0000
0000 0000 0000 0000 0000 0000 0000

The final 17 bytes are 0 padded.