%include "default.mgp"
%tab bullet icon arc "blue" 40
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%nodefault
%bgrad 0 0 32 0 1 "cyan" "magenta"
%xfont "utopia-bold-r", size 12

%center, fore "black"
cflowd 2.0
%size 6
Cisco flow-export collection

%xfont "courier-bold-r"




Daniel McRobb
dwm@caida.org

%size 5
http://www.caida.org/Tools/cflowd/

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

Background

%xfont "utopia-bold-r", fore "black", size 6
&bullet host software for collecting Cisco version 5 (and version 1)\
 flow-export data
&bullet aggregates data into tables for continuous collection of\
 summary data (in time-series)
&bullet stores raw flow data in rotating log files
&bullet provides client/server collection of tabular data in time series

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

Changes from 1.3b2 to 2.0

%xfont "utopia-bold-r", fore "black", size 6
&bullet all tables are per input interface (not per router)
&bullet support for version 1 flow-export
&bullet central collector included
&bullet uses arts++ package for data storage and aggregation
&bullet real-time flow-matching API (example `flowwatch' program included)
&bullet interface matrix
&bullet IP nexthop table

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%xfont "utopia-bold-r", fore "black"
%center

cflowd

%image "dataflow1.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"
%center

cflowd architecture

%image "dataflow2.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%xfont "utopia-bold-r", fore "black"

benefits of new architecture

%xfont "utopia-bold-r", fore "black", size 6
&bullet increased performance (shared memory packet queue vs. UNIX\
 domain sockets for cflowdmux to cflowd IPC)
&bullet hooks for real-time flow matching
&bullet source code easier to maintain due to heavy use of STL and\
 more modularization

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

flow export version 5

%xfont "utopia-bold-r", fore "black", size 6
&bullet sends UDP packets to a specified host address and port number
&bullet each UDP packet contains a number of flow entries

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%xfont "utopia-bold-r", fore "black"
%center

flow-export packets

%image "flowPacket.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%xfont "utopia-bold-r", fore "black"
%center

flow header

%image "flowHeader.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%xfont "utopia-bold-r", fore "black"
%center

flow entry

%image "flowEntry.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

cflowd aggregation

%xfont "utopia-bold-r", fore "black", size 6
&bullet cflowd primarily designed to aggregate flow data into\
 tabular data to be used for capacity planning
&bullet AS matrix, net matrix, port matrix, protocol table,\
 interface matrix and nexthop table aggregation across all flows

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

Why so much aggregation?

%xfont "utopia-bold-r", fore "black", size 6
&bullet data reduction
&bullet scale of intended use (backbone-wide)
&bullet disk and bandwidth requirements for raw flow data\
 in a backbone network.  We only transfer tabular data back to the\
 central collector.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

cfdcollect

%xfont "utopia-bold-r", fore "black", size 6
&bullet collects cflowd data at regular intervals
&bullet produces time-series data for each of the tabular data\
 types (AS matrix, net matrix, port matrix, protocol table, interface\
 matrix, nexthop table)
&bullet stores data in ARTS files (1 file per router per day)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"
%center

centralized collection

%image "centralCollection.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

arts++

%xfont "utopia-bold-r", fore "black", size 6
&bullet C++ class library for subset of ARTS data
&bullet supports reading/writing of ARTS data via iostreams and\
 UNIX file descriptors
&bullet supports simple time-domain aggregation for several data types
&bullet simple command-line utilities included for viewing ARTS data\
 files and time domain aggregation

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

ARTS data files

%xfont "utopia-bold-r", fore "black", size 6
&bullet efficient data archival (binary, simple size-reducing techniques)
&bullet data files are portable; arts++ class library is the interface
&bullet extensible for additional data types
&bullet versioning of data objects for different storage formats (typically\
 used for space/CPU tradeoff)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

ARTS data handled by arts++

%xfont "utopia-bold-r", fore "black", size 6
&bullet AS matrix (version 0)
&bullet net matrix (version 2)
&bullet port matrix (version 0)
&bullet selected port table (version 0)
&bullet port matrix (version 2)
&bullet selected port table (version 0)
&bullet interface matrix (version 0)
&bullet IP nexthop table (version 0)
&bullet protocol table (version 2)
&bullet forward IP path (version 0)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

AS matrix (version 0)

%xfont "utopia-bold-r", fore "black", size 6
&bullet counters for traffic (packets and bytes) from\
 source ASes to destination ASes

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

AS matrix data

%center, fore "black", xfont "utopia-bold-r"
%image "asMatrixData.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

ARTS net matrix (version 2)

%xfont "utopia-bold-r", fore "black", size 6
&bullet counters for traffic (packets and bytes) from\
 source networks to destination networks
&bullet networks are identified by network number and netmask length (a.b.c.d/n)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

net matrix data

%center
%xfont "utopia-bold-r", fore "black"

%image "netMatrixData.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

ARTS port matrix (version 2)

%xfont "utopia-bold-r", fore "black", size 6
&bullet counters for traffic (packets and bytes) from source transport layer\
 port number to destination port number

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

port matrix data

%center
%xfont "utopia-bold-r", fore "black"

%image "portMatrixData.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

selected port table data

%center
%xfont "utopia-bold-r", fore "black"
%image "selectedPorts.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

protocol table (version 2)

%xfont "utopia-bold-r", fore "black", size 6
&bullet counters (packets and bytes) versus IP protocol\
 (TCP, UDP, ICMP, IGMP, et. al.)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

protocol table data

%center
%xfont "utopia-bold-r", fore "black"
%image "protocols.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

interface matrix

%xfont "utopia-bold-r", fore "black"
&bullet counters (packets and bytes) for traffic from input interfaces\
 to output interfaces

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

interface matrix data

%center
%image "intfmatrix.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

nexthop table

%xfont "utopia-bold-r", fore "black"
&bullet counters (packets and bytes) for traffic from input interfaces\
 to each IP nexthop

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

IP nexthop table data

%center
%image "nexthops.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

IP forward path (version 0)

%xfont "utopia-bold-r", fore "black", size 6
&bullet contains IP addresses of hops in forward path from a\
 source to a destination
&bullet contains an RTT value for the source to destination
&bullet may be extended in the future to hold more information

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

forward IP path data

%center
%xfont "utopia-bold-r", fore "black"
%image "ipPathData.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

Time Domain Aggregation

%xfont "utopia-bold-r", fore "black", size 6
Time domain aggregation reduces time granularity in time-series data.

Examples:

&bullet convert 5-minute data to 1-hour data for a summary bar chart
&bullet convert 1-hour data to one large aggregate for a pie chart

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

Aggregation Utilities

%xfont "utopia-bold-r", fore "black", size 6
&bullet artsasagg - AS matrix
&bullet artsnetagg - net matrix
&bullet artsportmagg - port matrix
&bullet artsprotoagg - protocol table
&bullet artsintfmagg - interface matrix
&bullet artsnexthopagg - nexthop table

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

Simple display utilities

%xfont "utopia-bold-r", fore "black", size 6
&bullet artsdump
&bullet artsases
&bullet artsnets
&bullet artsportms
&bullet artsports
&bullet artsprotos
&bullet artsintfms
&bullet artsnexthops

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

Supported platforms

%xfont "utopia-bold-r", fore "black", size 6
&bullet FreeBSD 2.2.x Intel
&bullet FreeBSD 3.0 Intel
&bullet Linux 2.0.35 Intel
&bullet Sparc/Solaris 2.5.1
&bullet Sparc/Solaris 2.6

Other platforms (field reports)
&bullet BSDI BSD/OS 3.1
&bullet BSDI BSD/OS 4.0
&bullet Digital UNIX 4.0B

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

Future

%xfont "utopia-bold-r", fore "black", size 6
&bullet support for version 8 flow-export (soon)
&bullet plotting utilities using XRT/PDS
&bullet Web reporting tools (java-based)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r", size 6
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

Future: v8 flow-export

%xfont "utopia-bold-r", fore "black", size 6
Data aggregated on router:
&bullet  AS aggregation (AS matrix)
&bullet  destination prefix aggregation (like CEF accounting)
&bullet  prefix aggregation (net matrix and AS matrix)
&bullet  protocol and port aggregation
&bullet  source prefix aggregation (like CEF accounting)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r", size 6
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

v8 flow-export advantages: scalability

%xfont "utopia-bold-r", fore "black", size 6
&bullet less state information on router
&bullet configure and export only the data in which you're interested
&bullet less bandwidth consumed for export
&bullet more amenable to export over reliable transport (TCP)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

More Information

%xfont "utopia-bold-r", fore "black", size 6

http://www.caida.org/Tools/cflowd/

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

Example chart (traffic by port)

%center
%xfont "utopia-bold-r", fore "black"
%image "portPie.gif"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%xfont "utopia-bold-r"
%bgrad 0 0 32 0 0 "white" "cyan"
%fore "black"

Example chart (traffic vs. destination AS)

%center
%xfont "utopia-bold-r", fore "black"
%image "dstAsBar_weekly.gif"

