silk - monitoring & security analysis for Large-Scale Networks.

silk – monitoring & security analysis for Large-Scale Networks.

SiLK, the System for Internet-Level Knowledge, is a collection of traffic analysis tools developed by the CERT Network Situational Awareness Team (CERT NetSA) to facilitate security analysis of large networks. The SiLK tool suite supports the efficient collection, storage, and analysis of network flow data, enabling network security analysts to rapidly query large historical traffic data sets. SiLK is ideally suited for analyzing traffic on the backbone or border of a large, distributed enterprise or mid-sized ISP.

A SiLK installation consists of two categories of applications: the packing system and the analysis suite. The packing system collects IPFIX, NetFlow v9, or NetFlow v5 and converts the data into a more space efficient format, recording the packed records into service-specific binary flat files. The analysis suite consists of tools which read these flat files and perform various query operations, ranging from per-record filtering to statistical analysis of groups of records. The analysis tools interoperate using pipes, allowing a user to develop a relatively sophisticated query from a simple beginning.

The vast majority of the current code-base is implemented in C, Perl, or Python. This code has been tested on Linux, Solaris, OpenBSD, Mac OS X, and Cygwin, but should be usable with little or no change on other Unix platforms.

rwfilter silk

Analysis Suite:
The SiLK analysis suite is a collection of command-line tools for processing SiLK Flow records created by the SiLK packing system. These tools read binary files containing SiLK Flow records and partition, sort, and count these records. The most important analysis tool is rwfilter, an application for querying the central data repository for SiLK Flow records that satisfy a set of filtering options. The tools are intended to be combined in various ways to perform an analysis task. A typical analysis uses UNIX pipes and intermediate data files to share data between invocations of the tools.

The tools and plug-in modules that make up the analysis tools are listed below, roughly grouped by functionality.

Filtering, displaying, and sorting
+ rwfilter :Select SiLK Flow records form the data repository and partition the records into one or more ‘pass’ and/or ‘fail’ output streams.
+ rwcut :Print the attributes of SiLK Flow records in a delimited, columnar, human-readable format. Users can define new printable attributes using plug-ins written in C or PySiLK.
+ rwsort : Sort SiLK Flow records using a user-specified key comprised of record attributes, and write the records to the named output path or to the standard output. Users can define new key fields using plug-ins written in C or PySiLK.

SiLK Python Extension (PySiLK)
+ PySiLK: SiLK in Python: Read, manipulate, and write SiLK Flow records, IPsets, and Bags from within Python. PySiLK may be used in a stand-alone Python program or to write plug-ins for several SiLK applications. This document describes the objects, methods, and functions that PySiLK provides. The next entry describes using PySiLK from within a plug-in.
+ silkpython : Use PySiLK to define new partitioning rules for rwfilter, new key fields for rwcut, rwgroup, and rwsort, and new key or value fields for rwstats and rwuniq.

Counting, grouping, and mating
+ rcount: Summarize (aka group or bin) SiLK Flow records across time, producing textual output with counts of bytes, packets, and flow records for each time bin.
+ rwuniq: Summarize SiLK Flow records by a user-specified key comprised of record attributes and print columns for the total byte, packet, and/or flow counts for each bin. rwuniq can also count the number of distinct values for a field. Users can define new key fields and value fields using plug-ins written in C or PySiLK.
+ rwstats: Summarize SiLK Flow records just like rwuniq, but sort the results by a value field to generate a Top-N or Bottom-N list, and print the results.
+ rwtotal: Summarize SiLK Flow records by a specified key and print the sum of the byte, packet, and flow counts for flows matching the key. rwtotal uses a fixed amount of memory, and it will faster that rwuniq, but it has a limited set of keys.
+ rwaddrcount: Summarize SiLK flow records by the source or destination IPv4 address and print the byte, packet, and flow counts for each IP.
+ rgroup: Group SiLK flow records by a user-specified key comprised of record attributes, label the records with a group ID that is stored in the next-hop IP field, and write the resulting binary flows to the specified output path or to the standard output. rwgroup requires that its input is sorted by the user-specified key.
+ rwmatch: Match (mate) records as queries and responses, mark mated records with an ID that is stored in the next-hop IP field, and write the binary flow records to the output. rwmatch requires that its input files are sorted.

IPset, Bag, and Prefix Map manipulation
+ rwset: Read SiLK Flow records and generate binary IPset file(s) containing the source IP addresses or destination IP addresses seen on the flow records.
+ rwsetbuild: Read (textual) IP addresses in canonical form or in CIDR notation from an input file or from the standard input and write a binary IPset file.
+ rwsetcat: Print the contents of a binary IPset file as text. Additional information about the IPset file can be printed.
+ rwsetmember: Determine whether the IP address or CIDR block specified on the command line is contained in an IPset.
+ rwsettool: Perform union, intersection, difference, and sampling functions on the input IPset files, generating a new IPset file.
+ rwbag: Read SiLK Flow records and build binary Bag(s) containing key-count pairs. An example is a Bag containing the sum of the byte counts for each source port seen on the flow records.
+ rwbagbuild: Create a binary Bag file from a binary IPset file or from a textual input file.
+ etc…

Released Notes:
+ Add new IPset file format for IPv6 flow records.
+ Allow the installer of SiLK to choose the default IPset file format written by SiLK.
+ Allow rwsetcat to count the IPs in an IPset stream without loading the IPset into memory.
+ Add a work-around for NetFlow v9 records from a SonicWall device.
Usage and Download:

Source: https://tools.netsa.cert.org/