silk v3.16.0 - monitoring & security analysis for Large-Scale Networks.

silk v3.16.0 – monitoring & security analysis for Large-Scale Networks.

Changelog silk v3.16.0:
+ rwstats
– When the primary value is a distinct count, compute the number of distinct items across all bins and print each bin’s percentage of the total distinct count.
– Fix bugs that may occur when computing distinct counts and not all distinct counts fit into memory.
+ rwuniq
– Fix bugs that may occur when computing distinct counts and not all distinct counts fit into memory.
+ flowrate plug-in
– Change how the flowrate plug-in handles flow records whose duration is zero in order to fix bizarre looking output in rwstats. The plug-in now assumes each of these flow records has a duration of 400 microseconds (0.4 milliseconds).
– Add the –flowrate-zero-duration switch which allows the user to set the duration that the plug-in uses for flow records whose given duration is zero.
+ rwrandomizeip
– Read flow records from the standard input if the number of non-switch arguments is zero.
– Write the flow records to the standard output if the number of non-switch arguments is zero or one.
+ rwswapbytes
– Read flow records from the standard input if the number of non-switch arguments is zero.
– Write the flow records to the standard output if the number of non-switch arguments is zero or one.
+ rwflowpack, flowcap
— Change processing of NetFlow v9 records so that, when SiLK is compiled against libfixbuf 1.8.0, the OUT_BYTES and OUT_PKTS values are used when the IN_BYTES and IN_PKTS values are 0.
+ flowcap
— Print the probe definitions to the log file when the log-level is set to debug.
+ rwflowpack, rwflowappend, flowcap, rwsender, rwreceiver, rwpollexec
— Change how daemons invoke subprocesses in order to avoid creating subprocesses that deadlock and never complete.
— Modify start-up scripts to be more in line with the rules in the Linux Standard Base.
+ Plug-ins
— Add manual pages for the cutmatch, conficker-c, and app-mismatch plug-ins.
— No longer install the uniq-distproto plug-in since its functionality is available as –values=distinct:protocol.

silk v3.16.0

SiLK, the System for Internet-Level Knowledge, is a collection of traffic analysis tools developed by the CERT Network Situational Awareness Team (CERT NetSA) to facilitate security analysis of large networks. The SiLK tool suite supports the efficient collection, storage, and analysis of network flow data, enabling network security analysts to rapidly query large historical traffic data sets. SiLK is ideally suited for analyzing traffic on the backbone or border of a large, distributed enterprise or mid-sized ISP.

A SiLK installation consists of two categories of applications: the packing system and the analysis suite. The packing system collects IPFIX, NetFlow v9, or NetFlow v5 and converts the data into a more space efficient format, recording the packed records into service-specific binary flat files. The analysis suite consists of tools which read these flat files and perform various query operations, ranging from per-record filtering to statistical analysis of groups of records. The analysis tools interoperate using pipes, allowing a user to develop a relatively sophisticated query from a simple beginning.

The vast majority of the current code-base is implemented in C, Perl, or Python. This code has been tested on Linux, Solaris, OpenBSD, Mac OS X, and Cygwin, but should be usable with little or no change on other Unix platforms.

rwfilter silk

Analysis Suite:
The SiLK analysis suite is a collection of command-line tools for processing SiLK Flow records created by the SiLK packing system. These tools read binary files containing SiLK Flow records and partition, sort, and count these records. The most important analysis tool is rwfilter, an application for querying the central data repository for SiLK Flow records that satisfy a set of filtering options. The tools are intended to be combined in various ways to perform an analysis task. A typical analysis uses UNIX pipes and intermediate data files to share data between invocations of the tools.

The tools and plug-in modules that make up the analysis tools are listed below, roughly grouped by functionality.

Filtering, displaying, and sorting
+ rwfilter :Select SiLK Flow records form the data repository and partition the records into one or more ‘pass’ and/or ‘fail’ output streams.
+ rwcut :Print the attributes of SiLK Flow records in a delimited, columnar, human-readable format. Users can define new printable attributes using plug-ins written in C or PySiLK.
+ rwsort : Sort SiLK Flow records using a user-specified key comprised of record attributes, and write the records to the named output path or to the standard output. Users can define new key fields using plug-ins written in C or PySiLK.

SiLK Python Extension (PySiLK)
+ PySiLK: SiLK in Python: Read, manipulate, and write SiLK Flow records, IPsets, and Bags from within Python. PySiLK may be used in a stand-alone Python program or to write plug-ins for several SiLK applications. This document describes the objects, methods, and functions that PySiLK provides. The next entry describes using PySiLK from within a plug-in.
+ silkpython : Use PySiLK to define new partitioning rules for rwfilter, new key fields for rwcut, rwgroup, and rwsort, and new key or value fields for rwstats and rwuniq.

Counting, grouping, and mating
+ rcount: Summarize (aka group or bin) SiLK Flow records across time, producing textual output with counts of bytes, packets, and flow records for each time bin.
+ rwuniq: Summarize SiLK Flow records by a user-specified key comprised of record attributes and print columns for the total byte, packet, and/or flow counts for each bin. rwuniq can also count the number of distinct values for a field. Users can define new key fields and value fields using plug-ins written in C or PySiLK.
+ rwstats: Summarize SiLK Flow records just like rwuniq, but sort the results by a value field to generate a Top-N or Bottom-N list, and print the results.
+ rwtotal: Summarize SiLK Flow records by a specified key and print the sum of the byte, packet, and flow counts for flows matching the key. rwtotal uses a fixed amount of memory, and it will faster that rwuniq, but it has a limited set of keys.
+ rwaddrcount: Summarize SiLK flow records by the source or destination IPv4 address and print the byte, packet, and flow counts for each IP.
+ rgroup: Group SiLK flow records by a user-specified key comprised of record attributes, label the records with a group ID that is stored in the next-hop IP field, and write the resulting binary flows to the specified output path or to the standard output. rwgroup requires that its input is sorted by the user-specified key.
+ rwmatch: Match (mate) records as queries and responses, mark mated records with an ID that is stored in the next-hop IP field, and write the binary flow records to the output. rwmatch requires that its input files are sorted.

IPset, Bag, and Prefix Map manipulation
+ rwset: Read SiLK Flow records and generate binary IPset file(s) containing the source IP addresses or destination IP addresses seen on the flow records.
+ rwsetbuild: Read (textual) IP addresses in canonical form or in CIDR notation from an input file or from the standard input and write a binary IPset file.
+ rwsetcat: Print the contents of a binary IPset file as text. Additional information about the IPset file can be printed.
+ rwsetmember: Determine whether the IP address or CIDR block specified on the command line is contained in an IPset.
+ rwsettool: Perform union, intersection, difference, and sampling functions on the input IPset files, generating a new IPset file.
+ rwbag: Read SiLK Flow records and build binary Bag(s) containing key-count pairs. An example is a Bag containing the sum of the byte counts for each source port seen on the flow records.
+ rwbagbuild: Create a binary Bag file from a binary IPset file or from a textual input file.
+ etc…

Released Notes:
+ Add new IPset file format for IPv6 flow records.
+ Allow the installer of SiLK to choose the default IPset file format written by SiLK.
+ Allow rwsetcat to count the IPs in an IPset stream without loading the IPset into memory.
+ Add a work-around for NetFlow v9 records from a SonicWall device.
Usage and Download:

Source: https://tools.netsa.cert.org/ | Our Post Before