IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocols.

IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocols.

IntelMQ is a solution for IT security teams (CERTs, CSIRTs, abuse departments,…) for collecting and processing security feeds (such as log files) using a message queuing protocol. It’s a community driven initiative called IHAP (Incident Handling Automation Project) which was conceptually designed by European CERTs/CSIRTs during several InfoSec events. Its main goal is to give to incident responders an easy way to collect & process threat intelligence thus improving the incident handling processes of CERTs.

IntelMQ’s design was influenced by AbuseHelper, however it was re-written from scratch and aims at:
+ Reduce the complexity of system administration
+ Reduce the complexity of writing new bots for new data feeds
+ Reduce the probability of events lost in all process with persistence functionality (even system crash)
+ Use and improve the existing Data Harmonization Ontology
+ Use JSON format for all messages
+ Integration of the existing tools (AbuseHelper, CIF)
+ Provide easy way to store data into Log Collectors like ElasticSearch, Splunk, databases (such as PostgreSQL)
+ Provide easy way to create your own black-lists
+ Provide easy communication with other systems via HTTP RESTFUL API

intelmqdump

It follows the following basic meta-guidelines:
+ Don’t break simplicity – KISS
+ Keep it open source – forever
+ Strive for perfection while keeping a deadline
+ Reduce complexity/avoid feature bloat
+ Embrace unit testing
+ Code readability: test with unexperienced programmers
+ Communicate clearly

Changelog intelmq v1.0.0.dev6:
### General changes
– Dropped support for Python 2, Python >= 3.3 is needed
– Dropped startup.conf and system.conf. Sections in BOTS can be copied directly yo runtime.conf now.

### Bot changes
– ENH: added bots.collectors.rt.collector
– ENH: added bots.parsers.spamhaus.parser_cert
– ENH: added bots.parsers.fraunhofer.parser_dga
– ENH: added bots.experts.certat_contact.expert
– MAINT: renamed bots.parsers.spamhaus.parser to bots.parsers.spamhaus.parser_drop
– Dropped dragon research group feeds: discontinued
– changed configuration syntax for bots.experts.modify
– dropped bots.collectors.bitsight.collector in favor of bots.collectors.http.collector_http_stream

### Bug fixes
– FIX: all bots handle message which are None
– FIX: various encoding issues resolved in core and bots
– FIX: time.observation is generated in collectors, not in parsers

### Other enhancements and changes
– TST: testing framework for core and tests. Newly introduced components should always come with proper unit tests.
– ENH: intelmqctl has shortcut parameters and can clear queues
– STY: code obeys PEP8, new code should always be properly formatted
– ENH: More code is Python 3 compatible
– DOC: Updated user and dev guide
– Removed Message.contains, Message.update methods Message.add ignore parameter

###Configuration
– ENH: New parameter and field named accuracy to represent the accuracy of each feed
– Consistent naming “overwrite” to switch overwriting capabilities of bots (as opposed to override)
– Renamed http_ssl_proxy to https_proxy
– deduplicator bot has a new parameter to configure deduplication mode
– deduplicator bot key ignore_keys was renamed to filter_keys

### Harmonization
– ENH: Additional data types: integer, float and Boolean
– ENH: Added descriptions and matching types to all fields
– DOC: harmonization documentation has same fields as configuration, docs are generated from configuration
– ENH: New type LowercaseString and UppercaseString
– BUG: FQDNs are only allowed in IDN representation
– ENH: New fields feed.documentation and feed.provider

#### Most important changes:
(source|destination).bgp_prefix is now (source|destination).network
(source|destination).cc is now (source|destination).geolocation.cc
(source|destination).reverse_domain_name is (source|destination).reverse_dns
misp_id changed to misp_uuid
protocol.transport added
webshot_url removed
additional_information renamed to extra, must be JSON
os.name, os.version, user_agent removed in favor of extra

Install from source:

Source: https://github.com/certtools