Poor ListenBeats performance
In Uninett we have noticed quite poor performance by the ListenBeat processor at higher data rates. After switching to ListenHTTP and using Vector to send data, performance is much better. Vector.xml is a Nifi template showing how we collect Suricata logs from Vector in Nifi. The Vector config used is:
[sources.suricata_source]
type = "file"
ignore_older_secs = 600
include = ["/var/log/suricata/eve.*.json"]
read_from = "beginning"
[transforms.suricata_transform]
type = "remap"
inputs = ["suricata_source"]
source = '''
.message, err = parse_json(.message)
if err != null {
.malformed = true
log("Failed to parse common-log: " + err, level: "error")
} else {
.log_type= "suricata"
}
'''
[sinks.nifi_sink]
type = "http"
inputs = ["suricata_transform"]
compression = "none"
uri = "https://<nifi server>:5003/suricata"
tls.crt_file = "/etc/vector/client_certificate/cert.pem"
tls.key_file = "/etc/vector/client_certificate/key.pem"
encoding.codec = "json"