Commit d0fc936f authored by Kiril KJiroski's avatar Kiril KJiroski
Browse files

thehive integration with keycloak

parent 16fb12c7
Howto's
=======
Modify main NiFi pipeline
-------------------------
To make modifications to the main NiFi pipeline and add it to the Ansible playbook, do the following in the soctool directory:
* Make necesarry to the pipeline in the NiFi GUI
* Copy flow.xml.gz file from one of the NiFi containers:
`docker cp soctools-nifi-1:/opt/nifi/nifi-current/conf/flow.xml.gz .`
* Convert flowx.xml.gz to new template
`utils/flow2template.py flow.xml.gz roles/nifi/templates/flow.xml.j2`
Update configuration files in docker containers using Ansible
-------------------------------------------------------------
To update configuration files for all docker containers together, run the following command:
ansible-playbook -i inventories soctools.yml -t update-config
To update configuration files only for specific services, run the following commands:
ansible-playbook -i inventories soctools.yml -t update-keycloak-config
ansible-playbook -i inventories soctools.yml -t update-thehive-config
ansible-playbook -i inventories soctools.yml -t update-cortex-config
ansible-playbook -i inventories soctools.yml -t update-cassandra-config
ansible-playbook -i inventories soctools.yml -t update-haproxy-config
ansible-playbook -i inventories soctools.yml -t update-filebeat-config
ansible-playbook -i inventories soctools.yml -t update-nifi-config
ansible-playbook -i inventories soctools.yml -t update-odfees-config
ansible-playbook -i inventories soctools.yml -t update-odfekibana-config
Restart services inside docker containers using Ansible
-------------------------------------------------------
To restart services for all docker containers together, run the following command:
ansible-playbook -i inventories soctools.yml -t restart
To restart services only for specific docker containers, run the following commands:
ansible-playbook -i inventories soctools.yml -t restart-keycloak
ansible-playbook -i inventories soctools.yml -t restart-thehive
ansible-playbook -i inventories soctools.yml -t restart-cortex
ansible-playbook -i inventories soctools.yml -t restart-cassandra
ansible-playbook -i inventories soctools.yml -t restart-haproxy
ansible-playbook -i inventories soctools.yml -t restart-filebeat
ansible-playbook -i inventories soctools.yml -t restart-misp
ansible-playbook -i inventories soctools.yml -t restart-mysql
ansible-playbook -i inventories soctools.yml -t restart-nifi
ansible-playbook -i inventories soctools.yml -t restart-odfees
ansible-playbook -i inventories soctools.yml -t restart-odfekibana
Stop services inside docker containers using Ansible
----------------------------------------------------
To stop services for all docker containers together, run the following command:
ansible-playbook -i inventories soctools.yml -t stop
To stop services only for specific docker containers, run the following commands:
ansible-playbook -i inventories soctools.yml -t stop-keycloak
ansible-playbook -i inventories soctools.yml -t stop-thehive
ansible-playbook -i inventories soctools.yml -t stop-cortex
ansible-playbook -i inventories soctools.yml -t stop-cassandra
ansible-playbook -i inventories soctools.yml -t stop-haproxy
ansible-playbook -i inventories soctools.yml -t stop-filebeat
ansible-playbook -i inventories soctools.yml -t stop-misp
ansible-playbook -i inventories soctools.yml -t stop-mysql
ansible-playbook -i inventories soctools.yml -t stop-nifi
ansible-playbook -i inventories soctools.yml -t stop-odfees
ansible-playbook -i inventories soctools.yml -t stop-odfekibana
Restart services inside docker containers manually
--------------------------------------------------
To restart services inside docker containers after changes in configuration files:
1. Attache container: docker exec -it container_id_or_name bash (example: docker exec -it soctools-keycloak bash)
2. List services and their statuses: supervisorctl status
3. Restart service: supervisorctl restart supervisor_service_name (example: supervisorctl restart keycloak)
4. Detach from container: exit
......@@ -13,45 +13,36 @@ Log in and install ansible:
`yum -y install ansible git`
`ansible-galaxy collection install ansible.posix`
Clone soctools:
Temporary solution: Upload your ssh key to gitlab.geant.org
`git clone git@gitlab.geant.org:gn4-3-wp8-t3.1-soc/soctools.git`
Clone soctools:
`git clone https://scm.uninett.no/geant-wp8-t3.1/soctools.git`
`cd soctools`
Install soctools:
Edit group_vars/all/main.yml and change 'soctoolsproxy' so that it point to the FQDN of the server.
Edit group_vars/all/main.yml and change 'dslproxy' so that it point to the FQDN of the server.
`vi group_vars/all/main.yml`
Users are specified in the file:
`group_vars/all/users.yml`
The first entry in the soctools_users variable is the user with full admin privileges in NiFi and Kibana.
To configure the server running soctools, run the ansible playbook:
`ansible-playbook -i inventories soctools_server.yml`
`ansible-playbook -i soctools-inventory soctools_server.yml`
To build the Docker images needed, run the ansible playbook:
`ansible-playbook -i inventories buildimages.yml`
`ansible-playbook -i soctools-inventory buildimages.yml`
To build the CA needed for host and user certificates, run the ansible playbook:
`ansible-playbook -i inventories buildca.yml`
`ansible-playbook -i soctools-inventory buildca.yml`
If using soctools CA certificates provided with this installation, you first need to download and import root certificate found in secrets/CA/ca.crt
For Windows, CA certificate should be installed in Trusted Root Certification Authorities store.
User certificates are can be found in the directory secrets/certificates. Import into browser for authentication.
For Windows, user certificate should be installed in Personal store. Passwords for the certificates can be found in the directory secrets/passwords.
User certificates are can be found in the directory roles/ca/files/CA/private. Import into browser for authentication.
To start the cluster, run the ansible playbook soctools.yml:
`ansible-playbook -i inventories soctools.yml -t start`
`ansible-playbook -i soctools-inventory soctools.yml -t start`
To stop the cluster, run the ansible playbook soctools.yml:
`ansible-playbook -i inventories soctools.yml -t stop`
Web interfaces are available on the following ports:
* 9443 - NiFi
* 5601 - Kibana
* 6443 - Misp : Default user/password: admin@admin.test/test
* 9000 - The Hive : Default user/password: admin@thehive.local/secret
* 9001 - Cortex
* 12443 - Keycloak : Default user/password: admin/Pass005
`ansible-playbook -i soctools-inventory soctools.yml -t stop`
The NiFi interface should now be available on port 9443 on the server.
The OpenDistro for Elasticsearch interface should now be available on port 5601 on the server. To access preconfigured
index patterns you have to switch to Global tenant.
The Keycloak IdP interface should now be available on port 12443 on the server.
License
-------
......
---
- name: Build certification authority
hosts: soctoolsmain
hosts: dsldev
roles:
- ca
---
- name: Build docker images
hosts: soctoolsmain
hosts: dsldev
roles:
- build
---
soctoolsproxy: "<CHANGE_ME:hostname>"
maxmind_key: ""
docker_build_dir: "{{playbook_dir}}/build"
dslproxy: "dsoclab.gn4-3-wp8-soc.sunet.se"
# TheHive Button plugin
THEHIVE_URL: "https://hive.gn4-3-wp8-soc.sunet.se/"
THEHIVE_API_KEY: "5LymseWiurZBrQN8Kqp8O+9KniTL5cE0"
THEHIVE_OWNER: "admin"
# here enter API key for default admin user
THEHIVE_API_KEY: "bs2Jc3tGJqhVv0AYyX2NYlhMlorPz7mX"
# ID of the default admin user
THEHIVE_OWNER: "admin@thehive.local"
# TheHive Create Organisation and Users
# Login as default admin user and create API key, populate it here
# thehive_admin_api: "KoHrKbIJm8XMsJxA9nZLs6YemCu76o3u"
# thehive_writer: "[write]"
#THEHIVE_API_KEY: "1gFdNhmUSxO3BRe1SBB5JYEvkW9UOo6s"
THEHIVE_USERS:
- kiril:
username: "kiril"
name: "Kiril"
surname: "Kiroski"
roles: '["read", "write", "admin"]'
organization: "uninett.no"
- temur:
username: "temur"
name: "Temur"
surname: "Maisuradze"
roles: '["read", "write", "admin"]'
organization: "uninett.no"
soctools_netname: "soctoolsnet"
soctools_network: "172.22.0.0/16"
repo: soctools
repo: gn43-dsl
version: 7
suffix: a20201004
haproxy_name: "soctools-haproxy"
haproxy_name: "dsoclab-haproxy"
haproxy_version: "2.2"
haproxy_img: "{{repo}}/haproxy:{{version}}{{suffix}}"
HAPROXY_PROCESSES: "2"
FILEBEAT_VERSION: "7.9.3"
FILEBEAT_OUTPUT_HOST: "{{soctoolsproxy}}"
FILEBEAT_OUTPUT_PORT: "6000"
FILEBEAT_CERT: "/opt/filebeat/filebeat.crt"
FILEBEAT_KEY: "/opt/filebeat/filebeat.key"
HAPROXY_STATS_PASS: "eiph2Eepaizicheelah3tei+bae3ohgh"
temp_root: "/tmp/centosbuild"
openjdk_img: "{{repo}}/openjdk:{{version}}{{suffix}}"
zookeeper_name: "soctools-zookeeper"
zookeeper_name: "dsoclab-zookeeper"
zookeeper_img: "{{repo}}/zookeeper:{{version}}{{suffix}}"
misp_name: "soctools-misp"
misp_name: "dsoclab-misp"
misp_img: "{{repo}}/misp:{{version}}{{suffix}}"
misp_url: "https://{{soctoolsproxy}}:6443"
nifi_img: "{{repo}}/nifi:{{version}}{{suffix}}"
mysql_name: "soctools-mysql"
mysql_name: "dsoclab-mysql"
mysql_img: "{{repo}}/mysql:{{version}}{{suffix}}"
mysql_dbrootpass: "Pass006"
cassandra_name: "soctools-cassandra"
cassandra_name: "dsoclab-cassandra"
cassandra_img: "{{repo}}/cassandra:{{version}}{{suffix}}"
thehive_name: "soctools-thehive"
thehive_name: "dsoclab-thehive"
thehive_img: "{{repo}}/thehive:{{version}}{{suffix}}"
# GENERATED WITH cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1
thehive_secret_key: "LcnI9eKLo33711BmCnzf6UM1y05pdmj3dlADL81PxuffWqhobRoiiGFftjNPKpmM"
cortex_name: "soctools-cortex"
cortex_name: "dsoclab-cortex"
cortex_img: "{{repo}}/cortex:{{version}}{{suffix}}"
cortex_elasticsearch_mem: "256m"
# GENERATED WITH cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1
cortex_secret_key: "9CZ844IcAp5dHjsgU4iuaEssdopLcS6opzhVP3Ys4t4eRpNlHmwZdtfveLEXpM9D"
cortex_odfe_pass: "Pass009"
kspass: "Testing003"
tspass: "Testing003"
sysctlconfig:
- { key: "net.core.rmem_max", val: "4194304" }
- { key: "net.core.wmem_max", val: "4194304" }
- { key: "net.core.rmem_max", val: "2097152" }
- { key: "net.core.wmem_max", val: "2097152" }
- { key: "vm.max_map_count" , val: "524288" }
nifi_javamem: "1g"
odfe_javamem: "512m"
nifi_version: 1.12.1
nifi_version: 1.11.4
nifi_repo: "https://archive.apache.org/dist"
ca_cn: "SOCTOOLS-CA"
soctools_users:
- firstname: "Bozidar"
lastname: "Proevski"
username: "bozidar.proevski"
email: "bozidar.proevski@finki.ukim.mk"
DN: "CN=Bozidar Proevski"
CN: "Bozidar Proevski"
password: "Pass001"
- firstname: "Arne"
lastname: "Oslebo"
username: "arne.oslebo"
email: "arne.oslebo@uninett.no"
DN: "CN=Arne Oslebo"
CN: "Arne Oslebo"
password: "Pass002"
- firstname: "Kiril"
lastname: "Kjiroski"
username: "kiril.kjiroski"
email: "kiril.kjiroski@finki.ukim.mk"
DN: "CN=Kiril Kjiroski"
CN: "Kiril Kjiroski"
password: "Pass003"
odfees_img: "{{repo}}/odfees:{{version}}{{suffix}}"
odfekibana_img: "{{repo}}/odfekibana:{{version}}{{suffix}}"
# GENERATE 32-bit secure value
odfekibana_cookie: "iroAm0ueIV7w6CS1WcJTwIV6R4d5RIAt"
odfees_adminpass: "Pass004"
#elk_version: "oss-7.6.1"
elk_version: "oss-7.4.2"
#odfeplugin_version: "1.7.0.0"
......@@ -80,25 +129,16 @@ openid_scope: profile
openid_subjkey: preferred_username
keycloak_img: "{{repo}}/keycloak:{{version}}{{suffix}}"
keycloak_adminpass: "Pass005"
elastic_username: "admin"
misp_token: ""
misp_url: ""
maxmind_key: ""
misp_dbname: "mispdb"
misp_dbuser: "misp"
services:
- mysql
- haproxy
- openjdk
- zookeeper
- nifi
- elasticsearch
- kibana
- odfees
- odfekibana
- keycloak
- misp
- cassandra
- thehive
- cortex
misp_dbpass: "Pass007"
# misp_salt generated with: openssl rand -base64 32
misp_salt: "wa2fJA2mGIn32IDl+uKrCJ069Mg3khDdGzFNv8DOwM0="
---
soctools_users:
- firstname: "User1"
lastname: "SOC"
username: "user1"
email: "user1@soctools.test"
DN: "CN=User1Soctools"
CN: "User1Soctools"
- firstname: "User2"
lastname: "SOC"
username: "user2"
email: "user2@soctools.test"
DN: "CN=User2Soctools"
CN: "User2Soctools"
# Minimum one user is required
ODFE_ADMIN_USERS:
- user1
---
docker_image_path: images
base_image: python:2.7-stretch
all:
hosts:
nifi-image:
ansible_connection: docker
ansible_python_interpreter: /usr/bin/python
localhost:
ansible_python_interpreter: /usr/bin/python
ansible_connection: local
children:
nifi:
hosts:
localhost:
[cassandra]
soctools-cassandra ansible_connection=docker
[cortex]
soctools-cortex ansible_connection=docker
---
index: haproxy
scale: "{{ haproxy_scale | default('1')}}"
docker:
haproxy:
image: haproxy:latest
volumes:
- /usr/local/etc/haproxy/:/usr/local/etc/haproxy:ro
ports:
- "80:80"
source: pull
\ No newline at end of file
---
index: nifi
scale: "{{ nifi_scale | default('1')}}"
docker:
nifi:
# image: nifi-soctools #For nifi image built by soctools
# source: load
image: apache/nifi:latest
source: pull
command: /opt/nifi/nifi-current/scripts/start.sh
env:
NIFI_HOME: "/opt/nifi/nifi-current"
NIFI_LOG_DIR: "/opt/nifi/nifi-current/logs"
NIFI_PID_DIR: "/opt/nifi/nifi-current/run"
NIFI_CLUSTER_IS_NODE: "true"
NIFI_ZK_CONNECT_STRING: "zookeeper_1:2181"
NIFI_CLUSTER_NODE_PROTOCOL_PORT: "8082"
NIFI_ELECTION_MAX_WAIT: "1 min"
load_path: "{{ image_location }}/nifi-soctools.tar"
---
index: zookeeper
scale: "{{ zookeeper_scale | default('1')}}"
docker:
zookeeper:
image: zookeeper:latest
source: pull
\ No newline at end of file
all:
hosts:
host1:
ansible_ssh_user: debian
ansible_python_interpreter: /usr/bin/python
become: yes
children:
soctools_server:
hosts:
host1:
nifi:
hosts:
host1:
nifi_scale: 3
haproxy:
hosts:
host1:
zookeeper:
hosts:
host1:
zookeeper_scale: 3
\ No newline at end of file
[odfeescontainers]
soctools-odfe-1 ansible_connection=docker
soctools-odfe-2 ansible_connection=docker
[filebeat]
soctools-nifi-1 ansible_connection=docker FILEBEAT_FILES='["/opt/nifi/nifi-current/logs/nifi-app.log","/opt/nifi/nifi-current/logs/nifi-bootstrap.log","/opt/nifi/nifi-current/logs/nifi-user.log","/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="nifi" FILEBEAT_LOG_FORMAT="text"
soctools-nifi-2 ansible_connection=docker FILEBEAT_FILES='["/opt/nifi/nifi-current/logs/nifi-app.log","/opt/nifi/nifi-current/logs/nifi-bootstrap.log","/opt/nifi/nifi-current/logs/nifi-user.log","/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="nifi" FILEBEAT_LOG_FORMAT="text"
soctools-nifi-3 ansible_connection=docker FILEBEAT_FILES='["/opt/nifi/nifi-current/logs/nifi-app.log","/opt/nifi/nifi-current/logs/nifi-bootstrap.log","/opt/nifi/nifi-current/logs/nifi-user.log","/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="nifi" FILEBEAT_LOG_FORMAT="text"
soctools-misp ansible_connection=docker FILEBEAT_FILES='["/var/opt/rh/rh-php72/log/php-fpm/*.log","/var/opt/rh/rh-redis32/log/redis/redis.log","/var/log/httpd/*log","/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="misp" FILEBEAT_LOG_FORMAT="text"
soctools-odfe-1 ansible_connection=docker FILEBEAT_FILES='["/usr/share/elasticsearch/logs/soctools-cluster_server.json"]' FILEBEAT_LOG_TYPE="elasticsearch" FILEBEAT_LOG_FORMAT="json"
soctools-odfe-2 ansible_connection=docker FILEBEAT_FILES='["/usr/share/elasticsearch/logs/soctools-cluster_server.json"]' FILEBEAT_LOG_TYPE="elasticsearch" FILEBEAT_LOG_FORMAT="json"
soctools-kibana ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/kibana_stdout.log"]' FILEBEAT_LOG_TYPE="kibana" FILEBEAT_LOG_FORMAT="json"
soctools-keycloak ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="keycloak" FILEBEAT_LOG_FORMAT="json"
soctools-mysql ansible_connection=docker FILEBEAT_FILES='["/var/opt/rh/rh-mariadb103/log/mariadb/mariadb.log","/var/opt/rh/rh-mariadb103/lib/mysql/server_audit.log"]' FILEBEAT_LOG_TYPE="mysql" FILEBEAT_LOG_FORMAT="text"
soctools-haproxy ansible_connection=docker FILEBEAT_SYSLOG_PORT=9000 FILEBEAT_LOG_TYPE="haproxy" FILEBEAT_LOG_FORMAT="text"
soctools-zookeeper ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="zookeeper" FILEBEAT_LOG_FORMAT="text"
soctools-cortex ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="cortex" FILEBEAT_LOG_FORMAT="text"
soctools-thehive ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="thehive" FILEBEAT_LOG_FORMAT="text"
soctools-cassandra ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="cassandra" FILEBEAT_LOG_FORMAT="text"
[haproxy]
soctools-haproxy ansible_connection=docker
\ No newline at end of file
[keycloakcontainers]
soctools-keycloak ansible_connection=docker
[odfekibanacontainers]
soctools-kibana ansible_connection=docker
[mispcontainers]
soctools-misp ansible_connection=docker
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment