Commit 75be43b2 authored by Kiril KJiroski's avatar Kiril KJiroski
Browse files

Revert "thehive integration with keycloak"

This reverts commit d0fc936f
parent d0fc936f
Howto's
=======
Modify main NiFi pipeline
-------------------------
To make modifications to the main NiFi pipeline and add it to the Ansible playbook, do the following in the soctool directory:
* Make necesarry to the pipeline in the NiFi GUI
* Copy flow.xml.gz file from one of the NiFi containers:
`docker cp soctools-nifi-1:/opt/nifi/nifi-current/conf/flow.xml.gz .`
* Convert flowx.xml.gz to new template
`utils/flow2template.py flow.xml.gz roles/nifi/templates/flow.xml.j2`
Update configuration files in docker containers using Ansible
-------------------------------------------------------------
To update configuration files for all docker containers together, run the following command:
ansible-playbook -i inventories soctools.yml -t update-config
To update configuration files only for specific services, run the following commands:
ansible-playbook -i inventories soctools.yml -t update-keycloak-config
ansible-playbook -i inventories soctools.yml -t update-thehive-config
ansible-playbook -i inventories soctools.yml -t update-cortex-config
ansible-playbook -i inventories soctools.yml -t update-cassandra-config
ansible-playbook -i inventories soctools.yml -t update-haproxy-config
ansible-playbook -i inventories soctools.yml -t update-filebeat-config
ansible-playbook -i inventories soctools.yml -t update-nifi-config
ansible-playbook -i inventories soctools.yml -t update-odfees-config
ansible-playbook -i inventories soctools.yml -t update-odfekibana-config
Restart services inside docker containers using Ansible
-------------------------------------------------------
To restart services for all docker containers together, run the following command:
ansible-playbook -i inventories soctools.yml -t restart
To restart services only for specific docker containers, run the following commands:
ansible-playbook -i inventories soctools.yml -t restart-keycloak
ansible-playbook -i inventories soctools.yml -t restart-thehive
ansible-playbook -i inventories soctools.yml -t restart-cortex
ansible-playbook -i inventories soctools.yml -t restart-cassandra
ansible-playbook -i inventories soctools.yml -t restart-haproxy
ansible-playbook -i inventories soctools.yml -t restart-filebeat
ansible-playbook -i inventories soctools.yml -t restart-misp
ansible-playbook -i inventories soctools.yml -t restart-mysql
ansible-playbook -i inventories soctools.yml -t restart-nifi
ansible-playbook -i inventories soctools.yml -t restart-odfees
ansible-playbook -i inventories soctools.yml -t restart-odfekibana
Stop services inside docker containers using Ansible
----------------------------------------------------
To stop services for all docker containers together, run the following command:
ansible-playbook -i inventories soctools.yml -t stop
To stop services only for specific docker containers, run the following commands:
ansible-playbook -i inventories soctools.yml -t stop-keycloak
ansible-playbook -i inventories soctools.yml -t stop-thehive
ansible-playbook -i inventories soctools.yml -t stop-cortex
ansible-playbook -i inventories soctools.yml -t stop-cassandra
ansible-playbook -i inventories soctools.yml -t stop-haproxy
ansible-playbook -i inventories soctools.yml -t stop-filebeat
ansible-playbook -i inventories soctools.yml -t stop-misp
ansible-playbook -i inventories soctools.yml -t stop-mysql
ansible-playbook -i inventories soctools.yml -t stop-nifi
ansible-playbook -i inventories soctools.yml -t stop-odfees
ansible-playbook -i inventories soctools.yml -t stop-odfekibana
Restart services inside docker containers manually
--------------------------------------------------
To restart services inside docker containers after changes in configuration files:
1. Attache container: docker exec -it container_id_or_name bash (example: docker exec -it soctools-keycloak bash)
2. List services and their statuses: supervisorctl status
3. Restart service: supervisorctl restart supervisor_service_name (example: supervisorctl restart keycloak)
4. Detach from container: exit
......@@ -13,36 +13,45 @@ Log in and install ansible:
`yum -y install ansible git`
`ansible-galaxy collection install ansible.posix`
Clone soctools:
`git clone https://scm.uninett.no/geant-wp8-t3.1/soctools.git`
Clone soctools:
Temporary solution: Upload your ssh key to gitlab.geant.org
`git clone git@gitlab.geant.org:gn4-3-wp8-t3.1-soc/soctools.git`
`cd soctools`
Install soctools:
Edit group_vars/all/main.yml and change 'dslproxy' so that it point to the FQDN of the server.
Edit group_vars/all/main.yml and change 'soctoolsproxy' so that it point to the FQDN of the server.
`vi group_vars/all/main.yml`
The first entry in the soctools_users variable is the user with full admin privileges in NiFi and Kibana.
Users are specified in the file:
`group_vars/all/users.yml`
To configure the server running soctools, run the ansible playbook:
`ansible-playbook -i soctools-inventory soctools_server.yml`
`ansible-playbook -i inventories soctools_server.yml`
To build the Docker images needed, run the ansible playbook:
`ansible-playbook -i soctools-inventory buildimages.yml`
`ansible-playbook -i inventories buildimages.yml`
To build the CA needed for host and user certificates, run the ansible playbook:
`ansible-playbook -i soctools-inventory buildca.yml`
`ansible-playbook -i inventories buildca.yml`
User certificates are can be found in the directory roles/ca/files/CA/private. Import into browser for authentication.
If using soctools CA certificates provided with this installation, you first need to download and import root certificate found in secrets/CA/ca.crt
For Windows, CA certificate should be installed in Trusted Root Certification Authorities store.
User certificates are can be found in the directory secrets/certificates. Import into browser for authentication.
For Windows, user certificate should be installed in Personal store. Passwords for the certificates can be found in the directory secrets/passwords.
To start the cluster, run the ansible playbook soctools.yml:
`ansible-playbook -i soctools-inventory soctools.yml -t start`
`ansible-playbook -i inventories soctools.yml -t start`
To stop the cluster, run the ansible playbook soctools.yml:
`ansible-playbook -i soctools-inventory soctools.yml -t stop`
The NiFi interface should now be available on port 9443 on the server.
The OpenDistro for Elasticsearch interface should now be available on port 5601 on the server. To access preconfigured
index patterns you have to switch to Global tenant.
The Keycloak IdP interface should now be available on port 12443 on the server.
`ansible-playbook -i inventories soctools.yml -t stop`
Web interfaces are available on the following ports:
* 9443 - NiFi
* 5601 - Kibana
* 6443 - Misp : Default user/password: admin@admin.test/test
* 9000 - The Hive : Default user/password: admin@thehive.local/secret
* 9001 - Cortex
* 12443 - Keycloak : Default user/password: admin/Pass005
License
-------
......
---
- name: Build certification authority
hosts: dsldev
hosts: soctoolsmain
roles:
- ca
---
- name: Build docker images
hosts: dsldev
hosts: soctoolsmain
roles:
- build
---
dslproxy: "dsoclab.gn4-3-wp8-soc.sunet.se"
soctoolsproxy: "<CHANGE_ME:hostname>"
# TheHive Button plugin
THEHIVE_URL: "https://hive.gn4-3-wp8-soc.sunet.se/"
# here enter API key for default admin user
THEHIVE_API_KEY: "bs2Jc3tGJqhVv0AYyX2NYlhMlorPz7mX"
# ID of the default admin user
THEHIVE_OWNER: "admin@thehive.local"
# TheHive Create Organisation and Users
# Login as default admin user and create API key, populate it here
# thehive_admin_api: "KoHrKbIJm8XMsJxA9nZLs6YemCu76o3u"
# thehive_writer: "[write]"
#THEHIVE_API_KEY: "1gFdNhmUSxO3BRe1SBB5JYEvkW9UOo6s"
THEHIVE_USERS:
- kiril:
username: "kiril"
name: "Kiril"
surname: "Kiroski"
roles: '["read", "write", "admin"]'
organization: "uninett.no"
- temur:
username: "temur"
name: "Temur"
surname: "Maisuradze"
roles: '["read", "write", "admin"]'
organization: "uninett.no"
maxmind_key: ""
docker_build_dir: "{{playbook_dir}}/build"
# TheHive Button plugin
THEHIVE_URL: "https://hive.gn4-3-wp8-soc.sunet.se/"
THEHIVE_API_KEY: "5LymseWiurZBrQN8Kqp8O+9KniTL5cE0"
THEHIVE_OWNER: "admin"
soctools_netname: "soctoolsnet"
soctools_network: "172.22.0.0/16"
repo: gn43-dsl
repo: soctools
version: 7
suffix: a20201004
haproxy_name: "dsoclab-haproxy"
haproxy_name: "soctools-haproxy"
haproxy_version: "2.2"
haproxy_img: "{{repo}}/haproxy:{{version}}{{suffix}}"
HAPROXY_PROCESSES: "2"
HAPROXY_STATS_PASS: "eiph2Eepaizicheelah3tei+bae3ohgh"
FILEBEAT_VERSION: "7.9.3"
FILEBEAT_OUTPUT_HOST: "{{soctoolsproxy}}"
FILEBEAT_OUTPUT_PORT: "6000"
FILEBEAT_CERT: "/opt/filebeat/filebeat.crt"
FILEBEAT_KEY: "/opt/filebeat/filebeat.key"
temp_root: "/tmp/centosbuild"
openjdk_img: "{{repo}}/openjdk:{{version}}{{suffix}}"
zookeeper_name: "dsoclab-zookeeper"
zookeeper_name: "soctools-zookeeper"
zookeeper_img: "{{repo}}/zookeeper:{{version}}{{suffix}}"
misp_name: "dsoclab-misp"
misp_name: "soctools-misp"
misp_img: "{{repo}}/misp:{{version}}{{suffix}}"
misp_url: "https://{{soctoolsproxy}}:6443"
nifi_img: "{{repo}}/nifi:{{version}}{{suffix}}"
mysql_name: "dsoclab-mysql"
mysql_name: "soctools-mysql"
mysql_img: "{{repo}}/mysql:{{version}}{{suffix}}"
mysql_dbrootpass: "Pass006"
cassandra_name: "dsoclab-cassandra"
cassandra_name: "soctools-cassandra"
cassandra_img: "{{repo}}/cassandra:{{version}}{{suffix}}"
thehive_name: "dsoclab-thehive"
thehive_name: "soctools-thehive"
thehive_img: "{{repo}}/thehive:{{version}}{{suffix}}"
# GENERATED WITH cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1
thehive_secret_key: "LcnI9eKLo33711BmCnzf6UM1y05pdmj3dlADL81PxuffWqhobRoiiGFftjNPKpmM"
cortex_name: "dsoclab-cortex"
cortex_name: "soctools-cortex"
cortex_img: "{{repo}}/cortex:{{version}}{{suffix}}"
cortex_elasticsearch_mem: "256m"
# GENERATED WITH cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1
cortex_secret_key: "9CZ844IcAp5dHjsgU4iuaEssdopLcS6opzhVP3Ys4t4eRpNlHmwZdtfveLEXpM9D"
cortex_odfe_pass: "Pass009"
kspass: "Testing003"
tspass: "Testing003"
sysctlconfig:
- { key: "net.core.rmem_max", val: "2097152" }
- { key: "net.core.wmem_max", val: "2097152" }
- { key: "net.core.rmem_max", val: "4194304" }
- { key: "net.core.wmem_max", val: "4194304" }
- { key: "vm.max_map_count" , val: "524288" }
nifi_javamem: "1g"
odfe_javamem: "512m"
nifi_version: 1.11.4
nifi_version: 1.12.1
nifi_repo: "https://archive.apache.org/dist"
ca_cn: "SOCTOOLS-CA"
soctools_users:
- firstname: "Bozidar"
lastname: "Proevski"
username: "bozidar.proevski"
email: "bozidar.proevski@finki.ukim.mk"
DN: "CN=Bozidar Proevski"
CN: "Bozidar Proevski"
password: "Pass001"
- firstname: "Arne"
lastname: "Oslebo"
username: "arne.oslebo"
email: "arne.oslebo@uninett.no"
DN: "CN=Arne Oslebo"
CN: "Arne Oslebo"
password: "Pass002"
- firstname: "Kiril"
lastname: "Kjiroski"
username: "kiril.kjiroski"
email: "kiril.kjiroski@finki.ukim.mk"
DN: "CN=Kiril Kjiroski"
CN: "Kiril Kjiroski"
password: "Pass003"
odfees_img: "{{repo}}/odfees:{{version}}{{suffix}}"
odfekibana_img: "{{repo}}/odfekibana:{{version}}{{suffix}}"
# GENERATE 32-bit secure value
odfekibana_cookie: "iroAm0ueIV7w6CS1WcJTwIV6R4d5RIAt"
odfees_adminpass: "Pass004"
#elk_version: "oss-7.6.1"
elk_version: "oss-7.4.2"
#odfeplugin_version: "1.7.0.0"
......@@ -129,16 +80,25 @@ openid_scope: profile
openid_subjkey: preferred_username
keycloak_img: "{{repo}}/keycloak:{{version}}{{suffix}}"
keycloak_adminpass: "Pass005"
elastic_username: "admin"
misp_token: ""
misp_url: ""
maxmind_key: ""
misp_dbname: "mispdb"
misp_dbuser: "misp"
misp_dbpass: "Pass007"
# misp_salt generated with: openssl rand -base64 32
misp_salt: "wa2fJA2mGIn32IDl+uKrCJ069Mg3khDdGzFNv8DOwM0="
services:
- mysql
- haproxy
- openjdk
- zookeeper
- nifi
- elasticsearch
- kibana
- odfees
- odfekibana
- keycloak
- misp
- cassandra
- thehive
- cortex
---
soctools_users:
- firstname: "User1"
lastname: "SOC"
username: "user1"
email: "user1@soctools.test"
DN: "CN=User1Soctools"
CN: "User1Soctools"
- firstname: "User2"
lastname: "SOC"
username: "user2"
email: "user2@soctools.test"
DN: "CN=User2Soctools"
CN: "User2Soctools"
# Minimum one user is required
ODFE_ADMIN_USERS:
- user1
---
docker_image_path: images
base_image: python:2.7-stretch
all:
hosts:
nifi-image:
ansible_connection: docker
ansible_python_interpreter: /usr/bin/python
localhost:
ansible_python_interpreter: /usr/bin/python
ansible_connection: local
children:
nifi:
hosts:
localhost:
[cassandra]
soctools-cassandra ansible_connection=docker
[cortex]
soctools-cortex ansible_connection=docker
---
index: haproxy
scale: "{{ haproxy_scale | default('1')}}"
docker:
haproxy:
image: haproxy:latest
volumes:
- /usr/local/etc/haproxy/:/usr/local/etc/haproxy:ro
ports:
- "80:80"
source: pull
\ No newline at end of file
---
index: nifi
scale: "{{ nifi_scale | default('1')}}"
docker:
nifi:
# image: nifi-soctools #For nifi image built by soctools
# source: load
image: apache/nifi:latest
source: pull
command: /opt/nifi/nifi-current/scripts/start.sh
env:
NIFI_HOME: "/opt/nifi/nifi-current"
NIFI_LOG_DIR: "/opt/nifi/nifi-current/logs"
NIFI_PID_DIR: "/opt/nifi/nifi-current/run"
NIFI_CLUSTER_IS_NODE: "true"
NIFI_ZK_CONNECT_STRING: "zookeeper_1:2181"
NIFI_CLUSTER_NODE_PROTOCOL_PORT: "8082"
NIFI_ELECTION_MAX_WAIT: "1 min"
load_path: "{{ image_location }}/nifi-soctools.tar"
---
index: zookeeper
scale: "{{ zookeeper_scale | default('1')}}"
docker:
zookeeper:
image: zookeeper:latest
source: pull
\ No newline at end of file
all:
hosts:
host1:
ansible_ssh_user: debian
ansible_python_interpreter: /usr/bin/python
become: yes
children:
soctools_server:
hosts:
host1:
nifi:
hosts:
host1:
nifi_scale: 3
haproxy:
hosts:
host1:
zookeeper:
hosts:
host1:
zookeeper_scale: 3
\ No newline at end of file
[odfeescontainers]
soctools-odfe-1 ansible_connection=docker
soctools-odfe-2 ansible_connection=docker
[filebeat]
soctools-nifi-1 ansible_connection=docker FILEBEAT_FILES='["/opt/nifi/nifi-current/logs/nifi-app.log","/opt/nifi/nifi-current/logs/nifi-bootstrap.log","/opt/nifi/nifi-current/logs/nifi-user.log","/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="nifi" FILEBEAT_LOG_FORMAT="text"
soctools-nifi-2 ansible_connection=docker FILEBEAT_FILES='["/opt/nifi/nifi-current/logs/nifi-app.log","/opt/nifi/nifi-current/logs/nifi-bootstrap.log","/opt/nifi/nifi-current/logs/nifi-user.log","/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="nifi" FILEBEAT_LOG_FORMAT="text"
soctools-nifi-3 ansible_connection=docker FILEBEAT_FILES='["/opt/nifi/nifi-current/logs/nifi-app.log","/opt/nifi/nifi-current/logs/nifi-bootstrap.log","/opt/nifi/nifi-current/logs/nifi-user.log","/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="nifi" FILEBEAT_LOG_FORMAT="text"
soctools-misp ansible_connection=docker FILEBEAT_FILES='["/var/opt/rh/rh-php72/log/php-fpm/*.log","/var/opt/rh/rh-redis32/log/redis/redis.log","/var/log/httpd/*log","/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="misp" FILEBEAT_LOG_FORMAT="text"
soctools-odfe-1 ansible_connection=docker FILEBEAT_FILES='["/usr/share/elasticsearch/logs/soctools-cluster_server.json"]' FILEBEAT_LOG_TYPE="elasticsearch" FILEBEAT_LOG_FORMAT="json"
soctools-odfe-2 ansible_connection=docker FILEBEAT_FILES='["/usr/share/elasticsearch/logs/soctools-cluster_server.json"]' FILEBEAT_LOG_TYPE="elasticsearch" FILEBEAT_LOG_FORMAT="json"
soctools-kibana ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/kibana_stdout.log"]' FILEBEAT_LOG_TYPE="kibana" FILEBEAT_LOG_FORMAT="json"
soctools-keycloak ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="keycloak" FILEBEAT_LOG_FORMAT="json"
soctools-mysql ansible_connection=docker FILEBEAT_FILES='["/var/opt/rh/rh-mariadb103/log/mariadb/mariadb.log","/var/opt/rh/rh-mariadb103/lib/mysql/server_audit.log"]' FILEBEAT_LOG_TYPE="mysql" FILEBEAT_LOG_FORMAT="text"
soctools-haproxy ansible_connection=docker FILEBEAT_SYSLOG_PORT=9000 FILEBEAT_LOG_TYPE="haproxy" FILEBEAT_LOG_FORMAT="text"
soctools-zookeeper ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="zookeeper" FILEBEAT_LOG_FORMAT="text"
soctools-cortex ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="cortex" FILEBEAT_LOG_FORMAT="text"
soctools-thehive ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="thehive" FILEBEAT_LOG_FORMAT="text"
soctools-cassandra ansible_connection=docker FILEBEAT_FILES='["/var/log/supervisor/*.log"]' FILEBEAT_LOG_TYPE="cassandra" FILEBEAT_LOG_FORMAT="text"
[haproxy]
soctools-haproxy ansible_connection=docker
\ No newline at end of file
[keycloakcontainers]
soctools-keycloak ansible_connection=docker
[odfekibanacontainers]
soctools-kibana ansible_connection=docker
[mispcontainers]
soctools-misp ansible_connection=docker
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment