Deploying open-source SOC lab with red team simulation, at home. MISP, Cortex and TheHive (Part 2)
Sorry for posting this a bit late for those who are waiting for part 2. During my time writing part 1 and part 2, I was in rush to prepare for the new addition to my family, we (me, wife & family) have been blessed with our firstborn baby boy. So yeah, with the birth of my son, Ramadhan, and preparation for Aidilfitri all together at once, time is a little bit tight. Hehe
Thank you very much for all your wishes and thank you for still
following this series, so this time we will set up and install components for
MISP, Cortex, and TheHive. This includes installation and integration between
these 3 components as described in part 1.
Once you have completed all the installation and integration, this
server will tremendously help your analysis and automate a lot of things, you
can track your progress or investigation, manage case, alert, task, and IOCs
found. With a click of a button, you can receive reports and threat intel from
multiple sources.
This can be your great tool as a blue team.
Housekeeping
Installing these service will create a new group and user to manage the service. I would like to
advise you need to install the service first so that the group and user are
available to manage and configure the respective service.
Just in case your
configuration is not applied even after you restart the service, just reboot
your machine.
Let’s start!
Switch to an
interactive session as a root user so we don’t need to put sudo in every
command we run.
sudo -i |
Update and upgrade
apt update -y && apt upgrade -y |
Installing net-tools,
curl, apt-transport-https and OpenSSH-Server
apt install
net-tools curl apt-transport-https openssh-server -y |
Get the IP address of
your machine
ifconfig |
The best is to configure your server network configuration into static IP address, dynamic address might break your app during runtime when the IP address changed.
Enable OpenSSH-Server
on startup and start OpenSSH-Server
systemctl enable
openssh-server.service |
systemctl start
openssh-server.service |
Installing MISP
Download MISP install
script.
wget
https://github.com/MISP/MISP/blob/2.4/INSTALL/INSTALL.sh -O INSTALL.sh |
Give the script
appropriate permission
chmod 770 INSTALL.sh |
If you only run the
script without any parameters, it will show the installation options.
./INSTALL.sh |
We will install
everything, so we will use option -A. we need to save the output of the installation
because when MISP installation is complete, there is a set of important credentials
that we need.
./INSTALL.sh -A |
tee misp-install-output.txt |
Just wait until the
installation is complete.
By default, you can go
to MISP web page by navigate to https://<misp.ip.server> on your browser.
Default credential for
user is admin@admin.test
Default password is admin
As I remember. Hehe.
Then, just follow any instruction to initialize the service.
Create MISP
organization, User, Authentication Key and add Threat Feed.
For every new instance
of MISP, you can first login using default credential. Once you login, follow
this step to create or to administer respective item.
Organization
We will create new
organization, Go to Administration > Add organization.
Here filled in all the
details of our new organization.
For UUID, just click
on Generate UUID. When everything has been added, click “Submit”.
By the way, I already created this organization for my MISP. This is just step-by-step guide you can follow to create new MISP organization.
Organization User
Navigate to
Administration > Add User.
Here filled in all the
details of our user. Assigned the user to our newly created organization.
This user will have
the admin right and I will use this account to create authorization key a.k.a
API key to retrieve event of threat intel from MISP.
When everything is good. Create the user by clicking the Create user button.
Authentication Key
Navigate to
Administration > List Auth Keys
Here is some of the
list Authentication key that has been created. One
of the usages of Authentication Key is it can be used by third party services
to grab feed data from MISP.
So let me show how to
create new Authentication Key for Cortex.
Click on Add authentication
key.
I’ve set the User to
inherit the Auth key to admin@soc.local and only this server
IP can use this Auth Key. To make sure nothing can be done for managerial
purposes for this Auth Key, I’ve set this key as Read-only.
After everything is
good, click Submit.
After that, MISP will
prompt your Authentication Key. Save and store the key, it will only show
once in plain text. We will use this key for Cortex Analyzer.
Adding Feed
MISP has provided more
than 50 OSINT feeds you can use. Most of the feed is not added when you install
MISP instance. Let’s add and enable some of the feeds.
First, Login using admin@soc.local account.
Navigate to Sync
Actions > List Feeds
By defaults, you will
only see 2 feeds available. Now, navigate to Import Feeds from JSON
Now, we need to copy
list of Threat Intel feeds in JSON format and paste it here.
Copy the JSON from defaults.json available from MISP
Github.
Once you have pasted
the JSON in the text field, click Add.
Then, it will redirect
you back to List Feeds. You can enable, cache, and store available feed data, any
feed you like.
Installing Cortex
Download
TheHive-Project PGP
curl
https://raw.githubusercontent.com/TheHive-Project/TheHive/master/PGP-PUBLIC-KEY
| sudo apt-key add - |
Download
and add TheHive-Project repository
curl
https://raw.githubusercontent.com/TheHive-Project/TheHive/master/PGP-PUBLIC-KEY
| sudo apt-key add - |
Cortex require
Elasticsearch.
Download and install
Elasticsearch PGP Key
wget -qO -
https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor -o
/usr/share/keyrings/elasticsearch-keyring.gpg |
Saving Elastic
Repository to /etc/apt/sources.list.d/elastic-7.x.list
echo "deb
[signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/7.x/apt
stable main" | tee /etc/apt/sources.list.d/elastic-7.x.list |
Update
repository, download and install Elasticsearch
apt update -y && apt install
elasticsearch -y |
Once
completed, uncomment, edit and add the following in Elasticsearch configuration
file /etc/elasticsearch/elasticsearch.yml
Configure initial elasticsearch.yml
settings file.
nano /etc/elasticsearch/elasticsearch.yml |
# ---------------------------------- Cluster
----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: hive # # ------------------------------------ Node
------------------------------------ # # Use a descriptive name for the node: # node.name: node-1 # # ---------------------------------- Network
----------------------------------- # # By default Elasticsearch is only
accessible on localhost. Set a different # address here to expose this node on the
network: # network.host: 192.168.1.103 # # ---------------------------------
Discovery ---------------------------------- # # Pass an initial list of hosts to perform
discovery when this node is started: # The default list of hosts is
["127.0.0.1", "[::1]"] # #discovery.seed_hosts: ["host1",
"host2"] # # Bootstrap the cluster using an initial set
of master-eligible nodes: # cluster.initial_master_nodes:
["node-1"] # # ----------------------------------
Security ---------------------------------- # thread_pool.search.queue_size: 100000 |
Close
and save the file.
To enable Elasticsearch on startup and start Elasticsearch
systemctl
daemon-reload |
systemctl enable
elasticsearch.service |
systemctl start
elasticsearch.service |
Elasticsearch may take a while to start. Once the sessions are back, curl your IP address:9200 to check if Elasticsearch is running.
The output should be something like
this:
Now,
install Cortex.
apt install cortex -y |
Before
starting the server, to make sure that the cortex account can create log files
in /opt/cortex/logs. We need to run this command
sudo addgroup cortex |
sudo adduser --system cortex |
sudo cp /opt/cortex/package/cortex.service
/usr/lib/systemd/system |
sudo chown -R cortex:cortex /opt/cortex |
sudo chgrp cortex
/etc/cortex/application.conf |
sudo chmod 640 /etc/cortex/application.conf |
Once
completed, we need to add server secret key. To explain this better, this is
what they say in the documentation.
“The
only required parameter in order to start Cortex is the key of the server
(play.http.secret.key). This key is used to authenticate cookies that contain
data. If Cortex runs in cluster mode, all instances must share the same key.”
Run
this command
sudo mkdir /etc/cortex (cat << _EOF_ # Secret key # ~~~~~ # The secret key is used to secure
cryptographics functions. # If you deploy your application to several
instances be sure to use the same key! play.http.secret.key="$(cat
/dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1)" _EOF_ ) | sudo tee -a /etc/cortex/application.conf |
After
that done, you can tail /etc/cortex/application.conf to check if the key has
been successfully appended.
tail -n 10 /etc/cortex/application.conf |
Check
if configuration for Elasticsearch is correct in /etc/cortex/application.conf.
It should be like this.
## ElasticSearch search { #
Name of the index
index = cortex #
ElasticSearch instance address. #
For cluster, join address:port with ',':
"http://ip1:9200,ip2:9200,ip3:9200" #
example – uri = “http://<your-machine-ip-address>:9200” uri = "http://192.168.1.103:9200" |
If
everything is good, you can enable Cortex to start during start up, start
Cortex and restart Elasticsearch
systemctl enable cortex.service |
systemctl start cortex.service |
systemctl restart elasticsearch.service |
Installing
Cortex Analyzer
To install Cortex Analyzer, you may need to have python-pip2.
You can follow the
step from the tutorial here: Installing pip for Python2
Download all
dependency packages
apt install -y --no-install-recommends python-pip
python2.7-dev python3-pip python3-dev ssdeep libfuzzy-dev libfuzzy2
libimage-exiftool-perl libmagic1 build-essential git libssl-dev |
Install
Python setuptools and update pip/pip3
pip install -U pip setuptools && sudo pip3
install -U pip setuptools |
Clone the
Cortex Analyzer repository
git clone
https://github.com/TheHive-Project/Cortex-Analyzers |
Every
Analyzer have its own pip compatible requirements.txt file. Install the requirements
with the following commands
for I in $(find Cortex-Analyzers -name
'requirements.txt'); do sudo -H pip2 install -r $I; done && \ for I in $(find Cortex-Analyzers -name
'requirements.txt'); do sudo -H pip3 install -r $I || true; done |
Once
complete, check the Analyzer configuration in /etc/cortex/application.conf
file. We need to set where Cortex can find the Analyzer file. My Analyzer file
is in /opt/cortex/ Cortex-Analyzers/analyzers/
Edit the the
/etc/cortex/application.conf file
nano /etc/cortex/application.conf |
## ANALYZERS # analyzer { # analyzer
location # url can be
point to: # -
directory where analyzers are installed # - json
file containing the list of analyzer descriptions urls = [
"/opt/cortex/Cortex-Analyzers/analyzers/" ] |
Once completed,
restart Cortex service
systemctl restart cortex.service |
We’re done
with installing Cortex and its Analyzer, Cortex-Analyzer can be configured via
Web UI.
Once it is
started, you may browse from your browser and connect to http://YOUR_SERVER_ADDRESS:9001/
On first access, you need to update Cortex database. Just click “Update Database”, then you need to create a SuperAdmin account. Once complete you will be redirected to login page.
Login with the account you just created.
If you do not find any Cortex or Analyzers configuration as shown, restart your machine.
Cortex
Troubleshoot
If the login
page is blank and there is warning regarding Elasticsearch. Check the status of
Elasticsearch. If the service is down, restart Elasticsearch. Once
Elasticsearch back online, just refresh the page. You may use command below to
check status and restart Elasticsearch.
Check
Elasticsearch status
systemctl status elasticsearch.service |
Restart
Elasticsearch
systemctl restart elasticsearch.service |
Installing
TheHive
TheHive
require few components to be installed first before we can actually install TheHive.
We need to have Java Virtual Machine (JVM) and Cassandra as database. To
automate this step, just use this bash script or you can execute the command
line by line, your choice.
nano thehive-component.sh |
Paste this
script.
#! /bin/bash prinf "\n\ninstalling java and
cassandra\n\n" apt install -y openjdk-8-jre-headless -y echo
JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64" >>
/etc/environment export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64" curl -fsSL
https://www.apache.org/dist/cassandra/KEYS | apt-key add - echo "deb http://www.apache.org/dist/cassandra/debian
311x main" | tee -a /etc/apt/sources.list.d/cassandra.sources.list apt update - y apt install cassandra -y |
Ctrl + X to
close and save the buffer.
Give execute
permission to the script
chmod +x thehive-component.sh |
Execute the
script. Once the installation complete, continue to next step.
./ thehive-component.sh |
Start
Cassandra so we can edit some configuration.
cqlsh localhost 9042 |
Once the
session start, change the cluster_name to thp.
UPDATE system.local SET cluster_name = 'thp' where
key='local'; |
To exit, the
command is “exit”.
Next, run
this command.
nodetool flush |
Now, we need
to edit Cassandra configuration file, /etc/cassandra/cassandra.yaml
nano /etc/cassandra/cassandra.yaml |
Change according
to these settings.
# content from /etc/cassandra/cassandra.yaml cluster_name: 'thp' listen_address: 'xx.xx.xx.xx' # change to machine IP
address rpc_address: 'xx.xx.xx.xx' # change to machine IP address
for clients seed_provider: -
class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters: -
seeds: 'xx.xx.xx.xx' # change to machine IP address / self for the first node data_file_directories: -
'/var/lib/cassandra/data' commitlog_directory: '/var/lib/cassandra/commitlog' saved_caches_directory:
'/var/lib/cassandra/saved_caches' hints_directory: -
'/var/lib/cassandra/hints' |
Ctrl + X to
close the file and save the buffer.
Enable
Cassandra to start during start up and restart the service.
systemctl enable cassandra.service |
systemctl restart cassandra.service |
Next, we can install the hive!
apt install thehive4 |
After that installation complete, we need to create a dedicated folder to store file in the local storage and set thehive user as owner.
mkdir -p /opt/thp/thehive/files |
chown -R thehive:thehive /opt/thp/thehive/files |
Create a folder dedicated to host indexes for TheHive
mkdir /opt/thp/thehive/index |
chown thehive:thehive -R /opt/thp/thehive/index |
Now, we need to edit TheHive configuration file which at /etc/thehive/application.conf
nano /etc/thehive/application.conf |
Match and
edit the configuration to your machine.
## Database configuration db.janusgraph { storage { ##
Cassandra configuration # More
information at
https://docs.janusgraph.org/basics/configuration-reference/#storagecql backend:
cql #change to
your machine IP address hostname:
["192.168.1.103"] #
Cassandra authentication (if configured) #
username: "thehive" #
password: "password" cql {
cluster-name: thp
keyspace: thehive } } index.search
{ backend:
lucene directory:
/opt/thp/thehive/index # If
TheHive is in cluster ElasticSearch must be used: } ## For test
only ! # Comment
the two lines below before enable Cassandra database
storage.backend: berkeleyje
storage.directory: /opt/thp/thehive/database //
berkeleyje.freeDisk: 200 # disk usage threshold } ## Attachment storage configuration storage { ## Local
filesystem provider:
localfs
localfs.location: /opt/thp/thehive/files |
Ctrl + X to
close the file and save the buffer.
Now, we can start TheHive. Enable
TheHive to start during start up and start the service.
systemctl enable thehive.service |
systemctl start thehive.service |
I find sometimes
TheHive does take some time to start. Once the service started, you may browse
from your browser and connect to http://YOUR_SERVER_ADDRESS:9000/.
The default admin user is admin@thehive.local with password secret. It is recommended to change the default password.
Integrating
Cortex and TheHive
First, In
Cortex, we need to create new user and generate api for integration. To do
this, just follow this step.
Login to your Cortex. Let’s create new organization name
SOC-Org. As you can see, I already create the organization.
Then, select
the organization, and create user. This is step to create user. Click + Add
user.
Create the
following user and roles.
So, there are 3 user we just created which are self
explainatory. As orgadmin, you have full access in your organization, you can
add user and setup analyzer for your organization.
Follow
the same step to create organization and user in TheHive. Just create the same
organization.
Alright,
done for Organization and User creation.
In Cortex,
Once the user has been created, click botton “Create API Key” for api-user@soc.local. Once created, click Reveal, copy and store this value. We need this to
integrate TheHive with Cortex.
Now, edit /etc/thehive/application.conf
to add the API key.
nano /etc/thehive/application.conf |
Uncomment,
edit and add as the following format:
## CORTEX configuration # More information at https://github.com/TheHive-Project/TheHiveDocs/TheHive4/Administration/Connectors.md # Enable Cortex connector play.modules.enabled +=
org.thp.thehive.connector.cortex.CortexModule cortex { servers: [ { name:
"CORTEX" #
Cortex name url:
"http://192.xxx.xxx.xxx:9001" # URL of Cortex instance, change to http://<your-cortex-machine>:9001 auth { type:
"bearer" key:
"paste-api-key-here"
# change to Cortex API key you have copied } wsConfig
{} # HTTP client
configuration (SSL and proxy) } ] } |
Ctrl + X to
close and save the buffer.
Restart TheHive
systemctl restart thehive.service |
When
everything is done, login to thehive and go to About. It should be indicator
that show the integration with the hive is successful something like CORTEX
– CORTEX – X.X.X.X (OK)
So now the TheHive – Cortex integration is completed, we should be able to receive report from Cortex Analyzer that we enable.
Install
Analyzer Template
Let’s
install analyzer template in TheHive, you neet to login using admin@thehive.local to do this.
After login,
select Admin > Analyzer Template. This will redirect us to Analyzer template
management.
Next, download the official templates archive. There is hyperlink, click on “from here” hyperlink then it will download the template in zip file.
Once the download is complete, do not unzip the archive.
Import the templates by clicking on Import templates button, select the zip you have just downloaded.
After
selecting the archive, click “Yes, Import template archive”. That’s all.
Integrating
MISP and VirusTotal Get Report API for Cortex Analyzer.
In Cortex,
login using admin account from new organization you have created. In this case,
we created admin@soc.local.
We will
enable 2 Analyzer, Virustotal Get Report and MISP.
Then, navigate to Organization > Analyzers
VirusTotal Analyzer
First get your VirusTotal API key, this requires you to have VirusTotal account. Login or create your VirusTotal account and copy your API Key.
Here is
guide provided by VirusTotal to get you API key: VirusTotal API Key
We will
enable VirusTotal_GetReport_3.0, click enable and paste your API Key in key
text field. You may review and enable additional settings which ever you feel
suitable.
Field with asterisk is required field. Others are optional. Paste your VirusTotal API Key in Key field.
After you have review everything, click on “Save”
MISP
Analyzer
Search for
MISP analyzer, keyword is MISP. Once you have found it, to enable the analyzer,
click Enable.
Add the
details correspond to your MISP Server IP address.
For URL
field, add https and backslash in the address. For example: https://192.168.1.103/
Paste the Authorization
Key you have generated and copied from MISP in Key field.
Select False
for cert_check to make things lot easier for us. If you have dedicated CA used
for your MISP server. You may specific it in cert_path.
Once
everything is done, click Save.
Summary
When
everything is configured, you should be able to see in TheHive that you can
analyze observable using Cortex Analyzer that we have configured as shown in the picture below.
I think
that’s all for MISP Analyzer. With that, we have completed all configurations for this
MISP, Cortex, and TheHive server.
Thank you
and see you again in the next part!
Thanks for taking time out to write this post. This is very informational. I'm implementing this and I'm currently on the 1st part.
ReplyDeleteCome accross a lot of errors so I'm taking things slow. But planning on finishing this soon as and when I get time off from my office I love to get back in there and get my hands dirty 😁.
Please continue to write on this series. If not everyone atleast I'm eagerly waiting.
Thanks again.
Best,
Mashroor Alam