Deploying open-source SOC lab with red team simulation, at home. MISP, Cortex and TheHive (Part 2)

Sorry for posting this a bit late for those who are waiting for part 2. During my time writing part 1 and part 2, I was in rush to prepare for the new addition to my family, we (me, wife & family) have been blessed with our firstborn baby boy. So yeah, with the birth of my son, Ramadhan, and preparation for Aidilfitri all together at once, time is a little bit tight. Hehe

Thank you very much for all your wishes and thank you for still following this series, so this time we will set up and install components for MISP, Cortex, and TheHive. This includes installation and integration between these 3 components as described in part 1.

Once you have completed all the installation and integration, this server will tremendously help your analysis and automate a lot of things, you can track your progress or investigation, manage case, alert, task, and IOCs found. With a click of a button, you can receive reports and threat intel from multiple sources.

This can be your great tool as a blue team.

Housekeeping

Installing these service will create a new group and user to manage the service. I would like to advise you need to install the service first so that the group and user are available to manage and configure the respective service.

Just in case your configuration is not applied even after you restart the service, just reboot your machine.

Let’s start!

Switch to an interactive session as a root user so we don’t need to put sudo in every command we run.

sudo -i

Update and upgrade

apt update -y && apt upgrade -y

Installing net-tools, curl, apt-transport-https and OpenSSH-Server

apt install net-tools curl apt-transport-https openssh-server -y

Get the IP address of your machine

ifconfig

The best is to configure your server network configuration into static IP address, dynamic address might break your app during runtime when the IP address changed.

Enable OpenSSH-Server on startup and start OpenSSH-Server

systemctl enable openssh-server.service

systemctl start openssh-server.service

Now you can access this server by using SSH. I prefer this way. Most of the time, I SSH to this server via Windows terminal.

Installing MISP

Download MISP install script.

wget https://github.com/MISP/MISP/blob/2.4/INSTALL/INSTALL.sh -O INSTALL.sh

Give the script appropriate permission

chmod 770 INSTALL.sh

If you only run the script without any parameters, it will show the installation options.

./INSTALL.sh

We will install everything, so we will use option -A. we need to save the output of the installation because when MISP installation is complete, there is a set of important credentials that we need.

./INSTALL.sh -A | tee misp-install-output.txt

Just wait until the installation is complete.

By default, you can go to MISP web page by navigate to https://<misp.ip.server> on your browser.

Default credential for user is admin@admin.test

Default password is admin

As I remember. Hehe. Then, just follow any instruction to initialize the service.

Create MISP organization, User, Authentication Key and add Threat Feed.

For every new instance of MISP, you can first login using default credential. Once you login, follow this step to create or to administer respective item. 

Organization

We will create new organization, Go to Administration > Add organization.

Here filled in all the details of our new organization.

For UUID, just click on Generate UUID. When everything has been added, click “Submit”.

By the way, I already created this organization for my MISP. This is just step-by-step guide you can follow to create new MISP organization.

Organization User

Navigate to Administration > Add User.

Here filled in all the details of our user. Assigned the user to our newly created organization.

This user will have the admin right and I will use this account to create authorization key a.k.a API key to retrieve event of threat intel from MISP.

When everything is good. Create the user by clicking the Create user button.

Authentication Key

Navigate to Administration > List Auth Keys

Here is some of the list Authentication key that has been created. One of the usages of Authentication Key is it can be used by third party services to grab feed data from MISP.

So let me show how to create new Authentication Key for Cortex.

Click on Add authentication key.

I’ve set the User to inherit the Auth key to admin@soc.local and only this server IP can use this Auth Key. To make sure nothing can be done for managerial purposes for this Auth Key, I’ve set this key as Read-only.

After everything is good, click Submit.

After that, MISP will prompt your Authentication Key. Save and store the key, it will only show once in plain text. We will use this key for Cortex Analyzer.

 

Adding Feed

MISP has provided more than 50 OSINT feeds you can use. Most of the feed is not added when you install MISP instance. Let’s add and enable some of the feeds.

First, Login using admin@soc.local account.

Navigate to Sync Actions > List Feeds

By defaults, you will only see 2 feeds available. Now, navigate to Import Feeds from JSON

Now, we need to copy list of Threat Intel feeds in JSON format and paste it here.

Copy the JSON from defaults.json available from MISP Github.

Once you have pasted the JSON in the text field, click Add.

Then, it will redirect you back to List Feeds. You can enable, cache, and store available feed data, any feed you like.

Installing Cortex

Download TheHive-Project PGP

curl https://raw.githubusercontent.com/TheHive-Project/TheHive/master/PGP-PUBLIC-KEY | sudo apt-key add -

Download and add TheHive-Project repository

curl https://raw.githubusercontent.com/TheHive-Project/TheHive/master/PGP-PUBLIC-KEY | sudo apt-key add -

Cortex require Elasticsearch.

Download and install Elasticsearch PGP Key

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

Saving Elastic Repository to /etc/apt/sources.list.d/elastic-7.x.list

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/7.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-7.x.list

Update repository, download and install Elasticsearch

apt update -y && apt install elasticsearch -y

Once completed, uncomment, edit and add the following in Elasticsearch configuration file /etc/elasticsearch/elasticsearch.yml

Configure initial elasticsearch.yml settings file.

nano /etc/elasticsearch/elasticsearch.yml

# ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: hive # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: node-1 # # ---------------------------------- Network ----------------------------------- # # By default Elasticsearch is only accessible on localhost. Set a different # address here to expose this node on the network: # network.host: 192.168.1.103 # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.seed_hosts: ["host1", "host2"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # cluster.initial_master_nodes: ["node-1"] # # ---------------------------------- Security ---------------------------------- # thread_pool.search.queue_size: 100000

Close and save the file.

To enable Elasticsearch on startup and start Elasticsearch

systemctl daemon-reload

systemctl enable elasticsearch.service

systemctl start elasticsearch.service

Elasticsearch may take a while to start. Once the sessions are back, curl your IP address:9200 to check if Elasticsearch is running.

The output should be something like this:

Now, install Cortex.

apt install cortex -y

Before starting the server, to make sure that the cortex account can create log files in /opt/cortex/logs. We need to run this command

sudo addgroup cortex

sudo adduser --system cortex

sudo cp /opt/cortex/package/cortex.service /usr/lib/systemd/system

sudo chown -R cortex:cortex /opt/cortex

sudo chgrp cortex /etc/cortex/application.conf

sudo chmod 640 /etc/cortex/application.conf

Once completed, we need to add server secret key. To explain this better, this is what they say in the documentation.

“The only required parameter in order to start Cortex is the key of the server (play.http.secret.key). This key is used to authenticate cookies that contain data. If Cortex runs in cluster mode, all instances must share the same key.”

Run this command

sudo mkdir /etc/cortex (cat << _EOF_ # Secret key # ~~~~~ # The secret key is used to secure cryptographics functions. # If you deploy your application to several instances be sure to use the same key! play.http.secret.key="$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1)" _EOF_ ) | sudo tee -a /etc/cortex/application.conf

After that done, you can tail /etc/cortex/application.conf to check if the key has been successfully appended.

tail -n 10 /etc/cortex/application.conf

Check if configuration for Elasticsearch is correct in /etc/cortex/application.conf. It should be like this.

## ElasticSearch search {   # Name of the index   index = cortex   # ElasticSearch instance address.   # For cluster, join address:port with ',': "http://ip1:9200,ip2:9200,ip3:9200"   # example – uri = “http://<your-machine-ip-address>:9200”  uri = "http://192.168.1.103:9200"

If everything is good, you can enable Cortex to start during start up, start Cortex and restart Elasticsearch

systemctl enable cortex.service

systemctl start cortex.service

systemctl restart elasticsearch.service

Installing Cortex Analyzer

To install Cortex Analyzer, you may need to have python-pip2.

You can follow the step from the tutorial here: Installing pip for Python2

Download all dependency packages

apt install -y --no-install-recommends python-pip python2.7-dev python3-pip python3-dev ssdeep libfuzzy-dev libfuzzy2 libimage-exiftool-perl libmagic1 build-essential git libssl-dev

Install Python setuptools and update pip/pip3

pip install -U pip setuptools && sudo pip3 install -U pip setuptools

Clone the Cortex Analyzer repository

git clone https://github.com/TheHive-Project/Cortex-Analyzers

Every Analyzer have its own pip compatible requirements.txt file. Install the requirements with the following commands

for I in $(find Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip2 install -r $I; done && \ for I in $(find Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip3 install -r $I || true; done

Once complete, check the Analyzer configuration in /etc/cortex/application.conf file. We need to set where Cortex can find the Analyzer file. My Analyzer file is in /opt/cortex/ Cortex-Analyzers/analyzers/

Edit the the /etc/cortex/application.conf file

nano /etc/cortex/application.conf

## ANALYZERS # analyzer {   # analyzer location   # url can be point to:   # - directory where analyzers are installed   # - json file containing the list of analyzer descriptions   urls = [     "/opt/cortex/Cortex-Analyzers/analyzers/"   ]

Once completed, restart Cortex service

systemctl restart cortex.service

We’re done with installing Cortex and its Analyzer, Cortex-Analyzer can be configured via Web UI.

Once it is started, you may browse from your browser and connect to http://YOUR_SERVER_ADDRESS:9001/

On first access, you need to update Cortex database. Just click “Update Database”, then you need to create a SuperAdmin account. Once complete you will be redirected to login page.

Login with the account you just created.

If you do not find any Cortex or Analyzers configuration as shown, restart your machine.

Cortex Troubleshoot

If the login page is blank and there is warning regarding Elasticsearch. Check the status of Elasticsearch. If the service is down, restart Elasticsearch. Once Elasticsearch back online, just refresh the page. You may use command below to check status and restart Elasticsearch.

Check Elasticsearch status

systemctl status elasticsearch.service

Restart Elasticsearch

systemctl restart elasticsearch.service

Installing TheHive

TheHive require few components to be installed first before we can actually install TheHive. We need to have Java Virtual Machine (JVM) and Cassandra as database. To automate this step, just use this bash script or you can execute the command line by line, your choice.

nano thehive-component.sh

Paste this script.

#! /bin/bash prinf "\n\ninstalling java and cassandra\n\n" apt install -y openjdk-8-jre-headless -y echo JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64" >> /etc/environment export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64" curl -fsSL https://www.apache.org/dist/cassandra/KEYS | apt-key add - echo "deb http://www.apache.org/dist/cassandra/debian 311x main" | tee -a /etc/apt/sources.list.d/cassandra.sources.list apt update - y apt install cassandra -y

Ctrl + X to close and save the buffer.

Give execute permission to the script

chmod +x thehive-component.sh

Execute the script. Once the installation complete, continue to next step.

./ thehive-component.sh

Start Cassandra so we can edit some configuration.

cqlsh localhost 9042

Once the session start, change the cluster_name to thp.

UPDATE system.local SET cluster_name = 'thp' where key='local';

To exit, the command is “exit”.

Next, run this command.

nodetool flush

Now, we need to edit Cassandra configuration file, /etc/cassandra/cassandra.yaml

nano /etc/cassandra/cassandra.yaml

Change according to these settings.

# content from /etc/cassandra/cassandra.yaml   cluster_name: 'thp' listen_address: 'xx.xx.xx.xx' # change to machine IP address rpc_address: 'xx.xx.xx.xx' # change to machine IP address for clients seed_provider:     - class_name: org.apache.cassandra.locator.SimpleSeedProvider       parameters:           - seeds: 'xx.xx.xx.xx' # change to machine IP address / self for the first node data_file_directories:   - '/var/lib/cassandra/data' commitlog_directory: '/var/lib/cassandra/commitlog' saved_caches_directory: '/var/lib/cassandra/saved_caches' hints_directory:   - '/var/lib/cassandra/hints'

Ctrl + X to close the file and save the buffer.

Enable Cassandra to start during start up and restart the service.

systemctl enable cassandra.service

systemctl restart cassandra.service

Next, we can install the hive!

apt install thehive4

 After that installation complete, we need to create a dedicated folder to store file in the local storage and set thehive user as owner.

mkdir -p /opt/thp/thehive/files

chown -R thehive:thehive /opt/thp/thehive/files

 Create a folder dedicated to host indexes for TheHive

mkdir /opt/thp/thehive/index

chown thehive:thehive -R /opt/thp/thehive/index

 Now, we need to edit TheHive configuration file which at /etc/thehive/application.conf

nano /etc/thehive/application.conf

Match and edit the configuration to your machine.

## Database configuration db.janusgraph {   storage {     ## Cassandra configuration     # More information at https://docs.janusgraph.org/basics/configuration-reference/#storagecql     backend: cql     #change to your machine IP address     hostname: ["192.168.1.103"]     # Cassandra authentication (if configured)     # username: "thehive"     # password: "password"     cql {       cluster-name: thp       keyspace: thehive     }   }   index.search {     backend: lucene     directory: /opt/thp/thehive/index     # If TheHive is in cluster ElasticSearch must be used:   }   ## For test only !   # Comment the two lines below before enable Cassandra database   storage.backend: berkeleyje   storage.directory: /opt/thp/thehive/database   // berkeleyje.freeDisk: 200 # disk usage threshold } ## Attachment storage configuration storage {   ## Local filesystem   provider: localfs   localfs.location: /opt/thp/thehive/files

Ctrl + X to close the file and save the buffer.

Now, we can start TheHive. Enable TheHive to start during start up and start the service.

systemctl enable thehive.service

systemctl start thehive.service

I find sometimes TheHive does take some time to start. Once the service started, you may browse from your browser and connect to http://YOUR_SERVER_ADDRESS:9000/.

The default admin user is admin@thehive.local with password secret. It is recommended to change the default password.

Integrating Cortex and TheHive

First, In Cortex, we need to create new user and generate api for integration. To do this, just follow this step.

Login to your Cortex. Let’s create new organization name SOC-Org. As you can see, I already create the organization.

Then, select the organization, and create user. This is step to create user. Click + Add user.

Create the following user and roles.

So, there are 3 user we just created which are self explainatory. As orgadmin, you have full access in your organization, you can add user and setup analyzer for your organization.

Follow the same step to create organization and user in TheHive. Just create the same organization.

Alright, done for Organization and User creation.

In Cortex, Once the user has been created, click botton “Create API Key” for api-user@soc.local. Once created, click Reveal, copy and store this value. We need this to integrate TheHive with Cortex.

Now, edit /etc/thehive/application.conf to add the API key.

nano /etc/thehive/application.conf

Uncomment, edit and add as the following format:

## CORTEX configuration # More information at https://github.com/TheHive-Project/TheHiveDocs/TheHive4/Administration/Connectors.md # Enable Cortex connector  play.modules.enabled += org.thp.thehive.connector.cortex.CortexModule  cortex {   servers: [     {       name: "CORTEX"                # Cortex name       url: "http://192.xxx.xxx.xxx:9001" # URL of Cortex instance, change to http://<your-cortex-machine>:9001       auth {         type: "bearer"         key: "paste-api-key-here"                 # change to Cortex API key you have copied       }       wsConfig {}                  # HTTP client configuration (SSL and proxy)     }   ]  }

Ctrl + X to close and save the buffer.

Restart TheHive

systemctl restart thehive.service

When everything is done, login to thehive and go to About. It should be indicator that show the integration with the hive is successful something like CORTEX – CORTEX – X.X.X.X (OK)

So now the TheHive – Cortex integration is completed, we should be able to receive report from Cortex Analyzer that we enable.

Install Analyzer Template

Let’s install analyzer template in TheHive, you neet to login using admin@thehive.local to do this.

After login, select Admin > Analyzer Template. This will redirect us to Analyzer template management.

Next, download the official templates archive. There is hyperlink, click on “from here” hyperlink then it will download the template in zip file.

Once the download is complete, do not unzip the archive.

Import the templates by clicking on Import templates button, select the zip you have just downloaded.

After selecting the archive, click “Yes, Import template archive”. That’s all.

 

Integrating MISP and VirusTotal Get Report API for Cortex Analyzer.

In Cortex, login using admin account from new organization you have created. In this case, we created admin@soc.local.

We will enable 2 Analyzer, Virustotal Get Report and MISP.

Then, navigate to Organization > Analyzers

VirusTotal Analyzer

First get your VirusTotal API key, this requires you to have VirusTotal account. Login or create your VirusTotal account and copy your API Key.

Here is guide provided by VirusTotal to get you API key: VirusTotal API Key

We will enable VirusTotal_GetReport_3.0, click enable and paste your API Key in key text field. You may review and enable additional settings which ever you feel suitable.

Field with asterisk is required field. Others are optional. Paste your VirusTotal API Key in Key field. 

After you have review everything, click on “Save”

MISP Analyzer

Search for MISP analyzer, keyword is MISP. Once you have found it, to enable the analyzer, click Enable.

Add the details correspond to your MISP Server IP address.

For URL field, add https and backslash in the address. For example: https://192.168.1.103/

Paste the Authorization Key you have generated and copied from MISP in Key field.

Select False for cert_check to make things lot easier for us. If you have dedicated CA used for your MISP server. You may specific it in cert_path.

Once everything is done, click Save.

 

Summary

When everything is configured, you should be able to see in TheHive that you can analyze observable using Cortex Analyzer that we have configured as shown in the picture below.

I think that’s all for MISP Analyzer. With that, we have completed all configurations for this MISP, Cortex, and TheHive server.

Thank you and see you again in the next part!