swirl
Home Software Blog Wallpapers Webtools
Creating a two node Elastic Search cluster on Windows
Thursday 04, April 2024   |   Post link
Logo

Overview

In this blog post we will be setting up a 2 node Elastic Search cluster on a Windows 10 or 11 machine using HyperV. We will be running two Ubuntu servers, installing ElasticSearch on both and creating a cluster.

Pre-requisites

  • Enable virtualization in your BIOS.
  • Enable Hyper-V on your Windows 10 Prof or Windows 11 Prof PC.
  • Creating a network switch to allow static IPs for Hyper-V virtual machines.

Enable virtualization in the BIOS

The way this is done can different from one machine to another. Basically, look for any BIOS settings named 'Virtualization', 'Vt-d' or similar names and enable it. Unless enabled you will not be able to create virtual machines. Even WSL 2.0 (Windows Subsystem for Linux) requires virtualization to be turned on at the BIOS level.

Enable Hyper-V

Hyper-V is only available on professional edition of Windows 10 and Windows 11 operating system. To turn on Hyper-V type in 'Windows Features', you should get the 'Turn Windows features on or off' item, click it and select all Hyper-V features:

Enable Hyper-V screenshot

You may have to estart your PC after clicking OK. The next step is to add your user to the 'Hyper-V Administrators' group.

Enable Hyper-V screenshot

Search for 'Edit Local users and group' in the search app, select the 'Groups' item of the left and then double-click the 'Hyper-V Administrators', in the following dialog-box click the 'Add' button and enter the name of the current (your) user, click OK and then OK on 'Hyper-V Administrators Properties' dialog-box. You are now set to create the virtual machines.

Creating the network switch

The default network available in Hyper-V assigns a dynamic IP address to each virtual machine. This IP changes each time the virtual machine starts. For our purpose we would like to keep the IP address fixed. To use static IPs we create a network swtich which is then used by the virtual machines.

First, we need to make a list of the DNS servers on the PC, run the following command

ipconfig /all 

Look for entries against 'DNS Servers', note them down somewhere, we will need these later.

Open a Administrative Powershell Window.
Execute:

New-VMSwitch -SwitchName "StaticSwitch" -SwitchType Internal

Next run:

Get-NetAdapter

You'll see a list of network adapters present on the machine. You should see the new switch created in the previous step, now down the number mentioned in the ifIndex column:

Switches screenshot

Next run:

New-NetIPAddress -IPAddress 192.168.98.1 -PrefixLength 24 -InterfaceIndex <IFINDEX>
New-NetNat -Name MyNATnetwork -InternalIPInterfaceAddressPrefix 192.168.98.0/24

Replace <IFINDEX> with the ifIndex noted in the previous step. 192.168.98.0 will be the gateway IP address which will be used by the virtual machines.

Preparing the virtual machines

Download a copy of the Ubuntu Linux ISO, as of this time, it is available at https://ubuntu.com/download/server. You could even opt to use Ubuntu Desktop, in this post, Ubuntu Server is used. Note: Ubuntu Server does not come with a desktop GUI, you are limited to using the command line (Terminal) only.

Create the VM

Launch the Hyper-V Manager application. Click the 'New -> Virtual Machine' item from the Actions pane.

Hyper-V Manager

Enter the relevant details at each step as shown below. Ensure you select the 'Default Switch' and not the one created manually earlier. Ensure you select the correct amount of RAM and select the Ubuntu ISO file in the last step. Once all wizard steps are completed the virtual machine is ready to be started.

Create VM Step 1

Create VM Step 2

Create VM Step 3

Create VM Step 4

Create VM Step 5

Create VM Step 6

Double click the VM from the list in the Hyper-V Manager app, click the Start button. Go through the screens displayed by the Ubuntu installer, ensure you opt to installed 'Open SSH Server'. Once Ubuntu installed we need to configure the network to use static IPs instead of dynamic IPs.

Once VM installed, it is assigned a dynamic IP by Hyper-V. Before we change to using static IP, we need to execute the following commands in the Linux terminal:

sudo apt-get update
sudo apg-get upgrade
sudo apt-get install openvswitch-switch-dpdk
sudo apt-get install network-manager

We are not ready to configure the static IP. The configuration is specified in a file named as '00-XX--' present in the /etc/netplan/ directory.

Make a copy of this file in your home directory in case you want to revert any change. Next, Use your favorite editor to edit this file (as root). The new content should look like:

# This is the network config written by 'subiquity'
network:
  ethernets:
    eth0:
      dhcp4: false
      addresses: [192.168.98.10/16]
      routes:
      - to: 0.0.0.0/0
        via: 192.168.98.1
      nameservers:
        addresses: [8.8.8.8,1.1.1.1]
  version: 2

We are setting 192.168.98.10 as the static IP of this VM. The 'via: 192.168.98.1' specifies the gateway which is the switch we created earlier, the addresses under nameservers is the list of DNS servers we noted earlier. Save the file.

Select the new switch

Head back to the Hyper-V Manager application, select the runnig VM window, select File -> Settings, locate the 'Network Adapter' item on the left and select the 'StaticSwitch'. Click OK.

Now back to the Linux terminal, run:

sudo netplan apply

Your VM should now be configured to use the static IP. You can try by running:

ping google.com
ip address

You should see 192.168.98.10 as your IP address.

Repeat the same steps to create another VM with a different IP address (192.168.98.10).

Confiugreing Elastic Search

We will be configuring the following software:

  • The first Elastic Search node on the first Linux VM.
  • Kibana on Windows.
  • The second Elastic Search node on the second Linux VM.

Installing the first Elastic node

You can now SSH into the Ubuntu from Windows VM by running:

ssh yourusername@192.168.98.10

or you can use a software like MobaXTerm for doing SSH. Once in Linux exeucte the following commands:

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.13.1-linux-x86_64.tar.gz
tar -xvzf elasticsearch-8.13.1-linux-x86_64.tar.gz
sudo mkdir /usr/share/elasticsearch
sudo mv elasticsearch-8.13.1 /usr/share/elasticsearch/
/usr/share/elasticsearch/bin/elasticsearch

ElasticSearch should start running. At the end of the start up process, you'll see the autogenerated password for the 'elastic' admin user, information on how to connect Kibana and how to join new nodes to the cluster. Copy this output and store it for later use.

Installing Kibana

We are dedicating both Linux VMs for running Elastic Search, so, we'll install Kibana on the host Windows machine itself. Kibana was downloaded from here https://artifacts.elastic.co/downloads/kibana/kibana-8.12.2-windows-x86_64.zip for this blog post.

Installing Kibana is simple, unzip the files to a directory, run the kibana.bat in the bin folder. Kibana can take a couple of minutes to load, once ready the Kibana console output will show a message like "Kibana is now available". Kibana can be accessed at http://localhost:5601/ using a web browser.

The first thing we need to do is connect Kibana to our instance of Elastic Search. Copy the Kibana enrollment token provided by Elastic Search when it first started, note, this value is valid for a short time. Copy and paste this value in the textbox presented by Kibana and you should be set to go.

Installing the second Elastic Search node

We need to configure a couple of settings in the first Elastic Search node before installing second Elastic Search node. Stop the running instance of Elastic Search, edit the file /usr/share/elasticsearch/elasticsearch-8.13.1/config/elasticsearch.yml.

Uncomment these lines and specify the proper value:

cluster.name: MyElasticCluster
node.name: node1
network.host: 192.168.0.1
http.port: 9200
transport.host: 0.0.0.0

The cluster name is especially important to allow the 2nd Elastic Search node to join the cluster.

Restart Elastic Search. Run the following command:

cd /usr/share/elasticsearch/elasticsearch-8.13.1
bin/elasticsearch-create-enrollment-token -s node

This command will generate a token which will allow the second Elastic Search node to join the cluster.

On the second Ubuntu VM:

Install Elastic Search on the second Ubuntu VM. Just unzip the ElasticSearch tar.gz, do not run the elasticsearch shell script.

Edit the file /usr/share/elasticsearch/elasticsearch-8.13.1/config/elasticsearch.yml. Set the values for the following:

cluster.name: MyElasticCluster
node.name: node2
http.port: 9200
transport.host: 0.0.0.0

Save the file.

Next, run the following command:

cd /usr/share/elasticsearch/elasticsearch-8.13.1/bin
bin/elasticsearch --enrollment-token <token>

Replace <token> with the enrollment token created by running the 'elasticsearch-create-enrollment-token' on the first node. The second node should now become part of the cluster. You should be able to check this by running the following query in Kibana:

GET /_nodes

Which will output a result like:

Nodes

Hope you find this post useful!



Categories: Software (1)

Comments

Posts By Year

2024 (4)
2023 (5)
2022 (10)
2021 (5)
2020 (12)
2019 (6)
2018 (8)
2017 (11)
2016 (6)
2015 (17)
2014 (2)
2013 (4)
2012 (2)

Posts By Category

.NET (4)
.NET Core (2)
ASP.NET MVC (4)
AWS (5)
AWS API Gateway (1)
Android (1)
Apache Camel (1)
Architecture (1)
Audio (1)
Azure (2)
Book review (3)
Business (1)
C# (3)
C++ (2)
CloudHSM (1)
Containers (4)
Corporate culture (1)
Database (3)
Database migration (1)
Desktop (1)
Docker (1)
DotNet (3)
DotNet Core (2)
ElasticSearch (1)
Entity Framework (3)
Git (3)
IIS (1)
JDBC (1)
Java (10)
Kibana (1)
Kubernetes (1)
Lambda (1)
Learning (1)
Life (7)
Linux (2)
Lucene (1)
Multi-threading (1)
Music (1)
OData (1)
Office (1)
PHP (1)
Photography (1)
PowerShell (2)
Programming (28)
Python (1)
Rants (5)
SQL (2)
SQL Server (1)
Security (3)
Software (1)
Software Engineering (1)
Software development (2)
Solr (1)
Sql Server (2)
Storage (1)
T-SQL (1)
TDD (1)
TSQL (5)
Tablet (1)
Technology (1)
Test Driven (1)
Testing (1)
Tomcat (1)
Unit Testing (1)
Unit Tests (1)
Utilities (3)
VC++ (1)
VMWare (1)
VSCode (1)
Visual Studio (2)
Wallpapers (1)
Web API (2)
Win32 (1)
Windows (9)
XML (2)

Posts By Tags

.NET(6) API Gateway(1) ASP.NET(4) AWS(3) Adults(1) Advertising(1) Android(1) Anti-forgery(1) Asynch(1) Authentication(2) Azure(2) Backup(1) Beliefs(1) BlockingQueue(1) Book review(2) Books(1) Busy(1) C#(4) C++(3) CLR(1) CORS(1) CSRF(1) CTE(1) Callbacks(1) Camel(1) Certificates(1) Checkbox(1) Client authentication(1) CloudHSM(1) Cmdlet(1) Company culture(1) Complexity(1) Consumer(1) Consumerism(1) Containers(3) Core(2) Custom(2) DPI(1) Data-time(1) Database(4) Debugging(1) Delegates(1) Developer(2) Dockers(2) DotNetCore(3) EF 1.0(1) Earphones(1) Elastic Search(2) ElasticSearch(1) Encrypted(1) Entity framework(1) Events(1) File copy(1) File history(1) Font(1) Git(2) HierarchyID(1) Hyper-V(1) IIS(1) Installing(1) Intelli J(1) JDBC(1) JSON(1) JUnit(1) JWT(1) Java(3) JavaScript(1) Kubernetes(1) Life(1) LinkedIn(1) Linux(2) Localization(1) Log4J(1) Log4J2(1) Logging(1) Lucene(1) MVC(4) Management(2) Migration history(1) Mirror(1) Mobile Apps(1) Modern Life(1) Money(1) Music(1) NGINX(1) NTFS(1) NUnit(2) OData(1) OPENXML(1) Objects(1) Office(1) OpenCover(1) Organization(1) PHP(1) Paths(1) PowerShell(2) Processes(1) Producer(1) Programming(2) Python(2) QAAC(1) Quality(1) REDIS(2) REST(1) Runtimes(1) S3-Select(1) SD card(1) SLF4J(1) SQL(2) SQL Code-first Migration(1) SSH(2) SSL(1) Sattelite assemblies(1) School(1) Secrets Manager(1) Self reliance(1) Service(1) Shell(1) Solr(1) Sony VAIO(1) Spirituality(1) Spring(1) Sql Express(1) System Image(1) TDD(1) TSQL(3) Table variables(1) Tables(1) Tablet(1) Ubuntu(1) Url rewrite(1) VMWare(1) VSCode(1) Validation(2) VeraCode(1) Wallpaper(1) Wallpapers(1) Web Development(4) Windows(2) Windows 10(2) Windows 2016(2) Windows 8.1(1) Work culture(1) XML(1) Yii(1) iTunes(1) open file handles(1) renew(1) security(1) static ip address(1) ulimit(1)