swirl
Home Software Blog Wallpapers Webtools
Using Apache Camel with Spring to insert data into Elastic Search
Saturday 09, April 2022   |   Post link
Flow

Apache Camel is a neat framework if your application requires customizable workflows, components and a wide array of input and output adapters. This post demonstrates how to use Camel with Spring and XML to define routes.

Route

Flow

The dependencies

The following dependencies are needed for using Camel with Spring dependency injection along with with the ability to use XML files and Elastic search.

  • camel-core
  • camel-spring
  • camel-stream
  • camel-spring-xml
  • camel-elasticsearch-rest
  • spring-context
Here is the dependency section of the pom.

<dependencies>
	<dependency>
		<groupId>org.apache.camel</groupId>
		<artifactId>camel-core</artifactId>
		<version>${camel.version}</version>
	</dependency>
	<dependency>
		<groupId>org.apache.camel</groupId>
		<artifactId>camel-spring</artifactId>
		<version>${camel.version}</version>
	</dependency>
	<dependency>
		<groupId>org.apache.camel</groupId>
		<artifactId>camel-stream</artifactId>
		<version>${camel.version}</version>
	</dependency>
	<dependency>
		<groupId>org.apache.camel</groupId>
		<artifactId>camel-spring-xml</artifactId>
		<version>${camel.version}</version>
	</dependency>
	<dependency>
		<groupId>org.apache.camel</groupId>
		<artifactId>camel-elasticsearch-rest</artifactId>
		<version>${camel.version}</version>
	</dependency>
	<dependency>
		<groupId>org.springframework</groupId>
		<artifactId>spring-context</artifactId>
		<version>${spring.version}</version>
	</dependency>
	<dependency>
		<groupId>com.google.code.gson</groupId>
		<artifactId>gson</artifactId>
		<version>${gson.version}</version>
	</dependency>
</dependencies>

The route

The entire flow shown in the diagram is defined in the routes.xml file.

<?xml version = "1.0" encoding = "UTF-8"?>
<beans xmlns = "http://www.springframework.org/schema/beans"
		xmlns:xsi = "http://www.w3.org/2001/XMLSchema-instance"
		xsi:schemaLocation = "
		http://www.springframework.org/schema/beans
		http://www.springframework.org/schema/beans/spring-beans.xsd
		http://camel.apache.org/schema/spring
		http://camel.apache.org/schema/spring/camel-spring.xsd">
	<bean name="lowerCaser" class="com.sbytestream.processors.LowerCaser"></bean>
	<bean name="emailParser" class="com.sbytestream.processors.EmailParser"></bean>
	<bean name="jsonConvertor" class="com.sbytestream.processors.JsonConvertor"></bean>
	<bean name="spamIdentifier" class="com.sbytestream.processors.SpamIdentifier">
		<property name="blockedSenders">
			<list>
				<value>manish@email.com</value>
				<value>rajesh@email.com</value>
			</list>
		</property>
	</bean>
	<bean name="elasticsearch-rest" class="org.apache.camel.component.elasticsearch.ElasticsearchComponent">
		<property name="hostAddresses" value="localhost:9200"></property>
		<property name="user" value="elastic"></property>
		<property name="password" value="builder!12"></property>
	</bean>

	<camelContext xmlns = "http://camel.apache.org/schema/spring" id="camel">
		<route>
			<from uri="file://data/input" />
			<process ref="emailParser" />
			<process ref="spamIdentifier" />
			<choice>
				<when>
					<simple>${exchangeProperty.spam} == 'true'</simple>
					<to uri="file://data/spam" />
				</when>
				<otherwise>
					<process ref="jsonConvertor" />
					<to uri="file://data/output" />
					<to uri="elasticsearch-rest://elasticsearch?operation=INDEX&amp;indexName=mails" />
				</otherwise>
			</choice>
		</route>
	</camelContext>
</beans>

Reading emails

Input emails are stored in the 'data/input' folder in the current directory. The format of the email file is mocked, for the purpose of the demo, the file structure should be as shown below:

from:siddharth@email.com
to:neel@email.com
subject:hello world
content:this email is a test email for telling Hello World
Emails are read as plain files using the file:// protocol. Two sample email files 
are available in the data directory, these can be copied into the input folder for 
testing.

Processing emails

Email files are sent to the EmailParser class for converting the content to a POJO. The POJO is then sent to the SpamIdentifier. The SpamIdentifier class checks if the sender is one of the email addresses in its list of blocked senders. It adds a new property named "spam" to "true".

Output

The output of spam processing is sent to a check block where spam emails are stored in a folder named "spam" and other emails are sent to Elastic Search in an index named "mails" using the ElasticsearchComponent. You can query these items using the following request:

GET mails/_search
{
  "query": {
    "match_all": {}
  }
}

Wiring everything together

Components and wiring are defined in the routes.xml file.

Elastic search configuration

Make sure you have set the Elastic Search configuration to match your setup. This is defined in the routes.xml file as well. The sample code for this article is available for download at GitHub.




Comments

Posts By Year

2024 (4)
2023 (5)
2022 (10)
2021 (5)
2020 (12)
2019 (6)
2018 (8)
2017 (11)
2016 (6)
2015 (17)
2014 (2)
2013 (4)
2012 (2)

Posts By Category

.NET (4)
.NET Core (2)
ASP.NET MVC (4)
AWS (5)
AWS API Gateway (1)
Android (1)
Apache Camel (1)
Architecture (1)
Audio (1)
Azure (2)
Book review (3)
Business (1)
C# (3)
C++ (2)
CloudHSM (1)
Containers (4)
Corporate culture (1)
Database (3)
Database migration (1)
Desktop (1)
Docker (1)
DotNet (3)
DotNet Core (2)
ElasticSearch (1)
Entity Framework (3)
Git (3)
IIS (1)
JDBC (1)
Java (10)
Kibana (1)
Kubernetes (1)
Lambda (1)
Learning (1)
Life (7)
Linux (2)
Lucene (1)
Multi-threading (1)
Music (1)
OData (1)
Office (1)
PHP (1)
Photography (1)
PowerShell (2)
Programming (28)
Python (1)
Rants (5)
SQL (2)
SQL Server (1)
Security (3)
Software (1)
Software Engineering (1)
Software development (2)
Solr (1)
Sql Server (2)
Storage (1)
T-SQL (1)
TDD (1)
TSQL (5)
Tablet (1)
Technology (1)
Test Driven (1)
Testing (1)
Tomcat (1)
Unit Testing (1)
Unit Tests (1)
Utilities (3)
VC++ (1)
VMWare (1)
VSCode (1)
Visual Studio (2)
Wallpapers (1)
Web API (2)
Win32 (1)
Windows (9)
XML (2)

Posts By Tags

.NET(6) API Gateway(1) ASP.NET(4) AWS(3) Adults(1) Advertising(1) Android(1) Anti-forgery(1) Asynch(1) Authentication(2) Azure(2) Backup(1) Beliefs(1) BlockingQueue(1) Book review(2) Books(1) Busy(1) C#(4) C++(3) CLR(1) CORS(1) CSRF(1) CTE(1) Callbacks(1) Camel(1) Certificates(1) Checkbox(1) Client authentication(1) CloudHSM(1) Cmdlet(1) Company culture(1) Complexity(1) Consumer(1) Consumerism(1) Containers(3) Core(2) Custom(2) DPI(1) Data-time(1) Database(4) Debugging(1) Delegates(1) Developer(2) Dockers(2) DotNetCore(3) EF 1.0(1) Earphones(1) Elastic Search(2) ElasticSearch(1) Encrypted(1) Entity framework(1) Events(1) File copy(1) File history(1) Font(1) Git(2) HierarchyID(1) Hyper-V(1) IIS(1) Installing(1) Intelli J(1) JDBC(1) JSON(1) JUnit(1) JWT(1) Java(3) JavaScript(1) Kubernetes(1) Life(1) LinkedIn(1) Linux(2) Localization(1) Log4J(1) Log4J2(1) Logging(1) Lucene(1) MVC(4) Management(2) Migration history(1) Mirror(1) Mobile Apps(1) Modern Life(1) Money(1) Music(1) NGINX(1) NTFS(1) NUnit(2) OData(1) OPENXML(1) Objects(1) Office(1) OpenCover(1) Organization(1) PHP(1) Paths(1) PowerShell(2) Processes(1) Producer(1) Programming(2) Python(2) QAAC(1) Quality(1) REDIS(2) REST(1) Runtimes(1) S3-Select(1) SD card(1) SLF4J(1) SQL(2) SQL Code-first Migration(1) SSH(2) SSL(1) Sattelite assemblies(1) School(1) Secrets Manager(1) Self reliance(1) Service(1) Shell(1) Solr(1) Sony VAIO(1) Spirituality(1) Spring(1) Sql Express(1) System Image(1) TDD(1) TSQL(3) Table variables(1) Tables(1) Tablet(1) Ubuntu(1) Url rewrite(1) VMWare(1) VSCode(1) Validation(2) VeraCode(1) Wallpaper(1) Wallpapers(1) Web Development(4) Windows(2) Windows 10(2) Windows 2016(2) Windows 8.1(1) Work culture(1) XML(1) Yii(1) iTunes(1) open file handles(1) renew(1) security(1) static ip address(1) ulimit(1)