swirl
Home Software Blog Wallpapers Webtools
A case for Redis
Sunday 03, May 2020   |   Post link

As an architect, I have a very interesting job of having to understanding challenges in existing systems in multiple dimensions and still add new functionality. Recently I was approached with designing a system to store API usage for billing. It obviously came with a fine print, "it should not have any impact on the existing system". Now we all know there is no "free lunch" but how far really can we go with this?

To start with the simplest design, let's just introduce a database, say MySQL with a table which stores the Customer-Id, Api-Id & the Call-Count. The database is initialized with a set of rows covering all combinations of Api-Ids and Customer-Ids. Doing this allows us to ignore the existence/non-existence of a particular row.

Design 1

The flow of events is self-explanatory, a client makes an API call, the call is records by firing the SQL 'update apitracker set CallCount = CallCount + 1 where CustId = ? and ApiId = ?' after which we proceed to execute the existing business logic.

How did this peform?

Updates 2 Threads (ms) 4 Threads (ms) 8 Threads (ms) Avg Rate per sec
100 505 406 486 215
1000 5094 4549 5232 202
10000 55306 56644 54070 181

It looks like we have an average rate of updating 200 records per sec. In other words we are adding 0.2ms to every API call. Let's say we have 50 million API cals a day, we'll now be slower by 10,000 seconds or 3 hours.

Can we do better?

I decided to use Redis. The INCR functionlity of Redis which increments a key's value fits the requirement perfectly. Since Redis is not just an in-memory cache and can also persist it's state to disk, the information is also durable which is important in our scenario.

Design 2

Now, the requirement states that this data must be stored indefinitely, using a database is still a good option except that the database is going to be used in a slightly different manner. For the day's API recording, it's Redis that will do the job. We'll copy the data from Redis to MySql during off-peak hours every day.

How did this peform?

Updates 2 Threads (ms) 4 Threads (ms) 8 Threads (ms) Avg Rate per sec
100 16 5 15 8333
1000 47 47 38 22727
10000 326 267 255 35377

In this case, it looks like we have an average rate of updating 20,000 records per sec, actually 22,146 to be exact but who cares?. It's a staggering improvement of a 100 times. By using Redis, we should bring down the additional 3 hours to 2 minutes.

Note: Redis was configured with AOF with appendfsync every second as well as the default save options of save 900 1, save 300 10 and save 60 10000. Often moving to slower but more flexible system during off-peak hours really boosts performance of systems if the data we're dealing with is not needed immediately. This was my scenario and it could therefore be leveraged to maintain system performance.



Categories: Architecture (1)

Comments

Posts By Year

2023 (5)
2022 (10)
2021 (5)
2020 (12)
2019 (6)
2018 (8)
2017 (11)
2016 (6)
2015 (17)
2014 (2)
2013 (4)
2012 (2)

Posts By Category

.NET (4)
.NET Core (2)
ASP.NET MVC (4)
AWS (5)
AWS API Gateway (1)
Android (1)
Apache Camel (1)
Architecture (1)
Audio (1)
Azure (2)
Book review (3)
Business (1)
C# (3)
C++ (2)
CloudHSM (1)
Containers (4)
Corporate culture (1)
Database (3)
Database migration (1)
Desktop (1)
Docker (1)
DotNet (3)
DotNet Core (2)
ElasticSearch (1)
Entity Framework (3)
Git (3)
IIS (1)
JDBC (1)
Java (9)
Kibana (1)
Kubernetes (1)
Lambda (1)
Learning (1)
Life (7)
Linux (1)
Lucene (1)
Multi-threading (1)
Music (1)
OData (1)
Office (1)
PHP (1)
Photography (1)
PowerShell (2)
Programming (28)
Rants (5)
SQL (2)
SQL Server (1)
Security (2)
Software Engineering (1)
Software development (2)
Solr (1)
Sql Server (2)
Storage (1)
T-SQL (1)
TDD (1)
TSQL (5)
Tablet (1)
Technology (1)
Test Driven (1)
Unit Testing (1)
Unit Tests (1)
Utilities (3)
VC++ (1)
VMWare (1)
VSCode (1)
Visual Studio (2)
Wallpapers (1)
Web API (2)
Win32 (1)
Windows (9)
XML (2)

Posts By Tags

.NET(6) API Gateway(1) ASP.NET(4) AWS(3) Adults(1) Advertising(1) Android(1) Anti-forgery(1) Asynch(1) Authentication(2) Azure(2) Backup(1) Beliefs(1) BlockingQueue(1) Book review(2) Books(1) Busy(1) C#(4) C++(3) CLR(1) CORS(1) CSRF(1) CTE(1) Callbacks(1) Camel(1) Certificates(1) Checkbox(1) CloudHSM(1) Cmdlet(1) Company culture(1) Complexity(1) Consumer(1) Consumerism(1) Containers(3) Core(2) Custom(2) DPI(1) Data-time(1) Database(4) Debugging(1) Delegates(1) Developer(2) Dockers(2) DotNetCore(3) EF 1.0(1) Earphones(1) Elastic Search(1) ElasticSearch(1) Encrypted(1) Entity framework(1) Events(1) File copy(1) File history(1) Font(1) Git(2) HierarchyID(1) IIS(1) Installing(1) Intelli J(1) JDBC(1) JSON(1) JUnit(1) JWT(1) Java(3) JavaScript(1) Kubernetes(1) Life(1) LinkedIn(1) Linux(2) Localization(1) Log4J(1) Log4J2(1) Lucene(1) MVC(4) Management(2) Migration history(1) Mirror(1) Mobile Apps(1) Modern Life(1) Money(1) Music(1) NGINX(1) NTFS(1) NUnit(2) OData(1) OPENXML(1) Objects(1) Office(1) OpenCover(1) Organization(1) PHP(1) Paths(1) PowerShell(2) Producer(1) Programming(2) Python(2) QAAC(1) Quality(1) REDIS(2) REST(1) Runtimes(1) S3-Select(1) SD card(1) SLF4J(1) SQL(2) SQL Code-first Migration(1) SSH(2) Sattelite assemblies(1) School(1) Secrets Manager(1) Self reliance(1) Service(1) Shell(1) Solr(1) Sony VAIO(1) Spirituality(1) Spring(1) Sql Express(1) System Image(1) TDD(1) TSQL(3) Table variables(1) Tables(1) Tablet(1) Ubuntu(1) Url rewrite(1) VMWare(1) VSCode(1) Validation(2) VeraCode(1) Wallpaper(1) Wallpapers(1) Web Development(4) Windows(2) Windows 10(2) Windows 2016(2) Windows 8.1(1) Work culture(1) XML(1) Yii(1) iTunes(1) renew(1) security(1)