Why the need?

  • You need microseconds latency not even milliseconds now for most of the application performance requirements.
  • ElastiCache is hardened by AWS. That means Redis engine features and functionalities are constantly evaluated as per customer needs.
  • You can persist data structures to Redis (Strings, lists, hashes, sets, sorted sets, bitmaps, streams etc.)
  • Redis Use Cases: Caching, Real-time analytics, Gaming leaderboards, Geospatial, Media Streaming, Session State, Chat apps, Message queues, ML

Streams

  • Time Series Data with Redis
    • SortedSet allows you to use the Unix time as a score and sort the data based on that value. Values need to be unique.
    • List allows you to implement message processing.
    • Pub/Sub supports fanout where you can publish messages to a channel with subscribers. The problem on this one is data is not persistent. Not ideal for critical data.
  • Redis Streams
    • Producer: You can add values to the stream such as generate an ID etc.
    • Stream: Redis can iterate over the messages. Data is immutable, so it’s append only.
    • Consumer: You can then query the messages to get a specified data. Consumer Groups.

Scalability

  • Read Scalability
    • Let’s say you have EC2 and Lambda as clients and reads and writes are being sent to the Primary node. Primary node can replicate data onto Replica nodes with an asynchronous replication. You can enable failover as well. Referred as Shard as well. This will increase CPU. The more replicas, the more bandwidth this leaves primary node with more BW to accept read/writes.
  • Write Scalability
    • Data is partitioned across shards (up to 15). You can scale horizontally by adding more shards by distributing data across the shards and each shard having a subset of data. Cost-effective because it’s incremental, i.e. 1 shard at a time.
    • You can also scale vertically. You can provision a new cluster and install software etc. Once it’s ready, you can copy the data to new servers.
    • When there is a failover within Cluster mode, it’ll be detected and replica node will be promoted to primary with DNS propagation.

High Availability

  • Redis relies on replication to provide high availability. Read Replica would be promoted to Primary if Primary fails across 2-3 AZs for example.
  • Let’s say you have below architecture which includes 3 Availability Zones. ElastiCache creates the cluster with Primary and Replicas in different AZs and data gets replicated to the associated Replicas in other AZs. If you have a single shard, all of the data resides in that shard. But in this example; there are 3 shards in total. And each shard will have hash slots to store the data, automatically determined. In each AZ, primary nodes will have a different hash slots and replicates the data to the other replicas in other AZs.
  • When a failover of primary node occurs, ElastiCache determines which RR node has the lowest replication lag. Then it takes that RR and makes it primary. Newly promoted primary node replicates to the other AZs.
  • When AZ fails, ElastiCache removes it from the cluster. There will be temporary downtime for write but you will be able to read %100 of the data.
  • Deploy two RRs per shard.
  • Deploy primary nodes across AZs.
  • Deploy RRs in different AZs from their respective primary nodes.

My helpful screenshot

Security of ElastiCache

  • With Redis v6, AWS introduces RBAC functionality. Ability to create user account and passwords.
  • You can use this functionality for specific business units such as marketing, analytics etc.
  • User Groups including different users can be associated with certain ElastiCache Clusters as well.

Architecture Patterns

  • Caching: you can put in front of anything to speed up.
  • Real-time Sentiment Analysis use it as a buffer to support high-throughput for high-velocity ingestion of data. You can use Redis Streams as queue to be sent to workers/consumers.
  • IoT Data: You can have sensors and various types of data being ingested. You can have rules for incoming data to stream into different data stores.
  • Amazon Kinesis Filtering: using it for deduplication of incoming data or filtering raw data.
  • Mobile: mobile app with recommendation engine using geospatial data that would be hitting redis to execute commands and sending you back interested locations etc. using Lambda functions to keep your cache refreshed as well.