Memcached

What is Memcached?

Memcached is an in-memory key-value store optimized for caching frequently accessed data to reduce database load and accelerate application response times. Created in 2003 by Brad Fitzpatrick for LiveJournal, Memcached stores data entirely in RAM, enabling sub-millisecond read and write operations. The latest version, v1.6.39 (released July 2025), continues to refine performance and reliability. Memcached uses a simple protocol (get, set, delete) and implements LRU (Least Recently Used) eviction when memory is full, making it predictable and easy to reason about.

Unlike more complex systems like Redis, Memcached focuses exclusively on caching with a minimalist design philosophy. It stores only strings (byte arrays) as values—no complex data structures like lists, sets, or hashes. This simplicity translates to extremely high performance: production systems routinely handle 100,000+ operations per second per node, with optimized hardware reaching millions of ops/sec. Memcached's distributed architecture relies on client-side consistent hashing, allowing horizontal scaling by adding more nodes without centralized coordination. This makes Memcached ideal for web-scale applications requiring blazing-fast caching with minimal overhead.

Core Features and Architecture

Key Characteristics

• Sub-millisecond latency - Average response times <1ms for get/set operations
• High throughput - 100,000+ ops/sec typical, up to millions with tuned hardware
• Multithreaded - Leverages multiple CPU cores for concurrent request handling
• Simple key-value model - Stores strings/bytes only (no complex data structures)
• LRU eviction - Automatically removes least recently used items when memory full
• Client-side sharding - Distributed via consistent hashing in client libraries
• No persistence - All data lives in RAM; restart means data loss
• TTL support - Automatic expiration for time-sensitive cached data
• Binary and ASCII protocols - Support for both text and binary communication
• CAS (Check-And-Set) - Optimistic locking for concurrent updates

Operational Advantages

• Zero dependencies - Single standalone binary, no external libraries required
• Horizontal scalability - Add nodes linearly to increase capacity and throughput
• Minimal configuration - Few tunables; works well with defaults
• Predictable performance - O(1) operations with consistent latency
• Small memory footprint - Efficient memory usage with minimal overhead
• Battle-tested - 20+ years of production use at massive scale
• Cross-platform - Runs on Linux, macOS, Windows, BSD systems
• Wide language support - Client libraries for Python, PHP, Java, Node.js, Go, Ruby

Use Cases and Applications

Memcached excels in scenarios where simple caching dramatically improves performance:

• Database query caching - Cache expensive SQL query results to reduce DB load
• Session storage - Store user session data with automatic TTL expiration
• API response caching - Cache third-party API responses to reduce costs and latency
• Page fragment caching - Cache rendered HTML fragments for dynamic pages
• Computed results - Cache expensive calculations, aggregations, or transformations
• Rate limiting - Store API request counters with TTL for sliding windows
• Configuration data - Cache app configs, feature flags, service discovery info
• Authentication tokens - Cache OAuth tokens, JWT validation results
• LLM response caching - Cache AI/ML model outputs for identical inputs
• Content delivery - Cache static assets, images metadata, CDN content

Memcached for AI/ML Applications

Memcached serves as a high-speed caching layer in AI/ML architectures. For LLM applications, caching identical prompts can reduce API costs by 50-90% and improve response times from seconds to milliseconds. Memcached stores prompt hashes as keys and LLM responses as values, with TTL expiration for data freshness. For ML inference pipelines, Memcached caches preprocessed features, embeddings, and model predictions to avoid redundant computations. During model serving, frequently accessed model weights or metadata can be cached to reduce storage I/O.

Memcached is particularly effective for AI workloads with high cache hit rates. For example, customer support chatbots often encounter similar questions—caching responses for common queries dramatically reduces LLM API costs. Similarly, recommendation systems can cache user feature vectors or item embeddings for sub-millisecond retrieval. Memcached's simplicity and performance make it ideal for caching layers where complex data structures aren't needed, allowing AI teams to focus on model development rather than cache infrastructure.

Memcached vs Redis

Memcached and Redis are both in-memory stores, but with different philosophies. Memcached focuses exclusively on caching with a minimalist design: simple key-value storage, no persistence, and maximum throughput. Redis offers richer data structures (lists, sets, sorted sets, hashes), optional persistence (RDB snapshots, AOF logs), pub/sub messaging, Lua scripting, and replication. For pure caching workloads where simplicity and raw speed matter most, Memcached often edges out Redis with slightly lower latency and higher throughput due to its focused design.

However, Redis's versatility makes it suitable for many more use cases beyond caching: message queues, leaderboards, session stores with complex data, and even primary data storage with persistence. Memcached's lack of persistence means it's purely a cache—data loss on restart is expected. Choose Memcached when you need blazing-fast, simple caching with minimal operational overhead. Choose Redis when you need data structures, persistence, pub/sub, or more advanced features. Many organizations use both: Memcached for high-throughput caching, Redis for more complex caching and data storage needs.

Getting Started with Memcached

Install Memcached locally using package managers: `apt install memcached` (Ubuntu/Debian), `brew install memcached` (macOS), or `yum install memcached` (RHEL/CentOS). Start the service: `memcached -d -m 64 -p 11211` (runs as daemon with 64MB RAM on port 11211). Test with telnet: `telnet localhost 11211`, then try commands: `set mykey 0 0 5` (enter), `hello` (enter), then `get mykey` to retrieve. For production, use systemd/init scripts to manage the service with appropriate memory allocation.

For Python, install pymemcache: `pip install pymemcache`. For distributed setups, run multiple Memcached instances across servers, and client libraries handle sharding automatically. Managed services include AWS ElastiCache for Memcached, Google Cloud Memorystore (Memcached), and Azure Cache for Redis (supports Memcached protocol). These handle provisioning, monitoring, patching, and scaling. Monitor cache performance with `stats` command or tools like Nagios, Datadog, New Relic to track hit rates, memory usage, and evictions.

Code Example: LLM Response Caching with Memcached

This example demonstrates using Memcached to cache LLM API responses, reducing costs and latency for repeated prompts:

from pymemcache.client.base import Client
import hashlib
import json

# Connect to Memcached server
client = Client(('localhost', 11211))

def cache_llm_response(prompt: str, model: str = "gpt-4o-mini") -> str:
    """
    Cache LLM responses to reduce API costs and latency.
    Identical prompts return cached results in <1ms vs ~1-2s for API calls.
    """
    # Create cache key from prompt hash (MD5 sufficient for caching)
    cache_key = hashlib.md5(f"{model}:{prompt}".encode()).hexdigest()
    
    # Check cache first
    cached_response = client.get(cache_key)
    if cached_response:
        print(f"Cache HIT! Returning cached response (saved API call)")
        return cached_response.decode('utf-8')
    
    print(f"Cache MISS. Calling LLM API...")
    
    # Simulate LLM API call (replace with actual OpenAI/Anthropic call)
    from openai import OpenAI
    openai_client = OpenAI()
    response = openai_client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}]
    )
    result = response.choices[0].message.content
    
    # Store in cache with 1 hour TTL (3600 seconds)
    # TTL ensures fresh data for time-sensitive queries
    client.set(cache_key, result.encode('utf-8'), expire=3600)
    print(f"Response cached with 1h TTL")
    
    return result

# Example usage
if __name__ == "__main__":
    # First call: hits LLM API (~1-2 seconds)
    response1 = cache_llm_response("What is Memcached?")
    print(f"Response: {response1[:100]}...\n")
    
    # Second call: returns from cache (<1ms)
    response2 = cache_llm_response("What is Memcached?")
    print(f"Response: {response2[:100]}...\n")
    
    # Cache statistics
    stats = client.stats()
    print(f"Cache stats: {stats}")
    
    # Advanced: Cache with versioning for prompt engineering
    def cache_with_version(prompt: str, version: str = "v1") -> str:
        key = hashlib.md5(f"{version}:{prompt}".encode()).hexdigest()
        cached = client.get(key)
        if cached:
            return cached.decode('utf-8')
        # ... LLM API call ...
        result = "LLM response here"
        client.set(key, result.encode('utf-8'), expire=7200)  # 2h TTL
        return result
    
    # Use case: Session storage with automatic expiration
    def store_user_session(user_id: str, session_data: dict):
        key = f"session:{user_id}"
        client.set(key, json.dumps(session_data).encode('utf-8'), expire=1800)  # 30min
    
    def get_user_session(user_id: str) -> dict:
        key = f"session:{user_id}"
        data = client.get(key)
        return json.loads(data.decode('utf-8')) if data else None
    
    # Use case: Rate limiting
    def check_rate_limit(user_id: str, max_requests: int = 100) -> bool:
        key = f"rate:{user_id}"
        current = client.get(key)
        if current is None:
            client.set(key, b'1', expire=60)  # 1 minute window
            return True
        count = int(current)
        if count >= max_requests:
            return False
        client.incr(key, 1)
        return True

Integration with 21medien Services

21medien implements Memcached for high-performance caching in AI application architectures. We deploy Memcached as a caching layer for LLM API responses, reducing costs by 60-80% for applications with repeated queries. Our team provides Memcached consulting, architecture design (single vs multi-node deployments, sizing calculations), client library integration, and monitoring setup. We specialize in optimizing cache hit rates through key design strategies, TTL tuning, and eviction policy analysis. For AI inference pipelines, we implement Memcached for feature caching, embedding storage, and prediction result caching to achieve sub-10ms response times at scale.

Pricing and Deployment Options

Memcached is completely free and open-source (BSD license). Self-hosting costs are only infrastructure: cloud VMs starting at $10-50/month for small instances (1-4GB RAM), $100-500/month for medium workloads (16-32GB RAM), $500-2000/month for large production deployments (64-128GB RAM). For managed services, AWS ElastiCache for Memcached pricing: cache.t3.micro (0.5GB) ~$12/month, cache.m6g.large (6.38GB) ~$90/month, cache.r6g.xlarge (26.32GB) ~$280/month. Google Cloud Memorystore for Memcached: ~$0.049/GB-hour (~$35/month per GB). Azure Cache for Redis (Memcached protocol): Basic tier starts ~$15/month, Standard ~$55/month, Premium ~$450/month.

For AI applications, typical costs: small LLM caching layer $50-200/month, medium-scale inference caching $200-1000/month, enterprise feature stores $1000-5000/month. ROI is substantial—caching LLM responses can save 10-100x in API costs compared to uncached architectures. A $100/month Memcached deployment can easily offset $2000-5000/month in LLM API costs by achieving 80%+ cache hit rates. Memory sizing guideline: estimate working set size (frequently accessed data), multiply by 1.5-2x for headroom, and provision accordingly. Most applications start with 4-16GB RAM per node and scale horizontally by adding more nodes.