Serverless at Scale: Lessons Learned

Serverless computing promises effortless scaling and pay-per-use economics. After running Azure Functions in production for over three years across multiple projects, I can confirm that the promise is real, but only if you understand the constraints. Here are the lessons I have learned the hard way.

Lesson 1: Cold Starts Are Real and They Matter

Cold starts are the most discussed limitation of serverless. When a function has not been invoked recently, the platform needs to allocate resources, load your code, and initialize your dependencies. On the Consumption plan, this can add 1-3 seconds of latency.

Mitigation Strategies

For latency-sensitive workloads, use the Premium plan with pre-warmed instances:

{
  "type": "Microsoft.Web/serverfarms",
  "apiVersion": "2023-12-01",
  "sku": {
    "name": "EP1",
    "tier": "ElasticPremium"
  },
  "properties": {
    "maximumElasticWorkerCount": 20,
    "elasticScaleEnabled": true
  }
}

On the application side, keep your startup code lean. Avoid heavy initialization in the function constructor. Use lazy loading for expensive dependencies:

public class OrderFunction
{
    private static readonly Lazy<HttpClient> _httpClient = new(() =>
    {
        var client = new HttpClient();
        client.BaseAddress = new Uri(Environment.GetEnvironmentVariable("CATALOG_API_URL")!);
        return client;
    });

    [Function("ProcessOrder")]
    public async Task<IActionResult> Run(
        [HttpTrigger(AuthorizationLevel.Function, "post")] HttpRequest req)
    {
        var client = _httpClient.Value;
        // Process order...
    }
}

Lesson 2: Concurrency Limits Will Bite You

Each Azure Functions instance processes a limited number of concurrent requests. For HTTP triggers on the Consumption plan, the default is around 100 concurrent executions per instance. When traffic spikes, the platform scales out by adding instances, but this takes time.

I learned this when a marketing campaign drove a 10x traffic spike to our order processing function. The function scaled, but not fast enough. We saw a wave of 429 responses for about two minutes.

What Helped

• Set FUNCTIONS_WORKER_PROCESS_COUNT to utilize multiple worker processes per instance
• Configure maxConcurrentRequests in host.json to match your workload characteristics
• Use queue-based load leveling for background processing instead of synchronous HTTP calls

{
  "extensions": {
    "http": {
      "maxConcurrentRequests": 200,
      "routePrefix": "api"
    },
    "queues": {
      "maxPollingInterval": "00:00:02",
      "batchSize": 32,
      "newBatchThreshold": 16
    }
  }
}

Lesson 3: Durable Functions Change the Game

Standard Azure Functions are stateless by design. When you need to coordinate multi-step workflows, manage long-running processes, or fan out work across multiple functions, Durable Functions are indispensable.

We replaced a complex set of queue-triggered functions with a single orchestrator and the code became dramatically simpler:

[Function("OrderOrchestrator")]
public async Task RunOrchestrator(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var order = context.GetInput<Order>();

    // Step 1: Validate inventory
    var available = await context.CallActivityAsync<bool>(
        "CheckInventory", order.Items);

    if (!available)
    {
        await context.CallActivityAsync("NotifyCustomer",
            new Notification(order.CustomerId, "Items unavailable"));
        return;
    }

    // Step 2: Process payment
    var payment = await context.CallActivityAsync<PaymentResult>(
        "ProcessPayment", order.Payment);

    // Step 3: Ship order (fan out across warehouses)
    var shipmentTasks = order.Items
        .GroupBy(i => i.WarehouseId)
        .Select(g => context.CallActivityAsync<ShipmentResult>(
            "CreateShipment", g.ToList()));

    var shipments = await Task.WhenAll(shipmentTasks);

    // Step 4: Send confirmation
    await context.CallActivityAsync("SendConfirmation",
        new OrderConfirmation(order, payment, shipments));
}

Lesson 4: Monitoring Requires Different Tooling

Traditional APM tools expect long-running processes. Serverless functions are ephemeral, which means you need to think differently about monitoring.

Application Insights is essential. Enable it from day one and configure sampling appropriately. For high-throughput functions, adaptive sampling prevents drowning in telemetry data while still capturing enough to diagnose issues.

Key metrics I track for every serverless workload:

• Execution count and duration (P50, P95, P99)
• Failure rate with automated alerting above threshold
• Queue depth for async triggers (a growing queue signals a bottleneck)
• Instance count to understand scaling behavior

Lesson 5: Cost Can Surprise You

The Consumption plan is cheap for sporadic workloads, but costs can escalate quickly with high throughput. A function that processes 100 million events per month at 500ms average duration is not cheap.

Model your costs before going to production. The Azure pricing calculator helps, but also consider:

• Storage transactions for queue and blob triggers
• Application Insights ingestion costs (often the hidden budget killer)
• Networking costs for VNet-integrated functions

For predictable workloads, the Premium plan with reserved instances often costs less than Consumption at scale.

Conclusion

Serverless is a powerful paradigm, but it is not magic. Understanding cold starts, concurrency limits, orchestration patterns, monitoring requirements, and cost models will determine whether your serverless architecture thrives or struggles. Start small, measure everything, and scale with confidence.