Observability and Scale in Serverless — DevTalks Spring 2020

Tomi Hiltunen May 14 2020 8 AM | 5 min read

Over the years, we have loved to treat hard-boiled software engineers to DevTalks — an evening of tech talks and mingling at our office in downtown Helsinki. This spring, the coronavirus forced us to change the plan and take DevTalks fully online.

The sixth edition of DevTalks was all about serverless. We had two brilliant speakers: Yan Cui and Nicolas Moutschen, who gave talks on building observability into a serverless application and creating serverless microservices at scale.

Yan is an experienced engineer and a Serverless Hero who has run production workload at scale in AWS for 10 years. He has been an architect and principal engineer with a variety of industries ranging from banking, e-commerce, sports streaming to mobile gaming. He has worked extensively with AWS Lambda in production, and has been helping clients around the world adopt AWS and serverless as an independent consultant. Read more about Yan’s work at

Nicolas has been passionate about automating repetitive things through code since he was eight years old. In 2011, he went on to create a startup focused on helping SMBs improve the reliability of their IT infrastructure and think about how IT could help them transform their business. Since then, Nicolas has joined AWS as a Solutions Architect to help organizations of all sizes design cloud-native applications and leverage serverless technologies to focus on what matters.

Yan Cui: How to build observability into a serverless application

Yan outlines the challenges that the serverless paradigm introduces to existing tools used for observability, like nowhere to install monitoring agents/daemons, no background processing, higher concurrency to monitoring systems and lots more asynchronous invocations. Yan explains how you can build observability into a serverless application and touches on how to implement log aggregation and how to capture and forward correlation IDs through both synchronous (API Gateway) and asynchronous (SNS, Kinesis, S3, etc.) invocation events. He also discusses the need for structured logging and sampling debug logs in production, as well as how to capture performance traces to help debug performance issues.

Psst! Yan is coming to Helsinki in August to host a workshop on “Production-Ready Serverless”. You can get a 100€ discount to his workshop with the code “smartly-2020”.

After the talk, Yan answered attendees’ questions. The first question was what would be the best way to transfer custom metrics to third party platforms, from inside a Lambda function, considering +100ms -> 1% loss of sales relation Yan mentioned in his talk.

Here’s what Yan said “For user-facing APIs, your best bet might be to proxy through CW Logs, e.g. write what you want to send to CW Logs, and then have a function forward them afterwards (kinda similar to the embedded metric stuff I mentioned). Before the embedded metric came out, I actually had a function do that same job, and parse a statsd-esque string into a custom metric in CW, which you can apply the same technique for sending custom metrics to third-party. However, most vendors already support ingesting metrics from CloudWatch, so you can always piggyback off of that, and just use embedded metric format to generate them as custom metrics in CW, then have the third-party system ingest it from there (you'll end up paying for that extra overhead though, for both CW metrics, AND pulling from CW).”

The second question was about whether it is possible to process outside the request-response chain in Lambda? For example, to send a statsd call in the background after returning the response. This sparked a longer discussion between Yan, Nicolas and Heitor, the Principal Serverless Lead at AWS, who was among the attendees. Here are the key takeaways from the conversation:

  1. It is possible to use custom Lambda runtime to process outside the request-response chain, enabling the developer to perform some amount of background processing. This approach  has drawbacks, though. When calls are being made in the bootstrap, the Lambda container won’t be available for additional incoming requests. This increases the likelihood of another cold start. The time you have available for processing is not strictly documented and might change at any time without a notice.
  2. The original question played around with the idea of background processing but the example itself was about incrementing stats with statsd. For that use case, Heitor mentions that most users–Yan included–simply use canonical logs for creating metrics.

Nicolas Moutschen: Building serverless micro-services at scale

Building a single serverless service is a breeze with all the resources and tools available online. All you need is Amazon API Gateway, AWS Lambda, and an Amazon DynamoDB table. You can add AWS SAM or other tools like the Serverless Framework to define resources as code and you're all set. In his talk, Nicolas directs our attention at the bigger picture: How can we take those individual services and adopt serverless at scale within an organisation? How should we approach service discovery? How should we think about the developer experience and workflow in a serverless world? Drawing from the experiences of many customers he has worked with, Nicolas built a sample e-commerce platform using serverless technologies and walks us through the different design decisions and their implications.

Stay tuned for more

DevTalks will be back with more intriguing talks to boost your learning. Join the meetup group to stay up to date about upcoming events.

In the meantime, you can learn more about the challenges our engineering is working on in our Engineering blog, YouTube channel and Engineering Twitter account.

Learn more about being a developer at at

   Ebook: Actionable Creative Insights CHECK IT NOW
Tomi Hiltunen
Full Stack Developer at

Read Next