Case Study: Approaches to Logging Architecture

There are several approaches to design a logging architecture.

Best option depends obviously on the specific requirements.

In this blog post I won’t tell which logging architecture is best but I will give you options of what’s used in the industry.

Summary of approaches Native Option

First approache is to go all with AWS as Ancestry did ( https://www.youtube.com/watch?v=igcnes0PI10). With this option you will end up using kinesis for data streaming, which means you will probably also have Lambda functions (mapping to shards) and passing the results to another kinesis stream or storing them in Elasticsearch or S3.

Since this is AWS based, then API Gateway is needed for the Lambda Functions to work as Services.

Portable option

This is what companies like Pinterest do ( https://www.youtube.com/watch?v=DphnpWVYeG8). With this option usually kafka is used for data streamming as depicted in the image below.

For data processing there are few options, like Spark or Storm ( TODO: add sample) to read from kafka’s partititons as they do at Airbnb (link here).

Comparison

The following table summarizes the approaches (no duplicate entries).

References

kineses VS kafka: https://medium.com/faun/apache-kafka-vs-apache-kinesis-57a3d585ef78

Sample spark job: https://spark.apache.org/examples.html

Sample ElasticSearch with Java: https://www.baeldung.com/elasticsearch-java

Sample HBase with Java: https://www.baeldung.com/hbase

Sampel streamming with kafka and Spark: https://www.baeldung.com/kafka-spark-data-pipeline

Hive is a data warehouse software and HBase is a column-oriented database

Other options to Review later Generic
AWS Specific
https://aws.amazon.com/blogs/database/analyze-postgresql-logs-with-amazon-elasticsearch-service/
https://aws.amazon.com/blogs/containers/streaming-logs-from-amazon-eks-windows-pods-to-amazon-cloudwatch-logs-using-fluentd/
https://aws.amazon.com/blogs/containers/how-to-capture-application-logs-when-using-amazon-eks-on-aws-fargate/
https://aws.amazon.com/blogs/awsmarketplace/log-analysis-with-aws-control-tower-and-logz-io/
https://aws.amazon.com/blogs/architecture/stream-amazon-cloudwatch-logs-to-a-centralized-account-for-audit-and-analysis/
https://aws.amazon.com/blogs/apn/tag/elk-stack/
https://eng.uber.com/distributed-tracing/ Streaming, Flexible Log Parsing with Real-Time Application
https://engineering.fb.com/core-data/logdevice-a-distributed-data-store-for-logs/
https://blog.cloudera.com/real-time-log-aggregation-with-apache-flink-part-2/
https://netflixtechblog.com/edgar-solving-mysteries-faster-with-observability-e1a76302c71f
https://medium.com/@federicogaule/collecting-access-logs-into-elasticsearch-1a6f05288f8a
https://medium.com/logdna/redesigning-kafka-a-message-streaming-platform-built-for-logging-23ea1e54543b
https://medium.com/@pruthvikumar.123/centralized-logging-log-analytics-in-a-busy-distributed-environment-3df8a8d4549e
https://medium.com/@jonas_48080/distributed-tracing-and-aggregated-logging-98c8d6da8005

Originally published at http://jacace.wordpress.com on September 4, 2020.

Hands-on Sr Software Manager / Architect based in Ireland. Views are my own. Linkedin: https://ie.linkedin.com/in/jacace Twitter: https://twitter.com/jacace