How to query varnish logs from a Log analytics workspace

When using a varnish cache for a project in Azure, we would like to have an easy access to the varnish logs directly from the Azure portal. This can be done by creating a Custom Log containing the formatted varnish logs.

In this article we will see how to:

• Configure formatted logging in varnish
• Configure a log rotation for the varnish logs
• Configure a Custom Log in an Azure Log Analytics to easily access these logs

Prerequisites: – An Azure account – A varnish deployed in Azure

To configure our own format for the varnish logs, we used varnishncsa and used its format option by creating a file /etc/varnish/format-varnishncsa:

{
    "X-Forwarded-For" : "%{X-Forwarded-For}i",
    "timestamp" : "%t",
    "Host" : "%h",
    "HTTP-Method" : "%m",
    "URL" : "%U%q",
    "HTTP-response-code" : "%s",
    "Hit-Miss" : "%{Varnish:hitmiss}x",
    "handling" : "%{Varnish:handling}x",
    "Time-To-Serve-Request" : "%D",
    "X-Magento-Tags-Pattern" : "%{X-Magento-Tags-Pattern}i"
}

You can change the json to include whichever info best suits your needs. Once this file created, you can launch varnishncsa as a deamon or as a service, we chose to use it as a deamon with the command:

sudo varnishncsa -a -w /var/log/varnish/varnispid -f /etc/varnish/format-varnishncsa

You can create a file /etc/logrotate.d/varnish:

{
    "X-Forwarded-For" : "%{X-Forwarded-For}i",
    "timestamp" : "%t",
    "Host" : "%h",
    "HTTP-Method" : "%m",
    "URL" : "%U%q",
    "HTTP-response-code" : "%s",
    "Hit-Miss" : "%{Varnish:hitmiss}x",
    "handling" : "%{Varnish:handling}x",
    "Time-To-Serve-Request" : "%D",
    "X-Magento-Tags-Pattern" : "%{X-Magento-Tags-Pattern}i"
}

This will rotate the varnish logs daily for a week.

The final step is to configure these logs to appear in a log analytics, for that you need to navigate to the custom logs blade in a log analytics:

Click on Add custom log:

Then you have to follow the steps starting with uploading a sample of the logs. Once the custom logs are created you can query them directly from the log analytics workspace: