What is logged in Azure and how long are the logs retained?

This article gives an overview of what type of activity is logged into Azure and for what duration of time.

Through activity logs, you can determine:

  • what operations were taken on the resources in your subscription
  • who started the operation
  • when the operation occurred
  • the status of the operation
  • the values of other properties that might help you research the operation

The activity log contains all write operations (PUT, POST, DELETE) for your resources. It doesn't include read operations (GET). For a list of resource actions, see Azure Resource Manager Resource Provider operations. You can use the activity logs to find an error when troubleshooting or to monitor how a user in your organization modified a resource.

Activity logs are kept for 90 days. You can query for any range of dates, as long as the starting date isn't more than 90 days in the past. 
If a compliance policy requires logs to be kept for longer than 90 days, you must export them.

You can retrieve information from the activity logs through the portal, PowerShell, Azure CLI, Insights REST API, or Insights .NET Library.

Important: When exporting logs to event hub or a storage account, costs are likely to increase.

Archive Activity Log

Archiving the Activity Log to a storage account is useful if you would like to retain your log data longer than 90 days (with full control over the retention policy) for audit, static analysis, or backup. If you only need to retain your events for 90 days or less you do not need to set up archival to a storage account, since Activity Log events are retained in the Azure platform for 90 days.

Prerequisites

Storage account

If you're archiving your Activity Log, you need to create a storage account if you don't already have one. You should not use an existing storage account that has other, non-monitoring data stored in it so that you can better control access to monitoring data. If you are also archiving Diagnostic Logs and metrics to a storage account though, you may choose to use that same storage account to keep all monitoring data in a central location.

The storage account does not have to be in the same subscription as the subscription emitting logs as long as the user who configures the setting has appropriate RBAC access to both subscriptions.

Note: You cannot currently archive data to a storage account that is behind a secured virtual network.

Event Hubs

If you're sending your Activity Log to an event hub, then you need to create an event hub if you don't already have one. If you previously streamed Activity Log events to this Event Hubs namespace, then that event hub will be reused.

The shared access policy defines the permissions that the streaming mechanism has. Streaming to Event Hubs requires Manage, Send, and Listen permissions. You can create or modify shared access policies for the Event Hubs namespace in the Azure portal under the Configure tab for your Event Hubs namespace.

To update the Activity Log log profile to include streaming, you must have the ListKey permission on that Event Hubs authorization rule. The Event Hubs namespace does not have to be in the same subscription as the subscription that's emitting logs, as long as the user who configures the setting has appropriate RBAC access to both subscriptions and both subscriptions are in the same AAD tenant.

Stream the Activity Log to an Event Hub by creating a Log Profile.

Create a log profile

You define how your Azure Activity log is exported using a log profile. Each Azure subscription can only have one log profile. These settings can be configured via the Export option in the Activity Log blade in the portal. They can also be configured programmatically using the Azure Monitor REST API, PowerShell cmdlets, or CLI.

The log profile defines the following.

Where the Activity Log should be sent. Currently, the available options are Storage Account or Event Hubs.

Which event categories should be sent. The meaning of category in Log Profiles and Activity Log events is different. In the Log Profile, Category represents the operation type (Write, Delete, Action). In an Activity Log event, the category"* property represents the source or type of event (for example, Administration, ServiceHealth, and Alert).

Which regions (locations) should be exported. You should include all locations since many events in the Activity Log are global events.

How long the Activity Log should be retained in a Storage Account. A retention of zero days means logs are kept forever. Please select 365 days or 0. 

If retention policies are set, but storing logs in a storage account is disabled, then retention policies have no effect. Retention policies are applied per-day, so at the end of a day (UTC), logs from the day that is now beyond the retention policy are deleted. For example, if you had a retention policy of one day, at the beginning of the day today the logs from the day before yesterday would be deleted. The delete process begins at midnight UTC, but note that it can take up to 24 hours for the logs to be deleted from your storage account.

Warning

The format of the log data in the storage account changed to JSON Lines on Nov. 1st, 2018. See this article for a description of the impact and how to update your tooling to handle the new format.

Important

You may receive an error when creating a log profile if the Microsoft.Insights resource provider isn't registered. See Azure resource providers and types to register this provider.

Create log profile using the Azure portal

Create or edit a log profile with the Export to Event Hub option in the Azure portal.

  1. From the Monitor menu in the Azure portal, select Export to Event Hub.

    Export button in portal

  2. In the blade that appears, specify the following:

    • Regions with the events to export. You should select all regions to ensure that you don't miss key events since the Activity Log is a global (non-regional) log and so most events do not have a region associated with them.

    • If you want to write to storage account:

      • The Storage Account to which you would like to save events.
      • The number of days you want to retain these events in storage. A setting of 0 days retains the logs forever.  Please ensure that Retention (days) is set to 365 or 0.
    • If you want to write to event hub:

      • The Service Bus Namespace in which you would like an Event Hub to be created for streaming these events.

    • Export Activity Log blade

  3. NOTE: Please select the "all" option under the regions drop-down.
  4. Click Save to save these settings. The settings are immediately be applied to your subscription.

NOTE:  For whichever method you choose to export your logs, please ensure that the following settings are applied.

1.  Ensure that a Log Profile exists
  1. Go to Activity log
  2. Click on Export
  3. Configure the setting
  4. Click on Save
2.  Ensure that Activity Log Retention is set 365 days or greater
  1. Go to Activity log
  2. Select Export
  3. Set Retention (days)is set to 365 or 0
  4. Select Save
3.  Ensure the log profile captures activity logs for all regions including global
  1. Go to Activity log
  2. Select Export
  3. Select Subscription
  4. In Regions dropdown list, check Select all
  5. Select Save
4.  Ensure the storage container storing the activity logs is not publicly accessible
  1. In right column, Click service Storage Accounts to access Storage account blade
  2. Click on the storage account name
  3. In Section Blob Service click Containers. It will list all the containers in next blade
  4. Look for a record with container named as insight-operational-logs. Click ...from right most column to open Context menu
  5. Click Access Policy from Context Menu and set Public Access Level to Private (no anonymous access)
5.  Ensure the storage account containing the container with activity logs is encrypted with BYOK (Use Your Own Key)
  1. In right column, Click service Storage Accounts to access Storage account blade
  2. Click on the storage account name
  3. In Section SETTINGS click Encryption. It will show Storage service encryption configuration pane
  4. Check Use your own key which will expand Encryption KeySettings
  5. Use option Enter key URI or Select from Key Vault to set up encryption with your own key
6.  Ensure that logging for Azure KeyVault is 'Enabled'
  1.     Follow Microsoft Azure documentation and setup Azure Key Vault Logging.








Keywords:activity log, content, cloud, data center, linux, manage, microsoft, write operations   Doc ID:94593
Owner:Scott R.Group:University of Illinois at Chicago ACCC
Created:2019-09-20 13:35 CDTUpdated:2019-09-26 10:24 CDT
Sites:University of Illinois at Chicago ACCC
Feedback:  0   0