Office 365 Audit Logging and Email scams

by Rob Armstrong

Nov 30 2018

We’re coming across more and more instances of Office 365 accounts with suspicious activity. Unfortunately the logging defaults in Office 365 are unsatisfactory and a little additional configuration is required to improve the effectiveness of the logging, especially in regards to user activity.

This article is intended to provide some quick recommendations which will improve the usefulness of the information stored in the Office 365 and Exchange Online audit logs and record a couple of observations along the way.

Office 365 and it’s Azure AD and Exchange Online brethren are moving targets. This article was put together using a Small Business Suite Office 365 Business Premium subscription in November 2018. Things will probably change, you have been warned.

Audit Logging

The Office 365 and Exchange Online audit logs are of greatest interest when investigating user activity. Despite this, the Office 365 Audit Log is not enabled out of the box, and the free edition of Azure AD that backs the Office 365 instance does not provide access to sign-in event information. The Exchange Online Admin Audit Log is enabled by default, however no user mailbox activity events are recorded.

Microsoft has stated an intention to enable the Office 365 Audit Log by default in future but currently additional configuration is required to ensure useful information is being recorded.

This additional logging needs to be enabled well before you have a security incident!

What you get by default

Azure AD Risk Events - While the Azure AD Free instance does not provide access to the complete set of sign-in data, it will flag what it considers to be high risk sign-in events. The details of these sign-ins are limited to the user’s name, the originating IP address, the country to which that IP is registered, the time of the sign-in and whether the session is currently active.
Exchange Online Admin Audit Log - The Exchange Online admin audit log records events out of the box, however the events recorded largely concern actions taken by exchange administrators. Some information regarding user actions can be inferred from these events but it is far from a complete picture. These logs are kept for 90 days.

Enabling Office 365 Audit Log

The Office 365 audit log is disabled by default and must be manually enabled. This is a straight-forward click of a button and can be done via the Office 365 Security & Compliance Center at https://protection.office.com/#/unifiedauditlog:

Be warned however - it can take up to 24 hours to fully activate and start recording events:

Any events occurring prior to the audit log being enabled will not be recorded and events generated while the audit log is being prepared are unlikely to be recorded. The Office 365 Alert Policies located at https://protection.office.com/#/viewalerts require the audit log to be enabled in order to function. This even applies to the alert policies which come pre-configured in Office 365. See the ‘Detecting and Managing Malicious Mail Forwarding’ section of this post for more information.

Configuring Exchange Online Mailbox Auditing

Exchange Online does not log mailbox audit events or record actions taken by the mailbox owner by default. This is supposed to be changing soon.

The Exchange Online administrator audit log does record some events out of the box (more on that later).

The Exchange Online mailbox auditing is enabled via Exchange Online PowerShell. Microsoft has a write-up on connecting to Exchange Online with PowerShell here, and another on configuring the mailbox audit log here, however a quick summary of the steps involved is detailed below:

First up you’ll need to install the Exchange Online PowerShell Module to authenticate using MFA, because you’ve already configured MFA on your Office 365, right?

You can get the module from the ‘hybrid’ deployment area of the Exchange Admin Center:

This will do its magic and you’ll end up with a PowerShell session loaded up ready to connect to Exchange Online using Connect-EXOPSSession. You’ll also get an Exchange Online PowerShell Module icon on your desktop which you can use next time:

Once you’ve entered in your password and completed the MFA, mailbox auditing and the auditing of owner events can be enabled on a single mailbox (testuser in this example) with the following command:

Get-Mailbox -Identity testuser | Set-Mailbox -AuditEnable $true -AuditOwner "Create, Update, HardDelete, MailboxLogin, Move, MoveToDeletedItems, SoftDelete"

Realistically you’re going to want this audit logging enabled on all your mailboxes. The following command will enable auditing and specify that owner events be logged for all mailboxes in the organisation:

Get-Mailbox -ResultSize Unlimited -Filter {RecipientTypeDetails -eq "UserMailbox"} | Set-Mailbox -AuditEnable $true -AuditOwner "Create, Update, HardDelete, MailboxLogin, Move, MoveToDeletedItems, SoftDelete"

Note that this only enables and configures auditing on the organisation’s current mailboxes. The above command(s) should be incorporated into the process of setting up new user accounts or run on a regular schedule to ensure auditing is enabled before a security incident occurs. You can check whether auditing is enabled and what events are being audited with the following commands:

Get-Mailbox -Identity testuser | Select-Object Audit*
Get-Mailbox -Identity testuser | Select-Object -Expand AuditOwner
Get-Mailbox -Identity testuser | Select-Object -Expand AuditAdmin
Get-Mailbox -Identity testuser | Select-Object -Expand AuditDelegate

The first command should return output which looks roughly like this: The other three commands can be used to get the full list of events which will be recorded in the mailbox audit log.

Log Retention

The Office 365 audit log only retains the past 90 days of activity. This currently can’t be extended although Microsoft is “actively working on a plan to increase this limit”.

The Exchange Online admin audit log is limited to 90 days of activity and unlike on-premise Exchange this retention period cannot be extended.

The Exchange Mailbox audit logs are also retained for 90 days, but Exchange Online can be configured to retain events for longer using the Set-Mailbox cmdlet. The following example will configure mailbox audit logs to be retained for 365 days:

Set-Mailbox -Identity testuser -AuditLogAgeLimit 365.00:00:00

In short: The only way to get a meaningful log retention period is to archive your Office 365 logs locally.

Notes on Azure AD Free

The events recorded by the Azure AD that backs your Office 365 can be accessed via https://portal.azure.com. However, by default the Azure AD instance has only a ‘Free’ license, which does not provide much in the way of meaningful logging:

Complete sign-in data is not available to the ‘Azure AD Free’ license although it will flag sign-ins which Microsoft considers to be high risk, these can be accessed via the Risk events feature:

Commonly these are sign-in events from locations or IP addresses which have not been previously associated with user activity. More information about how Azure AD identifies risk events is available here

The risk event information available to users of free & basic Azure AD subscriptions consists of the user’s name, the IP address and location from which the sign-in originated, the time of the sign-in and whether it is still active or not.

Additional functionality to help manage these events is available to premium subscribers and more information on the risk event feature (previously called “risky sign-ins”) can be found here.

Along with the improved logging, other features of Premium Azure AD which may be of use are the user risk policy and conditional access components. A user risk policy and conditional access provides additional alerting and controls over access to Office 365 resources, such as denying access from unapproved devices or suspicious network locations.

Detecting and Managing Malicious Mail Forwarding

A common technique utilised by scammers who have compromised a mail account is to configure a rule to forward emails of interest. Most often this occurs where the targeted email account is used by the finance or accounting departments and the scammer uses a forwarding rule to alert them of any possible financial transactions. This knowledge of the transaction is then used to create a targeted phishing email which purports to be part of the legitimate email exchange but instead contains details of a bank account which is controlled by the scammer. This is one example of a malicious mail forward we have observed in the wild, though additional forwarding tricks are commonly used by scammers.

Due to the high occurrence of these scams, Office 365 comes pre-packaged with an alert policy which triggers when a rule is created to forward messages to an external address:

While the default inclusion of this alert is great, it appears that it will not trigger if the Office 365 audit log has not been enabled.

The following behavior was observed using a Small Business Suite Office 365 Business Premium subscription:

After purchasing the subscription, Office 365 Audit Logging was enabled at approximately 07:35PM 05/09/2018.
A forwarding rule, ‘test rule 1’, which sends emails to an external address was created at 08:42PM 05/09/2018.
No alert was raised by the O365 ‘Creation of forwarding/redirect rule’ alert policy.
Creation of the rule was not recorded in O365 audit log, even after the audit log had finished being prepared and a 24hr period had passed.

The good news is that the Exchange Online Admin audit log DOES record the creation of the forwarding rule in the default configuration:

Inbox rules which forward emails to external addresses that are created after the Office 365 audit log has finished preparing are successfully flagged by the pre-configured alert.

Office 365 can be configured to actively prevent inbox rules from forwarding email to external domains. The easiest way to do this is by using the O365 SecureScore tool to configure a mail flow rule for you. This is achieved by applying the ‘Enable Client Rules Forwarding Block’ action: This will create a mail flow rule for you in Exchange Online:

With this enabled, attempts to forward messages to external domains via a rule receive an Undeliverable message:

Using a mail flow rule like this to stop inbox rules from forwarding emails to external addresses is highly recommended. It’s also a really good idea to utilise the rest of the Secure Score tool to review your Office 365 configuration and apply it’s recommendations where applicable.

On a side note - inbox forwarding rules created via the Outlook Web App (OWA) can be differentiated from rules created via PowerShell (PS): when viewed in the Exchange Online admin audit log:

OWA-created rule:

Rules created using OWA include details of the O365 organisation in the Object Modified field.

PS-created rule:

Rules created via the New-InboxRule PowerShell cmdlet only record the mailbox and rule name in the Object Modified field. As your average user tends not to create forwarding rules via PowerShell this indicator may be of use in identifying potentially suspicious rules.

Testing indicates that email-forwarding rules created via the Outlook Desktop App DO NOT GET LOGGED in the O365 audit log/unified audit log/exchange online audit log.

The pre-configured ‘Creation of forwarding/redirect rule’ alert policy also does not fire when forwarding rules are created via the Outlook Desktop App.

Recommendations

Enable the Office 365 Audit Log ASAP
Enable mailbox auditing of owner actions on all existing mailboxes and ensure all new mailboxes have this additional auditing applied as part of the new user setup process.
Ensure the Office 365 audit logs are being regularly archived and retained locally.
Consider upgrading the Azure AD which supports O365 from ‘Free’ to one of the ‘Premium’ licenses.
Consider disabling Exchange Online PowerShell access for users who do not require it.

References / Further Reading

It wouldn’t have been possible to assemble this post without the efforts of others

and Microsoft is constantly-updating the O365/Exchange Online documentation

Follow us on LinkedIn