How to Integrate DataSunrise with AWS Database Activity Streams for Getting Auditing Results for AWS Aurora PostgreSQL

Overview of database activity auditing methods

Today, database auditing becomes more and more important because of numerous legal acts and regulations dedicated to sensitive data protection such as GDPR, KVKK, etc.

Technically, there are multiple database auditing methods exist, the most popular are:

• Database traffic proxying;
• Passive database traffic listening;
• Reading of logs collected by database native auditing tools;
• Integration with dedicated auditing services such as AWS DAS.

Each approach has its own pros and cons, limitations and opportunities. Amazon users in their turn may find the last two approaches most suitable for their needs. These methods enable AWS users to notify security personnel about specific database events, create reports on activity in Aurora PG cluster, etc.

AWS Database activity streams (DAS) provide a near-real-time stream of the activity in your DB cluster and give you the following advantages over database native logging mechanisms (DataSunrise’s Trailing DB audit logs functionality):

• DAS configuring process is much simpler than configuring “regular” trailing based on log files. You don’t need any extra storage space for storing logs as well;
• Increased protection against internal threats: database administrators don’t have access to the collection, transmission, storage, and processing of database activity streams;
• DAS provide more flexibility than “regular” trailing due to Synchronous and Asynchronous modes available.

The most critical con of DAS is incompatibility with some RDS Instance classes and Amazon Aurora versions.

The following diagram shows an Aurora PG cluster integrated with DataSunrise:

Here, an Aurora PG cluster with DAS enabled. Aurora pushes data on database activity to AWS Kinesis. The activity streams are encrypted using an AWS KMS encryption key at that. Kinesis in its turn passes database activity streams to DataSunrise which is used as a database monitoring tool. DataSunrise consumes the streams and saves the auditing results in its Audit storage. Then the captured database events are displayed in the Transactional trails section of the DataSunrise’s Web Console.

Prerequisites for Aurora PG cluster

Before configuring DAS, ensure that your AWS environment complies with the requirements listed below.

Supported Aurora PostgreSQL database versions:

• All 13 versions
• All 12 versions
• Version 11.6 and higher 11 versions
• Version 10.11 and higher 10 versions

Supported AWS RDS Instance classes:

• db.r6g
• db.r5
• db.r4
• db.x2g

Database activity streams are supported in all AWS Regions except the following ones:

• China (Beijing) Region, cn-north-1
• China (Ningxia) Region, cn-northwest-1
• AWS GovCloud (US-East), us-gov-east-1
• AWS GovCloud (US-West), us-gov-west-1

Miscellaneous:

• DAS require use of AWS Key Management Service (AWS KMS). AWS KMS is required because the activity streams are always encrypted
• Database activity streams require use of Amazon Kinesis

Read more: AWS Aurora DB Activity Streams – Version Requirements

Creating an AWS KMS key

Since the activity streams are always encrypted, you need to use an encryption key. Aurora uses the KMS key to encrypt the key that in turn encrypts database activity. If you don’t have a KMS key, create it.

Navigate to AWS’s Key Management System (KMS), click Create a key. Set the following options for your key:

• Key type: Symmetric
• Key material origin: KMS
• Regionality: Single-region key
• Key policy: default

Read more: AWS KMS – Creating Keys

Starting DAS for your Aurora PG cluster

Having prepared your RDS Aurora PG environment, you can start Database activity streams.

Navigate to RDS Managed Relational Database Service (RDS), select Databases, choose the DB cluster to enable an activity stream for, click Actions -> Start database activity stream:

Configure DAS in the following way:

• Master key: your KMS key
• Database activity stream mode: either Asynchronous or Synchronous. We recommend using Synchronous mode because it favors the accuracy of the activity stream over database performance. The difference between these two modes you can find here.
• Apply immediately: whether to apply the changes immediately or on schedule.
• You can access the stream at the Configuration section of your cluster’s settings (Database activity stream -> Kinesis stream): Read more: AWS Aurora DB Activity Streams – Enabling

Configuring DAS-based auditing in DataSunrise

Having started DAS for your Aurora cluster, you can proceed to configuring DataSunrise to consume database activity streams.

To enable DataSunrise to work with DAS, your IAM user needs access to some features of Kinesis, KMS and RDS. To grant the required permissions, navigate to Identity and Access Management (IAM) -> Users, and attach the following policy to your User:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
            "Kinesis:DescribeStreamSummary",
            "Kinesis:ListShards",
            "Kinesis:GetShardIterator",
            "Kinesis:GetRecords",
            "Kinesis:DescribeStreamSummary",
            "KMS:Decrypt",
            "RDS:DescribeDBClusters"],
            "Resource": ["<ARN of your Kinesis stream>","<ARN of your KMS key>","<ARN of your RDS DB>"]
        }
    ]
}

Create new Aurora PG Database Instance or use an existing one. Open DataSunrise’s Web Console and navigate to Configuration -> Databases.

• Provide connection details for your Aurora PG database. Specify the endpoint of your Aurora PG cluster in the Host field;
• At the Capture Mode section of the Instance page, select Trailing the DB Audit Logs from the Mode drop-down list;
• In the Format Type drop-down list, select Database activity stream;
• Fill out all the required fields. Save the Instance.

Create a data audit Rule for your Aurora PG Instance: navigate to Audit -> Rules and create a Rule

For auditing results, navigate to Audit -> Transactional Trails. Note that results might appear with a delay of about 5-10 minutes.

You can also adjust DataSunrise DAS-based auditing by changing the following parameters (System Settings -> Additional Parameters):

• TrailDASIntervalTime: period of time to get a list of events for (e.g. for last 5 minutes starting from latest record);
• TrailDASOffsetTime: time delay for getting events (required for synchronizing, seconds).