RSS memory, Oracle and Hugepages

Recently a customer called us that there is something wrong after upgrading Oracle Database to 19.15 on RedHat. Apparently The amount of RAM consumed by DBWR and process servers became really impressive – as a proof I got screenshots looking like this:

They said that before the migration it never looked like this and asked for explanation, so Radek Kut and I started to investigate the mysterious case of memory leak in Oracle 19.15. 😉

Before you go any further with this post, please get familiar with this document by Frits Hoogland:

https://fritshoogland.files.wordpress.com/2018/02/linux-memory-explained.pdf

Ok, now that you know what is going on with memory on Linux level and you understand what RSS memory is, let’s look at the screenshot once again – a column RES, actually means RSS. So it looks like that database writer processes are using more than 17G of RAM, but before migration they used not more than 200M!

Funny isn’t it?

If you read Frits’s article carefully you probable understand by now that RSS is not a statistic that actually tells us how much memory is consumed by a process and you correctly suspect that there is something wrong with those numbers.

After some investigation it has occurred that only one thing changed during an upgrade – previously the whole SGA was allocated in huge pages and now the database is using regular ones.

Let’s try to understand the difference.

First of all I want to be sure that my Oracle instance won’t be using large pages at all:

1SQL> alter system
2  2  set use_large_pages=FALSE
3  3  scope=spfile;
4 
5System altered.
6 
7SQL> startup force
8ORACLE instance started.
9 
10Total System Global Area 2147482384 bytes
11Fixed Size          9136912 bytes
12Variable Size         838860800 bytes
13Database Buffers     1291845632 bytes
14Redo Buffers            7639040 bytes
15Database mounted.
16Database opened.
17SQL>

Let’s start a new session:

1SQL> show con_name
2 
3CON_NAME
4------------------------------
5ORCLPDB1
6SQL> show user
7USER is "HR"

Now I will check SPID of my process server:

1SQL> select spid
2  2  from v$process p, v$session s
3  3  where p.addr = s.paddr
4  4  and   s.username='HR';
5 
6SPID
7------------------------
875149
9 
10SQL>

When we have SPID, we can check the RSS counters, related to Oracle shared memory allocated in operating system for SGA:

1[oracle@dbVictim ~]$ cat /proc/75149/smaps | grep SYSV000 -A 10 | grep Rss
2Rss:                   0 kB
3Rss:                 488 kB
4Rss:               17280 kB
5Rss:                  12 kB

Great, keep that in mind and run a query on some "big" table in our HR session:

1SQL> select count(distinct dbms_rowid.rowid_block_number(rowid)) as cnt
2  2  from hr.employees2;
3 
4       CNT
5----------
6     35839

35 839 blocks is ~ 280MB. Now let’s compare RSS counters for our process:

1[oracle@dbVictim ~]$ cat /proc/75149/smaps | grep SYSV000 -A 10  | grep Rss
2Rss:                   0 kB
3Rss:                 516 kB
4Rss:              336624 kB
5Rss:                  12 kB

Awesome! So Reading from buffer cache is accumulated in RSS statistics! But what if I turn on huge pages?

1SQL> alter system
2  2  set use_large_pages=only
3  3  scope=spfile;
4 
5System altered.
6 
7SQL> startup force
8ORACLE instance started.
9 
10Total System Global Area 2147482384 bytes
11Fixed Size          9136912 bytes
12Variable Size         838860800 bytes
13Database Buffers     1291845632 bytes
14Redo Buffers            7639040 bytes
15Database mounted.
16Database opened.

What would be shown in my Rss counters for session that issues the above SQL on EMPLOYEES2?

1[oracle@dbVictim ~]$ cat /proc/75666/smaps | grep SYSV000 -A 10  | grep Rss
2Rss:                   0 kB
3Rss:                   0 kB
4Rss:                   0 kB

Nothing, nada, nic!

The same for DBWR on any other Oracle RDBMS process!

1[oracle@dbVictim ~]$ cat /proc/`pidof ora_dbw0_ORCLCDB`/smaps | grep SYSV000 -A 10  | grep Rss
2Rss:                   0 kB
3Rss:                   0 kB
4Rss:                   0 kB

And this is not an Oracle-related "problem". Look at this simple code in C that can simulate the similar situation:

1#include <fcntl.h>
2#include <stdio.h>
3#include <stdlib.h>
4#include <unistd.h>
5#include <wait.h>
6#include <string.h>
7#include <sys/stat.h>
8#include <sys/shm.h>
9 
10enum {SEGMENT_SIZE = 0xA00000};
11const char *data = "dupa";
12 
13int main(int argc, char *argv[]) {
14 
15    int status;
16    int segment_id;
17    int huge_small;
18    printf("Hugepages (1) or regular pages (2):  ");
19    scanf("%d", &huge_small);
20    if(huge_small == 1) {
21        segment_id = shmget (0x42b33125, SEGMENT_SIZE,
22                IPC_CREAT | S_IRUSR | S_IWUSR | SHM_HUGETLB);
23    }
24    else {
25       segment_id = shmget (0x42b33125, SEGMENT_SIZE,
26                                IPC_CREAT | S_IRUSR | S_IWUSR);
27    }
28    char *sh_mem = (char *) shmat(segment_id, NULL, 0);
29 
30    printf("Segment ID %d\n", segment_id);
31    printf("Attached at %p\n\n\n", sh_mem);
32    int write_read;
33    while(1) {
34        printf("Want to write (1), read (2) or exit (0):  ");
35        scanf("%d", &write_read);
36        if(write_read == 1) {
37        int kb;
38        printf("How many kb to write:  ");
39        scanf("%d", &kb);
40        for(int i = 0; i < kb*256; i++) {
41            memmove(sh_mem+strlen(sh_mem), data, strlen(data)+1);
42        }
43        } else if (write_read == 2) {
44        printf("Read %d bytes\n", strlen(sh_mem));
45        } else if (write_read == 0) {
46        break;
47       }
48    }
49 
50    shmdt(sh_mem);
51    shmctl(segment_id, IPC_RMID, 0);
52    exit(EXIT_SUCCESS);
53}

Let’s see what will happen when we use it with regular – small pages:

In the first session I will write 10kb into shared memory:

1[root@dbVictim ~]# ./shared_memory
2Hugepages (1) or regular pages (2):  2
3Segment ID 48
4Attached at 0x7fb6585df000
5 
6 
7Want to write (1), read (2) or exit (0):  1
8How many kb to write:  10
9Want to write (1), read (2) or exit (0):

In the second one I will ask my program to read it:

1[root@dbVictim ~]# ./shared_memory
2Hugepages (1) or regular pages (2):  2
3Segment ID 48
4Attached at 0x7f33c0b5f000
5 
6 
7Want to write (1), read (2) or exit (0):  2
8Read 10240 bytes
9Want to write (1), read (2) or exit (0):

And those are the Rss counters for my second program which read the data from a shared memory segment:

`1`	`[root@dbVictim ~]`# cat /proc/`pidof -s shared_memory`/smaps \| grep 42b33125 -A 22 \| grep Rss

`2`	`Rss: 12 kB`

Great, let’s increase to 100kb:

1[root@dbVictim ~]# ./shared_memory
2Hugepages (1) or regular pages (2):  2
3Segment ID 50
4Attached at 0x7efd3aabd000
5 
6 
7Want to write (1), read (2) or exit (0):  1
8How many kb to write:  100

And read it:

1[root@dbVictim ~]# ./shared_memory
2Hugepages (1) or regular pages (2):  2
3Segment ID 50
4Attached at 0x7f520f835000
5 
6 
7Want to write (1), read (2) or exit (0):  2
8Read 102400 bytes
9Want to write (1), read (2) or exit (0):

What do counters say?

`1`	`[root@dbVictim ~]`# cat /proc/`pidof -s shared_memory`/smaps \| grep 42b33125 -A 22 \| grep Rss

`2`	`Rss: 104 kB`

Awesome! So now let’s repeat the test with Huge Pages but we will increase the size to 4MB!

Write:

1[root@dbVictim ~]# ./shared_memory
2Hugepages (1) or regular pages (2):  1
3Segment ID 52
4Attached at 0x7fbd24a00000
5 
6 
7Want to write (1), read (2) or exit (0):  1
8How many kb to write:  4096
9Want to write (1), read (2) or exit (0):

Read:

1[root@dbVictim ~]# ./shared_memory
2Hugepages (1) or regular pages (2):  1
3Segment ID 52
4Attached at 0x7fa3cd200000
5 
6 
7Want to write (1), read (2) or exit (0):  2
8Read 4194304 bytes
9Want to write (1), read (2) or exit (0):

And Rss?

1[root@dbVictim ~]# cat /proc/`pidof -s shared_memory`/smaps  | grep 42b33125 -A 22
27fa3cd200000-7fa3cdc00000 rw-s 00000000 00:0f 52                         /SYSV42b33125 (deleted)
3Size:              10240 kB
4KernelPageSize:     2048 kB
5MMUPageSize:        2048 kB
6Rss:                   0 kB
7Pss:                   0 kB
8Shared_Clean:          0 kB
9Shared_Dirty:          0 kB
10Private_Clean:         0 kB
11Private_Dirty:         0 kB
12Referenced:            0 kB
13Anonymous:             0 kB
14LazyFree:              0 kB
15AnonHugePages:         0 kB
16ShmemPmdMapped:        0 kB
17FilePmdMapped:        0 kB
18Shared_Hugetlb:     6144 kB
19Private_Hugetlb:       0 kB
20Swap:                  0 kB
21SwapPss:               0 kB
22Locked:                0 kB
23THPeligible:        0
24VmFlags: rd wr sh mr mw me ms de ht sd

Nothing! No Rss, no Pss – all clear! 🙂

So what is the conclusion? There was no problem at all! The only problem there was, was a problem with interpreting the statistics 🙂

And why they noticed the problem? Well, it was not the database version issue as you may suspect by now – they started a test database on that server, before the production database instance – and it consumed most of the huge pages, thus forcing production instance to use regular ones which caused a panic because of a high memory consumption by Oracle processes.

So next time when you say to a consultant "nothing changed" think twice – because in this situation thee change was the order of starting 2 instances of Oracle RDBMS on the same server.

RSS memory, Oracle and Hugepages

Search:

RICO3 – as a tool for thief

JAS-MIN

When security drives you crazy! (CBFUK)

How checkpoint queue actually works (the basic scenario)

[SOLVED!] ORA-39405: Oracle Data Pump does not support importing from a source database with TSTZ

Binary patching/hacking VM process

Last resort data recovery

Hacking database writer

Flashback query and cursor: pin S wait on X

Blockrecover in Standard Edition? Nope, but close enough.

How to change root password of running VM