[Babase] Tweaking papio so it does not crash
Karl O. Pinc
kop at meme.com
Wed May 9 12:17:29 EDT 2007
Hi Hunter,
The PostgreSQL server, and some of the Apache children
crashed a couple of times last night. Logs are
supplied at bottom.
What I think is happening is that we're running out of
swap. This is happening because I've configured PostgreSQL
to use a tad less than 1GB of shared memory, and I'm not
sure but I believe that shared memory must be backed by
swap. (At least until kernel 2.6.15, depending on the
shmget call.)
According to my rough calculations papio has about 5%
less than 1GB of swap.
I've cut the amount of shared memory PostgreSQL uses by
about 10%. This should keep us from crashing
in the future but it'd be nice to use that extra memory.
Could you please give papio 1GB (or probably more like
1.5G so there's extra to go around for regular usage)
swap?
I realize this is a grody task. So, while you're at it
I'd also like to increase the size of / by about 30-50%.
We should be able to get the extra from /disk, yes?
Please comment (and cc the babase at eeblistserv.princeton.edu
list). There may be better solutions to get that extra 10%,
probably involving spending money. Like buying another drive
or another computer. Or maybe we should not worry about it
and live with it, we may not see a noticeable performance
hit.
(If you can't cc babase at eeblistserv.princeton.edu please
let me know. Non-subscribers are supposed to be able to
post to the list but I've not tried it and we've had the
list reconfigured out from under us in the past.)
Karl <kop at meme.com>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein
------------</var/log/database.log>------------------
May 9 09:38:36 papio postgres[25379]: [4-1] LOG: server process (PID
31047) was terminated by signal 9
May 9 09:38:36 papio postgres[25379]: [5-1] LOG: terminating any
other active
server processes
May 9 09:38:36 papio postgres[30881]: [4-1] WARNING: terminating
connection because of crash of another server process
May 9 09:38:36 papio postgres[30881]: [4-2] DETAIL: The postmaster
has commanded this server process to roll back the current transaction
and exit, because another server
May 9 09:38:36 papio postgres[30881]: [4-3] process exited abnormally
and possibly corrupted shared memory.
May 9 09:38:36 papio postgres[30881]: [4-4] HINT: In a moment you
should be able to reconnect to the database and repeat your command.
May 9 09:38:36 papio postgres[25379]: [6-1] LOG: all server processes
terminated; reinitializing
May 9 09:38:36 papio postgres[25379]: [7-1] FATAL: could not create
shared memory segment: Cannot allocate memory
May 9 09:38:37 papio postgres[25379]: [7-2] DETAIL: Failed system
call was shmget(key=5432001, size=1042948096, 03600).
May 9 09:38:37 papio postgres[25379]: [7-3] HINT: This error usually
means that PostgreSQL's request for a shared memory segment exceeded
available memory or
swap space.
May 9 09:38:37 papio postgres[25379]: [7-4] To reduce the request
size (currently 1042948096 bytes), reduce PostgreSQL's shared_buffers
parameter (currently 124928) and/or
May 9 09:38:37 papio postgres[25379]: [7-5] its max_connections
parameter (currently 100).
May 9 09:38:37 papio postgres[25379]: [7-6] The PostgreSQL
documentation contains more information about shared memory
configuration.
May 9 09:38:37 papio postgres[25381]: [1-1] LOG: logger shutting down
------------------<dmesg>--------------------
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
Free pages: 2592kB (1664kB HighMem)
Active:279639 inactive:228868 dirty:0 writeback:0 unstable:0 free:648
slab:4121 mapped:483255 pagetables:1818
DMA free:16kB min:16kB low:32kB high:48kB active:6028kB inactive:6096kB
present:16384kB pages_scanned:13134 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:912kB min:936kB low:1872kB high:2808kB active:425820kB
inactive:426904kB present:901120kB pages_scanned:939543
all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:1664kB min:512kB low:1024kB high:1536kB active:686708kB
inactive:482472kB present:1179392kB pages_scanned:0 all_unreclaimable?
no
protections[]: 0 0 0
DMA: 0*4kB 2*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 16kB
Normal: 0*4kB 112*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 912kB
HighMem: 46*4kB 27*8kB 47*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB
0*1024kB 0*2048kB 0*4096kB = 1664kB
Swap cache: add 2242973, delete 2218821, find 9745435/9845659, race 0+0
0 bounce buffer pages
Free swap: 0kB
524224 pages of RAM
294848 pages of HIGHMEM
5278 reserved pages
84145 pages shared
24152 pages swap cached
Out of Memory: Killed process 30915 (httpd).
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
Free pages: 2504kB (1600kB HighMem)
Active:277463 inactive:231121 dirty:0 writeback:0 unstable:0 free:626
slab:4109 mapped:483248 pagetables:1818
DMA free:16kB min:16kB low:32kB high:48kB active:6024kB inactive:6100kB
present:16384kB pages_scanned:14355 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:888kB min:936kB low:1872kB high:2808kB active:416920kB
inactive:436016kB present:901120kB pages_scanned:996336
all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:1600kB min:512kB low:1024kB high:1536kB active:686908kB
inactive:482368kB present:1179392kB pages_scanned:0 all_unreclaimable?
no
protections[]: 0 0 0
DMA: 0*4kB 2*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 16kB
Normal: 0*4kB 109*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 888kB
HighMem: 6*4kB 29*8kB 52*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB
0*1024kB 0*2048kB 0*4096kB = 1600kB
Swap cache: add 2243034, delete 2218879, find 9745442/9845672, race 0+0
0 bounce buffer pages
Free swap: 0kB
524224 pages of RAM
294848 pages of HIGHMEM
5278 reserved pages
84079 pages shared
24155 pages swap cached
Out of Memory: Killed process 10510 (postmaster).
loop: loaded (max 8 devices)
RPC: bad TCP reclen 0x002e0100 (large)
RPC: bad TCP reclen 0x16030100 (non-terminal)
RPC: bad TCP reclen 0x16030000 (non-terminal)
RPC: bad TCP reclen 0x47455420 (non-terminal)
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
Free pages: 2520kB (1600kB HighMem)
Active:308063 inactive:201678 dirty:0 writeback:0 unstable:0 free:630
slab:3323 mapped:500245 pagetables:1470
DMA free:16kB min:16kB low:32kB high:48kB active:5960kB inactive:6276kB
present:16384kB pages_scanned:12540 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:904kB min:936kB low:1872kB high:2808kB active:432008kB
inactive:424584kB present:901120kB pages_scanned:956637
all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:1600kB min:512kB low:1024kB high:1536kB active:794284kB
inactive:375852kB present:1179392kB pages_scanned:0 all_unreclaimable?
no
protections[]: 0 0 0
DMA: 0*4kB 2*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 16kB
Normal: 88*4kB 9*8kB 26*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 904kB
HighMem: 200*4kB 14*8kB 11*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB
0*1024kB 0*2048kB 0*4096kB = 1600kB
Swap cache: add 3563570, delete 3554793, find 21054699/21196720, race
0+0
0 bounce buffer pages
Free swap: 0kB
524224 pages of RAM
294848 pages of HIGHMEM
5278 reserved pages
19547 pages shared
8777 pages swap cached
Out of Memory: Killed process 1578 (httpd).
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
Free pages: 2528kB (1600kB HighMem)
Active:327338 inactive:182387 dirty:0 writeback:0 unstable:0 free:632
slab:3317 mapped:500261 pagetables:1470
DMA free:16kB min:16kB low:32kB high:48kB active:5960kB inactive:6276kB
present:16384kB pages_scanned:14355 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:912kB min:936kB low:1872kB high:2808kB active:509296kB
inactive:347236kB present:901120kB pages_scanned:1343595
all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:1600kB min:512kB low:1024kB high:1536kB active:794096kB
inactive:376036kB present:1179392kB pages_scanned:0 all_unreclaimable?
no
protections[]: 0 0 0
DMA: 0*4kB 2*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 16kB
Normal: 104*4kB 2*8kB 26*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 912kB
HighMem: 184*4kB 16*8kB 14*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB
0*1024kB 0*2048kB 0*4096kB = 1600kB
Swap cache: add 3563641, delete 3554893, find 21054711/21196740, race
0+0
0 bounce buffer pages
Free swap: 0kB
524224 pages of RAM
294848 pages of HIGHMEM
5278 reserved pages
19574 pages shared
8748 pages swap cached
Out of Memory: Killed process 17455 (postmaster).
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
Free pages: 1592kB (640kB HighMem)
Active:255571 inactive:254960 dirty:0 writeback:0 unstable:0 free:398
slab:2943 mapped:506660 pagetables:1526
DMA free:16kB min:16kB low:32kB high:48kB active:6328kB inactive:6100kB
present:16384kB pages_scanned:13596 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:936kB min:936kB low:1872kB high:2808kB active:427896kB
inactive:430256kB present:901120kB pages_scanned:1307658
all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:640kB min:512kB low:1024kB high:1536kB active:588060kB
inactive:583484kB present:1179392kB pages_scanned:13266
all_unreclaimable? no
protections[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 16kB
Normal: 0*4kB 43*8kB 37*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 936kB
HighMem: 14*4kB 3*8kB 3*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB
0*1024kB 0*2048kB 0*4096kB = 640kB
Swap cache: add 5246648, delete 5246113, find 21582175/21827730, race
6+8
0 bounce buffer pages
Free swap: 0kB
524224 pages of RAM
294848 pages of HIGHMEM
5278 reserved pages
12539 pages shared
535 pages swap cached
Out of Memory: Killed process 17595 (httpd).
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
Free pages: 2592kB (1664kB HighMem)
Active:270314 inactive:239963 dirty:0 writeback:0 unstable:0 free:648
slab:2947 mapped:502454 pagetables:1525
DMA free:16kB min:16kB low:32kB high:48kB active:6400kB inactive:6032kB
present:16384kB pages_scanned:12672 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:912kB min:936kB low:1872kB high:2808kB active:432972kB
inactive:425332kB present:901120kB pages_scanned:965118
all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:1664kB min:512kB low:1024kB high:1536kB active:641884kB
inactive:528488kB present:1179392kB pages_scanned:0 all_unreclaimable?
no
protections[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 16kB
Normal: 16*4kB 0*8kB 1*16kB 0*32kB 13*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 912kB
HighMem: 284*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB
0*1024kB 0*2048kB 0*4096kB = 1664kB
Swap cache: add 5324818, delete 5318454, find 21590284/21838221, race
6+8
0 bounce buffer pages
Free swap: 0kB
524224 pages of RAM
294848 pages of HIGHMEM
5278 reserved pages
19328 pages shared
6364 pages swap cached
Out of Memory: Killed process 22462 (postmaster).
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
Free pages: 2608kB (1664kB HighMem)
Active:263704 inactive:246580 dirty:0 writeback:0 unstable:0 free:652
slab:2926 mapped:502458 pagetables:1525
DMA free:16kB min:16kB low:32kB high:48kB active:6400kB inactive:6032kB
present:16384kB pages_scanned:13926 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:928kB min:936kB low:1872kB high:2808kB active:406500kB
inactive:451800kB present:901120kB pages_scanned:2001978
all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:1664kB min:512kB low:1024kB high:1536kB active:641916kB
inactive:528488kB present:1179392kB pages_scanned:0 all_unreclaimable?
no
protections[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 16kB
Normal: 16*4kB 2*8kB 1*16kB 0*32kB 13*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 928kB
HighMem: 284*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB
0*1024kB 0*2048kB 0*4096kB = 1664kB
Swap cache: add 5324827, delete 5318456, find 21590284/21838222, race
6+8
0 bounce buffer pages
Free swap: 0kB
524224 pages of RAM
294848 pages of HIGHMEM
5278 reserved pages
19333 pages shared
6371 pages swap cached
Out of Memory: Killed process 17485 (postmaster).
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
Free pages: 2544kB (1600kB HighMem)
Active:360594 inactive:149545 dirty:0 writeback:0 unstable:0 free:636
slab:3122 mapped:462555 pagetables:1512
DMA free:16kB min:16kB low:32kB high:48kB active:6204kB inactive:6200kB
present:16384kB pages_scanned:12969 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:928kB min:936kB low:1872kB high:2808kB active:421280kB
inactive:435848kB present:901120kB pages_scanned:1353264
all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:1600kB min:512kB low:1024kB high:1536kB active:1014892kB
inactive:156132kB present:1179392kB pages_scanned:0 all_unreclaimable?
no
protections[]: 0 0 0
DMA: 0*4kB 2*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 16kB
Normal: 0*4kB 0*8kB 52*16kB 3*32kB 0*64kB 0*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 928kB
HighMem: 2*4kB 135*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB
0*1024kB 0*2048kB 0*4096kB = 1600kB
Swap cache: add 5358832, delete 5357741, find 21644003/21893866, race
6+8
0 bounce buffer pages
Free swap: 0kB
524224 pages of RAM
294848 pages of HIGHMEM
5278 reserved pages
23789 pages shared
1091 pages swap cached
Out of Memory: Killed process 24401 (postmaster).
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
Free pages: 2520kB (1600kB HighMem)
Active:304963 inactive:204706 dirty:0 writeback:0 unstable:0 free:630
slab:3597 mapped:259929 pagetables:1264
DMA free:16kB min:16kB low:32kB high:48kB active:8052kB inactive:4324kB
present:16384kB pages_scanned:14685 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:904kB min:936kB low:1872kB high:2808kB active:445928kB
inactive:409448kB present:901120kB pages_scanned:958221
all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:1600kB min:512kB low:1024kB high:1536kB active:765872kB
inactive:405052kB present:1179392kB pages_scanned:0 all_unreclaimable?
no
protections[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 16kB
Normal: 0*4kB 3*8kB 7*16kB 2*32kB 1*64kB 5*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 904kB
HighMem: 186*4kB 43*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB
0*1024kB 0*2048kB 0*4096kB = 1600kB
Swap cache: add 5419125, delete 5418379, find 21789844/22044571, race
6+8
0 bounce buffer pages
Free swap: 0kB
524224 pages of RAM
294848 pages of HIGHMEM
5278 reserved pages
19510 pages shared
746 pages swap cached
Out of Memory: Killed process 30874 (postmaster).
oom-killer: gfp_mask=0xd0
Mem-info:
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
Free pages: 2528kB (1600kB HighMem)
Active:266758 inactive:243452 dirty:0 writeback:0 unstable:0 free:632
slab:3039 mapped:504621 pagetables:1528
DMA free:16kB min:16kB low:32kB high:48kB active:6688kB inactive:5740kB
present:16384kB pages_scanned:13002 all_unreclaimable? yes
protections[]: 0 0 0
Normal free:912kB min:936kB low:1872kB high:2808kB active:444532kB
inactive:413188kB present:901120kB pages_scanned:1147971
all_unreclaimable? yes
protections[]: 0 0 0
HighMem free:1600kB min:512kB low:1024kB high:1536kB active:615812kB
inactive:554880kB present:1179392kB pages_scanned:0 all_unreclaimable?
no
protections[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 16kB
Normal: 0*4kB 4*8kB 5*16kB 1*32kB 0*64kB 6*128kB 0*256kB 0*512kB
0*1024kB 0*2048kB 0*4096kB = 912kB
HighMem: 254*4kB 7*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB
0*1024kB 0*2048kB 0*4096kB = 1600kB
Swap cache: add 5620644, delete 5616932, find 21816872/22091739, race
6+8
0 bounce buffer pages
Free swap: 0kB
524224 pages of RAM
294848 pages of HIGHMEM
5278 reserved pages
21788 pages shared
3712 pages swap cached
Out of Memory: Killed process 31047 (postmaster).
More information about the Babase
mailing list