May 18 14:02:19 oldweb auditd[1796]: Audit daemon rotating log files
May 18 14:12:22 oldweb kernel: INFO: task nginx:2829 blocked for more than 120 seconds.
May 18 14:12:22 oldweb kernel: Not tainted 2.6.32-573.el6.x86_64 #1
May 18 14:12:22 oldweb kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 18 14:12:22 oldweb kernel: nginx D 0000000000000000 0 2829 2780 0x00000080
May 18 14:12:22 oldweb kernel: ffff8804688a7e38 0000000000000082 ffff8804688a7dc8 ffff8804688a7eb8
May 18 14:12:22 oldweb kernel: ffff8804688a7de8 ffff8804688a7dc8 ffff880910840dc0 ffff8804688a7eb8
May 18 14:12:22 oldweb kernel: ffff8804688a7f48 000000007fffefff ffff8804774405f8 ffff8804688a7fd8
May 18 14:12:22 oldweb kernel: Call Trace:
May 18 14:12:22 oldweb kernel: [<ffffffff8153b2a5>] rwsem_down_failed_common+0x95/0x1d0
May 18 14:12:22 oldweb kernel: [<ffffffff8153b403>] rwsem_down_write_failed+0x23/0x30
May 18 14:12:22 oldweb kernel: [<ffffffff8129d2a3>] call_rwsem_down_write_failed+0x13/0x20
May 18 14:12:22 oldweb kernel: [<ffffffff8153a902>] ? down_write+0x32/0x40
May 18 14:12:22 oldweb kernel: [<ffffffff81158da0>] sys_brk+0x40/0x130
May 18 14:12:22 oldweb kernel: [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
May 18 14:14:22 oldweb kernel: INFO: task nginx:2829 blocked for more than 120 seconds.
May 18 14:14:22 oldweb kernel: Not tainted 2.6.32-573.el6.x86_64 #1
May 18 14:14:22 oldweb kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 18 14:14:22 oldweb kernel: nginx D 0000000000000000 0 2829 2780 0x00000080
May 18 14:14:22 oldweb kernel: ffff8804688a7e38 0000000000000082 ffff8804688a7dc8 ffff8804688a7eb8
May 18 14:14:22 oldweb kernel: ffff8804688a7de8 ffff8804688a7dc8 ffff880910840dc0 ffff8804688a7eb8
May 18 14:14:22 oldweb kernel: ffff8804688a7f48 000000007fffefff ffff8804774405f8 ffff8804688a7fd8
May 18 14:14:22 oldweb kernel: Call Trace:
May 18 14:14:22 oldweb kernel: [<ffffffff8153b2a5>] rwsem_down_failed_common+0x95/0x1d0
May 18 14:14:22 oldweb kernel: [<ffffffff8153b403>] rwsem_down_write_failed+0x23/0x30
May 18 14:14:22 oldweb kernel: [<ffffffff8129d2a3>] call_rwsem_down_write_failed+0x13/0x20
May 18 14:14:22 oldweb kernel: [<ffffffff8153a902>] ? down_write+0x32/0x40
May 18 14:14:22 oldweb kernel: [<ffffffff81158da0>] sys_brk+0x40/0x130
May 18 14:14:22 oldweb kernel: [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
May 18 14:36:22 oldweb kernel: INFO: task memcached:2871 blocked for more than 120 seconds.
May 18 14:36:22 oldweb kernel: Not tainted 2.6.32-573.el6.x86_64 #1
May 18 14:36:22 oldweb kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 18 14:36:22 oldweb kernel: memcached D 000000000000000b 0 2871 1 0x00000080
May 18 14:36:22 oldweb kernel: ffff880c7ab8fe18 0000000000000086 0000000000000001 0000000000000000
May 18 14:36:22 oldweb kernel: 0000000000000000 ffff880a00000040 ffff880c7ab8fdc8 0000000000000000
May 18 14:36:22 oldweb kernel: ffff880c7ab8fde8 ffff880c7ab8fdc8 ffff880c7a7afad8 ffff880c7ab8ffd8
May 18 14:36:22 oldweb kernel: Call Trace:
May 18 14:36:22 oldweb kernel: [<ffffffff811da550>] ? ep_send_events_proc+0x0/0x110
May 18 14:36:22 oldweb kernel: [<ffffffff8153b2a5>] rwsem_down_failed_common+0x95/0x1d0
May 18 14:36:22 oldweb kernel: [<ffffffff810a14b0>] ? autoremove_wake_function+0x0/0x40
May 18 14:36:22 oldweb kernel: [<ffffffff8153b403>] rwsem_down_write_failed+0x23/0x30
May 18 14:36:22 oldweb kernel: [<ffffffff8129d2a3>] call_rwsem_down_write_failed+0x13/0x20
May 18 14:36:22 oldweb kernel: [<ffffffff8153a902>] ? down_write+0x32/0x40
May 18 14:36:22 oldweb kernel: [<ffffffff8115a2d6>] sys_mprotect+0xe6/0x250
May 18 14:36:22 oldweb kernel: [<ffffffff810e88ae>] ? __audit_syscall_exit+0x25e/0x290
May 18 14:36:22 oldweb kernel: [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
May 18 14:52:43 oldweb auditd[1796]: Audit daemon rotating log files
May 18 15:01:06 oldweb kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
May 18 15:01:15 oldweb kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
May 18 15:36:21 oldweb auditd[1796]: Audit daemon rotating log files
解决来源https://bugs.centos.org/view.php?id=5716
看起来是由于一个内核级别的错误引起的。仔细看了下，所有的错误都是先由nginx进程挂起引起的。网上查看资料是由于内核锁造成的。
系统处于一个假死的状态，对于内核我也不熟悉，网上查查资料先，
然后今天运行了如下命令。再观察看看了
echo no > /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
