查看问题详情

编号项目分类查看权限最后更新
0000486Anolis OS 7Generalpublic2021-11-18 14:36
报告员wb-wpp899309 分派给yunqi-zwt  
优先级normal严重性minor出现频率always
状态 resolved处理状况open 
平台x86_64操作系统Anolis OS操作系统版本7
标题0000486: [Anolis OS 7.7] [x86_64]crashkernel为默认配置,生成的vmcore通过crash工具无法正常解析
描述[缺陷描述]:
 crashkernel=0M-2G:0M,2G-8G:192M,8G-:256M为默认配置,触发异常,可正常生成vmcore,但无法通过crash工具正常解析vmcore;

kdump实际预留内存为192M:
29000000-34ffffff : Crash kernel

crash、kexec-tools版本:
# rpm -qa |grep -E "crash|kexec-tools"
kexec-tools-2.0.15-33.an7.x86_64
crash-7.2.3-10.an7.x86_64

1、解析有报错
镜像:anolisos_7_7_x64_20G_anck_alibase_20211105.vhd
      实例:ecs.g7a.large、ecs.d2s.5xlarge

详细错误日志如下:
# crash vmcore /usr/lib/debug/usr/lib/modules/4.19.91-24.8.an7.x86_64/vmlinux

crash 7.2.3-10.an7
Copyright (C) 2002-2017 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [512MB]: patching 96191 gdb minimal_symbol values

please wait... (gathering module symbol data)
crash: invalid size request: -1067897492 type: "module gpl symbol strings"
   buf_1K_used: 468
   buf_2K_used: 1
   buf_4K_used: 1
   buf_8K_used: 0
  buf_32K_used: 5
    buf_1K_ovf: 0
    buf_2K_ovf: 0
    buf_4K_ovf: 0
    buf_8K_ovf: 0
   buf_32K_ovf: 0
 buf_1K_maxuse: 3 of 10
 buf_2K_maxuse: 1 of 10
 buf_4K_maxuse: 1 of 5
 buf_8K_maxuse: 0 of 5
buf_32K_maxuse: 1 of 1
  buf_inuse[5]: [3][0][0][0][0]
      smallest: -1067897492
       largest: 1769526533884
      embedded: 3
  max_embedded: 3
       mallocs: 0
         frees: 0
    reqs/total: 476/1768458861196
  average size: 3715249708

crash: cannot allocate any more memory!

2、输入解析命令无反应 crash vmcore vmlinux
镜像:anolisos_7_7_x64_20G_anck_alibase_20211105.vhd
      实例:ecs.g7a.32xlarge、ecs.d1.2xlarge
# crash vmcore /usr/lib/debug/usr/lib/modules/4.19.91-24.8.an7.x86_64/vmlinux


[重现概率]
必现

[重现环境]
内核:
# uname -r
4.19.91-24.8.an7.x86_64

# cat /etc/os-release
NAME="Anolis OS"
VERSION="7.7"
ID="anolis"
ID_LIKE="rhel fedora centos"
VERSION_ID="7.7"
PRETTY_NAME="Anolis OS 7.7"
ANSI_COLOR="0;31"
HOME_URL="https://openanolis.cn/"
BUG_REPORT_URL="https://bugs.openanolis.cn/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

内存:
# free -h
                      total used free shared buff/cache available
Mem: 7.4G 196M 4.0G 584K 3.2G 6.9G
Swap: 0B 0B 0B

# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 2
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 25
Model: 1
Model name: AMD EPYC 7T83 64-Core Processor
Stepping: 1
CPU MHz: 2545.216
BogoMIPS: 5090.43
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 32768K
NUMA node0 CPU(s): 0,1
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext invpcid_single vmmcall tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr rdpru wbnoinvd arat vaes vpclmulqdq rdpid fsrm


[期望结果]:
触发异常后echo c >/proc/sysrq-trigger,系统可正常生成vmcore,并且通过crash工具能正常解析

[实际结果]:
触发异常后echo c >/proc/sysrq-trigger,系统可正常生成vmcore,crash工具无法正常解析vmcore
问题重现步骤1、安装依赖包
 yum -y install crash kexec-tools
 systemctl restart kdump
需要安装配套的kernel-debuginfo和kernel-debuginfo-common包
手动配置yum源安装
[plus-debug]
name=plus-debug
baseurl=http://mirrors.openanolis.cn/anolis/7.7/Plus/x86_64/debug/
enabled=1

yum install kernel-debuginfo

2、触发异常
echo c >/proc/sysrq-trigger
3、检查系统是否正常启动,是否有vmcor生成,crash解析vmcore是否正常
标签没加标签.

活动

anolislw

2021-11-08 12:07

报告者   ~0000620

镜像:anolisos_7_7_x64_20G_anck_alibase_20211105.vhd
实例:ecs.g6.large,ecs.g6.26xlarge,ecs.g6a.large,ecs.g6a.32xlarge,ecs.g7.large,ecs.g7.32xlarge
存在相同问题

kangwen429

2021-11-10 13:53

报告者   ~0000653

镜像:anolisos_7_7_x64_20G_anck_alibase_20211105.vhd
实例规格:ecs.i1.xlarge、ecs.i2.xlarge、ecs.ebmg6.26xlarge、ecs.ebmhfg7.48xlarge也存在相同问题:crash无法解析vmcore。

yunqi-zwt

2021-11-16 22:25

开发人员   ~0000721

建议发布 crash-7.2.8 来修复该问题
image.png (736,041 字节)

wb-zh951434

2021-11-17 19:10

开发人员   ~0000743

实例镜像:
anolisos_7_7_x64_20G_anck_alibase_20211105.vhd
anolisos_7_7_arm64_20G_anck_alibase_20211105.vhd

实例规格:
x86_64 ecs.g6.large
aarch64 ecs.g6r.large

解决方案:
更新发布高版本软件包:
crash-7.2.8-3.an7
kexec-tools-2.0.15-51.1.an7.3

验证流程:
1. 安装新版本crash与kexec-tools软件包
2. systemctl restart kdump
3. 手动配置yum源并安装配套的kernel-debuginfo和kernel-debuginfo-common包
4. echo c >/proc/sysrq-trigger触发异常
5. 待重新连接后,通过crash对vmcore进行解析且解析正常

验证结果见图
image-2.png (97,465 字节)   
image-2.png (97,465 字节)   
image-3.png (76,033 字节)   
image-3.png (76,033 字节)   
image-4.png (95,639 字节)   
image-4.png (95,639 字节)   
image-5.png (99,433 字节)   
image-5.png (99,433 字节)   
image-6.png (425,131 字节)
image-7.png (92,638 字节)   
image-7.png (92,638 字节)   

问题历史

日期 用户名 字段 更改
2021-11-08 11:48 wb-wpp899309 新建问题
2021-11-08 11:50 wb-wpp899309 描述已修改
2021-11-08 12:07 anolislw 注释已添加: 0000620
2021-11-08 18:20 wb-wpp899309 描述已修改
2021-11-08 18:22 wb-wpp899309 描述已修改
2021-11-10 13:53 kangwen429 注释已添加: 0000653
2021-11-12 15:51 jacobwang 分派给 => yunqi-zwt
2021-11-12 15:51 jacobwang 状态 新建 => 已分配
2021-11-16 22:25 yunqi-zwt 注释已添加: 0000721
2021-11-16 22:25 yunqi-zwt 添加了以下文件:: image.png
2021-11-17 19:10 wb-zh951434 注释已添加: 0000743
2021-11-17 19:10 wb-zh951434 添加了以下文件:: image-2.png
2021-11-17 19:10 wb-zh951434 添加了以下文件:: image-3.png
2021-11-17 19:10 wb-zh951434 添加了以下文件:: image-4.png
2021-11-17 19:10 wb-zh951434 添加了以下文件:: image-5.png
2021-11-17 19:10 wb-zh951434 添加了以下文件:: image-6.png
2021-11-17 19:10 wb-zh951434 添加了以下文件:: image-7.png
2021-11-18 14:36 wb-zmy745940 状态 已分配 => 已解决