| Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback |
| Sun System Handbook - ISO 3.4 June 2011 Internal/Partner Edition | ||
|
|
||
 |
||||||||||||
Solution Type Technical Instruction Sure Solution 1006425.1 : Solaris[TM] x86, IA32, Initial Core Dump Analysis
PreviouslyPublishedAs 209009 Oracle Confidential (INTERNAL). Do not distribute to customers Reason: Migrated distribution from Sun Description Understanding the Solaris[TM] x86, IA32, core dump and performing initial analysis. Steps to Follow This document is intended to help those who are used to analysing core dumps on SPARC systems and want to learn about initial core dump analysis on IA32 (or 32bit-x86) platforms. For details on 64-bit AMD64 Solaris 10 core analysis, please refer document 87699. For Analysing core dumps on x86 we need to keep the following in mind : 1 - An x86 environment setup for analysis, 2 - adb (or MDB on Solaris[TM] 8 and above) and scat (on Solaris[TM] 10 and above) can be used, 3 - x86 architecture (registers, instructions etc) are different from SPARC! Once the right environment is available, run adb or mdb on the core files. To fo this, type the following (replace "suffix" with the suffix of the core to be analysed; so if your core files are "unix.3" and "vmcore.3" use "mdb 3" or "adb -k unix.3 vmcore.3"): $ mdb suffix or $ adb -k unix.suffix vmcore.suffix > $C f0115914 vpanic(fe8e4e18) f0115948 turnstile_block+0x21a(0, 0, e34cc8e8, fec03514, 0) f0115990 rw_enter_sleep+0x19f(e34cc8e8, 0) f01159bc ufs_scan_inodes+0xf8(0, fe909600, f01159e0, e18c66e0) f0115a34 setquota+0x3a6() f0115a6c quotactl+0xea(e196e498, 8047554, 102003, e3d0dc18) f0115ef0 ufs_ioctl+0x8d(e196e498, 30189, 8047554, 102003, e3d0dc18, f0115f7c) f0115f88 ioctl+0x15b() f0115fb4 _sys_call+0xe5() > > $? %cs = 0x0158 %eax = 0xf0115920 %ds = 0x0160 %ebx = 0xe3c402e0 %ss = 0x0160 %ecx = 0xfec9e2a8 turnstile_table+0x7ac %es = 0x0160 %edx = 0xffffff00 %fs = 0x01a8 %esi = 0xe24b10a0 %gs = 0x01b0 %edi = 0xe24b10a0 %eip = 0xfe80f46c vpanic %ebp = 0xf0115914 %esp = 0xf0115908 %eflags = 0x00000282 id=0 vip=0 vif=0 ac=0 vm=0 rf=0 nt=0 iopl=0x0 status=<of,df,IF,tf,SF,zf,af,pf,cf> %uesp = 0x00000000 %trapno = 0x0 %err = 0x0 > Note the registers %ebp and %esp, which are the frame and the stack pointers respectively. %eip is the instruction pointer. From the above register dumps : %ebp - 0xf0115914 <-- Frame Pointer %esp - 0xf0115908 <-- Stack Pointer %ebp+0x4 - return address or the instruction next to the calling instruction. - equivalent of %eip. %ebp+0x8 - list of arguments starts here Stack frames on x86 are demarcated using %ebp and %esp. %ebp, the frame pointer remains unchanged during the life of a function and marks the start of the frame and also the boundary between the called and the calling function. The %esp register, the stack pointer, changes as and when more stack space is used. Some SPARC equivalents : %esp == %sp %ebp == %fp or %i6 %ebp+0x08 == %i0 %ebp+0x0c == %i1 %ebp+0x10 == %i2 %ebp+0x14 == %i3 %ebp+0x18 == %i4 etc etc. All input variables are obtained from the stack, using the %ebp. %ebp+0x04 == %i7 (calling instruction) (or %ebp+0x4 is previous %eip) But unlike as in SPARC, %ebp+0x04 (a pointer to %eip) contains the instruction following the calling instruction. Please note this in the stack below. Lets take a look into the stack dump. > $C f0115914 vpanic(fe8e4e18) f0115948 turnstile_block+0x21a(0, 0, e34cc8e8, fec03514, 0) f0115990 rw_enter_sleep+0x19f(e34cc8e8, 0) f01159bc ufs_scan_inodes+0xf8(0, fe909600, f01159e0, e18c66e0) f0115a34 setquota+0x3a6() f0115a6c quotactl+0xea(e196e498, 8047554, 102003, e3d0dc18) f0115ef0 ufs_ioctl+0x8d(e196e498, 30189, 8047554, 102003, e3d0dc18, f0115f7c) f0115f88 ioctl+0x15b() f0115fb4 _sys_call+0xe5() > Note that the %eip is not the call instruction, but the instruction following the call instruction, for example : turnstile_block+0x215: call panic |___ actual calling Instruction turnstile_block+0x21a: addl $4,%esp |___ %eip will point here, will find this on stack. rw_enter_sleep+0x19a: call turnstile_block |___ actual calling Instruction rw_enter_sleep+0x19f: addl $0x14,%esp |___ %eip will point here, will find this on stack. So, we may need to dump the whole function to see what actually the calling instruction was, because instructions can be of different lengths. One more important point is that its always better to dump the whole function (use rw_enter_sleep::dis on mdb for example). For example, see the disassembly below : rw_enter_sleep+0x2a: orb $4,%al rw_enter_sleep+0x2c: movl %eax,-0xc(%ebp) rw_enter_sleep+0x2f: movl $-8,%ecx rw_enter_sleep+0x34: movl $3,%eax rw_enter_sleep+0x39: movl +8(%ebp),%edi rw_enter_sleep+0x3c: movl %ecx,-4(%ebp) rw_enter_sleep+0x3f: movl %eax,-8(%ebp) rw_enter_sleep+0x42: jmp +0xb <rw_enter_sleep+0x4d> Lets look an instruction address not seen above : > rw_enter_sleep+0x3d/ai rw_enter_sleep+0x3d: rw_enter_sleep+0x3d: decl %ebp The instruction rw_enter_sleep+0x3d is not part of the actual assembly dumped above, but still appears to be a valid instruction. So sometimes %eip or stack may point to an instruction which may not actually be part of the disassembled code, rw_enter_sleep+0x3d for instance. So, its always better that we dump the whole function and look for the panic instruction. Note also that x86 instructions are not necessarily 4 byte aligned as in SPARC, i.e x86 allows different instruction lengths. Lets look into some sample assemblies below taken from x86 and sparc systems, for example : On x86 : -------- rw_enter_sleep+1: movl %esp,%ebp ^^^^^^^^^^^^ rw_enter_sleep+3: subl $0x20,%esp ^^^^^^^^^^^^ rw_enter_sleep+6: pushl %ebx ^^^^^^^^^^^^ rw_enter_sleep+7: pushl %esi rw_enter_sleep+8: pushl %edi rw_enter_sleep+9: cmpl $1,+0xc(%ebp) rw_enter_sleep+0xd: jne +0x17 <rw_enter_sleep+0x24> ^^^^^^^^^^^^ rw_enter_sleep+0xf: movl $8,-0xc(%ebp) ^^^^^^^^^^^^ rw_enter_sleep+0x16: movl $6,%ecx ^^^^^^^^^^^^ rw_enter_sleep+0x1b: movl $1,%eax ^^^^^^^^^^^^ rw_enter_sleep+0x20: jmp +0x19 <rw_enter_sleep+0x39> ^^^^^^^^^^^^ On Sparc : ---------- rw_enter_sleep: save %sp, -0xb0, %sp ^^^^^^^^^^^^ rw_enter_sleep+4: sra %i1, 0, %i5 ^^^^^^^^^^^^ rw_enter_sleep+8: cmp %i5, 1 ^^^^^^^^^^^^ rw_enter_sleep+0xc: bne,pn %icc, +0x1c <rw_enter_sleep+0x28> ^^^^^^^^^^^^ rw_enter_sleep+0x10: mov %i0, %i4 ^^^^^^^^^^^^ rw_enter_sleep+0x14: mov 8, %l6 ^^^^^^^^^^^^ rw_enter_sleep+0x18: mov 6, %i0 ^^^^^^^^^^^^ rw_enter_sleep+0x1c: mov 1, %l7 ^^^^^^^^^^^^ Looking into the stack frames: ------------------------------ Frames are contained between %ebp and %esp, %ebp pointer is used to track the previous frame as %fp is used in sparc. Take the stack pointer from %esp, or even the addresses from the $C output. %esp - 0xf0115908 <-- Stack Pointer %ebp - 0xf0115914 <-- Frame Pointer > 0xf0115908,80/apn %esp -> 0xf0115908: 0xf0115908: panic+0xf <- panic instr 0xf011590c: 0xfe8e4e18 0xf0115910: 0xf0115920 %ebp -> 0xf0115914: *0xf0115948* ^^^^^^^^^^^ frame pointer, next %ebp, look for this below ... 0xf0115918: turnstile_block+0x21a ^^^^^^^^^^^^^^^^^^^^^ %eip that invoked panic 0xf011591c: 0xfe8e4e18 0xf0115920: 0xe34cc8e8 0xf0115924: 0x69fdb5b0 0xf0115928: 0 0xf011592c: 0 0xf0115930: 0 0xf0115934: 0 0xf0115938: turnstile_table+0x4f0 0xf011593c: 0x9e 0xf0115940: 0xe2feb018 0xf0115944: 0xe422a148 %ebp -> 0xf0115948: 0xf0115990 ^^^^^^^^^^ frame pointer, next %ebp, look for this below ... 0xf011594c: rw_enter_sleep+0x19f ^^^^^^^^^^^^^^ instruction that calls turnstile_block 0xf0115950: 0 0xf0115954: 0 0xf0115958: 0xe34cc8e8 0xf011595c: rw_sobj_ops 0xf0115960: 0 0xf0115964: 0xe34cc800 0xf0115968: 0xe34cc8e8 0xf011596c: 0xe36b61f0 0xf0115970: 0xffcee604 0xf0115974: 0xe24b10a4 0xf0115978: 0 0xf011597c: 4 0xf0115980: 0xffcee6fe 0xf0115984: 0xe3c402e4 0xf0115988: 3 0xf011598c: 0xfffffff8 %ebp -> 0xf0115990: 0xf01159bc ^^^^^^^^^^ frame pointer, next %ebp, look for this below ... 0xf0115994: ufs_scan_inodes+0xf8 ^^^^^^^^^^^^^^^ instruction that calls rw_enter_sleep 0xf0115998: 0xe34cc8e8 0xf011599c: 0 0xf01159a0: 0xe1a767e0 0xf01159a4: 0xe18c66e0 0xf01159a8: 0 0xf01159ac: 0 0xf01159b0: 0x30f 0xf01159b4: 0x1878 0xf01159b8: 0xe1213878 %ebp -> 0xf01159bc: 0xf0115a34 ^^^^^^^^^^ frame pointer, next %ebp, look for this below ... 0xf01159c0: setquota+0x3a6 ^^^^^^^^^^^^^^ instruction that calls ufs_scan_inodes 0xf01159c4: 0 0xf01159c8: setquota_scan_inode 0xf01159cc: 0xf01159e0 0xf01159d0: 0xe18c66e0 0xf01159d4: 0xe18c66e0 0xf01159d8: 0 0xf01159dc: 0xe18c66e0 0xf01159e0: 1 0xf01159e4: 0xe18c66e0 0xf01159e8: 0x346b 0xf01159ec: 0x4e06 0xf01159f0: 0xff2a0 0xf01159f4: 0xe1a767e0 0xf01159f8: 2 0xf01159fc: 2 0xf0115a00: 0 0xf0115a04: 0 0xf0115a08: 0 0xf0115a0c: 0 0xf0115a10: 0 0xf0115a14: 0 0xf0115a18: 0x9c0c00 0xf0115a1c: 0x68d60 0xf0115a20: 0xe196e4f8 0xf0115a24: 0xe1a76808 0xf0115a28: 0xe1a76828 0xf0115a2c: 0xe18c6708 0xf0115a30: 1 %ebp -> 0xf0115a34: 0xf0115a6c ^^^^^^^^^^ frame pointer, next %ebp, look for this below ... 0xf0115a38: quotactl+0xea ^^^^^^^^^^^^^ instruction that calls setquota 0xf0115a3c: 5 0xf0115a40: 0x346b 0xf0115a44: 0xe18c66e0 0xf0115a48: 0x80654dc 0xf0115a4c: 0xe3d0dc18 0xf0115a50: 0xe196e498 0xf0115a54: 0xe19cc000 0xf0115a58: 0xe18c66e0 0xf0115a5c: 5 0xf0115a60: 0x346b 0xf0115a64: 0x80654dc 0xf0115a68: 0xe19cc000 %ebp -> 0xf0115a6c: 0xf0115ef0 ^^^^^^^^^^ frame pointer, next %ebp, look for this below ... (this happens to be WAY below) 0xf0115a70: ufs_ioctl+0x8d ^^^^^^^^^^^^^^ instruction that calls quotactl 0xf0115a74: 0xe196e498 0xf0115a78: 0x8047554 0xf0115a7c: 0x102003 ...... ...... ...... 0xf0115af4: 0 0xf0115af8: 0 0xf0115afc: 0xf0115b30 0xf0115b00: hat_memload+0xbb 0xf0115b04: 0xe2c8e480 ...... ...... 0xf0115ee4: 0xf0115ee4: 0 0xf0115ee8: 0xf0115f04 0xf0115eec: getf+0x99 %ebp -> 0xf0115ef0: 0xf0115f88 ^^^^^^^^^^ frame pointer, next %ebp, look for this below ... 0xf0115ef4: ioctl+0x15b 0xf0115ef8: 0xe196e498 0xf0115efc: 0x30189 0xf0115f00: 0x8047554 0xf0115f04: 0x102003 0xf0115f08: 0xe3d0dc18 0xf0115f0c: 0xf0115f7c ...... ...... 0xf0115f78: open64+0x16 0xf0115f7c: 0 0xf0115f80: 0x8047554 0xf0115f84: 0x80654dc %ebp -> 0xf0115f88: 0xf0115fb4 ^^^^^^^^^^ frame pointer, next %ebp, look for this below ... 0xf0115f8c: _sys_call+0xe5 0xf0115f90: 4 0xf0115f94: 0x30189 0xf0115f98: 0x8047554 0xf0115f9c: 1 0xf0115fa0: 0xf0115fb4 0xf0115fa4: 0xf0115fb4 0xf0115fa8: _sys_call+0x120 0xf0115fac: 0x16 ...... 0xf0115fb4: 0 0xf0115fb8: 0 Whatever we traced above must match the $C output. Lets try to map the panic stack back to the memory dumped above. > $C f0115914 vpanic(fe8e4e18) f0115948 turnstile_block+0x21a(0, 0, e34cc8e8, fec03514, 0) f0115990 rw_enter_sleep+0x19f(e34cc8e8, 0) f01159bc ufs_scan_inodes+0xf8(0, fe909600, f01159e0, e18c66e0) f0115a34 setquota+0x3a6() f0115a6c quotactl+0xea(e196e498, 8047554, 102003, e3d0dc18) f0115ef0 ufs_ioctl+0x8d(e196e498, 30189, 8047554, 102003, e3d0dc18, f0115f7c) f0115f88 ioctl+0x15b() f0115fb4 _sys_call+0xe5() > Lets look at the stack to see what ufs_scan_inodes passes to rw_enter_sleep ... %ebp -> 0xf0115990: 0xf01159bc <-- next %ebp 0xf0115994: ufs_scan_inodes+0xf8 <-- %eip 0xf0115998: 0xe34cc8e8 <-- %i0, 1st Variable 0xf011599c: 0 <-- %i1, 2nd Variable 0xf01159a0: 0xe1a767e0 <-- the rest from here 0xf01159a4: 0xe18c66e0 <-- are local variable 0xf01159a8: 0 0xf01159ac: 0 0xf01159b0: 0x30f 0xf01159b4: 0x1878 0xf01159b8: 0xe1213878 Similarly lets see what setquota passes to ufs_scan_inodes .... %ebp -> 0xf01159bc: 0xf0115a34 <-- next %ebp 0xf01159c0: setquota+0x3a6 <-- %eip 0xf01159c4: 0 <-- %i0, 1st Variable 0xf01159c8: setquota_scan_inode <-- %i1, 2nd Variable 0xf01159cc: 0xf01159e0 <-- %i2, 3rd Variable 0xf01159d0: 0xe18c66e0 <-- %i3, 4th Variable 0xf01159d4: 0xe18c66e0 <-- the rest from here 0xf01159d8: 0 <-- are local variable 0xf01159dc: 0xe18c66e0 0xf01159e0: 1 0xf01159e4: 0xe18c66e0 0xf01159e8: 0x346b 0xf01159ec: 0x4e06 0xf01159f0: 0xff2a0 0xf01159f4: 0xe1a767e0 0xf01159f8: 2 0xf01159fc: 2 0xf0115a00: 0 0xf0115a04: 0 0xf0115a08: 0 0xf0115a0c: 0 0xf0115a10: 0 0xf0115a14: 0 0xf0115a18: 0x9c0c00 0xf0115a1c: 0x68d60 0xf0115a20: 0xe196e4f8 0xf0115a24: 0xe1a76808 0xf0115a28: 0xe1a76828 0xf0115a2c: 0xe18c6708 0xf0115a30: 1 Thus we can continue dumping the frames and respective stack areas of functions. The input variables can be found on the stack as no registers are used for argument passing. For further understanding you may refer to x86 architecture documents available at (though the links may change with time!) : [1]: http://www.caldera.com/developers/devspecs/abi386-4.pdf [2]: http://docs.sun.com/app/docs/doc/801-7045/ (can also be downloaded in PDF format) Product Sun Fire V65x Server Sun Fire V60x Server Solaris, x86, IA32, AMD64, Panic, Core, Dump, initial, analysis Previously Published As 78483 Change History Date: 2006-11-15 User Name: 71396 Action: Approved Comment: Performed final review of article. Updated product name and trademarking. Publishing. Version: 10 Date: 2006-11-15 User Name: 71396 Action: Accept Comment: Version: 0 Date: 2006-11-15 User Name: 112340 Action: Approved Comment: changes look fine. cleaned some links up. sending to the final review. Version: 0 Date: 2006-11-14 User Name: 89300 Action: Approved Comment: Cleaned up references to 64-bit and included references to 64bit core analysis document. Since the original author is different, it didnt allow me to save the changes till I changed the review data manually. I think it should be fine now. Here I go one more time. Version: 0 Date: 2006-11-14 User Name: 112340 Action: Rejected Comment: sending back to the reviewer per her request Version: 0 Date: 2006-11-14 User Name: 89300 Action: Approved Comment: Second attempt to send it to tech review. Version: 0 Date: 2006-11-14 User Name: 89300 Action: Rejected Comment: Included the changes/references. Sending this to the reviewer. Version: 0 Date: 2006-11-14 User Name: 89300 Action: Approved Comment: I am moving it to review stage to include the updates. Version: 0 Date: 2006-11-14 User Name: 89300 Action: Update Started Comment: Need to cleanup and link the 64bit AMD64 document (87699) to this one . Version: 0 Date: 2004-12-18 User Name: 89375 Action: Add Comment Comment: (misspellings in Document Body and potential better wording) - paragraph four, second line, needs correcting. the section currently reads: "Once the right environment is available, run adb or mdb on the core files. To fo this, type the following (replace "suffix" with the suffix of the core to..." -Another paragraph has superfluous speach; it says: "But unlike as in SPARC, %ebp+0x04 (a pointer to %eip) contains the instruction following the calling instruction. Please note this in the stack below." This *could say "But unlike SPARC, ..." Version: 0 Date: 2004-11-25 User Name: 25440 Action: Approved Comment: Excellent document! Publishing. Version: 5 Date: 2004-11-25 User Name: 25440 Action: Accept Comment: Version: 0 Date: 2004-11-24 User Name: 100346 Action: Approved Comment: Hi KE Team, please push this to sunsolve. The document is accurate, well-expressed and timely. The one thing I would request is that you add to the list of applicable platforms: - lx50 - v20z - v40z - generic x86/amd64-based PC. thanks and best regards, James C. McPherson Version: 0 Date: 2004-11-24 User Name: 116527 Action: Approved Comment: Updated document implementing feedback, please take a look Version: 0 Date: 2004-11-22 User Name: 38638 Action: Rejected Comment: Girish, Good doc! I do have some suggestions hope that you can consider to have them included: - Additional reference pointers for people who would like to find out more on the x86 and amd64 calling conventions would be good. E.g. i386 ABI and the amd64 ABI doc, as well as the Intel/AMD programming manuals. Sun's "x86 Assembly Language Reference Manual" also provides a good summary for the x86 instructions. - In the 'Internal Only' section, you mentioned Frank's slides without any location where one can gain access to the doc. A pointer to the actual location may be helpful. Thanks. Version: 0 Date: 2004-11-22 User Name: 38638 Action: Accept Comment: Version: 0 Date: 2004-11-22 User Name: 116527 Action: Approved Comment: Thanks to James for the comments, I have made some changes with the help of Frank, please let me know if anything needs ammendment. Thanks. Version: 0 Date: 2004-11-01 User Name: 100346 Action: Rejected Comment: Hi Girish, Sorry for the delay - I was on training last week. Overall I like the article -- could you add a phrase or two indicating that while the document is mostly ok with amd64, there is no guarantee that what you see with dumping out memory using .../apn will show valid function arguments because they generally get clobbered. Oh, for register windows like on sparc! Apart from that, it's great and I'll eagerly push it to final review once you've made that change. best regards, James C. McPherson Version: 0 Date: 2004-10-27 User Name: 100346 Action: Accept Comment: Version: 0 Date: 2004-10-21 User Name: 116527 Action: Approved Comment: Please write to me if there is anything that needs ammendment. This is my first document. Thanks. Version: 0 Date: 2004-10-03 User Name: 116527 Action: Created Comment: Version: 0 Product_uuid 079e9bbc-0ee3-11d7-8c50-f1061905b56f|Sun Fire V65x Server 03728cec-0ee3-11d7-9be8-dd41f651e0a3|Sun Fire V60x Server Attachments This solution has no attachment |
||||||||||||
|
||||||||||||