Sun Fire[TM] 6800/4810/4800/3800 CPU & Memory Document

Asset ID:	1-71-1003347.1
Update Date:	2012-07-30
Keywords:

Solution Type Technical Instruction Sure

Solution 1003347.1 : Sun Fire[TM] 6800/4810/4800/3800 CPU & Memory Document

Applies to:

Sun Fire E4900 Server - Version Not Applicable and later
Sun Fire 4800 Server - Version Not Applicable and later
Sun Fire 4810 Server - Version Not Applicable and later
Sun Fire 3800 Server - Version Not Applicable and later
Sun Fire 6800 Server - Version Not Applicable and later
All Platforms

Goal

Mapping Physical System Devices to Device Path Names for CPUs & Memory

Fix

Let's 1st review the Physical System Devices (CPUs/Memory):

Depending on the platform type; Sun Fire 6800/4810/4800/3800, a system can have 
up to 6 CPU/Memory boards (SB0 thru SB5).

Maximum number of CPU/Memory Boards in Each System

---------------------------------------------------------------------------------------
 System                                 Maximum Number of      Maximum Number of CPU
                                        CPU/Memory Boards      Processors
---------------------------------------------------------------------------------------
Sun Fire(TM) 6800 System                   6                        24
Sun Fire(TM) 4810 System                   3                        12
Sun Fire(TM) 4800 System                   3                        12
Sun Fire(TM) 3800R System                  2                        8
---------------------------------------------------------------------------------------


Each CPU/Memory board (SB) can have either 2 or 4 CPU processors, depending on your 
configuration. Each CPU/Memory board (SB) has up to 8 banks of memory. Each CPU 
supports 2 physical banks of memory on a system board.

A system board (SB) can have up to 4 CPUs & 32 DIMMs (memory). These DIMMs are 
broken into 4 groups, which have 8 DIMMs per group:
J13XXX: Reference Designator /P0 (CPU0)
J14XXX: Reference Designator /P1 (CPU1)
J15XXX: Reference Designator /P2 (CPU2)
J16XXX: Reference Designator /P3 (CPU3)

NOTE: Technically, P0 through P3 reference CPU0 through CPU3. Remember that a 
CPU needs to exist in order to populate a memory group. By default, the above 
CPU/Memory groups align themselves in this manner.

Therefore:
Each memory group (J13XXX through J16XXX) is managed by a CPU
Each memory group (J13XXX through J16XXX) contains 2 banks of memory
Each bank of memory contains 4 DIMMs

Each of the groups contain 2 physical banks (4 DIMMs per Bank):
Bank 0: Reference Designator /B0 (Even Bank DIMMs end with a "0")
Bank 1: Reference Designator /B1 (Odd Bank DIMMS end with a "1")

Each CPU controls 2 corresponding physical banks for a Group of DIMMs:
CPU0 J13XXX Bank 0 & 1 (Total of 8 DIMMs)
CPU1 J14XXX Bank 0 & 1 (Total of 8 DIMMs)
CPU2 J15XXX Bank 0 & 1 (Total of 8 DIMMs)
CPU3 J16XXX Bank 0 & 1 (Total of 8 DIMMs)

NOTE: Bank0 (/B0) needs to be fully populated (4-DIMMs) before Bank1 (B1) gets 
populated (4 DIMMs) within a memory group.

Each physical bank has 2 logical banks:
L0 & L2 for physical bank 0 (B0)
L1 & L3 for physical bank 1 (B1)

L0 represents the front side of all 4 DIMMs in physical bank 0 (Even Side)
L1 represents the front side of all 4 DIMMs in physical bank 1 (Odd Side)
L2 represents the back side of all 4 DIMMs in physical bank 0 (Even Side)
L3 represents the back side of all 4 DIMMs in physical bank 1 (Odd Side)

NOTE:Obviously a user cannot replace the front/back side of a specific DIMM. 
The entire DIMM would have to be replaced. This is just for improved error 
reporting.

So now we have the following:

CPU-0/P0/CPU-A:

J13600 (Bank 0,L0/L2)------\
J13601 (Bank 1,L1/L3)-------\---------------------------\ 
J13500 (Bank 0,L0/L2)--------\ BANK0(EVEN)               \ 
J13501 (Bank 1,L1/L3)--------/----------------------------\ BANK1(ODD)
J13400 (Bank 0,L0/L2)-------/                             /
J13401 (Bank 1,L1/L3)------/-----------------------------/
J13300 (Bank 0,L0/L2)-----/                             /
J13301 (Bank 1,L1/L3)----------------------------------/

CPU-1/P1/CPU-B:

J14601 (Bank 1,L1/L3)-------\---------------------------\ 
J14500 (Bank 0,L0/L2)--------\ BANK0(EVEN)               \ 
J14501 (Bank 1,L1/L3)--------/----------------------------\ BANK1(ODD)
J14400 (Bank 0,L0/L2)-------/                             /
J14401 (Bank 1,L1/L3)------/-----------------------------/
J14300 (Bank 0,L0/L2)-----/                             /
J14301 (Bank 1,L1/L3)----------------------------------/

Memory Controller Mapping

There are up to four banks of memory per system board.

Each bank is controlled by one memory controller (MMU).

The memory controller is co-packaged with its respective processor (CPU).

AID for memory controllers are the same as their associated processor AID, but with a different offset.

The offset is 0 for processors and 400000 for memory

Memory Controller Mapping Example

’/ssm@0,0/SUNW,memorycontroller@b,400000

                Memory AID------^    ^---Offset

--> @b   =b is hexadecimal & converted to decimal is 11

 So, to decode this CPU/Memory device path, do the following:

11/4 = 2 rm 3

 |     |    |

 |     |    |-------- CPU#-->CPU 10

 |     |------------- SB#-->system board 2

 |------------------- # of CPUs per BD--> always divide by 4; even if there's only 2 CPUs per bd --> ,400000 =Bank 0 (,600000 references Bank 1)

So the above path points to Bank 0 of CPU 10(P2) on System Board 2.

We also know that there are 4 DIMMS in Bank0 of the specified memory group (J15600/J15500/J15400/J15300).

The /var/adm/messages file would give the reference designator of the DIMM in 
that memory group. 
 


Also reference the following Technical Instruction:

Document 1008674.1: Sun Fire[TM] 3800, 4800, 4810, 6800, E4900, or E6900 Servers: Physical Device Mapping for I/O Boats

Keywords: memory, logical bank, physical bank, offset, AID, mapping

Attachments

This solution has no attachment