Sun Microsystems, Inc.  Sun System Handbook - ISO 3.4 June 2011 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-71-1019113.1
Update Date:2011-05-25
Keywords:

Solution Type  Technical Instruction Sure

Solution  1019113.1 :   Sun[TM] Sparc Enterprise Mx000 (OPL) Servers Memory Configuration Rules  


Related Items
  • Sun SPARC Enterprise M9000-64 Server
  •  
  • Sun SPARC Enterprise M9000-32 Server
  •  
  • Sun SPARC Enterprise M8000 Server
  •  
  • Sun SPARC Enterprise M3000 Server
  •  
  • Sun SPARC Enterprise M4000 Server
  •  
  • Sun SPARC Enterprise M5000 Server
  •  
Related Categories
  • GCS>Sun Microsystems>Servers>OPL Servers
  •  

PreviouslyPublishedAs
235101


Description
OPL memory subsystem description, configuration rules and restrictions.

Introduction

Introduction

In the Mx000 (OPL) Servers resources are located on 1 to 16 Physical System Boards (PSB). A Physical System Board (PSB) consists of up to four (4) CPU's, up to 32 DIMM's and one IOU resource (optional).
I/O resources on an IOU are tied to a specific CPU/Memory resource, and that CPU, Memory & I/O resource are assigned to a XSB (eXtended System Board). The XSB can then be assigned to a domain. To be usable for a domain, a PSB must first be configured in either UNI-XSB or QUAD-XSB mode.

A M3000 PSB contains a fixed number of pieces of hardware:
- A single CPU
- Up to 4 or 8 DIMM's
- IO

A M4000/M5000 - PSB contains of up to seven pieces of hardware:
- Up to two CPUM (each containing two CPU's for a total of four physical CPU's).
- Up to four MEMB's (each containing 4 or 8 DIMM's ; up to 32 total physical DIMM's)
- One IOU ( IOU's are needed to access internal devices – DVD, Tape, Disk, Network)
The M4000 has 1 PSB and the M5000 has 2.

A M8000/M9000 PSB contains up to two pieces of hardware:
- One CMU (each CMU has 2 or 4 CPU's and 16 or 32 DIMM's)
- One IOU (Optional, at least one required per domain)

In UNI-XSB mode, we get 1 XSB consisting of the whole PSB.
In QUAD-XSB mode, we get up to 4 XSB's, consisting each of one quarter of a PSB.
The M3000 (Ikkaku) can be viewed as a single QUAD-XSB (1 CPU, 8 DIMMs).

The Mx000 (OPL) Servers use DDR2 DIMMs. 1, 2, 4, or 8 GB types can be used.
OPL uses Memory Groups. XSBs can access 2 groups, Group A and Group B.
Group A must always be populated, Group B can optionally be populated to increase memory capacity.

OPL automatically configures memory interleaving to obtain the best performance.
Interleaving takes place within a XSB (UNI or QUAD).
All DIMMs within a XSB are included in a single interleaving scheme, there is only one interleave factor defined inside a XSB. This has the consequence that the number of DIMMS in a Group can only be 4, 8 or 16 and that the number of DIMMs in Group B must be either 0 or the same as the number of DIMMs in Group A
Populating Group B does not affect interleaving or mirroring, it only increases the memory capacity.



Interleave factor = 2

4 DIMMS in Group A

optionally 4 DIMMs in Group B

UNI-XSB or QUAD-XSB

Interleave factor = 4

8 DIMMS in Group A

optionally 8 DIMMs in Group B

UNI-XSB

Interleave factor = 8

16 DIMMS in Group A

optionally 16 DIMMs in Group B

UNI-XSB



Note: Interleave factor of 3 is not possible, therefore XSBs with 12 DIMMS (or 24 DIMMs ) are not permitted.
Note: By definition, a QUAD-XSB can only hold 4 DIMMs in Group A and optionally 4 DIMMS in Group B.

Memory can be configured in mirror mode for higher reliability. (except M3000)
In mirrored mode, both the the memory capacity and the the interleave factor are halved.

The minimal hardware required for a functioning XSB (UNI or QUAD) is to have at least 1 CPU and at least 4 DIMMs in Group A.

DIMM naming scheme and physical location

The DIMM names as output by the showhardconf command and as named on the FRUs is as follows:





Physical System Board nn (PSB)

UNI-XSB
mode

QUAD-XSB
mode

M3000 / M4000 / M5000

M8000 / M9000

XSB#

XSB#

Group A

Group B

Group A

Group B

nn-0

nn-0

M3000: only this row

MEMB#0
or
MEMB#4

MEM#0A
MEM#1A
MEM#2A
MEM#3A

MEMB#0
or
MEMB#4

MEM#0B
MEM#1B
MEM#2B
MEM#3B

MEM#00A
MEM#01A
MEM#02A
MEM#03A

MEM#00B
MEM#01B
MEM#02B
MEM#03B

On M3000:

MBU_A

On M3000:

MBU_A

nn-1

MEMB#1
or
MEMB#5

MEM#0A
MEM#1A
MEM#2A
MEM#3A

MEMB#1
or
MEMB#5

MEM#0B
MEM#1B
MEM#2B
MEM#3B

MEM#10A
MEM#11A
MEM#12A
MEM#13A

MEM#10B
MEM#11B
MEM#12B
MEM#13B

nn-2

MEMB#2
or
MEMB#6

MEM#0A
MEM#1A
MEM#2A
MEM#3A

MEMB#2
or
MEMB#6

MEM#0B
MEM#1B
MEM#2B
MEM#3B

MEM#20A
MEM#21A
MEM#22A
MEM#23A

MEM#20B
MEM#21B
MEM#22B
MEM#23B

nn-3

MEMB#3
or
MEMB#7

MEM#0A
MEM#1A
MEM#2A
MEM#3A

MEMB#3
or
MEMB#7

MEM#0B
MEM#1B
MEM#2B
MEM#3B

MEM#30A
MEM#31A
MEM#32A
MEM#33A

MEM#30B
MEM#31B
MEM#32B
MEM#33B





Applicable memory population rules

The rules are summarized in the 5 tables below and the additional Group rules.
The gray areas depict the hardware that can be installed.
It is important to understand that the rules apply per XSB.





M3000

PSB #

CPUM #

installed

Resulting

XSB #

MEMB #

required

# of DIMMs

Resulting interleave Factor


Group A

Group B

Normal

Mirrored

0

N/A

00-0

N/A

4

0

or

4

2

N/A




UNI-XSB mode M4000 / M5000

PSB #

CPUM #

installed

Resulting

XSB #

MEMB #

required

# of DIMMs on each MEMB

Resulting interleave Factor


Group A

Group B

Normal

Mirrored

0

0

00-0

0

4

0

or

4

2

1

0 and 1

0 and 1

4

2

0,1,2 and 3

8

4

1

2

01-0

4

4

0

or

4

2

1

2 and 3

4 and 5

4

2

4,5,6 and 7

8

4

Note: PSB1 only exists in M5000.



QUAD-XSB mode M4000 / M5000

PSB #

CPUM #

installed

Resulting

XSB #

MEMB #

required

# of DIMMs on each MEMB

Resulting interleave factor

Group A

Group B

Normal

Mirrored

0

0 *

00-0

0

4

0

4

2

1

00-1

1

4

0

4

2

1

1

00-2

2

4

0

4

2

1

00-3

3

4

0

4

2

1

1

2 *

01-0

4

4

0

4

2

1

01-1

5

4

0

4

2

1

3

01-2

6

4

0

4

2

1

01-3

7

4

0

4

2

1

*Note: Install these first.
Note: PSB1 only exists in M5000.
Note: Each PSB can individually be set to UNI-XSB or QUAD -XSB mode.





UNI-XSB M8000 / M9000 (2 and 4 CPU CMUs)

PSB nn

XSB #

DIMM quantity on each CMU

Resulting interleave factor

Group A

Group B

Normal

Mirrored

nn= 0..16

nn-0

16

0

8

4

16





QUAD-XSB M8000 / M9000 (4 CPU CMUs only)

PSB nn

XSB #

DIMM quantity on each CMU

Resulting interleave factor

Group A

Group B

Normal

nn= 0..16

nn-0
nn-1
nn-2
nn-3

16

0

2

16

Note: Each PSB can individually be set to UNI-XSB or QUAD -XSB mode.
Note: For architectural reasons, M8000 / M9000 do not allow memory mirroring in QUAD-XSB mode.

Additional Group Rules

within a XSB (UNI or QUAD), the following rules apply in addition to the above 4 tables:

  1. All DIMMs within a Group (A or B) must be of the same size / rank

  2. The size of the DIMMs in Group B must be smaller than or equal the size of the DIMMs in Group A.

  3. The number of DIMMs in Group B must be either 0 or the same as the number of DIMMs in Group A



Additional considerations

Implication of the above rules for M4000 / M5000

  1. Half populated MEMB, with only Group A populated are supported.

  2. In UNI-XSB mode, MEMBs must be installed in powers of 2 (1,2 or 4).
    3 MEMBs is not possible, this is due to the single interleaving scheme used on an XSB.

  3. It is permitted to have 3 MEMBs on a PSB, as long as the PSB is configured in QUAD-XSB mode.

  4. The minimum requirement for UNI-XSB mode is CPUM#0 / MEMB#0 (and CPUM#2 / MEMB#4)

  5. A CPU on a CPUM without its associated MEMB and vice-versa cannot be used in QUAD-XSB mode.
    The “setupfru” command will report “Operation has completed. However, a configuration error was detected”. If you configure a PSB in QUAD-XSB mode and one of the QUAD-XSB is made of a CPU and no corresponding MEMB or vice-versa.
    In this case, "showboards" will report the QUAD-XSB as "Unmount".

  6. XSBs can have different memory configurations.

  7. Only identically configured QUAD-XSBs can be reconfigured into one UNI-XSB

  8. Filling group B does not increase the interleaving factor, thus does not increase memory performance.
    Only the memory capacity is increased.

Memory upgrade

The rule requiring that Group A contains the DIMMs with the higher capacity may dictate that in case of a memory upgrade, DIMMs from Group A must be moved to Group B and the new higher capacity DIMMs be installed in Group A.

Performance considerations

UNI-XSB mode allows higher memory performance because it can use higher interleave factors. If all resources of a PSB are needed in one domain, then it is advisable to configure the PSB in UNI-XSB mode.

RAS considerations

Memory mirroring is an effective way to protect against memory failures. Configuring an XSB for memory mirroring halves the available memory capacity. It also halves the interleave factor. Memory mirroring should therefore best be used in UNI-XSB mode.

Best Practice

Try to avoid mixed configurations. It is advisable to have all identical MEMBs in a PSB of a M4000 / M5000. This allows to change between UNI-XSB and QUAD-XSB mode without loosing maximum performance or having systems containing unusable hardware.

If the capability to switch between UNI-XSB and QUAD-XSB mode is desired, then both the rules for UNI-XSB and those for QUAD_XSB must be followed. The supported M8000 / M9000 configurations enforce this.

How to identify DIMMs inside a system from XSCF

The showhardconf command run on the XSCF display the type of DIMMs

Type 1A = 1GB / rank1
Type 1B = 1GB / rank2
Type 2A = 2GB / rank1
Type 2B = 2GB / rank2
Type 4A = 4GB / rank1
Type 4B = 4GB / rank2
Type 8B = 8GB / rank2



Steps to Follow
Follow these steps to configure OPL memory.
A supported memory configuration must be built according to the above "Applicable memory population rules" section.
  1. For every PSB, use the appropriate table:
    • M3000
    • UNI-XSB mode M4000 / M5000
    • QUAD-XSB mode M4000 / M5000
    • UNI-XSB mode M8000 / M9000
    • QUAD-XSB mode M8000 / M9000
  2. For every PSB, follow the additional group rules.


Product
Sun SPARC Enterprise M3000 Server
Sun SPARC Enterprise M4000 Server
Sun SPARC Enterprise M5000 Server
Sun SPARC Enterprise M8000 Server
Sun SPARC Enterprise M9000 Server
Sun SPARC Enterprise M9000-32 Server
Sun SPARC Enterprise M9000-64 Server


Attachments
This solution has no attachment
  Copyright © 2011 Sun Microsystems, Inc.  All rights reserved.
 Feedback