Steps to ensure proper binding of resources to LDOM domains.

Asset ID:	1-71-1019983.1
Update Date:	2012-10-09
Keywords:

Solution Type Technical Instruction Sure

Solution 1019983.1 : Steps to ensure proper binding of resources to LDOM domains.

Applies to:

SPARC T4-1 - Version All Versions and later
Sun SPARC Enterprise M4000 Server - Version All Versions and later
All Platforms

Goal

Description
In order for resources to be assigned to the LDOM domains, they need to be bound to the appropriate domain. This process binds the virtual resources to actual physical resources available on the system.

If there is not enough physical resource to be allocated, failure will be reported during the binding process. This document list out the steps to confirm the resource bindings are set up correctly.

Symptoms:

One of more of the following are observed or described when one have issue binding resources:

Insufficient VCPUS resources to bind LDom
Not enough free memory present to meet this request
Could not bind requested memory for LDom
Only <number> physical crypto unit resource(s) available to bind to LDom
Didn't find a suitable vcc service in a bound service domain to bind guest <guestname> console

Purpose/Scope:

This document describe in the context during guest domain creation. Adding of resources to an active domain can also fail due to lack of additional resources to be assign to a domain. The corrective actions are the same as described here.

For more information on creation of LDoms guest domains, please refer to following references:

- Beginners Guide to LDOMs availabe at http://www.oracle.com/technetwork/articles/systems-hardware-architecture/beginners-vm-server-sparc-256946.pdf.
- Logical Domains (LDoms) X.X.X Administration Guides, which are available on http://www.oracle.com/technetwork/documentation/vm-sparc-194287.html .

Fix

We will provide an overview of how to review the bindings of the resources to LDOM and then follow by steps to ensure proper binding of resources that are critical to get a domain up and running (Memory/VCPU/MAUs/Console).

***NOTE*** Configuration changes to LDoms need to be save to the SP using the ldm(1M) add-config subcommand. If not, the configurations will be lost when the primary domain is rebooted.

1. Overview

When a guest domain is being created with resources assigned by ldm(1M) add-* or set-* commands, the actual corresponding physical resource bindings are only assigned when the ldm(1M) bind command is issue.

To review the existing bindings, use the ldm(1M) list-bindings -e command.

For example, below is a newly created LDom guest named guest01, with resources added, but not being bound yet. Notice that the LDC fields are blank, and there are no actual allocation of VCPUs and memory. The STATE of the guest is noted as inactive.

# ldm list-bindings -e guest01
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
guest01          inactive   ------          4     2G
NETWORK
NAME             SERVICE                     DEVICE     MAC               MODE   PVID VID
vnet1            primary-vsw0                           00:14:4f:f9:fc:3f        1
DISK
NAME             VOLUME                      TOUT DEVICE  SERVER         MPGROUP
vdisk1           vol01@primary-vds0
VLDCC
NAME             SERVICE                     DESC              LDC
ds               primary-vldc0@primary       domain-services      <LDC
primary-vcc0                5001      <


Once the guest is bound, the actual physical resources are bound,
and LDCs(Logical Domain Channels) are assigned. The domain STATE
will be shown as

bound

# ldm bind guest01
# ldm list-bindings -e guest01
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
guest01          bound      ------  5001    4     2G
MAC
00:14:4f:fb:09:19
HOSTID
0x84fb0919
VCPU
VID    PID    UTIL STRAND
0      4             100%
1      5             100%
2      6             100%
3      7             100%
MAU
ID     CPUSET
1      (4, 5, 6, 7)
MEMORY
RA               PA               SIZE
0x8000000        0x108000000      2G
NETWORK
NAME             SERVICE                     DEVICE     MAC               MODE   PVID VID
vnet1            primary-vsw0@primary        network@0  00:14:4f:f9:fc:3f        1
PEER                        MAC               MODE   PVID VID                  LDC
primary-vsw0@primary        00:14:4f:f8:b2:a0        1                         0x0
DISK
NAME             VOLUME                      TOUT DEVICE  SERVER         MPGROUP        LDC
vdisk1           vol01@primary-vds0               disk@0  primary                       0x1
VLDCC
NAME             SERVICE                     DESC              LDC
ds               primary-vldc0@primary       domain-services   0x2
VCONS
NAME             SERVICE                     PORT   LDC
guest01          primary-vcc0@primary        5001   0x3

2. Memory binding failure.

Memory binding failure is reported when there is no additional unallocated memory available for allocation. ldm(1M) bind will report the following error:

# ldm bind guest01
Not enough free memory present to meet this request Could not bind requested memory for LDom guest01

The corrective actions are to:

1. Review how much unallocated memory is available on the system using the ldm(1M) list-devices memory subcommand. Then assigned the actual free available memory to the domain.

For example, here we have 1920M of unallocated on this system and we proceed to assigned what is left to guest01:

# ldm list-devices memory
MEMORY
PA                   SIZE
0x188000000          1920M
# ldm set-memory 1920M guest01
# ldm bind guest01

2. Alternatively, reduce the amount of memory from another domain by using the ldm(1M) set-memory or remove-memory subcommand on the another domain. Then proceed to bind the domain.

For example, we know that we are lacking of 1GB of memory to successful by guest01, we can reduce the primary by 1GB and then bind the guest:

# ldm remove-memory 1g primary
# ldm bind guest01

3. VCPU binding failure.

VCPU binding failure is reported when there is no additional unallocated VCPU available for allocation. ldm(1M) bind will report the following error:

# ldm bind guest01
Insufficient VCPU resources to bind LDom guest01

The corrective actions are to:

1. Review how many unallocated VCPUs are available on the system using the ldm(1M) list-devices vcpu subcommand. Then assigned the actual free available VCPUs to the domain.

For example, here we have on 4 VCPUs unallocated on this system. Binding a new domain with 5 or more VCPUs would have failed.

# ldm list-devices vcpu
VCPU
PID     %FREE
28      100
29      100
30      100
31      100
# ldm set-vcpu 4 guest01
# ldm bind guest01

2. Alternatively, reduce the amount of VCPUs from another domain by using the ldm(1M) set-vcpu or remove-vcpu subcommand on the another domain. Then proceed to bind the domain.

For example, we know that we are lacking of 2 VCPUs to successful by guest01, we can reduce the primary by 2 VCPUs (One should really need to make sure that it is practical to reduce a number of VCPUs from another domain without impacting the services running on it before reducing the number of VCPUs on the domain) and then bind the guest:

# ldm remove-vcpu 2 primary
# ldm bind guest01

*NOTE* Do take note, if MAU from a core is assigned to the a domain, at least one of the VCPUs within the core needs to be assigned to the domain. Please refer to Beginners Guide to LDOMs,

4. MAU(Cryptographic Unit) binding failure

*NOTE* MAU units are on T1, T2 and T3 series systems only.
T4 systems do not have MAU's. On T4 systems, each core contains a stream processing unit (SPU) that provides cryptographic processing.

MAU binding failure is reported when there is no additional unallocated MAU available for allocation. Additionally requirement is that at least one of the VCPUs from the processor core where the MAU originated has to be assigned to the domain. Despite the MAU binding failure, the bind will proceed with the rest of resources. ldm(1M) bind will report the following error:

# ldm bind guest01
Only 1 physical crypto unit resource(s) available to bind to LDom guest01,
proceeding with binding 1 additional crypto unit(s)

The corrective actions are to:

1. Review how many unallocated MAUs are available on the system using the ldm(1M) list-devices mau subcommand. Then make sure at least one VCPUs from the associated core is assigned domain before assigning the available MAUs to the domain.

For example, here we have 6 MAUs unallocated on this system. To add an additional MAU, one have to assign one more VCPU so that at least one VCPU from CPUSET (8, 9, 10, 11) is assigned to guest01:

# ldm list-devices mau
MAU
ID     CPUSET
2      (8, 9, 10, 11)
3      (12, 13, 14, 15)
4      (16, 17, 18, 19)
5      (20, 21, 22, 23)
6      (24, 25, 26, 27)
7      (28, 29, 30, 31)
# ldm add-vcpu 1 guest01
# ldm add-mau 1 guest01
# ldm bind guest01

2. Alternatively, reduce the amount of MAUs from another domain by using the ldm(1M) set-mau or remove-mau subcommand on the another domain. Here, again one must take care to ensure one VCPU from the core where the MAU is originating from is assigned to the domain. Then proceed to bind the domain.

For example, we know that we need 1 MAU to be assigned guest01, we can reduce the primary by 1 MAU (One should really need to make sure that it is practical to reduce a number of MAUs/VCPUs from another domain without impacting the services running on it before unassign them from the domain) and then bind the guest:

# ldm remove-mau 1 primary

One may also need to remove 1 VCPU and assign to the guest to make sure one of the VCPUs from the same core as the MAU is assigned to the guest.

# ldm remove-vcpu 1 primary
# ldm add-vcpu 1 primary
# ldm bind guest01

*NOTE* Do take note, if MAU from a core is assigned to the a domain, at least one of the VCPUs within the core needs to be assigned to the domain. Please refer to Beginners Guide to LDOMs,

5. Virtual Console binding failure

Virtual console concentrator(VCC) and virtual console(VCONS) provide console access to the guest domain. It need to be bound a virtual console service created on the primary domain. Miscovered VCC can resulted in error during binding:

# ldm bind guest01
The virtual console concentrator service primary-vcc1 not exist or is not bound. Didn't find a suitable vcc service in a bound service domain to bind guest guest01 console

To ensure the VCC configurations are configured according, use the ldm(1M) list -l <domain> comand to review the configurations.

# ldm list -l guest01
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
guest01          inactive   ------          5     1920M
...
VCONS
NAME             SERVICE                     PORT
primary-vcc0                5001
# ldm list -l primary
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      4     4G       0.6%  2h 42m
VCC
NAME             PORT-RANGE
primary-vcc0     5001-5100
VCONS
NAME             SERVICE                     PORT   LDC
guest01          primary-vcc0@primary        5001   0x3

If not configured, please follow the following steps to configure the console service:

# ldm add-vconscon port-range=5000-5100 primary-vcc0 primary
# ldm set-vconsole port=5000 service=primary-vcc0@primary guest01

Then make sure that the LDom console SMF service is update and running:

# svcs vntsd
STATE          STIME    FMRI
offline        13:20:40 svc:/ldoms/vntsd:default
# svcadm enable vntsd
# svcs vntsd
STATE          STIME    FMRI
online         13:39:35 svc:/ldoms/vntsd:default

6. Remeber to run ldm add-config

Please take note that new configurations will not persist a primary domain reboot unless it is saved to the SP.

Run the ldm(1M) add-config <configname>subcommand to save the configurations:

# ldm add-config newconfig
# ldm list-config
factory-default
initial
newconfig [current]

7. Data Collection for further troubleshootings

If after the above steps fail to resolve the resource binding issues, please contact Oracle Support with the following information:

- The latest explorer output using the latest version of explorer script.
- The console session log of the attempt to reset the system configuration to factory-default.

Please refer to 1002383.1 Sun Explorer X.XX Data Collector for the latest Explorer Script.

Product
Logical Domain Manager

OVM Server for SPARC
Sun Fire T2000 Server
Sun Fire T1000 Server
Sun Netra T2000 Server
Sun Netra T5220 Server
Sun Netra T5440 Server

Internal Comments
This document contains normalized content and is managed by the the Domain Lead(s) of the respective domains. To notify content owners of a knowledge gap contained in this document, and/or prior to updating this document, please contact the domain engineers that are managing this document via the "Document Feedback" alias(es) listed below:

Normalization Lead: [email protected]

Normalized, ldom, vcpu, mau, ldm

Attachments

This solution has no attachment