SAP Performance Troubleshooting

SAP Performance Troubleshooting

SAP performance is very critical element and whenever you receive request for performance issue you need perform analysis of not only SAP system but its configured Hardware, the Database and the external process.

The initial checks for performance issue starts with checking the work process status in SM50. When you received a request for performance issue, please check if most of work processes (either of type DIA, UPD or BGD) are occupied and also some of the DIA work process in PRIV mode. If all are occupied then you need to check with users/application support team and discuss the scenario and depending on that you can kill the Work process.

You can also check ST22 for any particular kind of repetitive dump which might through some light on the performance issue (for example low memory). Along with ST22, SM21 will also help you to find out the cause of temporary performance issue. You can check the SM21 logs generated in particular duration as per the information given by the user.

Sometime significant performance issues are observed by long running background jobs. In that case also you need to check with the owner of the running job and decide if these can be killed to free up the resources (based on agreement with customer).

It very important to understand the nature of Performance problem which user is facing. You need to find out if it’s a General Performance problem or Specific Performance problem.

General Performance Problem: - This kind of problem results in poor response time and unsatisfactory throughput in all transactions. This can have a negative impact on business process and lead to financial loss.

Specific Performance Problem: - If the throughput or response time of individual transactions is unsatisfactory, then we can speak of specific performance problems.

After asking few questions to the affected end user/s, you can find out if it’s a general or specific performance problem. If it’s a General Performance problem, then you should ask some questions to user to understand the real situation.

-          What is the frequency of the problem?

-          Is the problem permanent or temporary?

-          Does this problem occur at regular time intervals – for example at particular times of the day?

-          Is it a non-recurring problem?

-          Does the problem occur following only specific system activities – for example, when background programs run on the system?

-          What times are high (CPU, Database, GUI etc) when the problem occurs?

-          Is it a region specific performance issue?

To examine these questions more closely, compare the workload statistics of recent days with each other. The detail of Workload Analysis is explained in subsequent sessions.

Workload Analysis:-

Workload analysis gives reliable data on throughput, load and response times for the SAP system and its components. It examines the various response times measured by the system. Workload analysis helps you to identify “bottlenecks”. Bottlenecks can critically affect production operation and therefore require speedy removal. In addition, workload analysis reveals the load distribution for each application’s programs or transactions and indicates which of these are placing the greatest load on the SAP system.

The Workload Monitor (ST03) has formed a part of SAP software since the 1^st release of SAP R/3. With SAP Basis 4.6C the workload monitor is now available with a completely reworked interface (ST03N).

Working with the Workload Monitor

Statistics such as response times, memory use, and database accesses are collected and stored for all transactions steps for all SAP components. These statics are organized in load profiles that can be displayed in the Workload Monitor.

Use Transaction Code ST03N to access the initial screen of the Workload monitor.

The main screen is divided into three windows.

In the upper left window you will find a button, with which you can choose a role. There are three roles

1)      Administrator: - This is the standard user mode. It offers fast access to the system load statistics of the current day and gives an overview of the system load distribution.

2)      Service Engineer: - This mode gives you the system load statistics for the current day and for the previous week as well as an overview of the history and distribution of the workload and a detailed analysis of system load.

3)      Expert: - This mode gives users all functions available in Transaction ST03N. You can display all the available data on the system load.

Furthermore, in the window in the upper left you can choose the SAP instance you wish to analyze or “Total”, if you want to analyze the entire SAP system. You can also select the period of time to be included in the analysis.

Further down you will see statistical data on the performance of the SAP system and information on possible causes of performance problems. If you select the Workload overview analysis view in the window in the lower left, you will see a breakdown of statistics on response times and throughput, according to different task types. For the most part, the task types corresponds to the work process types “Dialog”, “Update”,”Update2”,”Background”, and “Spool”. The “Dialog” work process type is further subdivided into the task types “Dialog”,”RFC”,”AutoABAP” and “Buffer Sync” .

Transaction step: - The 1^st important values are the number of transaction steps (Number of Steps column). In a dialog task, a transaction step corresponds to a screen change –that is, to a request that is executed for a user by the mySAP component.

Response Time: - The average response time for a transaction step in a dialog task (Av. Response Time) is seen by many SAP users as a criterion for the acceptable performance in an SAP component. Another generally accepted rule of thumb is that good performance in SAP R/3 is indicated by an average response time of 1 second or less.

Av. Wait Time: - The time for which request is waiting in dispatcher queue to get assigned free work process.

Roll-in, Roll-Out:- An SAP transaction normally consist of several transaction steps, during these steps ,data such as variable, internal tables and screen list it built up and stored in the main memory of the app. Server. This data is known as User context. At the beginning of a transaction step, the user context is made available to appropriate work process. This procedure is called Roll-in.

Roll-out saves the current user-context data to the virtual memory at the conclusion of a transaction step.

Please note Roll-out time is not part of response time of a transaction step. At roll-out, when the user context is copied from local memory to the roll memory, the processed data has already been returned to the presentation server.

Load Time: - All ABAP programs and screens which are required and are not yet available in the application server buffers must be loaded and possibly generated. The time it takes to do this is indicated as Av. Load+gen time.

Database Time: - Database time is measured from the moment of sending the database request to the database server and runs until the moment at which the data is returned to the application server.

Processing Time: - It is the total response time minus the sum of all time mentioned above.

After doing the initial checks we’ll see more detail analysis of Hardware, Database and SAP basis.

Monitoring Hardware, Database and SAP Basis:-

1)      Hardware :-

To Start OS Monitor for the server you are currently logged on to, enter the transaction code ST06. The main screen of the operating system monitor “Local OS Monitor” appears.

The main screen of OS Monitor lists the most important performance data for the OS and H/W. All the data is renewed every 10 seconds by program “saposcol”.

CPU

In the component CPU, it will show you the percentage of the total CPU capacity that is currently being used by user processes, the system and what percentage is not being consumed (idle).

For optimal performance idle CPU utilization should be at least 20%, optimally 35%.

The count field indicates number of processors / CPUs.

Memory

The Current RAM size is given in the field Physical memory available.

What is paging:-

Exchange of data between main memory and the overflow store in a paging file of Hard disk.

Paging occurs when main memory is not large enough for the context of running processes

Page In:-

Average number of page_ins per second.

 Page_in occurs if a process needs access to a page which is not in main memory.

Page_in count is critical for SAP with Windows OS.

Page out:-

Average number of page outs per second.

Page out count is critical for SAP with UNIX OS

Swap

Swap Space:-

Storage space on Hard disk on which data that is not currently required by main memory (RAM) is kept.

The Ideal Swap space would be 3*RAM

Disk

In Hard Disk monitor, a heavy load on the individual disk is indicated by the value > 50% in the column Utilizan %.

What are the causes of Hardware bottlenecks:-

The causes of H/W bottleneck would be

1)      Uneven workload distribution.

If you have detected bottleneck in a distributed system with multiple computers and issue is with at least on one computer while others have unused resources then the workload is probably not optimally distributed.

To improve the performance, redistribute SAP work processes and user logons.

2)      Individual processes that consumes too much CPU:-

With this data you can find out which processes are demanding / using more CPU.

Also check if any non sap / DB (external process) process is causing problem.

2)      Database:-

SAP systems support multiple relational databases, each of which has a different architecture. Many performance problem, however occur independently of the type of database system implemented. To help customers analyze and tune their databases, the SAP system has its own database performance monitor with basic functions.

Database Performance Monitor can be started with T.Code ST04

Analyzing Database Buffer:-

Every Database has various buffers that enable user data and administrative information from the database to be stored in main memory and as a result, reduce the number of accesses to the hard disk.

Accesses to these buffers in main memory are normally 10 to 100 times faster than accesses to hard disk. If the buffers are too small, the data volume is too large for the buffer. Data is then forced out of the buffer and has to be read from the hard disk. For this reason monitoring the buffer is an important element of performance analysis.

The most important buffer in Database is “data buffer”, which stores most recently read database tables and their indexes.

Quality of data buffer is the most important factor for database performance monitoring. Logical and Physical reads are two important elements for calculating the buffer quality.

Buffer quality = (logical reads – physical reads) / logical read * 100%.

The smaller the number of physical reads as compare to logical reads, higher the buffer quality. A buffer quality of 100% is ideal.

If the database instance has just started, the buffer will just have been loaded, and the hit ratio is low. Therefore, when you evaluate the buffer quality, ensure that the database has already been running for several hours.

In Production systems, the size of data buffer normally varies between 100 to 200 MB depending on the size of database. For large database the data buffer can be significantly larger.

Poor buffering normally has two possible causes –

·         Expensive SQL statements

·         Buffer pool is too small.

Please note these buffer quality values are only guideline values. In some cases, a database instance can still run well with an apparently low buffer quality. Please check the database response time from workload analysis before investing time in buffer optimization.

Checkpoints and Savepoints:-

Data buffers are not only useful for improving data read performance but also speed up database change operations. When a record is in buffer, database change operations initially perform changes to the respective data block in buffer pool. These changes are saved to hard disk asynchronously i.e. at a later point in time. The database must right all changed data blocks to hard disk within certain interval, defined by a “checkpoint or savepoint”.

For all database systems, you can strategically define the freq. of checkpoint by setting certain parameters. SAP’s default parameter settings should be changed only after consulting SAP.

Identifying Expensive SQL Statement:-

Expensive SQL statements are long-running statements and are one of the main causes of performance problems. In addition to causing long runtimes in the program in which they are called, they also indirectly cause performance problems for other transactions.

Expensive SQL statements can have the following effects on the entire system –

·         High CPU utilization and I/O load.

·         Block SAP work processes for a long time.

·         Data buffer overflow.

How will you find the expensive statement –

Transaction Code: - ST04                                                                                                                                             In SAP ECC 6.0 systems you will find the main screen will look like as below –

Double click on Shared Cursor Cache, it will show the list of SQL statements executed since database startup.

In Netweaver 04 the screen will look like

Press Detail analysis menu -> SQL request

This will also show the list of SQL statement executed since database startup.

Each SQL statement contain below details –

Executions – Number of times statement has been executed.

Disk Reads – Number of Physical reads required for all the execution of the statement

Read/Execution - Number of Physical reads required for one execution of the statement

Buffer gets – Number of logical read access required for all execution of the statement

Bgets/Exec – Number of logical read access required on an avg. for one execution of the statement.

Records processed – Number of rows read for all the executions of the statement.

Expensive SQL statements are indicated by high number of logical read accesses and physical read accesses.

3)      Analyzing SAP Memory management:-

To start SAP Memory configuration for the SAP Instance you are currently logged on to, use T.Code ST02

The main screen shows the configuration and utilization of SAP Buffer, SAP extended memory and the SAP heap memory. All the data shown corresponds to the period since the last startup of instance.

Details of various SAP buffers can be found in Buffer section.

It contain below listed columns -

Hit Ratio – Same as Database buffer

Allocated – space allocated to respective buffer.

Free Space – unoccupied space

Dir. Size – Max number of entries that can be stored in the buffer

Free directory – Difference bet current number of objects and max possible objects.

Swaps – Number of objects displaced from the buffer

For SAP System, SAP buffer also has to achieve a minimum buffer quality to ensure smooth operation.

If the buffers are too small, data is displaced, and the database has to be unnecessarily accessed to reload the data.

When monitoring SAP buffers, consider following guidelines –

·         The Hit Ratio for SAP buffers should generally be 98% or better. (Exception for program buffer, single record buffer and export/import buffer, lower hit ratios can be regarded as acceptable.)

·         There should be no swaps in the buffers of a production system. If there are swaps, the buffer size or the maximum number of entries should be increased. (exception again Program buffer)

·         To help avoid subsequent swaps, ensure that each buffer has sufficient memory and free entries.

If, in the memory configuration monitor, you see that there have been displacements in an SAP buffer, proceed as follows –

1)      First check whether the buffer is too small (free space field) or whether the maximum number of possible buffer entries is too small (Free directory field).

2)      Depending on the results of these checks, either increases the buffer size or the maximum number of allowed entries by 10 to 50%.

Analyzing SAP Extended memory, SAP Heap Memory and SAP Roll Memory:-

The SAP Memory section of the SAP Memory configuration monitor lists data on the SAP Memory areas Roll area, Paging area, Extended Memory and Heap Memory.

Current Use – Amount of memory currently in use in the respective memory area

Max. Use – Maximum amount of this memory area that has been used since the SAP instance started.

In Memory – The amount of main memory allocated to this area at system startup.

On Disk – For “Roll area” and the “Paging area” SAP roll files and SAP paging files are located on the hard disk of app. Server. The size of these files is indicated here.

When monitoring SAP Memory areas, consider below guidelines –

·         For Rollarea the Max. use value should not exceed the corresponding amount in the In Memory column

·         For Extended Memory the Max. Use value should be at least 20% smaller than the corresponding value in the Inmemorycolumn.

Please note that experience shows that performance is dramatically reduced when SAP extended memory is full, making it impossible to work productively in SAP instance.

Displaying the Allocated Memory:-

In order to analyze the performance, it is important to have an overview of current memory allocation.

Go to Transaction Code – ST02 -> Detail Analysis -> Additional Function-> Storage

This will take to below screen

Storage Shared bet Work Processes – The value for Allocated column in the Total row shows the total amount of memory allocated to SAP buffers.                                                                                                                 User Storage for all Work Processes – This shows the value of memory allocated to Work Processes.           Size of Extended Memory – memory allocated to SAP extended memory                                                    Virtual Memory allocated – This value is the size of total memory allocated at instance startup to SAP buffer, SAP work processes, and extended memory. This is critical value for accessing whether main memory bottlenecks are likely to occur.                                                                                                             Maximum heap area for all work processes – This value is the size of the SAP heap memory that can be allocated as local memory by SAP work processes, if required (parameterabap/heap_area_total).

Analyzing SAP Work Processes:-

To display data on the SAP work processes active on a particular application server, log on to that server and call SAP monitor. Enter the Transaction SM50. The Process Overview screen is displayed.

To display the work processes of any application server, call the server overview with Transaction SM51, place the cursor on the desired application server and choose Processes.

To see a system wide overview of all the work processes in SAP system call Transaction SM66. The System wide Work Process Overview screen is displayed.

If you are no longer able to check the work process from SAP system, you can start the program dpmon on the OS level as below –

Login as <SID>adm in the server.

Go to SAP profile directory (/usr/sap/<SID>/SYS/profile.

List all the profiles under the directory /usr/sap/<SID>/SYS/profile

Execute the command –

dpmon pf= <SID>_DVEBMGS<instance number>_server name

It will show the list of work process, same as shown in SM50.

In the next section we will brief the known performance issues and its recommended solutions

Checklists: -

In this section we will see the problem checklist – that is, short summaries for individual problems that are frequently identified in performance analysis.

1)      Problem - CPU Bottleneck due to High Resource Consumption From Individual Processes.

Indications – A Computer has less than 20% CPU capacity available. Check this using the Top CPU processes function in the Operating System Monitor. Transaction Code ST06 and select Detail analysis menu – Top CPU processes. The screen that appears shows individual processes that occupy considerable CPU over long period of time.

Solution – Use the Work Process Overview (SM50 or SM66) to identify the SAP work process, the program and the user; then analyze the program or reschedule it.

Identify the database processes in the Database Process Monitor and optimize the corresponding SQL statement.

2)      Problem – CPU Bottleneck due to Non-Optimal Load Distribution

Indication – In distributed system with multiple computers, you detect a hardware bottleneck on at least on computer, while other computers still have available, unused resources.

Solution – Redistribute the SAP work processes, after which you may need to reset the associated virtual memory areas, buffers and user distribution.

3)      Problem – Terminated Work Processes

Indication – In the Local Work Process Overview (Transaction SM50), if you detect numerous terminated work processes (indicated as complete in the column Status) and find that you cannot restart them, it is likely that there is a problem with the R/3 kernel or logging on to the database.

Solution – Check whether the SAP kernel version is up to date by calling Transaction SM51 and selecting Release Info. Refer to the SAP service marketplace for relevant SAP Notes or contact SAP support.

4)      Problem - Work Process stuck in Private Mode or in Roll-In or Roll-Out

Indication – More than 20% of the work processes are indicate in the Transaction SM50 or SM66 as being in PRIV mode or roll-in or roll-out. The problem is in SAP memory management.

Solution – Correctly set the parameter of SAP memory management – for example em/initial_size_MB and rdisp/ROLL_SHM, ztta/roll_extension.

5)      Problem – Deactivated Update Service

Indication – All update work processes are occupied. Transaction SM13 indicates that the update has been deactivated.

Solution – Call Transaction SM21 and check whether the update service has been deactivated. The system log contains an entry for the time, the user, and the reason for the deactivation. Resolve the reported problem – for example, a database error. Then reactivate the update service the Transaction SM13.

6)      Problem – Database Buffer is too small.

Indication – Call the Transaction ST04 and check whether the buffer quality and other key figures match the recommended values (>90%).

Solution – Increase the size of the respective buffer once by 25% and check if it improves the quality.

7)      Problem – Statistics for the Database Optimizer are Obsolete or Not Available.

Indication – Check whether the optimizer statistics are created regularly

Solution - call the Transaction DB13 and schedule the program for updating statistics.

8)      Problem – Extended memory is too small.

Indication – Call the Transaction ST02 and check if extended memory or the roll buffer is too small.

Solution – Correct SAP memory configuration parameters such as em/initial_size_MB and rdisp/ROLL_SHM, ztta/roll_extension. If you have sufficient main memory on the server, you can increase the memory size by 20 to 50%. Check whether this improves the situation.

9)      Problem – Displacements in SAP buffers.

Indication – Look for displacements in the SAP buffers in the column Swaps in the Transaction ST02. Displacements mean that the buffers are configured too small.

Solution – Increase the maximum number of buffer entries or increase the size of respective buffer, provided that the computer still has sufficient main memory reserves.

Parameters for SAP Memory management please refer SAP Notes 103747 and 88416.

Few important memory parameters which can directly affect the performance of the SAP system are –

Sr. No.	Parameter	Description
1	ztta/roll_area	Total local SAP roll area for all work processes
2	ztta/roll_first	Portion of the local SAP roll area allocated to a dialog work process before SAP extended memory is allocated.
3	rdisp/ROLL_SHM	Size of SAP roll buffer in shared memory
4	rdisp/PG_SHM	Size of ABAP paging buffer in shared memory
5	rdisp/ROLL_MAXFS	Size of the global SAP roll area, which comprises the SAP roll buffer plus the SAP roll file.
6	em/initial_size_MB	Initial size of the SAP extended memory
7	em/max_size_MB	Maximum size of the R/3 extended memory.
8	em/blocksize_KB	Size of a block in SAP extended memory
9	ztta/roll_extension	Maximum amount of SAP extended memory that can be allocated for each user context