SAPHDE Webinar-The-Art-Science-of-Tuning-HANA-Models-for-Performance Abani Pattanayak SAP-Nov 2015

The Arts & Science of Tuning HANA models
for Performance
Abani Pattanayak, SAP HANA CoE
Nov 12, 2015
Disclaimer
This presentation outlines our general product direction and should not be relied on in making a
purchase decision. This presentation is not subject to your license agreement or any other
agreement with SAP. SAP has no obligation to pursue any course of business outlined in this
presentation or to develop or release any functionality mentioned in this presentation. This
presentation and SAP's strategy and possible future developments are subject to change and may
be changed by SAP at any time for any reason without notice. This document is provided without a
warranty of any kind, either express or implied, including but not limited to, the implied warranties of
merchantability, fitness for a particular purpose, or non-infringement. SAP assumes no responsibility
for errors or omissions in this document, except if such damages were caused by SAP intentionally
or grossly negligent.
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
2
SAP HANA: The Platform for All Applications
SAP HANA PLATFORM
O N-P RE MI S E
| HY B RI D
Processing Services
Application Services
Web Server
| CL O UD
JavaScript
Integration & Quality Services
Spatial
Graph
Predictive
Search
Text
Analytics
Streaming
Analytics
Series
Data
Business
Functions
Data Virtualization
ELT &
Replication
ALM
Fiori UX
Graphic
Modeler
Application Lifecycle
Management
Data
Quality
Hadoop & Spark Remote Data
Integration
Sync
Database Services
Columnar
OLTP+OLAP
Multi-Core &
Parallelization
Advanced
Compression
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Multitenancy
Multi-Tier
Storage
Data
Modeling
Openness
Admin &
Security
High Availability &
Disaster Recovery
Public
3
Overview
What’s Model Performance?
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
•
SQL Query
•
Query Run Time
•
Memory Usage
•
CPU Usage
•
Concurrency
Public
5
What’s under the hood i.e. Horse Power?
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
•
Dynamic memory (at least 50%)
•
Enough CPU (1+ core/8GB)
•
Single node vs. Scale-out
•
Single/Multi-Tenant (Shared)
•
Cloud vs. In-premise
Public
6
SAP HANA: Performance Tools
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
•
Analysis of SQL query
•
Visual walkthrough of model
•
Explain plan (engine used)
•
Visualize plan
•
HDBAdmin
Public
7
Analysis of SQL query
•
•
•
SELECT EMPLOYEE FROM "_SYS_BIC"."abani/CV_SALES";
SELECT EMPLOYEE, SUM(SALES) FROM
(SELECT * FROM "_SYS_BIC"."abani/CV_SALES")
GROUP BY EMPLOYEE;
SELECT Table_1.EMPLOYEE, SUM(Table_1.SALES) FROM
"_SYS_BIC"."abani/CV_SALES" as Table_1 INNER JOIN
"_SYS_BIC"."abani/AT_DATE" as Table_2
ON Table_1.DATE_SQL = Table_2.DATE_SQL
WHERE Table_2.DATE = TO_CHAR('20150101','YYYY-MM-DD');
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
8
Analysis of SQL query
•
•
•
SELECT TOP 50 DISTINCT EMPLOYEE FROM
"_SYS_BIC"."abani/CV_SALES";
SELECT EMPLOYEE, SUM(SALES) FROM
(SELECT * FROM "_SYS_BIC"."abani/CV_SALES")
GROUP BY EMPLOYEE;
SELECT Table_1.EMPLOYEE, SUM(Table_1.SALES) FROM
"_SYS_BIC"."abani/CV_SALES_FINAL" as Table_1
WHERE Table_1.DATE_NEW = '20150101'
GROUP BY Table_1.EMPLOYEE;
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
9
Visual walkthrough of model
•
•
•
•
•
•
View properties
Review the joins
Calculated columns
Scripted views
Cardinality of dataset
…
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
10
Explain plan
Ÿ
Ÿ
Ÿ
Ÿ
EXPLAIN PLAN FOR
SELECT "MANAGER", SUM("AMOUNT_SOLD")
FROM "_SYS_BIC"."abani.efashion/AN_SALES"
WHERE MANAGER = 'Larry‘
GROUP BY "MANAGER";
HANA Academy: Using Explain Plans:
https://www.youtube.com/watch?v=YfkGutsz5Uo
https://help.sap.com/saphelp_hanaone/helpdata/en/20/d9ec5575191014a251e58ecf90997a/content.htm
http://sapbw.optimieren.de/hana/hana/html/sql_explain_plan.html
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
11
Visualize plan
SCN: Visualize Plan & Timeline
Ÿ
Ÿ
Ÿ
http://help.sap.com/saphelp_hanapl atform/helpdata/en/c 1/f281fbbb571014aaf38a264c0e12c4/frameset.htm
http://scn.sap.com/community/hana-in-memory/bl og/2015/09/18/littl e-trick-to-check-table-filtering-on- planviz
http://scn.sap.com/community/hana-in-memory/blog/2012/12/20/show-me-the-timelines-baby
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
12
HDB Admin
Ÿ
How to use HDBAdmin to analyze performance traces in SAP HANA
http://scn.sap.com/docs/DOC-51110
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
13
Performance Tuning in 30mins
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
•
Use left outer joins
•
Specify cardinality in joins (n:1 or 1:1)
•
Set optimize join = true (SP09)
•
Use table functions instead of scripted calculation
views (SP09).
•
Execute in SQL-engine* (HANA live)
Public
14
Deep Dive: the science
Performance Tuning: Basics
2bn scans /second /core & 16m aggregations /sec /core
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
•
No of columns (& rows) scanned
•
No of aggregations performed
•
Type of calculation required
•
No of tables participating in joins
•
Size of temporary (internal) tables
•
Amount of data transfer between engines
•
Degree of parallelization
Public
16
Performance Tuning: Basics
2bn scans /second /core & 16m aggregations /sec /core
•
•
•
•
•
•
•
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
No of columns (& rows) scanned
ü Filter & cardinality of dataset in DB
No of aggregations performed
ü Cardinality of selected columns
Type of calculation required
ü Sum vs. distinct count
No of tables participating in Joins
ü Type of joins & properties
Size of temporary (internal) tables
ü Push joins to the lowest level
Amount of data transfer between engines
ü Inefficient use of DB engines*
Degree of parallelization
ü Logical and physical partitioning*
Public
17
Filters
•
•
Reduce the dataset as early as possible. Use design time filters at the lowest level.
Input Parameter: Placeholders part of the models.
ü Can be used in Calculation. Can accept multiple values (SP09)
ü Can be derived from table. Can be derived from store procedure (SP09)
•
Ensure Variables (where clause) is pushed to the lowest level. Confirm using Visualization Plan.
•
Use analytical privilege and SQL analytical privilege (SP10) to filter data.
•
Avoid script-based calculation view, WHERE clause will not be pushed down.
•
Using Filter is better than using Inner Join to limit the dataset.
•
Avoid filters on calculated column (consider materializing these columns).
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
18
Dealing with high cardinality
•
Joins are the most expensive operation. Consider pushing most of the joins (at least on the tables
with high cardinality) to the lowest level.
ü Dimension : attribute views and
ü Star-schema : analytic views or the lowest-node of the calculation view.
•
Columns participating in the Joins are also selected (even if not requested) during query execution.
Hence direct impact on the cardinality of the dataset.
•
Use aggregation node after union nodes in calculation view.
•
Be careful with Keep Flag in aggregation node.
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
19
Type of calculation
•
“Distinct count” is a complex operation and very CPU intensive compared to a simple aggregation
like “sum” or “count”. CPU usage will exponentially high esp. with high cardinality counters like
“Transaction Count”. These counters must be computed in the Analytic View level (where possible).
•
Avoid calculation before aggregations (where possible).
•
Watch out for currency/unit conversion.
•
Attributes used in restricted or calculated measure are also requested and may increase the
cardinality of the dataset.
•
Calculation with data type conversion are expensive. Avoid where possible.
•
Watch out for calculations (like String_Args) which execute in the row-engine.
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
20
Join pruning
•
Specify “Cardinality” for all Joins (1:1, n:1 etc.). Avoid m:n (Cartesian product)
•
Consider using Left Outer or Referential Joins (vs. Inner Join)
•
Referential Join to attribute view (with design time filter) will behave as “Inner Join”
•
Specify “Optimize Join = True” for Left Outer and Text Joins (SP09 onwards)
•
Avoid Right Outer Joins
•
Temporal Join to attribute views (with design time filter) will always execute.
•
Avoid Joins on “Calculated Columns” (consider materializing these columns)
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
21
Model pruning
•
Consider using Constant Mapping in UNIONs for efficient model pruning.
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
22
Use of DB engines
•
Execute in SQL Engine
ü recommended for HANA live* models.
•
Use table function vs. scripted calculation views
ü migration option in studio (SP09)
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
23
Degree of parallelization
•
Consider using UNION in place of JOINS in calculation view, esp. while dealing with similar dataset
(plan & actual).
•
Implementing a logical partitioning model, may reduce query runtime but will increase CPU usage.
•
Avoid single datasource (or node) feeding multiple nodes in calculation view.
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
24
Reduce network data transfer
•
To reduce the network data transfer (i.e. to push the joins to each node) in a scale-out deployment
ü Collocate the master and fact data tables in the same node
ü Replicate master data to all nodes (if the fact table is distributed across multiple node)
ü Tables in _SYS_BI schema can’t be replicated to all nodes
•
Consider applying multi-level partitioning with hash-partitioning (on single field) in the first level. This
will ensure, calculations can be computed on each node and each partition.
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
25
Caching
•
Introduced in SP08. Improved in SP09.
ü for complex* queries only
ü an option to improve concurrency
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
26
Deep Dive: the art
Modeling approach: follow the measures
Dimension
OLAP Engine is the fastest
engine in HANA.
Dimension
Dimension
Fact
Table
Dimension
•
Consider building models with a star-schema/cube. The fact table (measures) is surrounded by various
dimensions. The star-schema can be built using analytic views or star-join calculation views (comparable
performance in SP09).
•
Dimensions should be built as attribute views or dimension-calculation views.
•
Only business logics (very minimal joins) on top of the star-schema.
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
28
Modeling approach: follow the measures
Before
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
After
Public
29
Building it right: the first time
•
•
•
•
Move Business Logic into HANA
Performance tuning is not a separate activity. Keep an eye of the
query performance as you model.
Test the model early and when small (after the first join).
Thoroughly test the model (all join fields & calculated columns)
before moving to the next. Almost complete is not good enough.
Ensure tables are pruned when not queried and where clause
pushed down to the table-level. Check and confirm using plan
visualization.
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
30
Transformation
•
Degree of transformation or materialization
•
Master data (dimension)
ü recommended to simplify models (fields used in joins or filters) (same or new table)
•
Transaction (fact) data:
ü recommended to simplify models (fields used in joins or filters) (same table)
ü acceptable to simplify models (calculation before aggregation, data type conversion) (same table)
ü avoid creating aggregates/materialized table (absolute last option)
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
31
SAP HANA Modeling health check
Data transfer
Minimize data transfer between Analytical views, Calculation
views & front end tools (or SAP HANA clients)
Avoid executing open ended queries (using KEYs)
Reducing the data set as early as possible
Enforce data aggregation (GROUP BYs) using fewer (and
more granular) dimensions
Use data filters (WHERE clauses, prompts, constraint filters
& Analytical privileges)
Joins
Use Unions (with constants) to combine large data sets
Join views (within Calculation Views, within Universes or
using SQL) with caution. Use WHERE clauses + data filters
that can be pushed down into each view
Define cardinality N:1 fact to dimension (star schema) &
enforce left-outer join where possible
Minimize long join chains and big data set joins. Where
possible move data foundation joins into Attribute views
Calculations
Imbed business logic calculations into modeled views instead of within SQL statements, Procedures, Universes or front end
Minimize the use of expensive calculations, row based expressions & data manipulation including Calculated attributes
Transform complex expressions before hand using SLT, Data Services or Generated Columns
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
32
HANA SP11 and beyond
•
Primary use of calculation views: In future releases
(SP11 and beyond), HANA optimizer will select
appropriate engines during query execution. This will
eliminate the need for creating different types of views for
different engines (i.e. star-join in calculation view will be
comparable to analytic view). However the model design
principles remains the same. We’ll still create dimensions
& star-schema views, but using calculation views.
•
Support of existing Views: HANA will be backyard
compatible and will still support analytic & attribute views.
Also a migration tool may be available to convert the
attribute and analytic views to calculation views.
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
33
Customer Workshop
Beverage Major
•
Stats
ü before: 5mins, after: 3sec
•
Changes (SP08)
ü
ü
ü
ü
ü
Move business logic to HANA layer
Efficient SQL query (some were 5000 lines)
Use appropriate filter
Avoid single node feeding multiple nodes.
Constant mapping in unions
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
35
Pension Plan Management
•
Stats
ü before: 30sec, after: 3sec
ü before: 2min (concurrency 5), after: 7 sec (concurrency 15)
•
Changes (SP08, HEC)
ü
ü
ü
ü
ü
ü
ü
Efficient SQL query
Implement star-schema (analytical view)
Push high cardinality joins to the lowest level (analytic & attribute views)
Use left outer joins. Set cardinality (n:1 or 1:1)
Convert joins to unions (esp. joins of analytical views)
Avoid single node feeding multiple nodes.
Avoid decision table and scripted-calculation view*
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
36
Grocery Retailer
•
Stats
Before
After
ü before: 19sec, after: 300ms (1 user)
•
Changes (SP08)
ü
ü
ü
ü
ü
Implement star-schema (analytical view)
Push high cardinality joins to the lowest level
Convert joins to unions.
Use left outer joins. Set cardinality (n:1 or 1:1)
Avoid single node feeding multiple nodes.
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
37
Audit Firm
•
Stats
•
Firm Stats
ü before: 19min, after: 8sec (1 user)
•
•
My Stats
ü before: 17sec, after: 1sec
ü before: 2min (concurrency 10), after 40sec
(concurrency 400)
Changes (SP09)
ü
ü
ü
ü
ü
Efficient SQL query
Use left outer joins. Set cardinality (n:1 or 1:1)
Set optimize join = true (SP09)
Scripted calculation views to table functions (SP09).
Push high cardinality joins to the lowest level
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
38
Questions & Answers
Please follow the blog post for Questions & Answers
https://scn.sap.com/community/hana-in-memory/blog/2015/11/03/the-art-and-science-of-hanaperformance-modeling
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
39
Thank you
Contact information:
Abani Pattanayak,
SAP HANA Distinguished Engineer
Principal Consultant, SAP HANA CoE
[email protected] / 1 647-406-3392
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company.
SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate
company) in Germany and other countries. Please see http://global12.sap.com/corporate-en/legal/copyright/index.epxfor additional trademark information and notices.
Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors.
National product specifications may vary.
These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP SE or its
affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP SE or SAP affiliate company products and
services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as
constituting an additional warranty.
In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop
or release any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future
developments, products, and/or platform directions and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time
for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forwardlooking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place
undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.
© 2015 SAP SE or an SAP affiliate company. All rights reserved.
Public
41