IT Answers » SQL query optimization

Challenge improving query performance on large tables

Sport2000 — Tue, 27 Sep 2011 14:21:07 +0000

Hi,

Here’s the structure of the db I’m using: * Table frs_ArticlesBDFournisseur which contains article main data.One row per article/color. Key is Refrefart (int). Size is over 220000 rows at the momemt. Only index is the PK (clusterred). * Table frs_ArticlesBDFournisseurTaille contains all the various sizes and barcode for the corresponding article/color from the previous table. Key is RefREFARTTAILLE (int). Size is over 1200000 rows at the moment. refrefart column also exists in this table to link with the previous table, altough there is no physical constraint linking the 2 tables. Indexes are PK (clustered), refrefart (non clustered) and refrefart,taille,codeean,codeeanfrs (non clustered).

Here’s what I want to do: I work at headquarters of franchise shops. I have to collect data from each selling point, match it with our own database (the one discribed above) to normalise the incoming data and complete it with more details. The best way to match an article is to use its barcode (when it exits in the shop data). We pack up to 2 barcodes with every article (callled CodeEan and CodeEanFrs from frs_ArticlesBDFournisseurTaille). I need to complete the information I have with the article size (called taille from frs_ArticlesBDFournisseurTaille), the article’s season (called Saison from frs_ArticlesBDFournisseur) and the number of different season the article has been existing in (to know whether it is a one season product or a permanent product).

Here’s the challenge for the query at hand: There is no constant data in frs_ArticlesBDFournisseurTaille with which I can do a GROUP BY. I need to get the latest product data (in case there is more than one season) and the total number of seasons.

Here’s the query I’ve designed: SELECT M.*, ART_REF, A.ART_PSEUDO, ART_NOM, ART_FEDAS, ART_ANNEECREATION, COU_CODE, COU_NOM, TGF_NOM, MRK_IDREF, MRK_NOM, MRK_NOM_ORI, CBI_EAN, CBI_EANFRS, B.RefRefArt, B.Taille AS sp2kTaille, CASE when B.CodeEan = ” then null else B.CodeEan END AS sp2kEan, CASE when B.CodeEanFrs = ” then null else B.CodeEanFrs END AS sp2kEanFrs, B.Saison, Coalesce(B.Pos, 0) as Permanent FROM gin_Mouvements M LEFT JOIN gin_Articles A ON M.MOV_ARTID=A.ART_ID LEFT JOIN gin_Couleurs C ON M.MOV_COUID=C.COU_ID LEFT JOIN gin_Tailles T ON M.MOV_TGFID=T.TGF_ID LEFT JOIN gin_Marques N ON A.ART_MRKID=N.MRK_ID LEFT JOIN gin_Ean E ON (M.MOV_ARTID=E.CBI_ARTID AND M.MOV_COUID=E.CBI_COUID AND M.MOV_TGFID=E.CBI_TGFID) OUTER APPLY (SELECT TOP 1 T.RefRefArt, CodeEan, CodeEANFrs, Taille, Art.Saison, Rank() over (order by T.RefREFART ASC) as Pos FROM frs_ArticlesBDFournisseurTaille T LEFT JOIN frs_ArticlesBDFournisseur Art ON T.RefREFART=Art.RefREFART WHERE (CodeEAN = E.CBI_EAN OR CodeEANFrs = CBI_EANFRS OR CodeEANFrs = CBI_EAN OR CodeEAN = CBI_EANFRS) ORDER BY RefREFART DESC) B

Query explained: All the first part, with all the left join, is to gather the data, relatively small, from the shops. There is no performance issue here. I’ve used an outer apply to avoid having to do a left join and the ON conditions which I couldn’t get to work to return at the same time the max(refrefart) and the total number of season; remember I can not make a group by.

Performances: This query runs at a 20row/s at best (nearly 20s to return 500 rows). The data in both tables described at the beginning is bound to increase twice a year by about 10000 new products and 6 times more for the barcodes. I’m afraid the performances will be degrading even more soon.

Advice ?: If anyone can see how to improve the query or rewrite the query differently or improve the db or else … please let me know.

Stats: Execution plan sum up: it is spending 57% on Nested Loops (Left Outer Join) and 40% on Index Scan (NonClustered)[sp2k_staging].[dbo].[frs_ArticlesBDFournisseurTaille].
Thank you.

Infinite recursion in SQL Server 2000

Gomsbk — Mon, 27 Sep 2010 06:52:18 +0000

consider a recursive query in SQL.How to handle infinite recursion to make the query evaluation optimized?

Joining 3 tables in SQL Server 2005

Macha1973 — Thu, 12 Aug 2010 12:37:01 +0000

Here are the 3 simple SELECT queries i have but i want to merge them all into 1 singl query.

Improving the initial speed of a SQL query

SQL Server Ask the Experts — Tue, 21 Jul 2009 18:11:54 +0000

I have optimized my SQL query to the maximum extent.

The first time I ran the query, it took 7 seconds; the next time it took half a second.

Why did this happen? How can I make the query run faster the first time around?

Sql Query in AS400

Mr. SEA — Mon, 20 Jul 2009 07:21:04 +0000

I am trying to execute a Query String but I receive following error
Query exceeded the specified time limit or storage limit
however my query length is normal, nevertheless i have executed larger query too.

How to solve this problem.

SQL Server 2005 Query Very Slow in the first run

Jsql — Fri, 17 Jul 2009 10:15:20 +0000

New Discussion Post by

Slow SQL Query

Mxaxyogesh2002 — Mon, 13 Jul 2009 10:00:04 +0000

when i run below all SQL Query simulataneously against 1500000 Rows its take
10 minutes
i put the non-clustered index on the columns TATCallType ,CallTo
the data type of both columns is varchar

select ID,AuditMaster_ID,CallTo,CallTypeTag into Auditdata_callto from auditdata where AuditData.TATCallType is null and substring(AuditData.CallTo,1,convert(int,0))=” and len(AuditData.CallTo)=10

select ID,AuditMaster_ID,CallTo,CallTypeTag into Auditdata_callto from auditdata where AuditData.TATCallType is null and substring(AuditData.CallTo,1,convert(int,1))=’0′ and len(AuditData.CallTo)=11

please help me its urgent

thanks in advance

Finding nth most recent action with SQL

Munona99 — Mon, 18 May 2009 22:11:41 +0000

I am trying to determine the date on which a certain number of things have happened cumulatively, eg I have a varying number of buyers each month. I want to find out on what date of that month, those buyers make their 3rd purchase. My SQL doesn’t seem to support TOP or ROWCOUNT so I would appreciate alternatives

Different Results on Insert Select than Select

Bem — Sun, 03 May 2009 23:36:10 +0000

I’m struggling, along with others, where I’m getting different results when doing an Insert select verses just the select.

Lots of joins to get data from various tables. Only want the first record from the uniqfield table so created the tmp_uniqfieldid table with that item fielded, creation_ts thinking it would help matters…..

Here is the query… The insert gets only 44 items where the same select gets the correct 166 items

insert into pack(select a.id, createdate,
createtime, d.user, to_char(b.received_ts, ‘YYYYMMDD’),
substr(to_char(b.received_ts, ‘HH24MISSFF’),1,8)

FROM uniqfield a, table b, table c, table d,
(Select id, min(creation_ts) creation_ts from uniquefield where fieldid in (select fieldid from pitest.tmp_uniqfieldid)
group by id) uniqfiled_id

where id >’ ‘ and ufield_id.id=a.id and
a.creation_ts = ufield_id.creation_ts and
createdate between 20040714 and 20040731 and
a.fieldid=b.fieldid and b.wk_id =c.wk_id and
trim(c.user_id)=trim(d.user_id))

—–

1) id, createdate, createtime from table uniqfiled a
2) user from table d where useid in table d=userid in table a
3) recvdata, recvdtime from table c need to get wk_id from table b where fieldid in table b =fielded in table a then get wk_id in table b to get recvdate and recvdtime from table c

SQL order by Issue

Rohit3312 — Sat, 02 May 2009 16:44:32 +0000

Please help me write this.
I need a SQL where the order by is done based on a variable passed to the query.SELECT *
FROM (SELECT distinct gl.group_id,
gl.group_name,
gl.group_description,
gl.status_code,
gl.member_count,
(SELECT grpp.group_name
FROM test_group_relationship grel JOIN test_group grpp
ON grel.parent_group_id = grpp.group_id
WHERE grel.child_group_id = gl.group_id
) AS parent_group_name,
gl.group_name_key,
gl.group_description_key
FROM test_group AS gl
WHERE gl.group_org_id = ’3909′
AND (gl.group_name_key LIKE ‘%GROUP%’)
) AS data_set
ORDER BY DECODE(:sort_key,
‘name’, ‘constant’,
‘description’, group_description_key,
‘memberCount’, LPAD(member_count, 4),
‘status’, LPAD(status_code, 4),
‘parentGroupName’, parent_group_name
)NULLS FIRST,
UPPER(SUBSTR(group_name, 1, 1)),
SUBSTR(group_name, 1, 1) DESC,
UPPER(group_name),
group_name DESC,
group_name_key,
;

When,sort_key :=’name’.As we can see it evaluates to a constant and then the remaining columns in the ORDER BY clause.

Now,when variable sort_key:= ‘description’ or ‘memberCount’….each of the values, then the ORDER BY LOGIC is different for each of them.

n brief,each of the options that the sort_key variable gets-the ORDER BY logic is different and i have to implement it in the same SQL only now.