IT Answers » Query optimization

Challenge improving query performance on large tables

Sport2000 — Tue, 27 Sep 2011 14:21:07 +0000

Hi,

Here’s the structure of the db I’m using: * Table frs_ArticlesBDFournisseur which contains article main data.One row per article/color. Key is Refrefart (int). Size is over 220000 rows at the momemt. Only index is the PK (clusterred). * Table frs_ArticlesBDFournisseurTaille contains all the various sizes and barcode for the corresponding article/color from the previous table. Key is RefREFARTTAILLE (int). Size is over 1200000 rows at the moment. refrefart column also exists in this table to link with the previous table, altough there is no physical constraint linking the 2 tables. Indexes are PK (clustered), refrefart (non clustered) and refrefart,taille,codeean,codeeanfrs (non clustered).

Here’s what I want to do: I work at headquarters of franchise shops. I have to collect data from each selling point, match it with our own database (the one discribed above) to normalise the incoming data and complete it with more details. The best way to match an article is to use its barcode (when it exits in the shop data). We pack up to 2 barcodes with every article (callled CodeEan and CodeEanFrs from frs_ArticlesBDFournisseurTaille). I need to complete the information I have with the article size (called taille from frs_ArticlesBDFournisseurTaille), the article’s season (called Saison from frs_ArticlesBDFournisseur) and the number of different season the article has been existing in (to know whether it is a one season product or a permanent product).

Here’s the challenge for the query at hand: There is no constant data in frs_ArticlesBDFournisseurTaille with which I can do a GROUP BY. I need to get the latest product data (in case there is more than one season) and the total number of seasons.

Here’s the query I’ve designed: SELECT M.*, ART_REF, A.ART_PSEUDO, ART_NOM, ART_FEDAS, ART_ANNEECREATION, COU_CODE, COU_NOM, TGF_NOM, MRK_IDREF, MRK_NOM, MRK_NOM_ORI, CBI_EAN, CBI_EANFRS, B.RefRefArt, B.Taille AS sp2kTaille, CASE when B.CodeEan = ” then null else B.CodeEan END AS sp2kEan, CASE when B.CodeEanFrs = ” then null else B.CodeEanFrs END AS sp2kEanFrs, B.Saison, Coalesce(B.Pos, 0) as Permanent FROM gin_Mouvements M LEFT JOIN gin_Articles A ON M.MOV_ARTID=A.ART_ID LEFT JOIN gin_Couleurs C ON M.MOV_COUID=C.COU_ID LEFT JOIN gin_Tailles T ON M.MOV_TGFID=T.TGF_ID LEFT JOIN gin_Marques N ON A.ART_MRKID=N.MRK_ID LEFT JOIN gin_Ean E ON (M.MOV_ARTID=E.CBI_ARTID AND M.MOV_COUID=E.CBI_COUID AND M.MOV_TGFID=E.CBI_TGFID) OUTER APPLY (SELECT TOP 1 T.RefRefArt, CodeEan, CodeEANFrs, Taille, Art.Saison, Rank() over (order by T.RefREFART ASC) as Pos FROM frs_ArticlesBDFournisseurTaille T LEFT JOIN frs_ArticlesBDFournisseur Art ON T.RefREFART=Art.RefREFART WHERE (CodeEAN = E.CBI_EAN OR CodeEANFrs = CBI_EANFRS OR CodeEANFrs = CBI_EAN OR CodeEAN = CBI_EANFRS) ORDER BY RefREFART DESC) B

Query explained: All the first part, with all the left join, is to gather the data, relatively small, from the shops. There is no performance issue here. I’ve used an outer apply to avoid having to do a left join and the ON conditions which I couldn’t get to work to return at the same time the max(refrefart) and the total number of season; remember I can not make a group by.

Performances: This query runs at a 20row/s at best (nearly 20s to return 500 rows). The data in both tables described at the beginning is bound to increase twice a year by about 10000 new products and 6 times more for the barcodes. I’m afraid the performances will be degrading even more soon.

Advice ?: If anyone can see how to improve the query or rewrite the query differently or improve the db or else … please let me know.

Stats: Execution plan sum up: it is spending 57% on Nested Loops (Left Outer Join) and 40% on Index Scan (NonClustered)[sp2k_staging].[dbo].[frs_ArticlesBDFournisseurTaille].
Thank you.

Infinite recursion in SQL Server 2000

Gomsbk — Mon, 27 Sep 2010 06:52:18 +0000

consider a recursive query in SQL.How to handle infinite recursion to make the query evaluation optimized?

AS400

CHARLESG — Tue, 20 Jul 2010 16:21:35 +0000

I am having problem with LF not reading correctly. Can you turn off QUERY Optimizer for one job or a program? If so how do you turn off Query Optimizer?

Calculating prior Query fields

Bostic87 — Tue, 12 May 2009 04:24:37 +0000

Inside my query, is four listed fields; date, printer, and a Start and End meter for daily meter readings to keep track of printer impressions. I’m trying to receive a return value by subtracting a meter reading from the START field in a current row and the END meter in the previous row depending on the printer selected in the PRINTER field. This will allow me to view the impressions from the shift before. I believe I need to use a DLOOKUP formula with a WHERE criteria but I’m not sure how to create it. Any help would be appreciated.

View is taking so much time for fetching data

440638 — Thu, 16 Apr 2009 09:05:55 +0000

i have created multilevel view (ie:- a view is inheritng data from multipe view)
and its taking so much time while fetching the data.
can i reduce time taking for fetching data.

Updating a field and counting unique field values from another fields all in the same query?

Grantp — Wed, 15 Apr 2009 10:59:15 +0000

How can I structure the following query to allow me to count the number of unique entries in a field named “amount” and using the same query, update another field named “unique”

Here is what I am doing:

SQL = “select amount, count(amount) from tblmessagein group by amount having count(*) = 1″
If rsMessageIn.State = adStateOpen Then
rsMessageIn.Close
rsMessageIn.Open SQL, cntblMessageIn, adOpenStatic, adLockOptimistic, -1
ElseIf rsMessageIn.State = adStateClosed Then
rsMessageIn.Open SQL, cntblMessageIn, adOpenStatic, adLockOptimistic, -1
End If
Dim TempID As String
‘Set the number of records that have been found to be unique – used for searching, sending and updating later
UniqueCount = rsMessageIn.RecordCount
‘Use the Record unique count to cycle through the original
‘amount values and then store them in unique message variables
For ii = 1 To UniqueCount
With rsMessageIn
UniqueMsgText(ii) = .Fields(0).OriginalValue
If UniqueCount > 1 Then .MoveNext
If UniqueCount = ii Then .MoveFirst
End With
Next ii
‘
For i = 1 To UniqueCount
SQLInbox3 = “select unique,amount from tblmessagein where amount = ‘” & UniqueMsgText(i) & “‘”
If rsMessageIn.State = adStateOpen Then
rsMessageIn.Close
rsMessageIn.Open SQL, cntblMessageIn, adOpenStatic, adLockOptimistic, -1
ElseIf rsMessageIn.State = adStateClosed Then
rsMessageIn.Open SQL, cntblMessageIn, adOpenStatic, adLockOptimistic, -1
End If
With rsMessageIn
.Fields(5) = 1 ‘Set record uniqueness status
.Update
End With
Next i
‘

This is what I would like to do, condense everything into one query instead of two as above, but I get “unique” is not part of the Aggregate function error:

SQL = “select unique,amount, count(amount) from tblmessagein group by amount having count(*) = 1″
If rsMessageIn.State = adStateOpen Then
rsMessageIn.Close
rsMessageIn.Open SQL, cntblMessageIn, adOpenStatic, adLockOptimistic, -1
ElseIf rsMessageIn.State = adStateClosed Then
rsMessageIn.Open SQL, cntblMessageIn, adOpenStatic, adLockOptimistic, -1
End If
Dim TempID As String
‘Set the number of records that have been found to be unique – used for searching, sending and updating later
UniqueCount = rsMessageIn.RecordCount
‘Use the Record unique count to cycle through the original
‘amount values and then store them in unique message variables
For ii = 1 To UniqueCount
With rsMessageIn
rsMessageIn.Fields(5) = 1 ‘Set record uniqueness status
rsMessageIn.Update
If UniqueCount > 1 Then .MoveNext
If UniqueCount = ii Then .MoveFirst
End With
Next ii

Basicly all I need to do is get all records that are unique based on the amount field, then update a field named unique with a 1 for each of the records found to have unique amount values! Can somebody please help me out on this one, I’m pulling my hair out already, thanks!

Need help with this Oracle Query

FidanSheza — Sun, 05 Apr 2009 09:08:00 +0000

BPC_APPLICATIONS_TRANSACTIONS (Table)
Application_no,Transaction_no,Transaction_date Step_id Dept_code Decision
20090045 1 01/14/2009 10:07:17 1 001 0
20090045 2 01/14/2009 10:07:17 70 007 0
20090045 3 01/14/2009 10:08:57 1 001 0
20090045 4 01/14/2009 10:08:57 60 006 0
20090045 5 01/14/2009 10:14:12 71 007 1
20090045 6 01/14/2009 10:14:41 61 006 1
20090045 7 01/18/2009 11:45:13 73 007 1
20090045 8 01/18/2009 11:46:26 74 007 1
20090045 9 01/18/2009 11:47:13 7 001 1
20090045 11 01/18/2009 11:48:43 70 007 0
20090045 10 01/18/2009 11:48:43 1 001 0
20090045 12 01/27/2009 11:22:16 1 001 0
20090045 13 01/28/2009 10:30:56 71 007 1
20090045 14 01/28/2009 10:31:37 73 007 1
20090045 15 01/28/2009 10:32:14 74 007 1
20090045 16 01/28/2009 10:35:44 7 001 1
20090045 17 01/28/2009 10:35:53 1 001 0
20090045 18 01/28/2009 10:35:53 70 007 0
20090045 19 02/01/2009 10:35:53 71 007 0
—————————————————————————————————–
select a.application_no,a.transaction_date dateIn,b.transaction_date dateout,
a.dept_code,b.decision
from bpc_applications_transactions a, bpc_applications_transactions b
where a.dept_code=007 and a.step_id in (71,73) and b.step_id in (71,73) and a.transaction_date in
(select transaction_date from bpc_applications_transactions
where step_id=71 and a.application_no=b.application_no )

and b.transaction_date in (select transaction_date from bpc_applications_transactions
where step_id=73 and a.application_no=b.application_no )
and
b.decision in (select nvl(decision,0) from bpc_applications_transactions
where step_id=73 and a.application_no=b.application_no )
and a.application_no=b.application_no;
———————————————————————————————————
When I am running the above query , i am getting the output below:

APPLICATION_NO DATEIN DATEOUT DECISION

20090045 1/28/2009 10:30:56 ص 1/18/2009 11:45:13 ص 1
20090045 1/28/2009 10:30:56 ص 1/28/2009 10:31:37 ص 1
20090045 1/14/2009 10:14:12 ص 1/28/2009 10:31:37 ص 1
20090045 1/14/2009 10:14:12 ص 1/18/2009 11:45:13 ص 1
20090045 02/01/2009 10:35:53 1/28/2009 10:31:37 ص 1
20090045 02/01/2009 10:35:53 1/18/2009 11:45:13 ص 1
—————————————————————————————————-
But I should get the output as:

APPLICATION_NO DATEIN DATEOUT DECISION
20090045 1/28/2009 10:30:56 ص 1/28/2009 10:31:37 ص 1
20090045 1/14/2009 10:14:12 ص 1/18/2009 11:45:13 ص 1
20090045 02/01/2009 10:35 0

Can anyone help me to tune my query above to get the desired result?

Thanks,
SK…

Matching External lists to your Database?

Nik14 — Wed, 01 Apr 2009 22:54:41 +0000

Hello everyone,
I work in a telco and manage the data in the company’s CRM. I’m often given lists of company names by our alliance team, and asked to match the list against our database to see which companies already exist in the CRM and which don’t. At the moment I can only accomplish this by one-to-one matching, using a LIKE “%part of company name%” search in Access, examining the results, and choosing the closest match. The lists given to me by Alliance often have hand type names, using ampersands, excluding the INC off the end, incorrect spelling, inappropriate punctuation etc – which makes a SELECT WHERE ‘company name’ is equal query completely fruitless. My question is this – can anyone recommend string matching software or an SQL query that will wash one list against another, finding the closest match with confidence intervals? I don’t know if this would involve fuzzy logic or not – but I’m convinced there needs to be a more efficient way to go about doing this. Any advice is greatly appreciated, thanks a lot for reading.
Nik

To retrieve records from SQL Server database

Pichha — Thu, 05 Feb 2009 03:35:23 +0000

Hello All,

I have a requirement with sql server. Please find below the same.

In database say we have a column named “Name” in a table. The records that are available in the table are say like

Column_Name = Names

Lean Metric
Sequence Lean
Operation Metric Lean
Cleaning and so on….

Now, through some application we are giving the input string as Project, so all the three records shown above should be retieved.

In the application i have filtered the condition like “Select name from table where name contains “Project”. so i am now getting all the three records. But i am also getting the record “Cleaning” as Lean is available as part of Cleaning. Now, I should get records like cleaning.

How can I frame a query for this?

The thing is that application should search for the name that are anywhere in the column(in our case it should etch records corresponding to Lean and Cleaning..

Thanks in Advance.
Regards
Arun

Need help with Oracle query

Xaviervicky — Sat, 24 Jan 2009 06:46:01 +0000

how to display the third highest salary from an employees table
thanks in advance