May 24, 2016
Recently I got a fun question about an “upsert” pattern as a “Dear SQL DBA” question. The question is about TSQL, so it lent itself to being answered in a blog post where I can show repro code and screenshots.
We have a lookup table, which is the parent table in a foreign key relationship with a child table. The child table has lots more rows. When data comes in, we need to:
- Check the Parent Table (the lookup table) to see if the ParentValue is already present.
- If ParentValue is not present, insert a row for the new ParentValue into the Parent Table. This happens very rarely, but we need to check for it.
- Then insert a row into the Child Table, using the ParentId.
The problem: blocking against the parent table
When run under a few sessions, locking and blocking issues were creeping up fast. A merge statement was being used to check for the values in the Parent Table and insert them when they weren’t present.
Here’s some sample code. This creates a database named upsert_test, then creates ParentTable and ChildTable objects in the database. Finally, a Foreign Key is created on ChildTable, referencing the ParentTable.
IF DB_ID('upsert_test') is not null BEGIN USE MASTER; ALTER DATABASE upsert_test SET SINGLE_USER WITH ROLLBACK IMMEDIATE DROP DATABASE upsert_test; END GO CREATE DATABASE upsert_test; GO use upsert_test; GO CREATE TABLE dbo.ParentTable ( ParentId int IDENTITY(1,1) NOT NULL, ParentValue varchar(50) NOT NULL, CONSTRAINT pk_ParentTable PRIMARY KEY CLUSTERED (ParentId ASC) ); GO CREATE TABLE dbo.ChildTable ( ChildId INT IDENTITY(1,1) NOT NULL, ParentId INT NOT NULL, ChildValue VARCHAR(50) NOT NULL, CreatedDate DATE NOT NULL CONSTRAINT DF_Work_created DEFAULT (getdate()), CONSTRAINT PK_Work PRIMARY KEY CLUSTERED (ChildId ASC) ); GO ALTER TABLE ChildTable ADD CONSTRAINT FK_Work_Source FOREIGN KEY (ParentId) REFERENCES ParentTable (ParentId); GO
The upsert (aka merge query)
A stored procedure is used to handle incoming values. It uses MERGE to look for matching rows in ParentTable, and insert when not matched.
IF OBJECT_ID('dbo.DoStuff') IS NULL EXEC ('CREATE PROCEDURE dbo.DoStuff as RETURN 0;'); GO ALTER PROCEDURE dbo.DoStuff ( @ParentValue varchar(50), @ChildValue varchar(50) ) AS MERGE ParentTable with (HOLDLOCK) as p USING (SELECT @ParentValue NewParentValue) as new ON p.ParentValue = new.NewParentValue WHEN NOT MATCHED THEN INSERT (ParentValue) VALUES (new.NewParentValue); INSERT INTO ChildTable (ParentId, ChildValue) SELECT p.ParentId, @ChildValue FROM ParentTable p WHERE p.ParentValue=@ParentValue; GO
Why is that HOLDLOCK hint in the merge query?
My reader quite rightly used this hint in their merge query. Although MERGE looks like a single query, it’s actually just “syntactic sugar”. Behind the scenes, merge can be implemented as a select and an insert in two separate commands. Developers are advised to use HOLDLOCK to avoid race conditions with MERGE.
I asked one clarifying question – was the lock wait type they were seeing “LCK_M_RS_U”?
This confirmed that HOLDLOCK and merge were slowing them down instead of helping them.
Let’s Populate Some Rows for Testing and Reproduce the Blocking
exec dbo.DoStuff @ParentValue='Stuff', @ChildValue='Things'; GO exec dbo.DoStuff @ParentValue='MoreStuff', @ChildValue='MoreThings'; GO exec dbo.DoStuff @ParentValue='EvenMoreStuff', @ChildValue='EvenMoreThings'; GO exec dbo.DoStuff @ParentValue='EvenMoreStuff', @ChildValue='EvenMoreThings x 2'; GO exec dbo.DoStuff @ParentValue='EvenMoreStuff', @ChildValue='EvenMoreThings x 3'; GO /* Create 5000 more ParentValues */ SET NOCOUNT ON; DECLARE @namevarchar varchar(50), @i int=1 BEGIN TRAN WHILE @i <= 5000 BEGIN SET @namevarchar= cast(RAND() AS VARCHAR(50)); EXEC dbo.DoStuff @ParentValue=@namevarchar, @ChildValue='Whatever'; SET @i=@i+1; END COMMIT GO
To see the blocking issue, just run the following code in three session windows at the same time. Note that we’re running this over and over with the same ParentValue, and the ParentValue of “Stuff” is already in the table. This will not have to insert any rows into ParentTable.
SET NOCOUNT ON; exec dbo.DoStuff @ParentValue='Stuff', @ChildValue='Things'; GO 1000000
Here’s what the blocking looks like in Adam Machanic’s sp_WhoIsActive:
HOLDLOCK = serializable isolation level = key range locks
The holdlock hint is a way to get serializable isolation level in SQL Server for a specific table, without having to change the isolation level for your entire session. Serializable is the highest isolation level in SQL Server using pessimistic locking.
When you “HOLDLOCK”, you tell SQL Server to protect any rows you read with a range lock– just in case someone comes along and tries to change one or sneak one in.
That means that even when you’re just reading ParentTable and not inserting a row, you’re taking out a key range lock. You’re willing to fight other users over those rows to protect your statement.
There’s two parts to getting around the blocking and making this faster…
Index the parent table (solution part 1)
Currently, the only index on ParentTable is on ParentId.
Even if ParentTable is tiny, if we’re frequently accessing the table and looking up a ParentValue, we’ll benefit from creating a nonclustered index on that column. We should also allow only unique values into ParentValue for data integrity reasons. A unique nonclustered index is exactly what we need:
CREATE UNIQUE NONCLUSTERED INDEX ix_ParentTable_ParentValue on dbo.ParentTable(ParentValue) GO
In my simple three session test, this makes the merge statement very efficient, and performance goes way up. You can no longer catch those LCK_M_RS_U waits in sp_WhoIsActive. However, I’m still concerned about them, and would still…
Ditch the merge (solution part 2)
The “merge” command in SQL Server is often a let-down for folks. The syntax is confusing, most folks find out about the race conditions/concurrency issues the hard way, and the biggest problem is that it often seems “better” than other TSQL options because it was introduced as an enhancement in SQL Server 2008… but it isn’t always the better choice.
In this case, ditching the merge gives me more granular control of when I want to use that high level lock on ParentTable. The code is longer mostly because of a lot of comments.
ALTER PROCEDURE dbo.DoStuff ( @ParentValue varchar(50), @ChildValue varchar(50) ) AS DECLARE @ParentId INT; /* ParentId is very rarely new, so check for it first with only a shared lock */ SELECT @ParentId=ParentId FROM dbo.ParentTable WHERE ParentValue=@ParentValue /* Insert the new value if we have to. */ /* Use the SELECT WITH UPDLOCK in case of race conditions */ /* Get the new ParentId so we don't have to rejoin back to the table */ IF @ParentId IS NULL BEGIN DECLARE @OutputVal TABLE (ParentId INT) INSERT dbo.ParentTable (ParentValue) OUTPUT inserted.ParentId INTO @OutputVal(ParentId) SELECT x.newval FROM (SELECT @ParentValue as newval) as x LEFT JOIN dbo.ParentTable as p WITH (UPDLOCK, HOLDLOCK) on x.newval=p.ParentValue WHERE p.ParentValue IS NULL; /* We are only ever going to have one row in @OutputVal */ SELECT @ParentId=ParentId FROM @OutputVal; END INSERT INTO dbo.ChildTable (ParentId, ChildValue) SELECT @ParentId, @ChildValue; GO
In our scenario, it’s rare for new ParentValues to come in. So I’ve used a pattern to try to use as many shared locks against ParentTable as possible, stick with the Read Committed Isolation level, and still protect against race conditions. The pattern is essentially this:
- Check if ParentValue already exists (shared read locks only)
- If this is the rare case that a ParentValue does not exist…
- Insert the row into ParentTable
- Protect against race conditions by inserting with a SELECT joining to ParentTable with UPDLOCK, HOLDLOCK in case the same new row happens to come in on two sessions at the same time
- Use the OUTPUT clause to get the new @ParentId so we don’t have to join to ParentTable again in the next step
- Insert into ChildTable
Measure it: do these changes help?
I didn’t do thorough load testing on this. I just ran the call to dbo.DoStuff above in three sessions in a 4 core VM on my MacBook and looked at the BatchRequests/sec performance counter in SQL Server. Here’s what I saw:
|Setup||Batch Requests/sec with No Nonclustered Index on dbo.Parent||Batch Requests/sec with Nonclustered Index on dbo.Parent|
|Insert with Left Join||495 avg||4000+|
In this test case, adding the nonclustered index makes a bigger difference than changing the TSQL. But I would still move away from merge, because I want to be able to control when anything tougher than a read lock is being taken out against ParentTable – that’s very attractive since new values come in rarely in this case. The more concurrent sessions that are running this, the more that will help.
Don’t forget to handle errors
Error handling is important! The code in this post doesn’t have it for simplicity reasons. If you need a starter guide for error handling, check out Erland Sommarskog’s excellent whitepaper.
Further tuning thoughts
This code can be tuned further, but I’d want to set up a really clean load test using application servers (not SSMS) against a proper instance (maybe NOT on a MacBook). I would look at:
- Whether validating ParentValue could be done in memory in the client application tier. Avoiding constant data access against dbo.ParentTable is attractive if that’s possible.
- Wait statistics during execution to point to the next thing to tune.
What do you think of merge?
Do you love or hate merge statements in SQL Server?