Bulk insert nested xml with foreign key as identity column of first table

nikhil

I have an xml as follows:

<Records>
  <Record>
    <Name>Best of Pop</Name>
    <Studio>ABC studio</Studio>
    <Artists>
      <Artist>
        <ArtistName>John</ArtistName>
        <Age>36</Age>            
      </Artist> 
      <Artist>
        <ArtistName>Jessica</ArtistName>
        <Age>20</Age>            
      </Artist>
    </Artists>
  </Record>
  <Record>
    <Name>Nursery rhymes</Name>
    <Studio>XYZ studio</Studio>
    <Artists>
      <Artist>
        <ArtistName>Judy</ArtistName>
        <Age>10</Age>            
      </Artist> 
      <Artist>
        <ArtistName>Rachel</ArtistName>
        <Age>15</Age>            
      </Artist>
    </Artists>
  </Record>
</Records>

This file may contain millions of records. My MS SQL database, running on Azure SQL Database, has the following 2 tables to store these records:

  1. Record (RecordId [PK, identity, auto-increment], Name, Studio)

  2. Artist (RecordId [Foreign Key refers Record.RecordId], ArtistName, Age)

Is it possible to bulk insert records into the Record table, get the RecordIds and then bulk insert the artist information into the Artist table in a single traversal of the xml using the xml nodes approach?

I have been searching for an efficient way to do this for a long time but in vain.

I have tried approaches similar to the ones described here and here, but I'm not able to get to the solution.

Any pointers in the direction of the solution will be of great help.

Update: @srutzky: Thanks for the solution. This works exactly as I wanted. But there is one catch. I have to use the nodes approach for the solution. I have changed the first part of the query. But I'm stuck in the 2nd half. Here's what I have got up to.

DECLARE @Record TABLE (RecordId INT NOT NULL IDENTITY(1, 1) PRIMARY KEY,
                       Name NVARCHAR(400) UNIQUE,
                       Studio NVARCHAR(400));
DECLARE @Artist TABLE (ArtistId INT NOT NULL IDENTITY(1, 1) PRIMARY KEY,
                       RecordId INT NOT NULL,
                       ArtistName NVARCHAR(400), Age INT);

INSERT INTO @Record (Name, Studio)
   SELECT  T.c.value(N'(Name/text())[1]', 'NVARCHAR(400)'),
           T.c.value(N'(Studio/text())[1]', 'NVARCHAR(400)')
 FROM @ImportData.nodes('/Records/Record') T(c);

SELECT * FROM @Record

Could you please help me out with the 2nd part? I'm new to this xml handling approach.

UPDATE2: And I got it.... I racked my brains for a couple of hours, tried a few things and finally arrived at the solution.

DECLARE @Record TABLE (RecordId INT NOT NULL IDENTITY(1, 1) PRIMARY KEY,
                       Name NVARCHAR(400) UNIQUE,
                       Studio NVARCHAR(400));
DECLARE @Artist TABLE (ArtistId INT NOT NULL IDENTITY(1, 1) PRIMARY KEY,
                       RecordId INT NOT NULL,
                       ArtistName NVARCHAR(400), 
                       Age INT);

INSERT INTO @Record (Name, Studio)
   SELECT  T.c.value(N'(Name/text())[1]', 'NVARCHAR(400)'),
           T.c.value(N'(Studio/text())[1]', 'NVARCHAR(400)')
 FROM @ImportData.nodes('/Records/Record') T(c);

INSERT INTO @Artist (RecordId, ArtistName, Age)
    SELECT  (SELECT RecordId FROM @Record WHERE Name=T.c.value(N'(../../Name/text())[1]', 'NVARCHAR(400)')),
            T.c.value(N'(ArtistName/text())[1]', 'NVARCHAR(400)'),
           T.c.value(N'(Age/text())[1]', 'INT')
 FROM @ImportData.nodes('/Records/Record/Artists/Artist') T(c);

 SELECT * FROM @Record
 SELECT * FROM @Artist

@srutzky: Thanks a ton for pointing me in the right direction. Any suggestions to improve this solution are welcome.

Solomon Rutzky

This can't be done in a single pass anyway as you can't insert into two tables in the same DML statement (well, outside of Triggers and the OUTPUT clause, neither of which would help here). But it can be done efficiently in two passes. The fact at the <Name> element within <Record> is unique is the key, as that allows us to use the Record table as the lookup table for the second pass (i.e. when we are getting the Artist rows).

First, you need (well, should) create a UNIQUE INDEX on Record (Name ASC). In my example below I am using a UNIQUE CONSTRAINT, but that is only due to my using a table variable instead of a temp table to make the example code more easily rerunnable (without needing an explicit IF EXISTS DROP at the top). This index will help the performance of the second pass.

The example uses OPENXML as that will most likely be more efficient that using the .nodes() function since the same document needs to be traversed twice. The last parameter for the OPENXML function, the 2, specifies that the document is "Element-based" since the default parsing is looking for "Attribute-based".

DECLARE @DocumentID INT, @ImportData XML;

SET @ImportData = N'
<Records>
  <Record>
    <Name>Best of Pop</Name>
    <Studio>ABC studio</Studio>
    <Artists>
      <Artist>
        <ArtistName>John</ArtistName>
        <Age>36</Age>            
      </Artist> 
      <Artist>
        <ArtistName>Jessica</ArtistName>
        <Age>20</Age>            
      </Artist>
    </Artists>
  </Record>
  <Record>
    <Name>Nursery rhymes</Name>
    <Studio>XYZ studio</Studio>
    <Artists>
      <Artist>
        <ArtistName>Judy</ArtistName>
        <Age>10</Age>            
      </Artist> 
      <Artist>
        <ArtistName>Rachel</ArtistName>
        <Age>15</Age>            
      </Artist>
    </Artists>
  </Record>
</Records>';


DECLARE @Record TABLE (RecordId INT NOT NULL IDENTITY(1, 1) PRIMARY KEY,
                       Name NVARCHAR(400) UNIQUE,
                       Studio NVARCHAR(400));
DECLARE @Artist TABLE (ArtistId INT NOT NULL IDENTITY(1, 1) PRIMARY KEY,
                       RecordId INT NOT NULL,
                       ArtistName NVARCHAR(400), Age INT);

EXEC sp_xml_preparedocument @DocumentID OUTPUT, @ImportData;

-- First pass: extract "Record" rows
INSERT INTO @Record (Name, Studio)
   SELECT Name, Studio
   FROM   OPENXML (@DocumentID, N'/Records/Record', 2) 
             WITH (Name    NVARCHAR(400)  './Name/text()', 
                   Studio  NVARCHAR(400)  './Studio/text()');


-- Second pass: extract "Artist" rows
INSERT INTO @Artist (RecordId, ArtistName, Age)
   SELECT rec.RecordId, art.ArtistName, art.Age
   FROM   OPENXML (@DocumentID, N'/Records/Record/Artists/Artist', 2) 
             WITH (Name        NVARCHAR(400)  '../../Name/text()',
                   ArtistName  NVARCHAR(400)  './ArtistName/text()', 
                   Age         INT  './Age/text()') art
   INNER JOIN @Record rec
           ON rec.[Name] = art.[Name];


EXEC sp_xml_removedocument @DocumentID;
-------------------

SELECT * FROM @Record ORDER BY [RecordID];
SELECT * FROM @Artist ORDER BY [RecordID];

References:

EDIT:
With the new requirement to use the .nodes() function instead of OPENXML, the following will work:

DECLARE @ImportData XML;

SET @ImportData = N'
<Records>
  <Record>
    <Name>Best of Pop</Name>
    <Studio>ABC studio</Studio>
    <Artists>
      <Artist>
        <ArtistName>John</ArtistName>
        <Age>36</Age>            
      </Artist> 
      <Artist>
        <ArtistName>Jessica</ArtistName>
        <Age>20</Age>            
      </Artist>
    </Artists>
  </Record>
  <Record>
    <Name>Nursery rhymes</Name>
    <Studio>XYZ studio</Studio>
    <Artists>
      <Artist>
        <ArtistName>Judy</ArtistName>
        <Age>10</Age>            
      </Artist> 
      <Artist>
        <ArtistName>Rachel</ArtistName>
        <Age>15</Age>            
      </Artist>
    </Artists>
  </Record>
</Records>';

IF (OBJECT_ID('tempdb..#Record') IS NOT NULL)
BEGIN
   DROP TABLE #Record;
END;
IF (OBJECT_ID('tempdb..#Artist') IS NOT NULL)
BEGIN
   DROP TABLE #Artist;
END;

CREATE TABLE #Record (RecordId INT NOT NULL IDENTITY(1, 1) PRIMARY KEY,
                      Name NVARCHAR(400) UNIQUE,
                      Studio NVARCHAR(400));
CREATE TABLE #Artist (ArtistId INT NOT NULL IDENTITY(1, 1) PRIMARY KEY,
                      RecordId INT NOT NULL,
                      ArtistName NVARCHAR(400),
                      Age INT);


-- First pass: extract "Record" rows
INSERT INTO #Record (Name, Studio)
   SELECT col.value(N'(./Name/text())[1]', N'NVARCHAR(400)') AS [Name],
          col.value(N'(./Studio/text())[1]', N'NVARCHAR(400)') AS [Studio]
   FROM   @ImportData.nodes(N'/Records/Record') tab(col);


-- Second pass: extract "Artist" rows
;WITH artists AS
(
   SELECT col.value(N'(../../Name/text())[1]', N'NVARCHAR(400)') AS [RecordName],
          col.value(N'(./ArtistName/text())[1]', N'NVARCHAR(400)') AS [ArtistName],
          col.value(N'(./Age/text())[1]', N'INT') AS [Age]
   FROM   @ImportData.nodes(N'/Records/Record/Artists/Artist') tab(col)
)
INSERT INTO #Artist (RecordId, ArtistName, Age)
   SELECT rec.RecordId, art.ArtistName, art.Age
   FROM artists art
   INNER JOIN #Record rec
           ON rec.[Name] = art.RecordName;

-- OR --
-- INSERT INTO #Artist (RecordId, ArtistName, Age)
   SELECT rec.RecordId,
          col.value(N'(./ArtistName/text())[1]', N'NVARCHAR(400)') AS [ArtistName],
          col.value(N'(./Age/text())[1]', N'INT') AS [Age]
   FROM   @ImportData.nodes(N'/Records/Record/Artists/Artist') tab(col)
   INNER JOIN #Record rec
           ON rec.Name = col.value(N'(../../Name/text())[1]', N'NVARCHAR(400)');

-------------------

SELECT * FROM #Record ORDER BY [RecordID];
SELECT * FROM #Artist ORDER BY [RecordID];

There are two options for inserting into #Artist shown above. The first uses a CTE to abstract the XML extraction away from the INSERT / SELECT query. The other is a simplified version, similar to your query in UPDATE 2 of the question.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Bulk insert nested xml with foreign key as identity column of first table

From Dev

Bulk insert with destination table identity column

From Dev

Bulk Insert into SQL Table with unique ID column that is not Identity

From Dev

Insert foreign key into table

From Dev

Foreign key in the first table

From Dev

Xml bulk insert in oracle table

From Dev

How to add a Foreign key in Customer table (CreatedBy column) for AspNetUser table (Id column) in ASP.NET MVC 5 Identity 2.0 using Code First

From Dev

Insert row into table with only one column, primary key and identity

From Dev

"Cannot insert explicit value for identity column in table when IDENTITY_INSERT is set to OFF" with composite key

From Dev

"Cannot insert explicit value for identity column in table when IDENTITY_INSERT is set to OFF" with composite key

From Dev

Insert data into foreign key column

From Dev

Insert a new column into the Foreign Key table when a particular column in the primary key table is Update (Using Triggers)

From Dev

C# Code first foreign key. The foreign key refers to the non valid column in table with references

From Dev

Mysql insert in two tables where the first table has foreign key and update the first table

From Dev

How to insert Primary Key value of the primary table to the Foreign Key column of the child table in MySQL?

From Dev

insert rows into table with identity column

From Dev

Unable to insert into table with Identity Column

From Dev

Remove table name prefix in EF code first foreign key table column

From Dev

How to improve the speed of bulk_update on a foreign key of a large table?

From Dev

INSERT FOREIGN KEY in another table with executemany() in PostgreSQL

From Dev

Insert random data into a table that has foreign key

From Dev

SQL insert data into table has foreign key

From Dev

SQL insert data into table has foreign key

From Dev

How to insert data in a table with foreign key

From Dev

Insert values in table with foreign key in php

From Dev

Get AutoIncrement ID and insert into Foreign Key Table

From Dev

insert data into table having foreign key

From Dev

Insert values into a table with a foreign key using mysql

From Dev

INSERT conflicted with foreign key constraint for lookup table

Related Related

  1. 1

    Bulk insert nested xml with foreign key as identity column of first table

  2. 2

    Bulk insert with destination table identity column

  3. 3

    Bulk Insert into SQL Table with unique ID column that is not Identity

  4. 4

    Insert foreign key into table

  5. 5

    Foreign key in the first table

  6. 6

    Xml bulk insert in oracle table

  7. 7

    How to add a Foreign key in Customer table (CreatedBy column) for AspNetUser table (Id column) in ASP.NET MVC 5 Identity 2.0 using Code First

  8. 8

    Insert row into table with only one column, primary key and identity

  9. 9

    "Cannot insert explicit value for identity column in table when IDENTITY_INSERT is set to OFF" with composite key

  10. 10

    "Cannot insert explicit value for identity column in table when IDENTITY_INSERT is set to OFF" with composite key

  11. 11

    Insert data into foreign key column

  12. 12

    Insert a new column into the Foreign Key table when a particular column in the primary key table is Update (Using Triggers)

  13. 13

    C# Code first foreign key. The foreign key refers to the non valid column in table with references

  14. 14

    Mysql insert in two tables where the first table has foreign key and update the first table

  15. 15

    How to insert Primary Key value of the primary table to the Foreign Key column of the child table in MySQL?

  16. 16

    insert rows into table with identity column

  17. 17

    Unable to insert into table with Identity Column

  18. 18

    Remove table name prefix in EF code first foreign key table column

  19. 19

    How to improve the speed of bulk_update on a foreign key of a large table?

  20. 20

    INSERT FOREIGN KEY in another table with executemany() in PostgreSQL

  21. 21

    Insert random data into a table that has foreign key

  22. 22

    SQL insert data into table has foreign key

  23. 23

    SQL insert data into table has foreign key

  24. 24

    How to insert data in a table with foreign key

  25. 25

    Insert values in table with foreign key in php

  26. 26

    Get AutoIncrement ID and insert into Foreign Key Table

  27. 27

    insert data into table having foreign key

  28. 28

    Insert values into a table with a foreign key using mysql

  29. 29

    INSERT conflicted with foreign key constraint for lookup table

HotTag

Archive