I have about a million of records in the list. And foreach in the list I query the database to get the modified date based on id. Once I get it, it's test in the condition to either insert it or update it .. but is there any way to do it as bulk (at once or in batches) insertion or update..??
foreach (No_kemi no_list in newforSQL)
{
DateTime eModifiedDate = no_list.ModifiedDate.Value;
string Id = no_list.ID.ToString();
String selectQuery = "SELECT ModifiedDate FROM NO_table WHERE ID = '" + Id + "'";
DateTime? dbmoddate = null;
using (SqlCommand selectCommand = new SqlCommand(selectQuery, connection))
{
// Use the above SqlCommand object to create a SqlDataReader object.
using (SqlDataReader rdr = selectCommand.ExecuteReader())
{
while (rdr.Read())
{
dbmoddate = (DateTime)rdr["ModifiedDate"];
}
}
}
string eType = null;
string SubGUID = null;
string trimSubKey = no_list.SubscriberKey;
try
{
if ((trimSubKey.Length > 3) && (trimSubKey != ""))
{
if (trimSubKey.Contains(","))
{
string[] values = trimSubKey.Split(',');
SubGUID = values[0];
eType = values[1];
}
else
{
SubGUID = trimSubKey;
eType = "";
}
}
else
{
SubGUID = "";
eType = "";
}
}
catch (Exception ex)
{
logger.Error("Error : " + ex.Message );
}
if (dbmoddate == null)
{
DateTime no_listinsetdate = DateTime.Now;
String insertQuery = "INSERT INTO NO_table (CreatedDate, ModifiedDate, ID, eType, SubGUID, DbDate) VALUES (@CreatedDate, @ModifiedDate, @ID, @eType, @SubGUID, @DbDate,)";
using (SqlCommand insertCommand = new SqlCommand(insertQuery, connection))
{
insertCommand.Parameters.AddWithValue("@CreatedDate", no_list.CreatedDate);
insertCommand.Parameters.AddWithValue("@ModifiedDate", no_list.ModifiedDate);
insertCommand.Parameters.AddWithValue("@ID", no_list.ID);
insertCommand.Parameters.AddWithValue("@eType", eType);
insertCommand.Parameters.AddWithValue("@SubGUID", SubGUID);
insertCommand.Parameters.AddWithValue("@DbDate", no_listinsetdate);
insertCommand.ExecuteNonQuery(); // execute the query
}
}
else if (eModifiedDate > dbmoddate)
{
DateTime no_listinsetdate = DateTime.Now;
String updateQuery = "UPDATE NO_table SET CreatedDate = @CreatedDate, ModifiedDate = @ModifiedDate, ID = @ID, eType = @eType, SubGUID = @SubGUID, DbDate = @DbDate WHERE ID = '" + Id + "'";
using (SqlCommand updateCommand = new SqlCommand(updateQuery, connection))
{
updateCommand.Parameters.AddWithValue("@CreatedDate", no_list.CreatedDate);
updateCommand.Parameters.AddWithValue("@ModifiedDate", no_list.ModifiedDate);
updateCommand.Parameters.AddWithValue("@ID", no_list.ID);
updateCommand.Parameters.AddWithValue("@eType", eType);
updateCommand.Parameters.AddWithValue("@SubGUID", SubGUID);
updateCommand.Parameters.AddWithValue("@DbDate", no_listinsetdate);
updateCommand.ExecuteNonQuery(); // execute the query
}
}
}
any way to speed up the process... it takes lot of time to finish the operations... or some time I get timeout error ...
thx
This is way to broad to actually have a coded solution here but you should be able to create one yourself based on general advice:
You could probably save on a lot of network roundtrips if you wrote a SQL procedure that does this. Further optimization would result in this SQL procedure taking batches of records instead of a single one so you can for example send 1000 records, the procedure will loop over them and for each do the query and resulting update/insert. That means the results do not have to go back and forth across the network, resulting in less traffic and shorter runtime.
Maybe your actions can be postulated as a single MERGE command instead of query and update/insert. This depends on your table structure, data and intention.
Terms to google for: stored procedure
, table type
, MERGE
.
この記事はインターネットから収集されたものであり、転載の際にはソースを示してください。
侵害の場合は、連絡してください[email protected]
コメントを追加