我正在从API提取JSON数据,输出如下:
[[{'employeeId': 1, 'lastName': 'Smith'}, {'employeeId': 2, 'lastName': 'Flores'}]]
列表中大约有25万个对象。我能够遍历列表中的对象并以这种方式update_one
通过PyMongo进行操作:
json_this = json.dumps(json_list[0])
json_that = json.loads(json_this)
for x in json_that:
collection.update_one({"employeeId": x['employeeId']},{"$set": x},upsert=True)
但是,如果有25万条记录,则需要花费很长时间。我正在尝试使用update_many
但无法弄清楚如何正确转换/格式化此JSON列表以使用该update_many
功能。任何指导将不胜感激。
将250K文档更新/插入数据库可能是一项艰巨的任务,您不能使用update_many
筛选查询,并且每个字典之间的更新值都会改变。因此,使用下面的查询至少可以避免多次调用数据库,但是我不确定这对您的情况有多好,请注意,我是python的初学者,这是一个基本的代码,可以帮助您:
对于批量操作,最好的办法是PyMongo-bulk,由于.bulkWrite()的局限性,我们将250K条记录分成多个块:
from pymongo import UpdateOne
from pprint import pprint
import sys
json_this = json.dumps(json_list[0])
json_that = json.loads(json_this)
primaryBulkArr = []
secondaryBulkArr = []
thirdBulkArr = []
## Here we're splicing 250K records into 3 arrays, in case if we want to finish a chunk at a time,
# No need to splice all at once - Finish end - to - end for one chunk & restart the process for another chunk from the index of the list where you left previously
for index, x in enumerate(json_that):
if index < 90000:
primaryBulkArr.append(
UpdateOne({"employeeId": x['employeeId']}, {'$set': x}, upsert=True))
elif index > 90000 and index < 180000:
secondaryBulkArr.append(
UpdateOne({"employeeId": x['employeeId']}, {'$set': x}, upsert=True))
else:
thirdBulkArr.append(
UpdateOne({"employeeId": x['employeeId']}, {'$set': x}, upsert=True))
## Reason why I've spliced into 3 arrays is may be you can run below code in parallel if your DB & application servers can take it,
# At the end of the day irrespective of time taken only 3 DB calls are needed & this bulk op is much efficient.
try:
result = collection.bulk_write(bulkArr)
## result = db.test.bulk_write(bulkArr, ordered=False)
# Opt for above if you want to proceed on all dictionaries to be updated, even though an error occured in between for one dict
pprint(result.bulk_api_result)
except:
e = sys.exc_info()[0]
print("An exception occurred ::", e) ## Get the ids failed if any & do re-try
本文收集自互联网,转载请注明来源。
如有侵权,请联系[email protected] 删除。
我来说两句