Updating all documents in a collection (million+) with Date object

emifenac

I'm trying to update all documents in a collection. (~20m documents). Essentially the date string is stored as a month/day/year, and I want to instead store a date object so that I can sort based on that later. I've tried doing:

for document in collection.find({}, no_cursor_timeout=True):
date = document['Date']
dateobj = datetime.strptime(date, "%m/%d/%Y")
 doc = col.find_one_and_update(
     {"_id": document["_id"]},
     {"$set":
      {"DateObj_Sort": dateobj}
      }, upsert=False
 )
count = count + 1
print("Count is ", count)

But the problem is, the cursor gives me no cursor id found at around 180k documents

pymongo.errors.CursorNotFound: cursor id 361490694651426388 not found, full error: {'operationTime': Timestamp(1606772819, 18), 'ok': 0.0, 'errmsg': 'cursor id 361490694651426388 not found', 'code': 43, 'codeName': 'CursorNotFound', '$clusterTime': {'clusterTime': Timestamp(1606772819, 18), 'signature': {'hash': b'\xf7\xca*^\xea\xa8\xc3\xda\xa1\xa6\xf6\xec\x0cAQT\xa34T\x88', 'keyId': 6867170402951495684}}}

How can I update all the documents in a collection when there are soo many?

Oluwafemi Sule

Offload conversion to the Database server. Afterall date conversion gives similar results as new Date in Javascript.

> new Date('04/01/2020');

<- Wed Apr 01 2020 00:00:00 GMT+0200 (Mitteleuropäische Sommerzeit) {}

In an Aggregation pipeline, you can merge results to documents in existing collection like so:

date_conversion_stage = {
    "$addFields": {
        "DateObj_Sort": { "$toDate": "$Date" }
    }
}

merge_stage = {
    "$merge": {
        "into": "<replace_with_collection_name>",
        "whenMatched": "merge"
    }
}

collection.aggregate([
    date_conversion_stage,
    merge_stage,
])

この記事はインターネットから収集されたものであり、転載の際にはソースを示してください。

侵害の場合は、連絡してください[email protected]

編集
0

コメントを追加

0

関連記事

分類Dev

Get All documents in a Collection CosmosDb

分類Dev

Find all documents in a collection with mongo go driver

分類Dev

Get All 'documents' from MongoDB 'collection'

分類Dev

How to get all documents from a collection in FaunaDB?

分類Dev

What is the preferred way to add many fields to all documents in a MongoDB collection?

分類Dev

Elastic - prevent updating documents

分類Dev

How to get all documents from a firestore collection and return them in an array list?

分類Dev

Get all documents from mongo collection using a nested list comprehension in Python

分類Dev

Get all documents from mongo collection using a nested list comprehension in Python

分類Dev

Saving and updating nested documents with MongoEngine

分類Dev

Working with Firestore documents and collection data

分類Dev

when i try adding an object to the collection, all the object values are changed to the current object, how?

分類Dev

Updating already published collection in Meteor

分類Dev

Updating array element in a collection in Meteor

分類Dev

Object in array is not updating in JavaScript

分類Dev

Updating Mongoose Object

分類Dev

Updating an object in an array if the object exists

分類Dev

How to stop insertion of Duplicate documents in a mongodb collection

分類Dev

Get the count of the number of documents in a Collection Mongodb

分類Dev

firestore: arrays vs sub collection of documents performance

分類Dev

Fast query and deletion of documents of a large collection in MongoDB

分類Dev

How to modify the structure of this object by using the date as key obj and pair to it all activity completed on the same date

分類Dev

Wpf Observable collection and DataGrid not updating changes

分類Dev

Find the sum of all odd Fibonacci numbers less than 2 million

分類Dev

Upsert issue when updating multiple documents using an array of IDs with $in

分類Dev

Updating existing documents in ElasticSearch (ES) while using rollover API

分類Dev

Angular 2 Updating Array object

分類Dev

Updating an object attribute from a method

分類Dev

Rotating object AABBs updating incorrectly

Related 関連記事

  1. 1

    Get All documents in a Collection CosmosDb

  2. 2

    Find all documents in a collection with mongo go driver

  3. 3

    Get All 'documents' from MongoDB 'collection'

  4. 4

    How to get all documents from a collection in FaunaDB?

  5. 5

    What is the preferred way to add many fields to all documents in a MongoDB collection?

  6. 6

    Elastic - prevent updating documents

  7. 7

    How to get all documents from a firestore collection and return them in an array list?

  8. 8

    Get all documents from mongo collection using a nested list comprehension in Python

  9. 9

    Get all documents from mongo collection using a nested list comprehension in Python

  10. 10

    Saving and updating nested documents with MongoEngine

  11. 11

    Working with Firestore documents and collection data

  12. 12

    when i try adding an object to the collection, all the object values are changed to the current object, how?

  13. 13

    Updating already published collection in Meteor

  14. 14

    Updating array element in a collection in Meteor

  15. 15

    Object in array is not updating in JavaScript

  16. 16

    Updating Mongoose Object

  17. 17

    Updating an object in an array if the object exists

  18. 18

    How to stop insertion of Duplicate documents in a mongodb collection

  19. 19

    Get the count of the number of documents in a Collection Mongodb

  20. 20

    firestore: arrays vs sub collection of documents performance

  21. 21

    Fast query and deletion of documents of a large collection in MongoDB

  22. 22

    How to modify the structure of this object by using the date as key obj and pair to it all activity completed on the same date

  23. 23

    Wpf Observable collection and DataGrid not updating changes

  24. 24

    Find the sum of all odd Fibonacci numbers less than 2 million

  25. 25

    Upsert issue when updating multiple documents using an array of IDs with $in

  26. 26

    Updating existing documents in ElasticSearch (ES) while using rollover API

  27. 27

    Angular 2 Updating Array object

  28. 28

    Updating an object attribute from a method

  29. 29

    Rotating object AABBs updating incorrectly

ホットタグ

アーカイブ