How to use unordered bulk inserting with Mongoskin?

jeebface

I'm having trouble using Mongoskin to perform bulk inserting (MongoDB 2.6+) on Node.

var dbURI = urigoeshere;
var db = mongo.db(dbURI, {safe:true});
var bulk = db.collection('collection').initializeUnorderedBulkOp();

for (var i = 0; i < 200000; i++) {
    bulk.insert({number: i}, function() {
        console.log('bulk inserting: ', i);
    });
}

bulk.execute(function(err, result) {
    res.json('send response statement');
}); 

The above code gives the following warnings/errors:

(node) warning: possible EventEmitter memory leak detected. 51 listeners added. Use emitter.setMaxListeners() to increase limit.
TypeError: Object #<SkinClass> has no method 'execute'
(node) warning: possible EventEmitter memory leak detected. 51 listeners added. Use emitter.setMaxListeners() to increase limit.
TypeError: Object #<SkinClass> has no method 'execute'

Is it possible to use Mongoskin to perform unordered bulk operations? If so, what am I doing wrong?

Neil Lunn

You can do it but you need to change your calling conventions to do this as only the "callback" form will actually return a collection object from which the .initializeUnorderedBulkOp() method can be called. There are also some usage differences to how you think this works:

var dbURI = urigoeshere;
var db = mongo.db(dbURI, {safe:true});
db.collection('collection',function(err,collection) {
    var bulk = collection.initializeUnorderedBulkOp();
    count = 0;

    for (var i = 0; i < 200000; i++) {
        bulk.insert({number: i});
        count++;

        if ( count % 1000 == 0 )
            bulk.execute(function(err,result) {
               // maybe do something with results
               bulk = collection.initializeUnorderedBulkOp(); // reset after execute
            });      

    });

    // If your loop was not a round divisor of 1000
    if ( count % 1000 != 0 )
        bulk.execute(function(err,result) {
          // maybe do something here
        });
});

So the actual "Bulk" methods themselves don't require callbacks and work exactly as shown in the documentation. The exeception is .execute() which actually sends the statements to the server.

While the driver will sort this out for you somewhat, it probably is not a great idea to queue up too many operations before calling execute. This basically builds up in memory, and though the driver will only send in batches of 1000 at a time ( this is a server limit as well as the complete batch being under 16MB ), you probably want a little more control here, at least to limit memory usage.

That is the point of the modulo tests as shown, but if memory for building the operations and a possibly really large response object are not a problem for you then you can just keep queuing up operations and call .execute() once.

The "response" is in the same format as given in the documentation for BulkWriteResult.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Bulk insert efficiency for inserting a list with one row

From Dev

Inserting multiple document into MongoDB with Mongoskin

From Dev

How to use unordered_map in cython?

From Dev

How do I access (check) the element of a std::unordered_map without inserting into it?

From Dev

How to get a list of all collections with mongoskin

From Dev

I want to use runCommand in mongoDB while using mongoskin

From Dev

'undefined is not a function' when using unordered bulk upsert

From Dev

How to use Dapper for bulk insert with foreign keys?

From Dev

How to use unordered_set with compare function?

From Dev

how to choose the right batch size while bulk inserting

From Dev

how to choose the right batch size while bulk inserting

From Dev

How to use read preferences in mongoskin?

From Dev

PHP PDO inserting bulk records or update if exists

From Dev

Bulk inserting specific values using sql

From Dev

Bulk inserting to MongoDB on Replicaset

From Dev

I want to use runCommand in mongoDB while using mongoskin

From Dev

How to use unordered bulk inserting with Mongoskin?

From Dev

Bulk inserting documents into couchbase database with php - how to?

From Dev

How to use ttl in elasticsearch bulk api (NEST)

From Dev

Bulk inserting in database unique codes

From Dev

How to use Dapper for bulk insert with foreign keys?

From Dev

How to use unordered_set with compare function?

From Dev

How many rows affected in a second, while SQL Server bulk inserting?

From Dev

How can I use bulk insert in the case

From Dev

Inserting bulk data into table by checking a condition

From Dev

How to Use Mongo Pojos in Bulk Upsert

From Dev

SQL Bulk Insert in C# not inserting values

From Dev

Bulk inserting in Oracle PL/SQL

From Dev

How to use bulk upsert in a loop?

Related Related

  1. 1

    Bulk insert efficiency for inserting a list with one row

  2. 2

    Inserting multiple document into MongoDB with Mongoskin

  3. 3

    How to use unordered_map in cython?

  4. 4

    How do I access (check) the element of a std::unordered_map without inserting into it?

  5. 5

    How to get a list of all collections with mongoskin

  6. 6

    I want to use runCommand in mongoDB while using mongoskin

  7. 7

    'undefined is not a function' when using unordered bulk upsert

  8. 8

    How to use Dapper for bulk insert with foreign keys?

  9. 9

    How to use unordered_set with compare function?

  10. 10

    how to choose the right batch size while bulk inserting

  11. 11

    how to choose the right batch size while bulk inserting

  12. 12

    How to use read preferences in mongoskin?

  13. 13

    PHP PDO inserting bulk records or update if exists

  14. 14

    Bulk inserting specific values using sql

  15. 15

    Bulk inserting to MongoDB on Replicaset

  16. 16

    I want to use runCommand in mongoDB while using mongoskin

  17. 17

    How to use unordered bulk inserting with Mongoskin?

  18. 18

    Bulk inserting documents into couchbase database with php - how to?

  19. 19

    How to use ttl in elasticsearch bulk api (NEST)

  20. 20

    Bulk inserting in database unique codes

  21. 21

    How to use Dapper for bulk insert with foreign keys?

  22. 22

    How to use unordered_set with compare function?

  23. 23

    How many rows affected in a second, while SQL Server bulk inserting?

  24. 24

    How can I use bulk insert in the case

  25. 25

    Inserting bulk data into table by checking a condition

  26. 26

    How to Use Mongo Pojos in Bulk Upsert

  27. 27

    SQL Bulk Insert in C# not inserting values

  28. 28

    Bulk inserting in Oracle PL/SQL

  29. 29

    How to use bulk upsert in a loop?

HotTag

Archive