Python fastest way to read a large number of small files into memory?

DJJ

I'm trying to read a few thousands html files stored on disk.

Is there any way to do better than;

for files in os.listdir('.'):
    if files.endswith('.html') :
        with (open) files as f:
            a=f.read()
            #do more stuffs
Thomas8

For a similar problem I have used this simple piece of code:

import glob
for file in glob.iglob("*.html"):
    with open(file) as f:
        a = f.read()

iglob doesn't stores all file simultaneously, this is perfect with a huge directory.
Remenber to close files after you have finished, the construct "with-open" make sure for you.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Python fastest way to read a large number of small files into memory?

From Dev

Fastest way to sort a large number of arrays in python

From Dev

The fastest way to read binary files by bytes in Python

From Dev

What is the fastest way to find the exact value of the factorial of a large number in python?

From Dev

What is the fastest compression method for a large number of files?

From Dev

Copy large files in fastest way possible

From Dev

Fastest Way to Move Large Files on NAS

From Dev

Memory problems when compressing and transferring a large number of small files (1TB in total)

From Dev

Python memory errors when hashing large number of files in sequence

From Dev

Fastest way to write large CSV with Python

From Dev

Is there a really efficient (FAST) way to read large text files in python?

From Dev

Adding a large number with a small number in Python

From Dev

Fastest way to transfer a lot of small files on server (without git)

From Dev

Fastest way to read and process string from large file in java?

From Dev

Fastest way to read very large text file in C#

From Dev

What is the fastest way to format a large number of date/time objects in Android?

From Dev

Fastest way to Group a large number of Elements in C++

From Dev

Fastest way to download thousand files using python?

From Dev

Fastest way to download a set of files in Python

From Dev

Performance: fastest way of reading in files with Python

From Dev

Fastest way to extract tar files using Python

From Dev

Performance: fastest way of reading in files with Python

From Dev

How to read large number of files in raw in ANDROID?

From Dev

Faster way to delete large number of files

From Dev

Simple way to resize large number of image files

From Dev

Fastest way to read in 100,000 .dat.gz files

From Dev

Can a large number of (small) files degrade the performance of a filesystem?

From Dev

What's the bottleneck in transfer of a large number of small files?

From Dev

The fastest way to count the number of files in a directory (including subdirectories)

Related Related

  1. 1

    Python fastest way to read a large number of small files into memory?

  2. 2

    Fastest way to sort a large number of arrays in python

  3. 3

    The fastest way to read binary files by bytes in Python

  4. 4

    What is the fastest way to find the exact value of the factorial of a large number in python?

  5. 5

    What is the fastest compression method for a large number of files?

  6. 6

    Copy large files in fastest way possible

  7. 7

    Fastest Way to Move Large Files on NAS

  8. 8

    Memory problems when compressing and transferring a large number of small files (1TB in total)

  9. 9

    Python memory errors when hashing large number of files in sequence

  10. 10

    Fastest way to write large CSV with Python

  11. 11

    Is there a really efficient (FAST) way to read large text files in python?

  12. 12

    Adding a large number with a small number in Python

  13. 13

    Fastest way to transfer a lot of small files on server (without git)

  14. 14

    Fastest way to read and process string from large file in java?

  15. 15

    Fastest way to read very large text file in C#

  16. 16

    What is the fastest way to format a large number of date/time objects in Android?

  17. 17

    Fastest way to Group a large number of Elements in C++

  18. 18

    Fastest way to download thousand files using python?

  19. 19

    Fastest way to download a set of files in Python

  20. 20

    Performance: fastest way of reading in files with Python

  21. 21

    Fastest way to extract tar files using Python

  22. 22

    Performance: fastest way of reading in files with Python

  23. 23

    How to read large number of files in raw in ANDROID?

  24. 24

    Faster way to delete large number of files

  25. 25

    Simple way to resize large number of image files

  26. 26

    Fastest way to read in 100,000 .dat.gz files

  27. 27

    Can a large number of (small) files degrade the performance of a filesystem?

  28. 28

    What's the bottleneck in transfer of a large number of small files?

  29. 29

    The fastest way to count the number of files in a directory (including subdirectories)

HotTag

Archive