I am scraping a webpage. The webpage consists of 50 entries. After 50 entries it gives a Load more reults button. I need to automatically select it. How can I do it. For scraping I am using Python, Lxml.
Even JavaScript is using http requests to get the data, so one method would be to investigate, what requests are providing the data when user asks to "Load more results" and emulate these requests.
This is not traditional scraping, which is based on plain or rendered html content and detecting further links, but can be working solution.
Next actions:
Note, that the data do not necessarily come in HTML or XML form, but could be in JSON. But Python provide enough tools to process this format too.
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments