How to get all information with all images in site

mikezang Published at Dev

mikezang

I want to get all information with all images, is there any simple way to use bash on Mac? I want to get data as below:

"product": "8020"
"simage": "/uploadfile/201281616171259157_.GIF"
"image": "/uploadfile/201281616171259157.GIF"
"name": "Taipei 101"

"product": "8019"
"simage": "/uploadfile/201432010288118198_.jpg"
"image": "/uploadfile/201432010288118198.jpg"
"name": "TianTan"

This doesn't work, also I need product and name and so on, there are not in src attribute...

baseurl=$(echo $url | egrep -o "https?://[a-z.]+")

curl --silent $url | egrep -o "src=[^>]*(\.jpg|\.gif|\.png)" | sed 's/src=\"\(.*\)/\1/g' > /tmp/$$.list
sed -i "s|^/|$baseurl/|" /tmp/$$.list

while read filename;
do
    curl -s -O "$baseurl/$filename"
done < /tmp/$$.list

The site contents of product.asp?cxsort=10001

....
<ul id="small" >
    <li><a href="product.asp?cxsort=10001">Military1</a></li>
    <li><a href="product.asp?cxsort=10021">Military2</a></li>
    <li><a href="product.asp?cxsort=10101">Military3</a></li>
....
</ul>

....
<table cellpadding="0" cellspacing="0">
    <tr>
        <td>Product:8020</td>
        <td><div class="set"><img  src="/uploadfile/201281616171259157_.GIF" width="94" height="69"  style="display:block" class="/uploadfile/201281616171259157.GIF" alt="TianTan" /></div></td>
    </tr>
</table>
....
<table cellpadding="0" cellspacing="0">
    <tr>
        <td>Product:8019</td>
        <td><div class="Set"><img  src="/uploadfile/201432010288118198_.jpg" width="94" height="69"  style="display:block" class="/uploadfile/201432010288118198.jpg" alt="Taipei 101" /></div></td>
    </tr>
</table>
....

SLePort

You can try this :

sed -n '
/Product/ {
        s/[ \t]*<[^>]*>//g
        s/Product:\([0-9]*\)/"product": "\1"/p
        n
        s/.*img  *src="\([^"]*\)".*class="\([^"]*\).*alt="\([^"]*\).*/"simage": "\1"\n"image": "\2"\n"name": "\3"\n/p
}
' file.html

It works with your example and should do the trick on your html if the code related to product and image is always structured the same way.

But a web-scraping library like BeautifulSoup in python would be a better choice.

A BeautifulSoup python code looks like this :

from bs4 import BeautifulSoup

f = file('file.html', 'r')
soup = BeautifulSoup(f)

all_img = soup.find_all('img')
for img in all_img:
        print '%s : %s' % (img['alt'], img['src'])

Collected from the Internet

Please contact [email protected] to delete if infringement.