Trouble grabbing text from a website using Jsoup

debugcn Published at Dev

Nate

I'm trying to grab a price from an amazon link.

Here's the html I'm focusing on:

<div class="buying" id="priceBlock">
    <table class="product">
        <tbody>
            <tr id="actualPriceRow">
                <td class="priceBlockLabelPrice" id="actualPriceLabel">Price:</td>
                <td id="actualPriceContent">
                    <span id="actualPriceValue">
                        <b class="priceLarge">
                                $1.99
                        </b>
                    </span>

                </td>
            </tr>
        </tbody>
    </table>
</div>

I'm trying to grab that $1.99 text.

Here's my code that is trying to grab it.

protected Void doInBackground(Void... params) {
            try {
                // Connect to the web site
                Document document = Jsoup.connect(url).get();
                // Get the html document title
                Elements trs = document.select("table.product");



                for (Element tr : trs)
                {
                    Elements tds = tr.select("b.priceLarge");
                    Element price1 = tds.first();
                    String str1 = price1.text();
                    System.out.println(str1);
                    String str2 = str1.replaceAll( "[$,]", "" );
                    double aInt = Double.parseDouble(str2);
                    System.out.println("Price: " + aInt);

                }

            } catch (IOException e) {
                e.printStackTrace();
            }

            return null;
        }

Why isn't this code working?

Alkis Kalogeris

You have to use a user agent so the site won't reject you as a bot. You should also add some timeout limit in order to override the default one, which might be too short for you. Three seconds is a good option but feel free to change it at will. timeout(0) will wait as long as the server needs to give some response. If you don't want a limit use that. There is also some weird DOM parsing you are doing, which is causing a NullPointerException. Try this

String url = "http://www.amazon.com/dp/B00H2T37SO/?tag=stackoverfl08-20";
Document doc = Jsoup
                .connect(url)
                .userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36")
                .timeout(3000)
                .get();

Elements prices = doc.select("table.product b.priceLarge");
for (Element pr : prices)
{
    String priceWithCurrency = pr.text();
    System.out.println(priceWithCurrency);
    String priceAsText = priceWithCurrency.replaceAll( "[$,]", "" );
    double priceAsNumber = Double.parseDouble(priceAsText);
    System.out.println("Price: " + priceAsNumber);
}

Collected from the Internet

Please contact [email protected] to delete if infringement.