I am trying to get the content of an HTML page, using this code:
String malSearch = "http://myanimelist.net/anime.php?letter=" + firstLetter;
URL url = new URL(malSearch);
URLConnection con = url.openConnection();
InputStream in = con.getInputStream();
String encoding = con.getContentEncoding();
encoding = encoding == null ? "UTF-8" : encoding;
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buf = new byte[8192];
int len = 0;
while ((len = in.read(buf)) != -1) {
baos.write(buf, 0, len);
}
String body = new String(baos.toByteArray(), encoding);
It works fine, but it doesn't give me what I really want. It gives me this:
<html>
<head>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
<meta name="format-detection" content="telephone=no">
<meta name="viewport" content="initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
</head>
<body style="margin:0px">
<iframe src="/_Incapsula_Resource?CWUDNSAI=9&xinfo=6-122029399-0 0NNN RT(1404149034204 2) q(0 -1 -1 -1) r(0 -1) B12(4,315,0) U1&incident_id=124001330081285077-564449081699338326&edet=12&cinfo=4ee46646c753833e04000000" frameborder=0 width="100%" height="100%" marginheight="0px" marginwidth="0px">Request unsuccessful. Incapsula incident ID: 124001330081285077-564449081699338326</iframe>
</body>
</html>
when it should give me the whole page (approximately 800 lines).
I think it's due to the fact this is a website using PHP, but I'm not really sure. Can someone tell me how I could get the whole HTML content?
Here's the page I'm trying to get the content from: http://myanimelist.net/anime.php?letter=A
This site uses a service called Incapsula. The website admins configured Incapsula to prevent bots from accessing it's content.
I suggest you contact the admins and ask to be whitelisted, Trying to bypass the system will likely get you banned and blacklisted.
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments