使用BeautifulSoup从单个博客存档页面提取多个帖子，无需脚本

ChristinaP 发表于 Dev

克里斯蒂娜·P

我试图从一系列WordPress和Blogger博客存档页面中抓取作者，标题，日期和帖子内容。我已将页面保存下来，因此不会重复ping服务器。我已经完成了其他部分的工作，但是我似乎既无法从每个文件中获取所有帖子，也无法从底部获取“添加到任何文件”或“可社交”或其他凌乱的脚本。我在这里

import urllib2
from bs4 import BeautifulSoup
import re

file_list = open ("hafiles.txt", "r")
posts_file = open ("haposts.txt","w")


for indurl in file_list:
    indurl = indurl.rstrip("\n")
    with open(indurl,"r") as ha_file:
     soup_ha = BeautifulSoup(ha_file)

    #works the second find gets rid of the sociable crap
    # this is the way it looks on the page <div class='post-body'>

    posts = soup_ha.find("div", class_="post-body").find_all("p")


    #tried a trick i saw on http://stackoverflow.com/questions/24458353/cleaning-text-string-after-getting-body-text-using-beautifulsoup
    #no joy
    #posts = soup_ha.find("div", class_="post-body")
    #text = [''.join(s.findAll(text=True))for s in posts.findAll('p')] 
    text = str(posts) + "\n" + "\n"
    posts_file.write (text)

print ("All done!")



file_list.close()
posts_file.close()

因此，如果我执行一次find_all并获得所有帖子（甚至不确定我是否实际上已全部获得），那么我将获得脚本。如果我仅使用find，至少可以通过两种方法获得没有脚本的漂亮帖子。我有一个文件列表，每个文件都有多个要提取的帖子。我在这里在stackoverflow和网络上进行了搜索。

eta：输入是一个非常混乱的网页，顶部是一堆脚本，页面上的所有css定义，然后

<div id='main-wrapper'>
<div class='main section' id='main'><div class='widget Blog' id='Blog1'>
<div class='blog-posts'>
<h2 class='date-header'>27 February, 2007</h2>
<div class='post uncustomized-post-template'>
<a name='edit'></a>
<h3 class='post-title'>
<a href='http:// edited for anon.html'>edit</a>
</h3>
<div class='post-header-line-1'></div>
<div class='post-body'>
<style>span.fullpost{display:none;}</style>
<p>edit this is post text - what i want</p>
<script type='text/javascript'>
          var permlink='edit';
          var title='edit';

          var spans = document.getElementsByTagName('span');
          var number = 0;
          for(i=0; i <spans.length; i++){
                var c = " " + spans[i].className + " ";
                if (c.indexOf("fullpost") != -1) {
                number++;
                }
                }

                if(number != memory){document.write('<p></p><a href=' + permlink + '>"'+ title + '" continues...</a>') }
           memory = number;
           </script>
<div style='clear: both;'></div>
</div>
<div class='post-footer'>
<p class='post-footer-line post-footer-line-1'>
<span class='post-author'>
Posted by
this is the author name, also want, have way to get
</span>
<span class='post-timestamp'>
at
<a class='timestamp-link' href='http://edit' title='permanent link'>2:53 pm</a>
</span>
<span class='post-comment-link'>
<a class='comment-link' href='edit' onclick=''>1 comments</a>
</span>
<span class='post-backlinks post-comment-link'>
<a class='comment-link' href='edit'>Links to this post</a>
</span>
<span class='post-icons'>
<span class='item-control blog-admin pid-edit'>
<a href='edit' title='Edit Post'>
<img alt='' class='icon-action' height='18' src='http://img2.blogblog.com/img/icon18_edit_allbkg.gif' width='18'/>
</a>
</span>
</span>
</p>
<p class='post-footer-line post-footer-line-2'>
<span class='post-labels'>
Labels:
<a href='edit' rel='tag'>edi</a>
</span>
</p>
<p class='post-footer-line post-footer-line-3'></p>
</div>
</div>
<h2 class='date-header'>26 February, 2007</h2>
<div class='post uncustomized-post-template'>
<a name='5518681505930320089'></a>
<h3 class='post-title'>
<a href='edit'>edit</a>
</h3>
<div class='post-header-line-1'></div>
<div class='post-body'>
<style>span.fullpost{display:none;}</style>
<p>edit post text, what I want.</p>
<script type='text/javascript'>
          var permlink='http://edit';
          var title='edit';

          var spans = document.getElementsByTagName('span');
          var number = 0;
          for(i=0; i <spans.length; i++){
                var c = " " + spans[i].className + " ";
                if (c.indexOf("fullpost") != -1) {
                number++;
                }
                }

                if(number != memory){document.write('<p></p><a href=' + permlink + '>"'+ title + '" continues...</a>') }
           memory = number;
           </script>
<div style='clear: both;'></div>
</div>
<div class='post-footer'>
<p class='post-footer-line post-footer-line-1'>
<span class='post-author'>
Posted by
edit author name
</span>
<span class='post-timestamp'>
at
<a class='timestamp-link' href='edit' title='permanent link'>9:00 am</a>
</span>
<span class='post-comment-link'>
<a class='comment-link' href='edit' onclick=''>5
comments</a>
</span>
<span class='post-backlinks post-comment-link'>
<a class='comment-link' href='edit'>Links to this post</a>
</span>
<span class='post-icons'>
<span class='item-control blog-admin pid-edit'>
<a href='edit' title='Edit Post'>
<img alt='' class='icon-action' height='18' src='http://img2.blogblog.com/img/icon18_edit_allbkg.gif' width='18'/>
</a>
</span>
</span>
</p>
<p class='post-footer-line post-footer-line-2'>
<span class='post-labels'>
Labels:
<a href='edit' rel='tag'>edit</a>,
<a href='edit' rel='tag'>edit</a>
</span>
</p>
<p class='post-footer-line post-footer-line-3'></p>
</div>
</div>
<h2 class='date-header'>22 February, 2007</h2>
<div class='post uncustomized-post-template'>
<a name='edit'></a>

！所以我可能有20个左右的文件，每个文件中都有1到10个帖子（这有2个）...会很可爱的是csv或excel文件，就像日期作者标题postcontent

列，每行一行。我将使用一个仅包含帖子内容的文件，每个帖子之间有一些空间。我可以使用文本中的某些链接，粗体和列表以及其他功能，但是我不希望所有凌乱的脚本。谢谢

这是一个包含多个帖子的单个页面的示例：

from bs4 import BeautifulSoup


soup = BeautifulSoup(open('test.html'))
posts = []
for post in soup.find_all('div', class_='post'):
    title = post.find('h3', class_='post-title').text.strip()
    author = post.find('span', class_='post-author').text.replace('Posted by', '').strip()
    content = post.find('div', class_='post-body').p.text.strip()
    date = post.find_previous_sibling('h2', class_='date-header').text.strip()

    posts.append({'title': title,
                  'author': author,
                  'content': content,
                  'date': date})
print posts

对于您发布的html，它打印：

[{'content': u'edit this is post text - what i want', 
  'date': u'27 February, 2007', 
  'author': u'this is the author name, also want, have way to get', 
  'title': u'edit'}, 
 {'content': u'edit post text, what I want.', 
  'date': u'26 February, 2007', 
  'author': u'edit author name', 
  'title': u'edit'}]

本文收集自互联网，转载请注明来源。

如有侵权，请联系[email protected] 删除。

编辑于2021-02-11

我来说两句

0条评论

登录后参与评论

上一篇：将NodeEntity直接添加到Neo4j-Spatial图层中？

来自分类Dev

Related 相关文章

文章