PowerShell: Remove an xml node if its childnodes matches specific criteria

I have an xml file which is created by FileZilla and contains connection details for multiple ftp servers.

XML:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<FileZilla3>
    <Servers>
        <Server>
            <Host>host</Host>
            <Port>1</Port>
            <Protocol>3</Protocol>
            <Type>0</Type>
            <User>SomeUser</User>
            <Pass>p455w0rd</Pass>
            <Logontype>1</Logontype>
            <TimezoneOffset>0</TimezoneOffset>
            <PasvMode>MODE_DEFAULT</PasvMode>
            <MaximumMultipleConnections>0</MaximumMultipleConnections>
            <EncodingType>Auto</EncodingType>
            <BypassProxy>0</BypassProxy>
            <Name>Server1</Name>
            <Comments>Comment</Comments>
            <LocalDir />
            <RemoteDir />
            <SyncBrowsing>0</SyncBrowsing>Server1
        </Server>
        <Server>
            <Host>host</Host>
            <Port>1</Port>
            <Protocol>3</Protocol>
            <Type>0</Type>
            <User>SomeOtherUser</User>
            <Pass>passw0rd</Pass>
            <Logontype>1</Logontype>
            <TimezoneOffset>0</TimezoneOffset>
            <PasvMode>MODE_DEFAULT</PasvMode>
            <MaximumMultipleConnections>0</MaximumMultipleConnections>
            <EncodingType>Auto</EncodingType>
            <BypassProxy>0</BypassProxy>
            <Name>Server2</Name>
            <Comments />
            <LocalDir />
            <RemoteDir />
            <SyncBrowsing>0</SyncBrowsing>Server2
        </Server>
    </Servers>
</FileZilla3>

Now I'm working on a script to select ftp accounts which then get removed from this xml file. Here's what I have so far:

$SiteManager = "C:\Temp\SiteManager.xml"

[XML]$SiteManagerXMLContent = Get-Content $SiteManager -Encoding UTF8

#Account that gets removed
$FTPAccName     = "Server1"
$FTPAccUserName = "SomeUser"
$FTPAccPassWord = "p455w0rd"
$FTPAccComment  = "Comment"

ForEach($Server in $SiteManagerXMLContent.FileZilla3.Servers)
{
    $XMLServerName = $Server.SelectSingleNode("//Name[.='$FTPAccName']")
    $XMLUserName = $Server.SelectSingleNode("//User[.='$FTPAccUserName']")
    $XMLPassWord = $Server.SelectSingleNode("//Pass[.='$FTPAccPassWord']")
    $XMLComment = $Server.SelectSingleNode("//Comment[.='$FTPAccComment']")

    if($XMLServerName.'#text' -eq $FTPAccName -and $XMLUserName.'#text' -eq $FTPAccUserName -and $XMLPassWord.'#text' -eq $FTPAccPassWord -and $XMLComment.'#text' -eq $FTPAccComment)
    {
        $XMLPassWord.ParentNode.RemoveAll()
    }
}

$SiteManagerXMLContent.Save($sitemanager)

This removes all child nodes of the selected server but not the parent node: which is what I'm aiming for. I would like to delete the entire node.

After running this script my xml looks like so:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<FileZilla3>
  <Servers>
    <Server>
    </Server>
    <Server>
      <Host>host</Host>
      <Port>1</Port>
      <Protocol>3</Protocol>
      <Type>0</Type>
      <User>SomeOtherUser</User>
      <Pass>passw0rd</Pass>
      <Logontype>1</Logontype>
      <TimezoneOffset>0</TimezoneOffset>
      <PasvMode>MODE_DEFAULT</PasvMode>
      <MaximumMultipleConnections>0</MaximumMultipleConnections>
      <EncodingType>Auto</EncodingType>
      <BypassProxy>0</BypassProxy>
      <Name>Server2</Name>
      <Comments />
      <LocalDir />
      <RemoteDir />
      <SyncBrowsing>0</SyncBrowsing>Server2
        </Server>
  </Servers>
</FileZilla3>

The problem I'm facing is that every server has the same tag which is why I need to identify the correct server by its child nodes.

Thanks for any help.