Is there an RDD transform function that looks at neighboring elements?

Probably the most efficient approach simplest approach is to convert a RDD to data frame and use lag:

case class NameValue(name: String, value: Int)
val rdd = sc.parallelize(
    NameValue("a", 3) ::  NameValue("b", 6) :: 
    NameValue("c", 4) ::  NameValue("d", 20) :: Nil)

val df = sqlContext.createDataFrame(rdd)
df.registerTempTable("df")
sqlContext.sql("""SELECT name, value,
                  value - lag(value) OVER (ORDER BY name, value) lag
                  FROM df""").show

Unfortunately at this moment window functions without PARTITION BY clause move all data to a single partition so it is particularly useful if you have large dataset.

Using low level operations you could use zipWithIndex followed by flatMap and groupByKey:

case class NameValueWithLag(name: String, value: Int, lag: Int)
val cnt = rdd.count() - 1

rdd.
    zipWithIndex.
    flatMap{case (x, i) => (0 to 1).map(lag => (i - lag, (i, x)))}.
    groupByKey.
    filter{ case (k, v) => k != cnt}.
    values.
    map(vals => {
        val sorted = vals.toArray.sortBy(_._1).map(_._2)
        if (sorted.length == 1) {
            NameValueWithLag(sorted(0).name, sorted(0).value, sorted(0).value)
        } else {
            NameValueWithLag(
               sorted(1).name, sorted(1).value,
               sorted(1).value - sorted(0).value
            )
        }
    })

Edit:

If you don't mind using developers API there you can try RDDFunctions.sliding but it requires manual processing

import org.apache.spark.mllib.rdd.RDDFunctions._

val first = rdd.first match {
  case NameValue(name, value) => NameValueWithLag(name, value, value)
}

sc.parallelize(Seq(first)).union(rdd
  .sliding(2)
  .map(a => NameValueWithLag(a(1).name, a(1).value, a(1).value - a(0).value)))