Saving an item as "Favorite" and showing it across other recyclerviews

I have 1 viewpager with 2 pages main, favorites. Both have RecyclerViews, with endless scrolls, after 15 items it loads another 15 from server and so on. I want to implement an option to add an item from recyclerView to favorite and refresh both recyclerViews. I can add them to favorite or remove them if they are added. I already have this implemented and it works, perfectly.

The problem is, that if I add item to favorite, it won't refresh other RecyclerView. I had an idea, that for each time I add or remove from favorites, the other fragment restarts, so it request another query from server and you just load them in recycler view. It worked, but you were always put on the top, which is not a good UX. You can open item and get more information about that item, and you can add them to favorites inside that activity. So when i returned from that activity everything stayed the same, like you didn't add them to favorites. But actually you did.

I don't know how to solve this problem :/ other asking another query from server, and loading it inside of RecylerView. If i didn't had the endless scroll implemented i could probably share the items on both sides, but I have to have endless scroll, too much of data would be wasted.

This is how my Adapter for RecyclerView looks like:

        customViewHolder.llLike.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View v) {
            final String fontAwesomeFullHeart = mContext.getString(R.string.fa_heart);
            final String getLikeText = customViewHolder.tvPicLike.getText().toString();
            if (getLikeText == fontAwesomeFullHeart) {
                MyFunctions.showNotToFavoriteHeart(mContext, customViewHolder.tvPicLike, customViewHolder.tvLike);
                API_Network_Service.serverRemoveFromFavorites(mContext, feedItem.getId(), logged_in, customViewHolder.tvPicLike);
            } else {
                MyFunctions.showFavoriteHeart(mContext, customViewHolder.tvPicLike, customViewHolder.tvLike);
                API_Network_Service.serverAddToFavorites(mContext, feedItem.getId(), logged_in, customViewHolder.tvPicLike);
            }
        }
    });

And this is from API_SERVICE_CLASS:

   public static void serverAddToFavorites(final Context mContext, String apartment_id, RealmUser_Personal user, final TextView tvLike) {
    String user_id = user.getId();
    HashMap<String, String> map = new HashMap<>();
    map.put("user_id", user_id);
    HashMap<String, Object> mapMeta = new HashMap<>();
    mapMeta.put("meta", map);
    mapMeta.put("entry_id", apartment_id);
    final ApiUporabnik apiService = ApiClient.getAPIUser().create(ApiUporabnik.class);
    Call<FavoriteAdd> saveFavorite = apiService.favoriteAdd(API_KEY, mapMeta);
    saveFavorite.enqueue(new Callback<FavoriteAdd>() {
        @Override
        public void onResponse(Call<FavoriteAdd> call, Response<FavoriteAdd> response) {
            Log.d("Saved FAV", "");
            //TODO Refresh item everywhere
        }

        @Override
        public void onFailure(Call<FavoriteAdd> call, Throwable t) {
            Log.e("E: Failed FAV", t.toString());
            tvLike.setText(mContext.getString(R.string.fa_heart_o));
        }
    });
}

Any ideas, advice, solution?