Bootstrap Row spacing issue

So basically I want to have 2 extra panels show up on the left and right on md and higher, but hide it when on sm and xs. The hiding works fine, but not the showing.

They do show up on the left and right, but the problem is the spacing between the crumbs and navigation. The crumbs get pushed down as I add content to the right panel. When I add to the left panel it gets pushed to the side..

Nothing on the left but something on the right is the same as the first picture

Here's what I mean:

I need it to be on the third picture so that crumbs stay right below nav.

Here's my code:

        <div class="container">
        <h1>Bootstrap grid (Bootstrap)</h1>
        <div class="site-box">
            <div class="container-fluid">

                <div class="row">

                    <!-- HEADER -->
                    <!-- Logo -->
                    <div class="col-xs-4 col-md-3">
                        logo
                    </div>

                    <!-- Search -->
                    <div class="col-xs-8 col-md-6">
                        search
                    </div>

                    <!-- Control Panel -->
                    <div class="col-xs-12 col-md-3">
                        control panel
                    </div>

                </div>

                <div class="row">

                    <div class="col-md-3 hidden-xs hidden-sm">
                        extra panel
                    </div>

                    <!-- NAVIGATION -->
                    <div class="col-xs-12 col-md-6">
                        nav
                    </div>


                    <div class="col-md-3 hidden-xs hidden-sm">
                        extra panel

                    </div>

                    <div class="clearfix visible-md-block"></div>

                    <!-- Crumbs -->
                    <div class="col-xs-12 col-md-6 col-md-offset-3">
                        crumbs
                    </div>


                </div>
            </div>
        </div>
    </div>