Starting at 16:30, after Hood's Brigades took over Devil's Den, Kershaw and Anderson attacked along Rose's Woods and the Wheatfield. The Union

General Meade had earlier specified that BG John C. Caldwell's Division of the II Corps to be sent to reinforce Sykes' position. At about 17:30, Caldwell arrived just as the Confederates drove the III and the V Corps from their positions. Caldwell's Division had barely arrived to reinforce the position (in fact the division arrived with with the rear ranks forward) when it came under heavy attack. It was enough to drive the Confederates temporarily back.

The Confederates soon counter-attacked - led by BG William T. Wofford 's GA Brigade. The Confederates broke the Union line at the Peach Orchard and began attacking down Wheatfield Road - outflanking Caldwell's Division which now now fell back in disorder. As the Confederates continued their attack, Col. Jacob B. Sweitzer's Brigade was sent into the Wheatfield to halt the Confederate advance. Fierce hand-to-hand fighting resulted but Sweitzer was unable to hold the Wheatfield.

As Sweitzer's men were forced to retreat, two brigades of BG Romeyn B. Ayres' Division entered east of the Wheatfield. They sought to delay the Confederates long enough so that the retreating Union could set up defensive positions on the ridge line just north of Little Round Top (the same ridge where Sickles should've been placed to begin with). In delaying the Confederates, the Union brigades took over 800 casualties. The Confederates reached Plum Run at the base of Little Round Top, but were unable to advance any further. A brigade of PA Reserves charged the Confederates and drove them back across the Wheatfield, but by this time it was dark.