The afternoon traffic will be heavy going TOWARD New Jersey, not toward Manhattan.
Then, if you can get to the George Washington Bridge by 3:00 p.m., take it.
What I recommend instead is the New Jersey Turnpike all the way from Delaware, since it almost always moves quickly outside of rush hour. US Route 206 is a slow road, and the detour north to the Tappan Zee Bridge will prolong your trip unnecessarily. Don't take I-95, US Route 206, or the Tappan Zee Bridge. As a veteran driver up and down the Northeast Corridor, here is my advice.