Phillip Rivers had a down season last year but that appears to be an anomaly if you compare it to other years. I think it is a compliment to Rivers to say last year he had a down year as he gained 4,600 yards as and made 27 touchdowns.
Therefore my list of the top quarterbacks who are most the best in the league would go:
Obviously Brady, Brees and Rodgers rank above Rivers and Roethlisberger can also go in the list above due to his consistent success with the Steelers. Peyton Manning would have been above Rivers if he hadn't just had neck surgery. Apart from that I can't honestly say (and I'm not being biased) that another QB is better. Yes Manning won the Superbowl but has he really been a better quarterback than Rivers? Rivers has the fifth best passer rating of all time which i value highly and the reason Romo is on this list is because he is second.
All these quarterbacks are special players, some have performed better on the big stage than others and if you were grading it on playoff and Superbowl performances Romo and Rivers probably wouldn't even make the list while Eli would probably be top dog.