Think like a librarian for functionality. Group like series together. In addition to being attractive, a bookshelf needs to be arranged in a logical order so you can find your books. So if you have several books by the same author, group those together. Then go through your remaining books to create piles of books by the same similar genre, or of a similar topic. For example, I have one of my 4 shelves dedicated to teaching reference materials, and a separate shelf dedicated to reading for pleasure.
Arrange your books by height. Sort each pile you made in step 1 by height. This will make your sections look much neater and cleaner. If you have a pile of several books that are the same size, you could even further sort those books by color to make the piles look less "busy" (to achieve this, think of the color wheel: put reds, then oranges, yellows, then greens, blues, then purples, black and finally brown). Don't worry about sorting books by "depth." For a uniform look, pull out smaller books away from the back of the shelf or the wall so that they are flush with the edge of your larger books.
Try a new direction. Be creative! Now that you have well planned out piles of books organized by series, height, and color, instead of placing all of your piles vertically, place a few stacks horizontally. This will create more visual interest than having books all going the same way on a bookshelf. For those horizontal stacks, be sure to keep the largest books at the bottom to give the stack a solid base and avoid warping.
Another tip for your vertical sets of books is to have your largest books against the shelf, with your shorter books in the middle of a set for a "step down" effect into the center of your shelf. This brings the eye inward and keeps the focus on the design elements you will be adding to your bookshelf.