Rough Trade has a VERY small used section now. 99% is new records. Academy is large and has a great used section as well as new records. If you hit up Academy in greenpoint, you could also walk over to Record Grouch and if you really want to dig, go to The Thing.
Norm's Sound and Vision has a great used selection. And I'll +1 for Academy, TTL, and Rough Trade. Honestly as someone who lived in NYC for six years up until recently, they have alot of record shops I like but none I love. The prices may have something to do with it.
Academy Records on Oak Street in Brooklyn is a short walk from Rough Trade. One of my favorite shops in the city. Stop and grab a drink at Dolly's Swing & Dive on Kent between stops
@wynn72 I find Norm's to be INSANELY expensive. It's right by my work and every time I walk in and expect them to have lowered prices, they always astound me on how high their prices are. They were selling RSD items at flipper prices last time I went in. Academy on Oak has way better prices and a good selection of used stuff IMO.