All we can do is show you what there is along the way and then you can decide. You certainly have time to go there. You do not need to go quite so far as Brier Island if you don't want.
There are lots of things to see and do along the way. That is why we are so popular with tourists. You could see our Atlantic Coastline, the UNESCO Heritage town of Lunenburg, the beautiful Annapolis Valley with the wonderful towns of Annapolis Royal and Wolfville. There are scenic drives around there, wineries and microbreweries. Grand Pre, near Wolfville is another UNESCO World Heritage site for its Acadian history.
Here are some of our relevant Top Questions to help you see what we have to see and do and help you decide.
http://www.tripadvisor.ca/Travel-g154967-c212434/Nova-Scotia:Canada:Planning.Your.Trip.To.Nova.Scotia.html
http://www.tripadvisor.ca/Travel-g154967-c219541/Nova-Scotia:Canada:Learning.About.The.Bay.Of.Fundy.html
http://www.tripadvisor.ca/Travel-g154967-c219895/Nova-Scotia:Canada:Learning.About.The.South.Shore.html
http://www.tripadvisor.ca/Travel-g154967-d629037/Nova-Scotia:Canada:Annapolis.Valley.html
So you could go one way along some of the South Shore and come back along the Annapolis Valley. You could spend Thursday night in the Lunenburg area perhaps and then Friday night in Annapolis Royal or somewhere close to where you decide to take the whale watching trip. Saturday you could go whale watching and whatever else you have time for. Saturday night could be spent in the same place or around Wolfville if you have time to get there. Sunday you could spend time in the Wolfville area and then head towards Amherst, perhaps staying near Truro or along the Minas Basin shore. Monday you could finish your drive to Amherst.
Halifax to Lunenburg on the back roads for the scenery: https://goo.gl/maps/T854cLAVcey
Lunenburg to Annapolis Royal and on to Brier Island: https://goo.gl/maps/eYmNaKrj5S22
Annapolis Royal to Wolfville and area: https://goo.gl/maps/sdYRpNGYLCS2
Wolfville to Amherst: https://goo.gl/maps/VSyjQSUjrGH2