Tom should be allowed to find love again, not stuck at Downton just because the same thing happened to Mary. Someone has to have romance upstairs. I hope Rose will improve.

As for the regular cast:
It is just too bad for Dan Stevens, Burning bridges with DA was handled badly, and it may have done more to hurt his career than help it.
The scene at the end of the cricket match of Robert with Tom and Matthew was one of the best from S3. I knew the rumors about Stevens’ leaving and had hoped it was a ploy to bump his salary and keep viewers on edge pending the beginning of S4. It would have been nice to see the storyline of Matthew’s vision for making the estate self-sustaining continue; there is a lot of potential for that aspect of DA.
Stevens was offered the chance to film a few scenes to keep the Matthew alive. With Matthew and Mary happily married, there was no UST to build upon, but there was the chance to see them “… build their kingdom…” and the chance to see Matthew become more prestigious. With Mary and Matthew secure enough in their relationship that he could accept a career with an embassy or a seat in Parliament would have been an ideal way to cover Stevens’ absence for most of S4.
Over Series 2 and 3, Mary learned to think beyond herself and care for others. Let us hope that Mary doesn’t spend S4 in constant mourning. The storyline of Matthew’s guilt over Lavinia’s death dragged on too long.
I like Tom Branson. He’s a much better man than Robert realized; he’s honest, he’s down-to-earth, and Tom has learned to fit into the family without having to compromise too much on his values. Tom had a lot of class when he told off his brother in front of the downstairs staff. (Mr. Carson, thank you for recognizing that.) Tom matured over S3, and I believe Robert will come to value Tom as much as he valued Matthew.
With the Downton succession in place, it will be Robert and Tom’s job to keep the estate in place. Having Mary work through her grief with the loss of Matthew by becoming involved in the workings of the estate would seem a good idea. It would also be a good way to build a closer alliance between Mary and Tom. Building a romance between Mary and Tom would seem a bit far-fetched, though.
While I may not be fond of Edith, I do think she’s been hurt enough and she doesn’t need another doomed relationship. Her becoming involved with Michael would be as pitiful as her involvement with Anthony Strallan. Edith should be allowed the chance to find a man without baggage – Strallan by his age and his injury; Michael by his inescapable marriage – neither is/was a decent prospect. Maybe Robert should give Edith the “… to go to America and … find a good man …” talk.
It’s time for Daisy to realize that Mr. Mason is offering her a chance to better herself beyond the confines of the kitchen. It would be good to Daisy “wise up” with Mr. Mason’s advice and become a successful, enterprising young woman.
Mrs. Hughes turned down a chance to rekindle an old flame, so her heart is at Downton. She knows the ol’ boy cares; how much more of a push does Mr. Carson need?
As for Thomas Barrow, it’s interesting how an unusual alliance with Bates was formed. Anna’s no fool, so it won’t take long for her to figure out the significance of “… her ladyship’s soap.” Thomas won’t allow himself to be made a target again. Should O’Brien persist in targeting him, she may find herself in hot water for a change.
Jimmy and Alfred can either learn to work with Thomas as the “under-butler”, or they can jolly well take a hike. Unlike William, I won’t miss either of them.

I want to see more of Thomas! I think I’m developing a crush on that guy actually. I hope he finally gets some love scenes. He’s been deprived too long.

What I do NOT want to see is a romance between Mary and Branson. That would be unholy or immoral or something. They should get close, be friends, raise the kids together but a romantic relationship would just undo all the love they had for Sybil and Matthew.