Is there a reason you aren't resistance training? It might make a huge difference for you. The metabolic benefits are numerous.
You mentioned that you go for walks, which is great, but we need to define what is activity, and what is exercise. Walking is activity. It's just part of our daily function as a human. Exercise is when we are .... well..... exercising our body beyond what is a normal daily activity.
We want to keep activity high
And we want to keep exercise high
Incorporating resistance training would be a great way to do that.
Also, keep in mind the difference between scale weight and bodyfat. The scale represents all the mass of our body, and all the water the tissues hold. If you are eating full calories on the weekend, you are likely holding a lot of water from the increased carbohydrate intake.