The children were accustomed of playing freely every afternoon in the Giant’s garden during his absence. This made them very happy. Unfortunately, they were all driven out when the Giant returned to his castle. So in both cases, the children are referring to the garden where they were later prohibited to enter.