On one occasion the children said. “How happy we are here!” Later they said; “How happy we were there!” What are they referring to in both the cases?

The children were accustomed of playing freely every afternoon in the Giant’s garden during his absence. This made them very happy. Unfortunately, they were all driven out when the Giant returned to his castle. So in both cases, the children are referring to the garden where they were later prohibited to enter.