Prophet Ismail was a baby at the time of Zamzam well incident according to the popular Muslim folklore. However, he may have been much older as you point out. The Qur’an mentions Ismail being a boy/youth—not fully grown up but just old enough to help his father in his chores—when Prophet Ibrahim mentions his dream to Ismail. This is vividly described in the Qur’an (37:100-109). However, the Qur’an also mentions that Prophet Ibrahim left his family (“some of my offspring”) with seminal prayers in the wilderness— near the Ka’ba or Makkah—the desolate desert (14:37-38). According to the Muslim legend, this occurs early on in Ismail’s infantile life. But that’s true, then the question is “When did Ismail go back to his father for the dream story to happen?”. The verses 37:100-101 seem to imply that the baby Ismail reaches the youth age in his father’s company. That being the case, the ‘boy/youth’ Ismail must have accompanied his father to the Makkan desert AFTER the dream story.
I would love to hear other views.