The organisation, which interviewed 500 women for its survey, says it is a common problem with nearly 60% of infertile women claiming to have felt "judged by their peers for leaving it late".
About 40% are too embarrassed to talk about fertility, especially with family and friends, often the biggest source of pressure.
Infertility viewed as failure'
Susan Seenan from the Network says this stigma can often prevent women from seeking help for fertility problems.
"Trying for a baby is a very personal thing which people don't always want to talk about, but there is constant pressure from families saying 'Isn't it about time...?'," said Ms Seenan.
"Unfortunately, infertility is still a taboo subject. When women are labelled as infertile they feel a failure, because they have let themselves and their partner down.
"Their basic biological instinct to have a child is kicking in - and at that point everyone seems to have babies, but they can't."
Ahead of National Infertility Awareness Week, starting on 28 October, Ms Seenan suggests that for many women in this position, it is easier to talk about mental health problems than infertility problems.
Women waiting too late to talk about fertility
Tim Child, medical director at the Oxford Fertility Unit at the University of Oxford agrees that people are leaving it too long before before going to see their GP about their fertility problems.
"When couples start talking about their fertility, that's the point to speak to a healthcare professional," he said.
"Good advice can be given early on about weight, diet, alcohol intake etc which could help, but many couples try for years before seeking help and before they know it they are in their late 30s - and in some areas that's too old."
He added that women wrongly assume that IVF is a good fallback solution when in fact the success rates are 40-50% for the under-35s, dropping to 20% for the under-40s and just 5% for women aged up to 43.