This is a good question, and I've seen a number of different approaches taken to measuring a game's (or app's) marketability in pre-production or before launch.
The most popular method is to just throw some ads up using StoreMaven and get a sense for how your target audience responds to those ads. This will help you understand what your acquisition costs might look like, but it won't help you understand if your game monetizes, or to what scale your game can grow, etc., so this isn't a comprehensive business analysis. You can benchmark these performance metrics (CTR, CPI, etc.) against other games at similar scale to get a feel for how the new game / game concept compares, though.
But to address the "marketability is weak" comment, this is probably just a dodge: the publisher doesn't feel strongly enough about your game to run a test against it. No serious publisher would sign a game without first running a marketing test against it. The publisher would want to see advertising metrics (CTR, CPI, delivery, CPM, etc.) as well as funnel metrics (CTI, install rate) and early-stage engagement metrics (D1-D3 retention, early monetization, etc.). You can't sign a game without understanding those metrics: run away from any publisher that doesn't want to see those before signing your game.
And if a publisher is passing on your game before even conducting a test, it simply means that they don't believe the game has the potential to be competitive: they likely think it won't be marketable because: too niche, too obscure, space too crowded, game too undifferentiated, etc.