In the Swoyer paper, he describes a problem for abstract objects:
"Epistemology is the Achilles' heel of realism about abstracta. We are biological organisms thoroughly ensconced in the natural, spatiotemporal causal order. Abstract entities, by contrast, are atemporal, non-spatial, and causally inert, so they cannot affect our senses, our brains, or our instruments for measuring and detecting"
A couple paragraphs down he elaborates:
"Indeed, even if abstracta did exist, it is difficult to see how they could make any difference to our cognitive processes. Things would seem just the same whether they existed or not" P.27
I have two responses:
On page 24 he states:
"We can rarely explain much with the bald assertions that numbers exist or that properties exist. These claims are typically part of a longer story, a philosophical theory, that tells us something about what the relevant abstract entity is like. The theory also needs to explain how the entity is related to other things ... The account also needs to tell us how its abstracta are related to the phenomena around us that led us to postulate them in the first place"
Any theory worth its salt will have these components. This means that the abstracta are in fact related to the (observable) phenomena around us. If properties are posited to explain why certain things are similar to other things, then properties will have a special relationship with those things that the theory must elaborate on. Once it does, a very real tie is established between the abstracta and something observable. I'm betting that in most cases this can easily be exploited to shed light on the epistemic problem.
Consider properties. They don't cause anything. However, objects do instantiate properties. There are observable differences between things that instantiate some properties rather than others. We can come to know of these abstracta by observing such differences.
Consider numbers. They must be in some way related to collections of objects if we're to say there are a certain number of objects in a collection. Moreover, how many objects there are in a collection is an observable fact, even though the number itself is not. Given strict relationships between numbers, once we come to know of a few it's not hard to gain knowledge of the rest.
A side note about the second quote. He states that if abstracta didn't exist then things would seem exactly the same to us. This is far from obvious. For instance, if abstract objects called properties are in fact responsible for the differences between objects and the relationships between them, then if there were none then no objects would ever be related to have any properties. You can sure bet that if that was the world we were in, we'd be experiencing it a heck of a lot differently.
He actually describes an epistemic rout to abstracta in his paper. We argue for abstracta by considering the merits of theories which posit them. How is that not a good way to gain knowledge of them? Why doesn't that qualify? Why must we posit an entity with great explanitory value, and then explain how we could have gained knowledge of it in another way?