Ugh, and no sooner do I post this than do I notice the "peer_address" optional parameter on recv... no idea how I missed that when looking around yesterday :S
Still curious about constructing a network address with port, but pretty low priority, thanks :)