Cool. Yeah, an interesting approach that we've avoided for obvious complexity reasons, would be to allow explicit visibility rules about who can see what. Explicit control, but tough to administer. It's fun to work through the requirements though!
Right. Got it.
The first half is easily handled by making the guest the agenda item owner. That allows the person to add items to the agenda, etc. This permission / capability is already there for you.
The second half, limiting the participant's view of the agenda to only their item is new though. There's some work to do there, but I can see a path toward that.
It may get a bit tricky when the guest joins a running meeting, as the agenda for the entire meeting is visible to all participants during the meeting. That will require a bit more thought. Sometimes people invite guests for a single agenda item; sometimes they invite guests for the entire meeting. Those different approaches change how the guest interacts with the meeting content.