Mux Real-Time Video has been sunset and is unavailiable for new usage. Existing access will end on December 31, 2023. We recommend migrating your application to our partner, LiveKit. Please reach out to real-time-video@mux.com if you need more help or details.
This guide contains instructions for setting up screen sharing with the Mux Spaces Swift SDK for iOS. By the end of the guide you'll have a working app that will be able to connect to a space and share the video of your screen with other Participants.
Introduction
How screen sharing is different from other situations.
Understand core abstractions
Understand the concepts of working with Mux Real-Time Video.
Application setup
Set up your iOS application with Xcode.
Join a space
Join a Mux Space.
Share your screen to a Space
Publish a local participant's screen contents to a Mux Space.
In this guide, you will learn how to share the screen of an iOS or iPadOS device as a video track into a Space so that remote participants can view it on their devices. This is a separate tutorial from the microphone and camera publishing guide. The key difference is that you'll be setting up a custom video source that captures video from ReplayKit and forwards it onto the SDK. Aside from this, you'll be using the same APIs and following the same as when publishing video from the camera.
While screen sharing does not actually require access to the microphone or camera there is a known issue in the Spaces Swift SDK for iOS that requires microphone permission when joining a space. We are currently working on resolving this issue so that joining a space does not require any additional permissions. Please reach out at real-time-video@mux.com if you would like to know when this issue is fixed.
Every third-party application on iOS is sandboxed. This is a security measure to prevent an application from accessing data or making changes to the device without appropriate entitlements. For more information, refer to Apple's official documentation.
Sandboxing restrictions extend to the device screen. Getting access to screen content from outside of the application sandbox, requires developers to create and deploy a Broadcast Extension alongside their app to the App Store. This includes screen recording a different app, for example a game or a document editor like Pages. In this guide we'll only transmit the content of what is available in the current sandbox. A future version of this guide will include a walkthrough on creating a Broadcast Extension.
Screen sharing uses quite a lot of system resources and upstream network bandwidth, and while we constantly strive to optimize our code appropriately you will need to have realistic expectations about how well it will work as it is very dependent on your target device. The system is tuned to prioritize keeping the resolution slightly higher at the cost of framerate and latency.
That said, streaming sophisticated 3D rendering from games at good resolutions and framerates is very achievable on modern devices with good network connectivity. If you encounter a situation where you think it should be better please reach out to us at real-time-video@mux.com.
A space is the basic abstraction for creating real-time communications with Mux. In order for clients to authenticate to a space, they need to provide a signed JSON Web Token, or JWT. See the Sign a JWT section of the Real-Time Video guide for more details.
A participant
is an abstraction of a single user in a space. A participant
can be a subscriber-only, or a publisher who sends one or more streams of audio or video media in the form of a track
.
A track
is a single stream of media (audio or video). A participant
can publish one or more tracks of media.
A space must be created either through the Mux dashboard or via the Mux API. See the Create a space section of the Real-Time Video guide for more details about creating a space.
To join a space, we will need a JSON Web Token (JWT) that is generated by a server. See the Sign a JWT section of the Real-Time Video guide for more details.
To complete this example, you should have experience with iOS development, iOS development tools (Xcode) and, optionally, a device to test on.
As you're going through these steps, we recommend keeping open the SDK API documentation in Xcode to look up more information about the SDK APIs you'll be using.
Create a new Xcode project. We'll select the iOS platform and App application template.
Enter the name of your app, and make sure to select Storyboard as the Interface and Swift as the Language.
The recommended way to install the MuxSpaces
SDK for iOS is via Swift Package Manager
In your project settings select Package Dependencies then click the plus button highlighted below:
This should pop open a search window, enter the URL for the Spaces Swift SDK:
https://github.com/muxinc/mux-spaces-sdk-swift-distribution
Specify what version of the SDK Swift Package Manager will pull down. To always fetch the latest, select Branch and enter main
as the branch name. To pin a specific version of the SDK select Exact Version
and enter the pinned version. See Apple's guidelines about deciding on package requirements for a deeper explanation of all Swift Package Manager options.
Click on Add Package to begin package download and installation.
Add the MuxSpaces
and MuxSpacesReplayKit
Package Products to your app target. The SDK is now installed and available to use after adding import MuxSpaces
and import MuxSpacesReplayKit
, respectively, wherever you make calls to SDK APIs.
Due to a known issue in the Spaces Swift SDK for iOS, microphone permissions are required by your app when joining a space, even if your app is not publishing a microphone. Until the issue is resolved, you will need to include the NSMicrophoneUsageDescription
key in your app's Info.plist
file. Please refer to Apple's documentation about this property for more info. If you would like to be notified when this issue is resolved, please let us know at real-time-video@mux.com.
Next, we'll add a view controller that will display a button to join a space and a toggle for enabling and disabling screensharing. Create a Swift file called SpaceViewController.swift
within your app directory and copy the following code into it.
import ReplayKit
import MuxSpaces
import MuxSpacesReplayKit
class SpaceViewController: UIViewController {
var joinSpaceButton: UIButton!
var screensharingLabel: UILabel!
var screensharingToggle: UISwitch!
override func viewDidLoad() {
super.viewDidLoad()
joinSpaceButton = UIButton()
joinSpaceButton.translatesAutoresizingMaskIntoConstraints = false
joinSpaceButton.setTitle(
NSLocalizedString(
"Join Space",
comment: "Button that triggers connecting and joining a space"
),
for: .normal
)
joinSpaceButton.setTitleColor(
.systemBlue,
for: .normal
)
joinSpaceButton.titleLabel?.textAlignment = .center
joinSpaceButton.tintColor = .systemBlue
joinSpaceButton.addAction(
UIAction(
handler: { [weak self] _ in
guard let self = self else {
return
}
self.joinSpaceButton.isEnabled = false
//self.space.join()
self.screensharingLabel.isHidden = false
self.screensharingToggle.isHidden = false
}
),
for: .touchUpInside
)
view.addSubview(joinSpaceButton)
view.addConstraints([
joinSpaceButton.centerXAnchor.constraint(equalTo: view.safeAreaLayoutGuide.centerXAnchor),
joinSpaceButton.centerYAnchor.constraint(equalTo: view.safeAreaLayoutGuide.centerYAnchor, constant: -100)
])
screensharingLabel = UILabel()
screensharingLabel.translatesAutoresizingMaskIntoConstraints = false
screensharingLabel.textAlignment = .center
screensharingLabel.text = "Share screen:"
screensharingLabel.isHidden = true
view.addSubview(screensharingLabel)
view.addConstraints([
screensharingLabel.centerXAnchor.constraint(equalTo: view.safeAreaLayoutGuide.centerXAnchor),
screensharingLabel.topAnchor.constraint(equalTo: joinSpaceButton.bottomAnchor, constant: 40)
])
screensharingToggle = UISwitch()
screensharingToggle.translatesAutoresizingMaskIntoConstraints = false
screensharingToggle.isOn = false
screensharingToggle.isEnabled = true
screensharingToggle.isHidden = true
screensharingToggle.addAction(
UIAction(
handler: { [weak self] _ in
guard let self = self else {
return
}
/*
if self.screensharingToggle.isOn {
self.handleScreenRecordingStart()
} else {
self.handleScreenRecordingStop()
}
*/
}
),
for: .valueChanged
)
view.addSubview(screensharingToggle)
view.addConstraints([
screensharingToggle.centerXAnchor.constraint(equalTo: view.safeAreaLayoutGuide.centerXAnchor),
screensharingToggle.topAnchor.constraint(equalTo: screensharingLabel.bottomAnchor, constant: 10)
])
}
}
Let's also add SpaceViewController
to Main.storyboard
so that it is visible when launching the app.
The storyboard file is pre-seeded with a single scene, click on the scene and select the Custom Class tab on the Identity Inspector control panel on the right side of the Xcode window. Enter SpaceViewController
as the Class. Make sure Inherit Module From Target check box is checked and the textfield right above that checkbox matches the name of your app target.
To join a space, we will need a signed JWT. See the Sign a JWT section of the Real-Time Video guide for more details.
Add these properties to the view controller:
var space = try! Space(
token: "YOUR_JWT"
)
var videoTrack: VideoTrack?
Replace "YOUR JWT"
in this example with a signed JWT. When running in production, your application should make a request to fetch a signed JWT from your own authentication server.
Uncomment self.space.join()
in the handler for addAction
of joinSpaceButton
. You should now be able to join a space.
Now that our app is running and we're connected to a space, let's setup and publish our screensharing track. To create a screensharing track, your application will need to provide an object that conforms to CustomSampleBufferSource
. This object will be managed by the SDK and will provide the video that gets consumed and transmitted by the track.
iOS applications use ReplayKit to capture screen content. The MuxSpaces SDK comes with a supporting package containing a ReplayKit helper class that conforms to CustomSampleBufferSource
, we'll use this helper class here. If necessary, feel free to make changes or reuse the content of the helper package in your applications.
Add the following property to SpaceViewController.swift
:
var sampleBufferSource: ReplayKitRecorderSampleBufferSource = ReplayKitRecorderSampleBufferSource()
Now let's add methods to the view controller to start and stop screensharing.
func handleScreenRecordingStart() {
let videoTrack = space.makeScreenCaptureVideoTrack(
customSource: sampleBufferSource
)
self.videoTrack = videoTrack
space.publishTrack(
videoTrack
) { error in
}
}
func handleScreenRecordingStop() {
guard let videoTrack else {
return
}
space.unpublishTrack(videoTrack)
}
Finally uncomment out this snippet in the handler for addAction
of screenshareToggle
:
if self.screensharingToggle.isOn {
self.handleScreenRecordingStart()
} else {
self.handleScreenRecordingStop()
}
Build and run the application, you'll now be able to share your screen by first joining the space as you did before and then enabling the toggle.
Yes, you can publish camera and microphone tracks while screen sharing all at the same time. Follow this guide to learn how to build an app that publishes microphone and camera tracks.
We do not support publishing device audio to a Space with the Swift SDK for iOS at this time. If you need this functionality, please e-mail us with your use case at real-time-video@mux.com.