Share the screen of an iOS device from an application

1Introduction

In this guide, you will learn how to share the screen of an iOS or iPadOS device as a video track into a Space so that remote participants can view it on their devices. This is a separate tutorial from the microphone and camera publishing guide. The key difference is that you'll be setting up a custom video source that captures video from ReplayKit and forwards it onto the SDK. Aside from this, you'll be using the same APIs and following the same as when publishing video from the camera.

While screen sharing does not actually require access to the microphone or camera there is a known issue in the Spaces Swift SDK for iOS that requires microphone permission when joining a space. We are currently working on resolving this issue so that joining a space does not require any additional permissions. Please reach out at real-time-video@mux.com if you would like to know when this issue is fixed.

Screen sharing on iOS

Every third-party application on iOS is sandboxed. This is a security measure to prevent an application from accessing data or making changes to the device without appropriate entitlements. For more information, refer to Apple's official documentation.

Sandboxing restrictions extend to the device screen. Getting access to screen content from outside of the application sandbox, requires developers to create and deploy a Broadcast Extension alongside their app to the App Store. This includes screen recording a different app, for example a game or a document editor like Pages. In this guide we'll only transmit the content of what is available in the current sandbox. A future version of this guide will include a walkthrough on creating a Broadcast Extension.

Screen sharing uses quite a lot of system resources and upstream network bandwidth, and while we constantly strive to optimize our code appropriately you will need to have realistic expectations about how well it will work as it is very dependent on your target device. The system is tuned to prioritize keeping the resolution slightly higher at the cost of framerate and latency.

That said, streaming sophisticated 3D rendering from games at good resolutions and framerates is very achievable on modern devices with good network connectivity. If you encounter a situation where you think it should be better please reach out to us at real-time-video@mux.com.

2Understand core abstractions

Space

A space is the basic abstraction for creating real-time communications with Mux. In order for clients to authenticate to a space, they need to provide a signed JSON Web Token, or JWT. See the Sign a JWT section of the Real-Time Video guide for more details.

Participant

A participant is an abstraction of a single user in a space. A participant can be a subscriber-only, or a publisher who sends one or more streams of audio or video media in the form of a track.

Track

A track is a single stream of media (audio or video). A participant can publish one or more tracks of media.

Creating a space

A space must be created either through the Mux dashboard or via the Mux API. See the Create a space section of the Real-Time Video guide for more details about creating a space.

Authenticating into a space

To join a space, we will need a JSON Web Token (JWT) that is generated by a server. See the Sign a JWT section of the Real-Time Video guide for more details.

3Application setup

Prerequisites for this example

To complete this example, you should have experience with iOS development, iOS development tools (Xcode) and, optionally, a device to test on.

Xcode Setup

As you're going through these steps, we recommend keeping open the SDK API documentation in Xcode to look up more information about the SDK APIs you'll be using.

Create a new Xcode project. We'll select the iOS platform and App application template.

Enter the name of your app, and make sure to select Storyboard as the Interface and Swift as the Language.

Install the SDK

The recommended way to install the MuxSpaces SDK for iOS is via Swift Package Manager

Install the SDK using Swift Package Manager (recommended)

In your project settings select Package Dependencies then click the plus button highlighted below:

This should pop open a search window, enter the URL for the Spaces Swift SDK:

https://github.com/muxinc/mux-spaces-sdk-swift-distribution

Specify what version of the SDK Swift Package Manager will pull down. To always fetch the latest, select Branch and enter main as the branch name. To pin a specific version of the SDK select Exact Version and enter the pinned version. See Apple's guidelines about deciding on package requirements for a deeper explanation of all Swift Package Manager options.

Click on Add Package to begin package download and installation.

Add the MuxSpaces and MuxSpacesReplayKit Package Products to your app target. The SDK is now installed and available to use after adding import MuxSpaces and import MuxSpacesReplayKit, respectively, wherever you make calls to SDK APIs.

Request microphone permissions

Due to a known issue in the Spaces Swift SDK for iOS, microphone permissions are required by your app when joining a space, even if your app is not publishing a microphone. Until the issue is resolved, you will need to include the NSMicrophoneUsageDescription key in your app's Info.plist file. Please refer to Apple's documentation about this property for more info. If you would like to be notified when this issue is resolved, please let us know at real-time-video@mux.com.

Create a view controller for displaying the app user interface

Next, we'll add a view controller that will display a button to join a space and a toggle for enabling and disabling screensharing. Create a Swift file called SpaceViewController.swift within your app directory and copy the following code into it.

import ReplayKit

import MuxSpaces
import MuxSpacesReplayKit

class SpaceViewController: UIViewController {
    var joinSpaceButton: UIButton!
    var screensharingLabel: UILabel!
    var screensharingToggle: UISwitch!

    override func viewDidLoad() {
        super.viewDidLoad()

        joinSpaceButton = UIButton()
        joinSpaceButton.translatesAutoresizingMaskIntoConstraints = false
        joinSpaceButton.setTitle(
            NSLocalizedString(
                "Join Space",
                comment: "Button that triggers connecting and joining a space"
            ),
            for: .normal
        )
        joinSpaceButton.setTitleColor(
            .systemBlue,
            for: .normal
        )
        joinSpaceButton.titleLabel?.textAlignment = .center
        joinSpaceButton.tintColor = .systemBlue

        joinSpaceButton.addAction(
            UIAction(
                handler: { [weak self] _ in

                    guard let self = self else {
                        return
                    }

                    self.joinSpaceButton.isEnabled = false

                    //self.space.join()

                    self.screensharingLabel.isHidden = false
                    self.screensharingToggle.isHidden = false
                }
            ),
            for: .touchUpInside
        )

        view.addSubview(joinSpaceButton)
        view.addConstraints([
            joinSpaceButton.centerXAnchor.constraint(equalTo: view.safeAreaLayoutGuide.centerXAnchor),
            joinSpaceButton.centerYAnchor.constraint(equalTo: view.safeAreaLayoutGuide.centerYAnchor, constant: -100)
        ])

        screensharingLabel = UILabel()
        screensharingLabel.translatesAutoresizingMaskIntoConstraints = false
        screensharingLabel.textAlignment = .center
        screensharingLabel.text = "Share screen:"
        screensharingLabel.isHidden = true
        view.addSubview(screensharingLabel)
        view.addConstraints([
            screensharingLabel.centerXAnchor.constraint(equalTo: view.safeAreaLayoutGuide.centerXAnchor),
            screensharingLabel.topAnchor.constraint(equalTo: joinSpaceButton.bottomAnchor, constant: 40)
        ])

        screensharingToggle = UISwitch()
        screensharingToggle.translatesAutoresizingMaskIntoConstraints = false
        screensharingToggle.isOn = false
        screensharingToggle.isEnabled = true
        screensharingToggle.isHidden = true
        screensharingToggle.addAction(
            UIAction(
                handler: { [weak self] _ in

                    guard let self = self else {
                        return
                    }

                    /*
                    if self.screensharingToggle.isOn {
                        self.handleScreenRecordingStart()
                    } else {
                        self.handleScreenRecordingStop()
                    }
                    */
                }
            ),
            for: .valueChanged
        )

        view.addSubview(screensharingToggle)
        view.addConstraints([
            screensharingToggle.centerXAnchor.constraint(equalTo: view.safeAreaLayoutGuide.centerXAnchor),
            screensharingToggle.topAnchor.constraint(equalTo: screensharingLabel.bottomAnchor, constant: 10)
        ])
    }
}

Let's also add SpaceViewController to Main.storyboard so that it is visible when launching the app.

The storyboard file is pre-seeded with a single scene, click on the scene and select the Custom Class tab on the Identity Inspector control panel on the right side of the Xcode window. Enter SpaceViewController as the Class. Make sure Inherit Module From Target check box is checked and the textfield right above that checkbox matches the name of your app target.

4Join a space

To join a space, we will need a signed JWT. See the Sign a JWT section of the Real-Time Video guide for more details.

Setup reference to a space

Add these properties to the view controller:

var space = try! Space(
    token: "YOUR_JWT"
)
var videoTrack: VideoTrack?

Replace "YOUR JWT" in this example with a signed JWT. When running in production, your application should make a request to fetch a signed JWT from your own authentication server.

Uncomment self.space.join() in the handler for addAction of joinSpaceButton. You should now be able to join a space.

5Share your screen to a Space

Now that our app is running and we're connected to a space, let's setup and publish our screensharing track. To create a screensharing track, your application will need to provide an object that conforms to CustomSampleBufferSource. This object will be managed by the SDK and will provide the video that gets consumed and transmitted by the track.

iOS applications use ReplayKit to capture screen content. The MuxSpaces SDK comes with a supporting package containing a ReplayKit helper class that conforms to CustomSampleBufferSource, we'll use this helper class here. If necessary, feel free to make changes or reuse the content of the helper package in your applications.

Add the following property to SpaceViewController.swift:

var sampleBufferSource: ReplayKitRecorderSampleBufferSource = ReplayKitRecorderSampleBufferSource()

Now let's add methods to the view controller to start and stop screensharing.

func handleScreenRecordingStart() {
  let videoTrack = space.makeScreenCaptureVideoTrack(
      customSource: sampleBufferSource
  )

  self.videoTrack = videoTrack

  space.publishTrack(
      videoTrack
  ) { error in

  }
}

func handleScreenRecordingStop() {
  guard let videoTrack else {
      return
  }

  space.unpublishTrack(videoTrack)
}

Finally uncomment out this snippet in the handler for addAction of screenshareToggle:

if self.screensharingToggle.isOn {
  self.handleScreenRecordingStart()
} else {
  self.handleScreenRecordingStop()
}

Build and run the application, you'll now be able to share your screen by first joining the space as you did before and then enabling the toggle.

FAQ

Can I share the camera and microphone at the same time as screen sharing?

Yes, you can publish camera and microphone tracks while screen sharing all at the same time. Follow this guide to learn how to build an app that publishes microphone and camera tracks.

Can I publish device audio too?

We do not support publishing device audio to a Space with the Swift SDK for iOS at this time. If you need this functionality, please e-mail us with your use case at real-time-video@mux.com.