This guide contains instructions for setting up screen sharing with the Mux Spaces Android SDK. By the end of the guide you'll have a working app that will be able to connect to a Space and share the video of your screen with other Participants.
Introduction
How screen sharing is different from other situations.
Understand core abstractions
Understand the concepts of working with Mux Real-Time Video.
Application setup
Set up your Android application with Android Studio.
Join a space and subscribe to participants
Join a Mux Space and receive media from other participants.
Share your screen to a Space
Publish a local participant's screen contents to a Mux Space.
Other considerations
In this guide, you will learn how to share the screen of an Android device as a video track into a Space so that remote participants can view it on their devices. This is a separate tutorial from the microphone and camera publishing guide as some of the important details are surprisingly different.
The key difference is that the Android operating system handles permission to share the screen in a fundamentally different way than accessing the camera or microphone, and this has implications for the API. We have worked to keep things as consistent as possible.
If you have no need to access the microphone or camera you will not need to request permission for them, and so we will not do so in this guide, although we will subscribe to remote camera and microphone feeds. A subscriber-only app would not need additional permissions to view video or hear audio.
Screen sharing uses quite a lot of system resources and upstream network bandwidth, and while we constantly strive to optimize our code appropriately you will need to have realistic expectations about how well it will work as it is very dependent on your target device. The system is tuned to prioritize keeping the resolution slightly higher at the cost of framerate and latency.
That said, streaming sophisticated 3D rendering from games at good resolutions and framerates is very achievable on modern devices with good network connectivity. If you encounter a situation where you think it should be better please reach out to us at real-time-video@mux.com.
A Space is the basic abstraction for creating real-time communications with Mux. In order for clients to authenticate to a space, they need to provide a signed JSON Web Token, or JWT. See the Sign a JWT section of the Real-Time Video guide for more details.
A participant is an abstraction of a single user in a space. A participant can be a subscriber-only, or a publisher who sends one or more streams of audio or video media in the form of a track.
A track is a single stream of media (audio or video). A participant can publish one or more tracks of media.
A Space must be created either through the Mux dashboard or via the Mux API. See the Create a Space section of the Real-Time Video guide for more details about creating a space.
If you already have Mux Access Tokens setup and just want to create a space from the terminal, use this command.
curl https://api.mux.com/video/v1/spaces \
-H "Content-Type: application/json" \
-X POST \
-u "${MUX_TOKEN_ID}:${MUX_TOKEN_SECRET}"To join a Space, we will need a signed JWT. See the Sign a JWT section of the Real-Time Video guide for more details.
To complete this example, you should have experience with Android development, Android development tools (Android Studio, Gradle etc.) and a device to test on.
Here you can view the API documentation as Javadoc.
Note: Our example will be in Java, but the SDK is compatible with Kotlin
MuxSpaces SDK:settings.gradle add the following two items to dependencyResolutionManagement → repositories (typically under google() and mavenCentral()):
maven { url 'https://jitpack.io' }
maven { url 'https://muxinc.jfrog.io/artifactory/default-maven-release-local' }app/build.gradle in the dependencies section of the app module build.gradle script:
implementation "com.mux.spaces:sdk:1.0.0"AndroidManifest.xml) ensure android:allowBackup="false" in the application element, e.g.:...
<application
android:allowBackup="false"
android:icon="@mipmap/ic_launcher">
...launchMode "singleTop", e.g.:...
<activity
android:launchMode="singleTop"
android:name=".MainActivity"
android:exported="true">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
...layout/activity_main.xml .LinearLayout. See below for an example with a white background that makes for a good backdrop for what you'll add next:<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:background="#FFFFFFFF"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:orientation="vertical">
</LinearLayout>TextView and will need a couple of buttons to start and stop screen sharing, so add these to the layout:...
<TextView
android:id="@+id/activity_main_status"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="Not initialized"
android:textColor="#FF000000" />
<Button
android:id="@+id/activity_main_start_screensharing"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="Start screensharing" />
<Button
android:id="@+id/activity_main_stop_screensharing"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="Stop screensharing" />
...com.mux.sdk.webrtc.spaces.views.TrackRendererSurfaceView (this is a SurfaceView subclass). At this time audio tracks are automatically played and mixed when you are subscribed to them.LinearLayout:<com.mux.sdk.webrtc.spaces.views.TrackRendererSurfaceView
android:id="@+id/activity_main_local_renderer"
android:layout_width="640dp"
android:layout_height="360dp" /><ScrollView
android:layout_width="match_parent"
android:layout_height="0dp"
android:layout_weight="1">
<LinearLayout
android:orientation="vertical"
android:id="@+id/activity_main_remote_renderers"
android:layout_width="match_parent"
android:layout_height="wrap_content">
</LinearLayout>
</ScrollView>Build and run the app, you should see the status, buttons, and a view where our screen share video will appear when we're screen sharing.
To join a Space, we will need a JSON Web Token (JWT) that is generated by a server. See the Sign a JWT section of the Real-Time Video guide for more details. We will use the JWT to assemble this into a SpaceConfiguration in a new method in MainActivity and call it from the onCreate method. We will also gather references to the views we created and keep them in class variables for which the different View classes will need to be imported:
private TrackRendererSurfaceView localRenderView;
private TextView status;
private Button startScreenshare;
private Button stopScreenshare;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
localRenderView = findViewById(R.id.activity_main_local_renderer);
status = findViewById(R.id.activity_main_status);
startScreenshare = findViewById(R.id.activity_main_start_screensharing);
stopScreenshare = findViewById(R.id.activity_main_stop_screensharing);
joinSpace();
}
private void joinSpace() {
status.setText("Joining space");
final String jwt = "YOUR JWT";
try {
SpaceConfiguration spaceConfiguration = SpaceConfiguration.newBuilder()
.setJWT(jwt)
.build();
} catch (Exception e) {
status.setText("Error with configuration: "+e.getMessage());
e.printStackTrace();
return;
}
}Replace "YOUR JWT" in this example with the JWT that you create server-side. In a production application, your application should make a request to fetch the JWT from a server.
Next, create class level variables for Spaces (the SDK) and Space, and when importing Space make sure to import "com.mux.sdk.webrtc.spaces.Space".
private Spaces spaces;
private Space space;Before calling joinSpace in onCreate, get the Spaces instance and assign it to the variable:
spaces = Spaces.getInstance(this);We then get Spaces to retrieve the Space at the bottom of joinSpace after the try/catch block:
space = spaces.getSpace(spaceConfiguration);If there is an existing instance of the same configuration that will be returned. Simply having an empty Space doesn’t do anything, you must join it, and for the Android SDK it is a requirement to add a Space.Listener when you join, so we will create a class level variable holding our Space.Listener:
private final Space.Listener spaceListener = new Space.Listener() {};This is where all events relevant to the Space will occur, and we provide empty default implementations. All callbacks occur on the UI thread so you won’t have to worry about accessing UI components directly. (The SDK itself is multithreaded and should not ever block the UI thread. This is also why you receive callbacks on Space.Listener for Participant and Track events on Android: this way the events arrive in exact order and it’s harder to lose them when you’re doing something else)
To test this out, let's trigger a Toast when the Space is joined, and let's add an onError override within the Listener:
private final Space.Listener spaceListener = new Space.Listener() {
@Override
public void onJoined(Space space, LocalParticipant localParticipant) {
Toast.makeText(MainActivity.this, "Joined space "+space.getId()+" as "+localParticipant.getId(), Toast.LENGTH_LONG).show();
status.setText("Joined");
}
@Override
public void onError(Space space, MuxError muxError) {
Toast.makeText(MainActivity.this, "Error! "+muxError.toString(), Toast.LENGTH_LONG).show();
}
};To join the space, add this line at the bottom of the try/catch in the onCreate function:
space.join(spaceListener);Finally, to prevent joining a Space multiple times, add a statement to return if we already have a reference to a Space:
private void joinSpace() {
if(space != null) {
return;
}
...Now that our app is running and we're connected to a space, we can subscribe to remote participants. There are events in Space.Listener that deal with this:
onParticipantTrackPublished: This will fire when a remote participant publishes a track.onParticipantTrackSubscribed: This will fire when we have chosen to actively subscribe to a track and our subscription has been accepted. As of now the SDK will automatically subscribe to all tracks that are added.We now need to create TrackRendererSurfaceViews and set their tracks as things appear, so let’s override onParticipantTrackSubscribed in our spaceListener.
@Override
public void onParticipantTrackSubscribed(Space space, Participant participant, Track track) {
// We can only add video type tracks to views or we'll get an IllegalArgumentException
if(track.trackType == Track.TrackType.Video) {
TrackRendererSurfaceView trackRendererSurfaceView = new TrackRendererSurfaceView(MainActivity.this);
// Evil sizing hard coding to keep things on point
trackRendererSurfaceView.setLayoutParams(new LinearLayout.LayoutParams(320, 240));
trackRendererSurfaceView.setTrack(track);
((ViewGroup) findViewById(R.id.activity_main_remote_renderers)).addView(trackRendererSurfaceView);
}
}Note about audio: You will receive events when audio track subscriptions start and stop, however, right now all subscribed audio is automatically mixed and played without you needing to do anything.
Right now in your app, when remote participants disconnect you will be left with a frozen view. To handle that, we will add a HashMap class member variable to keep track of the View associated with each Track:
private HashMap<Track, TrackRendererSurfaceView> remoteViews;Create it in onCreate immediately after setting the content view:
remoteViews = new HashMap<>();When we add views in onParticipantTrackSubscribed, we also put them in the map immediately following trackRendererSurfaceView.setTrack(track):
remoteViews.put(track, trackRendererSurfaceView);We can then remove the view when the track unsubscribes:
@Override
public void onParticipantTrackUnsubscribed(Space space, Participant participant, Track track) {
TrackRendererSurfaceView view = remoteViews.get(track);
if(view != null) {
((ViewGroup) findViewById(R.id.activity_main_remote_renderers)).removeView(view);
remoteViews.remove(view);
}
}We're almost in the place to publish our screen remotely. However, as mentioned permissions are very different when screen sharing than when using the camera and microphone.
onActivityResult. The SDK handles the bulk of this for you, but needs you to pass the appropriate events from your Activity to the SDK for processing.After retrieving the Spaces instance with Spaces.getInstance(this) we need to call the following:
spaces.setActivity(this);This call allows the SDK to use our Activity to make the appropriate system request to start screen sharing. For handling the result we now need to override onActivityResult in our Activity and pass the result to the SDK:
@Override
protected void onActivityResult(int requestCode, int resultCode, Intent data) {
spaces.onActivityResult(requestCode, resultCode, data);
}We considered providing an Activity subclass which does this for you. However, the approach described in this guide makes it easier to integrate with other libraries or frameworks.
Screen sharing is represented as a separate Track on the LocalParticipant received when joining a Space. (As long as the Space is active the same LocalParticipant is available via space.getLocalParticipant()). This means we can always access the Track for the screen sharing at space.getLocalParticipant().getScreenCaptureTrack().
The SDK will initiate the flow to request and start screen sharing when either that Track is published to a Space or attached to a TrackRendererSurfaceView via setTrack. It will stop capturing the screen only when the Track has been unpublished and any TrackRendererSurfaceView displaying it has been changed to display something else, or nothing. The video will only be transmitted remotely while it is published.
We are going to use the buttons we created earlier to trigger these situations. The visibility state of the buttons will need to change based on the situation, and we will also want to update the status text and contents of our localRenderView. The easiest way to do this is to create an updateUI method in our Activity which we will then call on the relevant events. It should look like:
private void updateUI() {
if(space == null) {
status.setText("Not connected");
startScreenshare.setVisibility(View.GONE);
stopScreenshare.setVisibility(View.GONE);
localRenderView.setTrack(null);
return;
}
if(space.getLocalParticipant().getScreenCaptureTrack().isPublished()) {
startScreenshare.setVisibility(View.GONE);
stopScreenshare.setVisibility(View.VISIBLE);
localRenderView.setTrack(space.getLocalParticipant().getScreenCaptureTrack());
} else {
startScreenshare.setVisibility(View.VISIBLE);
stopScreenshare.setVisibility(View.GONE);
localRenderView.setTrack(null);
}
}Now call this from a few different places! Call it in onCreate before joinSpace, and in the Space.Listener in onJoined and onError, and because we're going to be publishing and unpublishing we will listen for those too, so add these to the listener:
@Override
public void onParticipantTrackPublished(Space space, Participant participant, Track track) {
updateUI();
}
@Override
public void onParticipantTrackUnpublished(Space space, Participant participant, Track track) {
updateUI();
}This takes advantage of the fact all callbacks from the SDK to the application are on the main thread so you don't need to worry about accessing Views from inside the listener.
Because the system will launch another Activity when requesting permission we may get paused, and then return. We will use what we have to override our Activity onResume:
protected void onResume() {
super.onResume();
joinSpace();
updateUI();
}Finally we're ready to actually trigger the interactions. This is a relatively simple matter of attaching click listeners to the buttons, which we will do in onCreate after getting the View references:
status.setText("onCreate called");
startScreenshare.setVisibility(View.GONE);
stopScreenshare.setVisibility(View.GONE);
startScreenshare.setOnClickListener(new View.OnClickListener() {
@Override
public void onClick(View view) {
if(space != null) {
space.getLocalParticipant().publish(space.getLocalParticipant().getScreenCaptureTrack());
updateUI();
}
}
});
stopScreenshare.setOnClickListener(new View.OnClickListener() {
@Override
public void onClick(View view) {
if(space != null) {
space.getLocalParticipant().unpublish(space.getLocalParticipant().getScreenCaptureTrack());
updateUI();
}
}
});This sets the button visibility to GONE, which will hide them (updateUI() will take care of showing the appropriate button if needed). The important lines are those which call publish and unpublish on the Track returned from getScreenCaptureTrack. These will trigger the permissions request and publishing flow.
In the case the user refuses permission the Space.Listener onError will be called with a LocalParticipant.Errors.ScreensharePermissionDenied error, but if all goes well the onParticipantTrackPublished and onParticipantTrackUnsubscribed will be called. updateUI() will inspect the state of the screen capture Track to see if it is published and update the state of everything accordingly.
If you attempt to run the app at this point you should find that it works. You should be able to see RemoteParticipants appear as they publish their camera or screen share videos in the Space, and you should be able to share your screen.
If you have run the app you will notice that while the screen is being captured a Notification appears in the system notification area. This is a requirement of the Android operating system for all applications that capture the screen. The SDK provides a very bare bones notification, but it will not be specific to your app.
It is important that the user knows an application is screensharing, knows which application is screensharing, and can quickly return to the location in that application which would allow them to stop screen sharing. We will need to make a Notification that tells the user it's our app and when pressed takes them to our Activity where they can stop screensharing.
The magic code in the example app to enable this follows the familiar pattern of using Notification.Builder to construct the notification in our onCreate right before getting the Spaces instance:
// We can customize the notification the SDK displays while screen sharing
// This serves two important purposes:
// 1. It refers to this specific application as doing the screen sharing
// 2. We create a PendingIntent which returns to this Activity instance when selecting the notification so the user can get
// back to a screen where they can stop the screensharing
NotificationManager nm = (NotificationManager) getSystemService(Context.NOTIFICATION_SERVICE);
Notification.Builder builder = null;
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
NotificationChannel channel = new NotificationChannel(
"screen_capture_channel_id",
"Screen Capture",
NotificationManager.IMPORTANCE_LOW
);
nm.createNotificationChannel(channel);
builder = new Notification.Builder(this, channel.getId());
} else {
builder = new Notification.Builder(this);
}
// For this to work the launchMode of MainActivity must be set to "singleTop" in AndroidManifest.xml
// This ensures Activity instances are unique and so the notification won't do things like launch a new activity
// If you have more complex use cases which require things like preserving a large back stack you will need to use one of the
// normal approaches for doing that, but it will be highly specific to your application.
Intent intent = new Intent(this, MainActivity.class);
PendingIntent pendingIntent = PendingIntent.getActivity(this, 1, intent, PendingIntent.FLAG_MUTABLE);
// The setSmallIcon is necessary or the operating system will silently disregard the notification, and so the whole screen sharing request
// setOngoing ensures the user can't remove the notification while screensharing
builder.setPriority(Notification.PRIORITY_DEFAULT)
.setContentIntent(pendingIntent)
.setSmallIcon(R.mipmap.ic_launcher)
.setOngoing(true)
.setContentTitle("Mux Spaces SDK Screen sharing example")
.setContentText("Your screen is being shared");
...There are several non obvious pieces here:
setSmallIcon is necessary (you can use another icon but without it the operating system will not recognize the notification)setOngoing(true) is required to make sure the user can't remove the notificationlaunchMode of the Activity being called in the PendingIntent must be singleTop.That last point isn't always strictly true, however, it's assumed if you're doing anything more complex such as using a framework that preserves the back stack for a task then you are able to modify the above to suit your situation.
Now we have our Notification, we need to give it to the SDK right after getting the Spaces instance. This leads the end of our onCreate implementation to look like:
...
Notification notification = builder.build();
updateUI();
spaces = Spaces.getInstance(this);
spaces.setScreenShareNotification(notification);
spaces.setActivity(this);
joinSpace();One unfortunate aspect of sharing your screen is you probably want to share the display of other applications and not just your own. This works but makes our Activity lifecycle a little more complex than the situation with a camera. For instance we don't want our application to quit if we're screensharing, but otherwise if we're put into the background it would be a good idea to leave the Space. To achieve this the Activity method we will override is onStop:
@Override
protected void onStop() {
if(space != null && space.getLocalParticipant().getScreenCaptureTrack().isPublished()) {
// Don't do anything!
// We want to keep the app running when in the background
} else {
if (space != null) {
// This is so that when the screen rotates we don't disconnect and reconnect to the server
// When the device is being rotated isChangingConfigurations will return true, and this flag
// will ensure that the space stays connected for a short grace period (about half a second)
// so that when the Activity requests the space with spaces.getSpace(spaceConfiguration); it
// actually gets exactly the same instance, in the same connected state.
space.leave(spaceListener, isChangingConfigurations());
space = null;
}
if (!isChangingConfigurations()) {
finish();
}
}
super.onStop();
}First it uses a familiar pattern (as in our implementation of updateUI) to check if we're currently publishing the screen, and if we are we will take no particular action. Otherwise we will handle the call with similar variation for if we are in the middle of a configuration change (a screen rotation, keyboard appearing etc.) In the event that something else has led to onStop and we're not publishing we do the right thing and request the app be terminated.
At this point the screen share example is complete. To see the complete solution, refer to the examples repo.
Yes, you can publish camera and microphone tracks while screen sharing all at the same time. Follow this guide to learn how to build an app that publishes microphone and camera tracks.
We do not support publishing device audio to a Space with the Android SDK at this time. If you need this functionality, please e-mail us with your use case at real-time-video@mux.com.