@fishjam-cloud/video-effects

WebGPU background effects and person-segmentation providers for Fishjam video sources.

Installation

yarn add @fishjam-cloud/video-effects

React Native applications using the TypeGPU provider also need compatible WebGPU and Worklets integrations. The provider requires React Native Worklets 0.12.1 or newer and the New Architecture.

Background blur

import { useBackgroundBlur } from "@fishjam-cloud/video-effects/background-blur";
import { typeGpuPersonSegmentation } from "@fishjam-cloud/video-effects/segmentation/typegpu";

const segmentation = typeGpuPersonSegmentation();
const effect = useBackgroundBlur({ segmentation, radius: 18 });

Pass effect to a compatible Fishjam video-source adapter. The TypeGPU provider consumes GPU-texture input and ships with the model in assets/selfie_segmenter.ssgbin.

Background image

import { useBackgroundImage } from "@fishjam-cloud/video-effects/background-image";

const effect = useBackgroundImage({
  segmentation,
  image: { uri: "https://example.com/background.jpg" },
  fit: "cover",
});

With fit: "cover" the image covers the full output. When viewers see the output cropped around its center, for example a portrait video in a 4:3 tile with objectFit: "cover", most of the image is outside the visible part. coverAspectRatio (width / height) fits the image to the largest centered part of the output that has this shape:

const effect = useBackgroundImage({
  segmentation,
  image,
  coverAspectRatio: 4 / 3,
});

The camera picture does not change. Outside that part the image continues as a mirror copy, so a viewer that crops to a different shape sees no bars.

Fishjam React Native camera

@fishjam-cloud/video-effects/fishjam-react-native runs an effect on the camera track Fishjam already publishes, through useCamera's camera-track middleware. No separate camera and no custom track: the app keeps using useCamera, and remote peers keep seeing peer.cameraTrack.

import { createBackgroundBlurEffect } from "@fishjam-cloud/video-effects/background-blur";
import { createCameraEffectMiddleware } from "@fishjam-cloud/video-effects/fishjam-react-native";
import { typeGpuPersonSegmentation } from "@fishjam-cloud/video-effects/segmentation/typegpu";

const segmentation = typeGpuPersonSegmentation({ modelUrl });
export const backgroundBlur = createCameraEffectMiddleware(
  createBackgroundBlurEffect(() => ({ segmentation, radius: 24 })),
);

const { currentCameraMiddleware, setCameraTrackMiddleware } = useCamera();
const isBlurOn = currentCameraMiddleware === backgroundBlur;
<Button
  onPress={() => setCameraTrackMiddleware(isBlurOn ? null : backgroundBlur)}
/>;

The middleware lives in Fishjam's camera state, so it stays applied across screens until it is cleared with null. Switch it on once the camera is on. Pass onStatus in the options to follow loading and errors.

The frame worklet works on a copy of the effect options, taken when the middleware starts on a track. To change coverAspectRatio of a background image while the effect runs, pass a Synchronizable from react-native-worklets. It is read on every frame:

import { createSynchronizable } from "react-native-worklets";

const tileAspectRatio = createSynchronizable(4 / 3);
export const backgroundImage = createCameraEffectMiddleware(
  createBackgroundImageEffect(() => ({
    segmentation,
    image,
    coverAspectRatio: tileAspectRatio,
  })),
);

// When the layout shows the video in a different shape:
tileAspectRatio.setBlocking(2 / 3);

For a component-scoped version there is useFishjamCameraEffect(effect | null), the twin of the web hook; it clears the effect when the component unmounts.

The model file ships in the package: require("@fishjam-cloud/video-effects/assets/selfie_segmenter.ssgbin") through expo-asset gives the modelUrl. Add ssgbin to Metro's resolver.assetExts.

The app must have these installed and linked: @fishjam-cloud/react-native-client, @fishjam-cloud/react-native-webrtc (0.30.2 or newer), @fishjam-cloud/react-native-worklets, react-native-worklets (0.12 or newer, with its Babel plugin) and react-native-webgpu, with the New Architecture on. Android needs API 26.

Entry points

• @fishjam-cloud/video-effects — provider and effect contracts
• @fishjam-cloud/video-effects/background-blur — blur factory and React hook
• @fishjam-cloud/video-effects/background-image — image-replacement factory and React hook
• @fishjam-cloud/video-effects/segmentation/typegpu — experimental GPU-only segmentation provider
• @fishjam-cloud/video-effects/web — browser track middleware
• @fishjam-cloud/video-effects/fishjam-react — Fishjam React adapter
• @fishjam-cloud/video-effects/fishjam-react-native — Fishjam React Native camera-track session

Effects improve presentation; they are not privacy or security boundaries.