<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Dyte - Medium]]></title>
        <description><![CDATA[Real-time audio &amp; video SDKs, ready to launch. - Medium]]></description>
        <link>https://medium.com/dyte?source=rss----9477120405bc---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Dyte - Medium</title>
            <link>https://medium.com/dyte?source=rss----9477120405bc---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:27:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/dyte" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Live Proctoring System to Detect Participant Switching]]></title>
            <link>https://medium.com/dyte/building-a-live-proctoring-system-to-detect-participant-switching-7173cce6969c?source=rss----9477120405bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/7173cce6969c</guid>
            <category><![CDATA[video-sdk]]></category>
            <category><![CDATA[proctoring-software]]></category>
            <category><![CDATA[video-api]]></category>
            <dc:creator><![CDATA[Ravindra Rathor]]></dc:creator>
            <pubDate>Tue, 02 Jan 2024 11:48:45 GMT</pubDate>
            <atom:updated>2024-01-02T11:48:45.361Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v5Hh8cIQtHL-VWXPpkfYzw.png" /></figure><h3>Introduction</h3><p>To stop students from cheating during tests, we often assign a proctor to observe them. But we can’t have someone in-person watching each student for online tests. This is where live automatic proctoring helps.</p><p>With a webcam and microphone, this method monitors students while they take online tests. It uses computer tools to spot any cheating. 👀</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*MQwNBxi8wTD-Whdm.gif" /></figure><p>In <a href="https://dyte.io/blog/live-proctoring-system/">the previous blog</a>, we built a live proctoring system to detect whether there are multiple people in the candidate’s frame.</p><p>In this tutorial, we will build a live proctoring system using <a href="https://dyte.io/">Dyte</a> APIs that will allow an admin to monitor if the candidate who was supposed to be in the video is still the same candidate during the whole duration in real-time. ✨</p><h3>High-Level Design of the application</h3><p>We aim to notify the proctor if the candidate changes during the session. 🔍</p><p>The proctor would get the candidate’s details and a photograph from their webcam as proof in his meeting sidebar. 📸✨</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tl7gh53ZIrdpxXck.png" /></figure><ul><li>In this project, we will use React with <a href="https://dyte.io/blog/custom-ui-kit-sdk/">Dyte UI kit</a> and <a href="https://www.npmjs.com/package/@dytesdk/react-web-core">Dyte React Web Core</a> packages for the frontend.</li><li>For the backend, we will use <a href="https://fastapi.tiangolo.com/lo/">FastApi</a> (Python 3).</li><li>We will also use database as a service by <a href="https://www.elephantsql.com/">ElephantSQL</a> (PostgreSQL).</li><li>Lastly, we will use <a href="https://apidocs.imgur.com/">Imgur</a> for storing screenshots. And the image metadata will then be saved to the same database.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5S3ZIlliEoHKUD6P.png" /></figure><h3>Folder Structure</h3><p>After completing the tutorial, the folder structure will look like this. 👇</p><pre>dyte-proctoring/<br>├── frontend<br>│   ├── README.md<br>│   ├── package.json<br>│   ├── public<br>│   ├── src<br>│   │   ├── App.css<br>│   │   ├── App.jsx<br>│   │   ├── Heading.jsx<br>│   │   ├── ImageInput.jsx<br>│   │   ├── Meet.jsx<br>│   │   ├── Proctor.jsx<br>│   │   ├── index.css<br>│   │   ├── index.jsx<br>│   │   ├── logo.svg<br>│   │   ├── react-app-env.d.ts<br>│   │   ├── reportWebVitals.ts<br>│   │   ├── setupTests.ts<br>│   │   ├── stage.jsx<br>│   │   └── utils.js<br>│   ├── tsconfig.json<br>├── venv<br>├── app.py<br>├── imgur.py<br>└── requirements.txt</pre><h3>Step 0: Configurations and Setup</h3><p>🧑‍💻 Before building our live proctoring system, we must set up a Dyte account.</p><p>We can create a free account by clicking the “Start Building” button on Dyte.io and signing up using Google or GitHub 🎉.</p><p>Once signed up, we can access our Dyte API keys from the “API Keys” tab in the left sidebar. We will keep these keys secure as we will use them later.🔑🤫</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/0*o-LrFQHeTl0xK0pC.gif" /></figure><p>For our live proctoring system, we will use <a href="https://react.dev/">React</a> for the frontend and <a href="https://fastapi.tiangolo.com/lo/">FastAPI</a> for building the Backend and APIs.</p><p>We will begin by creating a new directory for our project, called dyte-proctoring and navigating into it using the following commands:</p><pre>mkdir dyte-proctoring<br>cd dyte-proctoring</pre><p>Please note:</p><p>We will also require accounts on the following platforms:</p><ul><li>Imgur: Create an account on Imgur and create an API key. Here is a <a href="https://apidocs.imgur.com/">step-by-step guide</a></li><li>ElephantSQL: Here is a <a href="https://www.elephantsql.com/docs/index.html">step-by-step guide</a> to creating a db on ElephantSQL.</li></ul><p>Now back to the tutorial.</p><h3>Step 1: Setting up the frontend</h3><p>Let’s start setting up our frontend project using React and Dyte! ✨</p><p>We will create a boilerplate React app using create-react-app. We can do this with the following command:</p><pre>yarn create react-app frontend</pre><p>This will initialize a new React app in the frontend directory. 📁</p><p>Then, we will go ahead and install the dyte react-web-core, dyte react-ui-kit and react-router packages in this project using the following command 👇</p><pre>yarn add @dytesdk/react-web-core @dytesdk/react-ui-kit react-router react-router-dom</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/0*DUTgSuv_DxKEsUWk.gif" /></figure><h3>Step 2: Setting up the backend</h3><p>Let’s get started with setting up our FastAPI backend now. 🙌</p><p>We will go back to the root directory of our project and initiate our project here itself for the ease of hosting:</p><pre>cd ..</pre><p>First of all, we will go ahead and create our requirements.txt file in the root directory itself with the following content 👇</p><p>requirements.txt</p><pre>cmake<br>fastapi<br>uvicorn<br>face_recognition<br>numpy<br>python-multipart<br>psycopg2-binary<br>httpx<br>python-dotenv<br>pydantic<br>requests</pre><p>After this, we will go ahead and create our virtual environment with venv and install the dependencies.</p><pre>python -m venv venv<br>source venv/bin/activate # for linux/mac<br>venv\Scripts\activate.bat # for windows<br>pip install -r requirements.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JrFVU2hELvVzlCbI.gif" /></figure><p>We will also create an environment variable file .env, for storing our credentials.</p><p>.env</p><pre>DYTE_ORG_ID=&lt;ID&gt;<br>DYTE_API_KEY=&lt;KEY&gt;<br>IMGUR_CLIENT_ID=&lt;ID&gt;<br>DB_USER=&lt;ID&gt;<br>DB_PASSWORD=&lt;PASSWORD&gt;<br>DB_HOST=&lt;HOST&gt;</pre><h3>Step 3: Setting up image upload with Imgur</h3><p>Let’s create a new file named imgur.py and add the following code. This will help us upload our screenshots to Imgur. 🌅</p><p>Here we are connecting the Imgur API using the CLIENT_ID that we can get from our Imgur API Dashboard and using it to upload the suspicious candidate’s image and get back the link to it. 🔗👇</p><pre>import base64<br>from fastapi import FastAPI, UploadFile, HTTPException<br>from httpx import AsyncClient<br>from dotenv import load_dotenv<br><br>load_dotenv()<br><br>app = FastAPI()<br>IMGUR_CLIENT_ID = os.getenv(&quot;IMGUR_CLIENT_ID&quot;)<br><br>async def upload_image(img_data):<br>    headers = {<br>        &quot;Authorization&quot;: f&quot;Client-ID {IMGUR_CLIENT_ID}&quot;<br>    }<br>    data = {<br>        &quot;image&quot;: img_data<br>    }<br><br>    async with AsyncClient() as client:<br>        response = await client.post(&quot;https://api.imgur.com/3/image&quot;, headers=headers, data=data)<br><br>    if response.status_code != 200:<br>        raise HTTPException(status_code=500, detail=&quot;Could not upload image.&quot;)<br><br>    print(response.json())<br>    return response.json()[&quot;data&quot;][&quot;link&quot;]</pre><h3>Step 4: Setting up our backend application</h3><p>Now, we will create a new file named app.py and add our 🐘 ElephantSQL PostgreSQL database connection and code for our APIs, including face detection logic.</p><p>In this file, we would need to create the following routes:</p><p>GET / - Root route</p><p>POST /is_admin/ - Check if the user is an admin</p><p>POST /suspicious_list/ - This route retrieves a list of participants who are detected as suspicious</p><p>POST /same_faces/ - Detect if the face in the screenshot is the same as the face in the reference image of the candidate.</p><p>POST /upload_photo/ - Upload a reference photo of the candidate</p><p>POST /meetings - Create a new meeting</p><p>POST /meetings/{meetingId}/participants - This route is responsible for adding a participant to a specific meeting identified by meetingId</p><p>So let’s get started 👇</p><p>app.py</p><pre>import base64<br>import io<br>import logging<br>import random<br>import requests<br><br>import uvicorn<br>from fastapi import FastAPI<br>from fastapi.middleware.cors import CORSMiddleware<br>from pydantic import BaseModel<br>from imgur import upload_image<br>import face_recognition<br>import psycopg2<br><br>import os<br>import base64<br>from fastapi import FastAPI, HTTPException<br>from pydantic import BaseModel<br>from dotenv import load_dotenv<br>from httpx import AsyncClient<br>import uuid<br><br>load_dotenv()<br><br>DYTE_API_KEY = os.getenv(&quot;DYTE_API_KEY&quot;)<br>DYTE_ORG_ID = os.getenv(&quot;DYTE_ORG_ID&quot;)<br><br>API_HASH = base64.b64encode(f&quot;{DYTE_ORG_ID}:{DYTE_API_KEY}&quot;.encode(&#39;utf-8&#39;)).decode(&#39;utf-8&#39;)<br><br>DYTE_API = AsyncClient(base_url=&#39;https://api.cluster.dyte.in/v2&#39;, headers={&#39;Authorization&#39;: f&quot;Basic {API_HASH}&quot;})<br><br>logger = logging.getLogger(__name__)<br>logging.basicConfig(level=logging.INFO)<br><br>fh = logging.FileHandler(&quot;app.log&quot;)<br>fh.setLevel(logging.DEBUG)<br>formatter = logging.Formatter(&quot;%(asctime)s - %(name)s - %(levelname)s - %(message)s&quot;)<br>fh.setFormatter(formatter)<br>logger.addHandler(fh)<br><br><br>class ParticipantScreen(BaseModel):<br>    base64_img: str<br>    participant_id: str<br>    meeting_id: str<br>    ref_image_url: str<br>    participant_name: str<br><br>class ProctorPayload(BaseModel):<br>    meeting_id: str<br>    admin_id: str<br><br>class AdminProp(BaseModel):<br>    meeting_id: str<br>    admin_id: str<br><br>class Meeting(BaseModel):<br>    title: str<br><br>class Participant(BaseModel):<br>    name: str<br>    preset_name: str<br>    meeting_id: str<br><br>origins = [<br>    # allow all<br>    &quot;*&quot;,<br>]<br><br>app = FastAPI()<br><br># enable cors<br>app.add_middleware(<br>    CORSMiddleware,<br>    allow_origins=origins,<br>    allow_credentials=True,<br>    allow_methods=[&quot;*&quot;],  # allow all<br>    allow_headers=[&quot;*&quot;],  # allow all<br>)<br><br>def connect_to_db():<br>    conn = psycopg2.connect(<br>            dbname=os.getenv(&#39;DB_USER&#39;),<br>            user=os.getenv(&#39;DB_USER&#39;),<br>            password=os.getenv(&#39;DB_PASSWORD&#39;),<br>            host=os.getenv(&#39;DB_HOST&#39;),<br>            port=5432<br>    )<br>    return conn<br><br>@app.get(&quot;/&quot;)<br>async def root():<br>    return {&quot;message&quot;: &quot;Hello World&quot;}<br><br>@app.post(&quot;/is_admin/&quot;)<br>async def is_admin(admin: AdminProp):<br>    conn = connect_to_db()<br>    cur = conn.cursor()<br>    cur.execute(&quot;SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s&quot;, (admin.meeting_id, admin.admin_id,))<br><br>    count = cur.fetchone()[0]<br><br>    if(count &gt; 0):<br>        return { &quot;admin&quot;: True }<br>    else:<br>        return { &quot;admin&quot;: False }<br><br>@app.post(&quot;/suspicious_list/&quot;)<br>async def suspicious_list(meeting: ProctorPayload):<br>    conn = connect_to_db()<br>    cur = conn.cursor()<br>    cur.execute(&quot;SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s&quot;, (meeting.meeting_id, meeting.admin_id,))<br><br>    count = cur.fetchone()[0]<br><br>    if(count &gt; 0):<br>        cur.execute(&quot;SELECT * FROM meeting_proc_details WHERE meeting_id = %s ORDER BY ts DESC&quot;, (meeting.meeting_id,))<br>        rows = cur.fetchall()<br>        conn.commit()<br>        cur.close()<br>        conn.close()<br>        return rows<br>    else:<br>        conn.commit()<br>        cur.close()<br>        conn.close()<br>        raise HTTPException(status_code=401, detail=&quot;Participant dose not has admin role&quot;)<br><br>@app.post(&quot;/same_faces/&quot;)<br>async def same_faces(participant: ParticipantScreen):<br>    conn = connect_to_db()<br>    cur = conn.cursor()<br><br>    cur.execute(&quot;CREATE TABLE IF NOT EXISTS meeting_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), img_url VARCHAR(255), verdict VARCHAR(255))&quot;)<br><br>    img_url = participant.ref_image_url<br><br>    response = requests.get(img_url)<br>    ref_img_data = io.BytesIO(response.content)<br><br>    img_data = participant.base64_img.split(&quot;,&quot;)[1]<br>    img_data_dc = base64.b64decode(participant.base64_img.split(&quot;,&quot;)[1])<br><br>    file_obj = io.BytesIO(img_data_dc)<br>    unknown_image = face_recognition.load_image_file(file_obj)<br>    known_image = face_recognition.load_image_file(ref_img_data)<br><br>    biden_encoding = face_recognition.face_encodings(known_image)[0]<br>    unknown_encoding = face_recognition.face_encodings(unknown_image)[0]<br><br>    results = face_recognition.compare_faces([biden_encoding], unknown_encoding)<br>    is_same_face = True<br>    for res in results:<br>        is_same_face = is_same_face and res<br><br><br>    if not is_same_face:<br>        logger.info(<br>            f&quot;Detected different faces for participant {participant.participant_id}&quot;<br>        )<br><br>        verdict = f&quot;Participant Name: {participant.participant_name} &lt;&gt; Anomaly: Different Faces Detected &lt;&gt; Participant ID: {participant.participant_id}&quot;<br>        cur.execute(&quot;SELECT count(1) FROM meeting_proc_details WHERE meeting_id=%s AND participant_id=%s AND ts &gt;= (current_timestamp - INTERVAL &#39;10 minutes&#39;)&quot;, (participant.meeting_id, participant.participant_id))<br>        count = cur.fetchone()[0]<br><br>        if count == 0:<br>            upload_resp = await upload_image(img_data)<br>            cur.execute(&quot;INSERT INTO meeting_proc_details (ts, meeting_id, participant_id, img_url, verdict) VALUES (current_timestamp, %s, %s, %s, %s)&quot;,<br>                (participant.meeting_id, participant.participant_id, upload_resp, verdict)<br>            )<br><br>        conn.commit()<br>        cur.close()<br>        conn.close()<br><br>        if count == 0:<br>            return { &quot;id&quot;: participant.participant_id, &quot;different_faces_detected&quot;: True, &quot;url&quot;: upload_resp }<br>        return { &quot;id&quot;: participant.participant_id, &quot;different_faces_detected&quot;: True, &quot;url&quot;: &quot;not uploaded&quot; }<br><br>    return {&quot;id&quot;: participant.participant_id, &quot;different_faces_detected&quot;: False}<br><br>@app.post(&quot;/upload_photo/&quot;)<br>async def upload_photo(participant: ParticipantScreen):<br>    img_data = participant.base64_img.split(&quot;,&quot;)[1]<br><br>    conn = connect_to_db()<br>    cur = conn.cursor()<br><br>    cur.execute(&quot;CREATE TABLE IF NOT EXISTS meeting_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), img_url VARCHAR(255), verdict VARCHAR(255))&quot;)<br><br>    verdict = f&quot;REF_IMAGE&quot;<br><br>    upload_resp = await upload_image(img_data)<br>    cur.execute(&quot;INSERT INTO meeting_proc_details (ts, meeting_id, participant_id, img_url, verdict) VALUES (current_timestamp, %s, %s, %s, %s)&quot;,<br>        (participant.meeting_id, participant.participant_id, upload_resp, verdict)<br>    )<br><br>    conn.commit()<br>    cur.close()<br>    conn.close()<br><br>    return {&quot;id&quot;: participant.participant_id, &quot;img_url&quot;: upload_resp, &quot;reference_image_uploaded&quot;: True}<br><br>@app.post(&quot;/meetings&quot;)<br>async def create_meeting(meeting: Meeting):<br>    response = await DYTE_API.post(&#39;/meetings&#39;, json=meeting.dict())<br>    if response.status_code &gt;= 300:<br>        raise HTTPException(status_code=response.status_code, detail=response.text)<br>    admin_id = &#39;&#39;.join(random.choices(&#39;abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789&#39;, k=32))<br>    resp_json = response.json()<br>    resp_json[&#39;admin_id&#39;] = admin_id<br>    meeting_id = resp_json[&#39;data&#39;][&#39;id&#39;]<br><br>    conn = connect_to_db()<br>    cur = conn.cursor()<br>    cur.execute(&quot;INSERT INTO meeting_host_info (ts, meeting_id, admin_id) VALUES (CURRENT_TIMESTAMP, %s, %s)&quot;, (meeting_id, admin_id))<br>    conn.commit()<br>    cur.close()<br>    conn.close()<br><br>    return resp_json<br><br><br>@app.post(&quot;/meetings/{meetingId}/participants&quot;)<br>async def add_participant(meetingId: str, participant: Participant):<br>    client_specific_id = f&quot;react-samples::{participant.name.replace(&#39; &#39;, &#39;-&#39;)}-{str(uuid.uuid4())[0:7]}&quot;<br>    payload = participant.dict()<br>    payload.update({&quot;client_specific_id&quot;: client_specific_id})<br>    del payload[&#39;meeting_id&#39;]<br>    resp = await DYTE_API.post(f&#39;/meetings/{meetingId}/participants&#39;, json=payload)<br>    if resp.status_code &gt; 200:<br>        raise HTTPException(status_code=resp.status_code, detail=resp.text)<br>    return resp.text<br><br>if __name__ == &quot;__main__&quot;:<br>    uvicorn.run(&quot;app:app&quot;, host=&quot;localhost&quot;, port=8000, log_level=&quot;debug&quot;, reload=True)</pre><p>This code defines a ⚡️ FastAPI application with an endpoint /detect_faces, which takes in a base64 encoded image and returns a boolean value indicating if the face matches the reference image of the candidate.</p><p>It uses the <a href="https://github.com/ageitgey/face_recognition">face recognition</a> library to detect faces in the image received.</p><p>The code also makes use of the Dyte API for meeting-related operations. 📹</p><p>We can start the backend server simply by using the following command 🧑‍💻:</p><pre>python app.py</pre><p>This Python server helps us create and join meetings, detect different faces, and get the list of suspicious candidates. 🕵️</p><p>Here, when we hit the /same_faces endpoint with an image file encoded as a base64 string, the different_faces_detected key of the response would be set to True if the face of the candidate from the reference image doesn&#39;t match the face in the screenshot, else it will be set to False.</p><p>We can call this from our frontend with the participant’s webcam feed.</p><p>With this sorted, let’s return to our React application and create our UI. ✨</p><h3>Step 5: Setting up the Meeting UI</h3><p>First, let us add our CSS file, create a new file, frontend/src/App.css and paste the following code.</p><pre>.App {<br> text-align: center;<br>}<br><br>.App-logo {<br> height: 40vmin;<br> pointer-events: none;<br>}<br><br>@media (prefers-reduced-motion: no-preference) {<br> .App-logo {<br>  animation: App-logo-spin infinite 20s linear;<br> }<br>}<br><br>.App-header {<br> background-color: #282c34;<br> min-height: 100vh;<br> display: flex;<br> flex-direction: column;<br> align-items: center;<br> justify-content: center;<br> font-size: calc(10px + 2vmin);<br> color: white;<br>}<br><br>.App-link {<br> color: #61dafb;<br>}<br><br>.heading-proctor {<br> font-size: x-large;<br> font-weight: bolder;<br> color: #fff;<br>}<br><br>@keyframes App-logo-spin {<br> from {<br>  transform: rotate(0deg);<br> }<br> to {<br>  transform: rotate(360deg);<br> }<br>}<br><br>.file-input label {<br> margin-top: 20px;<br> display: block;<br> position: relative;<br> width: 200px;<br> height: 50px;<br> border-radius: 10px;<br> background-color: #2160fd;<br> display: flex;<br> align-items: center;<br> justify-content: center;<br> flex-direction: row;<br> color: #fff;<br> font-weight: bold;<br> cursor: pointer;<br> transition: transform 0.2s ease-out;<br>}<br><br>.file {<br> opacity: 0;<br> width: 0.1px;<br> height: 0.1px;<br> position: absolute;<br>}</pre><p>Next, we will add the initial Dyte Meeting component to our app. We can do this by replacing the contents of frontend/src/App.jsx with the following code:</p><pre>import { useEffect, useState } from &quot;react&quot;;<br>import Home from &quot;./Home&quot;;<br>import { BrowserRouter, Routes, Route, Link } from &quot;react-router-dom&quot;;<br>import &quot;./App.css&quot;;<br>import Stage from &quot;./Stage&quot;;<br><br>const SERVER_URL = process.env.SERVER_URL || &quot;http://localhost:8000&quot;;<br><br>function App() {<br> const [meetingId, setMeetingId] = useState();<br><br> const createMeeting = async () =&gt; {<br>  const res = await fetch(`${SERVER_URL}/meetings`, {<br>   method: &quot;POST&quot;,<br>   body: JSON.stringify({ title: &quot;Joint Entrance Examination&quot; }),<br>   headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br>  });<br>  const resJson = await res.json();<br>  window.localStorage.setItem(&quot;adminId&quot;, resJson.admin_id);<br>  setMeetingId(resJson.data.id);<br> };<br><br> useEffect(() =&gt; {<br>  window.localStorage.removeItem(&quot;refImgUrl&quot;);<br>  const id = window.location.pathname.split(&quot;/&quot;)[2];<br>  if (!!!id) {<br>   createMeeting();<br>  }<br> }, []);<br><br> return (<br>  &lt;BrowserRouter&gt;<br>   &lt;Routes&gt;<br>    &lt;Route path=&quot;/&quot; element={&lt;Home meetingId={meetingId} /&gt;}&gt;&lt;/Route&gt;<br>    &lt;Route path=&quot;/meeting/:meetingId&quot; element={&lt;Stage /&gt;}&gt;&lt;/Route&gt;<br>   &lt;/Routes&gt;<br>  &lt;/BrowserRouter&gt;<br> );<br>}<br><br>export default App;</pre><p>This component will create a Dyte meeting link and an adminId for the admin. We will store the adminId secretly in localstorage. The adminId will be used later for accessing any sensitive data.</p><h3>Home component</h3><p>The home component renders the / route. Create a file as frontend/src/Home.jsx.</p><pre>import { Link } from &quot;react-router-dom&quot;;<br>function Home({ meetingId }) {<br>	return (<br>		&lt;div<br>			style={{<br>				height: &quot;100vh&quot;,<br>				width: &quot;100vw&quot;,<br>				fontSize: &quot;x-large&quot;,<br>				display: &quot;flex&quot;,<br>				justifyContent: &quot;center&quot;,<br>				alignItems: &quot;center&quot;,<br>			}}<br>		&gt;<br>			{meetingId &amp;&amp; !window.location.pathname.split(&quot;/&quot;)[2] &amp;&amp; (<br>				&lt;Link to={`/meeting/${meetingId}`}&gt;Create and Join Meeting&lt;/Link&gt;<br>			)}<br>		&lt;/div&gt;<br>	);<br>}<br><br>export default Home;</pre><h3>Heading component</h3><p>Now we will create a file as frontend/src/Heading.jsx.</p><pre>const Heading = ({ text }) =&gt; {<br>	return (<br>		&lt;div<br>			className=&quot;heading-proctor&quot;<br>			style={{<br>				padding: &quot;10px&quot;,<br>				textAlign: &quot;center&quot;,<br>				backgroundColor: &quot;#000&quot;,<br>				borderBottom: &quot;solid 0.5px gray&quot;,<br>			}}<br>		&gt;<br>			{text}<br>		&lt;/div&gt;<br>	);<br>};<br><br>export default Heading;</pre><p>Let us now build the component to upload a reference image of the candidate.<br>Create a file frontend/src/ImageInput.jsx</p><pre>import { useState } from &quot;react&quot;;<br><br>function ImageUploader({ setImgUploadedStatus }) {<br> const [selectedFile, setSelectedFile] = useState(null);<br> const [uploading, setUploading] = useState(false);<br> const [uploadError, setUploadError] = useState(null);<br> const [uploadSuccess, setUploadSuccess] = useState(false);<br><br> const handleFileChange = (event) =&gt; {<br>  setSelectedFile(event.target.files[0]);<br>  setUploadSuccess(false);<br>  setUploadError(null);<br> };<br><br> const handleSubmit = async (event) =&gt; {<br>  event.preventDefault();<br>  setUploading(true);<br>  setUploadError(null);<br><br>  try {<br>   const formData = new FormData();<br>   formData.append(&quot;image&quot;, selectedFile);<br><br>   const response = await fetch(&quot;https://api.imgur.com/3/image&quot;, {<br>    method: &quot;POST&quot;,<br>    headers: {<br>     Authorization: &quot;Client-ID 48f0caef7256b40&quot;,<br>    },<br>    body: formData,<br>   });<br><br>   if (!response.ok) {<br>    throw new Error(&quot;Failed to upload image&quot;);<br>   }<br><br>   const data = await response.json();<br>   window.localStorage.setItem(&quot;refImgUrl&quot;, data.data.link);<br>   setUploadSuccess(data.data.link);<br>   setImgUploadedStatus(true);<br>  } catch (error) {<br>   setUploadError(error.message);<br>  } finally {<br>   setUploading(false);<br>  }<br> };<br><br> return (<br>  &lt;div<br>   style={{<br>    display: &quot;flex&quot;,<br>    justifyContent: &quot;center&quot;,<br>    flexDirection: &quot;column&quot;,<br>    alignItems: &quot;center&quot;,<br>    fontSize: &quot;x-large&quot;,<br>   }}<br>  &gt;<br>   &lt;div&gt;Hey Candidate 👋&lt;/div&gt;<br>   &lt;div style={{ marginBottom: &quot;30px&quot; }}&gt;<br>    Please upload a image of yours to proceed<br>   &lt;/div&gt;<br>   &lt;input<br>    type=&quot;file&quot;<br>    id=&quot;file&quot;<br>    style={{ border: &quot;0.5px white solid&quot; }}<br>    onChange={handleFileChange}<br>   /&gt;<br>   &lt;form<br>    onSubmit={handleSubmit}<br>    style={{<br>     display: &quot;flex&quot;,<br>     flexDirection: &quot;column&quot;,<br>     alignItems: &quot;center&quot;,<br>     marginTop: &quot;1rem&quot;,<br>    }}<br>   &gt;<br>    &lt;button<br>     type=&quot;submit&quot;<br>     disabled={!selectedFile || uploading || uploadSuccess}<br>     style={{<br>      marginTop: &quot;10px&quot;,<br>      padding: &quot;0.5rem 1rem&quot;,<br>      backgroundColor: &quot;#3460f3&quot;,<br>      color: &quot;#fff&quot;,<br>      border: &quot;none&quot;,<br>      borderRadius: &quot;0.25rem&quot;,<br>      cursor: &quot;pointer&quot;,<br>     }}<br>    &gt;<br>     {uploading ? &quot;Uploading...&quot; : &quot;Upload&quot;}<br>    &lt;/button&gt;<br>   &lt;/form&gt;<br>   {uploadError &amp;&amp; (<br>    &lt;div style={{ marginTop: &quot;10px&quot;, color: &quot;red&quot; }}&gt;{uploadError}&lt;/div&gt;<br>   )}<br>  &lt;/div&gt;<br> );<br>}<br><br>export default ImageUploader;</pre><p>It will look like this 👇</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vlriKpiqIVK7c5k3.png" /></figure><p>Let’s create a staging area for participants joining the meeting. Admin bypasses the staging area, but the candidate will be asked to upload a reference image of himself on this page.</p><p>For this, we will create another file, frontend/src/Stage.jsx</p><pre>import { useState, useEffect } from &quot;react&quot;;<br>import ImageUploader from &quot;./ImageInput&quot;;<br>import Meet from &quot;./Meet&quot;;<br><br>const SERVER_URL = process.env.REACT_APP_SERVER_URL || &quot;http://localhost:8000&quot;;<br><br>const Stage = () =&gt; {<br> const [isAdminBool, setAdminBool] = useState(null);<br> const [isRefImgUploaded, setImgUploadedStatus] = useState(null);<br> const meetingId = window.location.pathname.split(&quot;/&quot;)[2];<br><br> const isAdmin = async (id) =&gt; {<br>  const res = await fetch(`${SERVER_URL}/is_admin`, {<br>   method: &quot;POST&quot;,<br>   body: JSON.stringify({<br>    admin_id: window.localStorage.getItem(&quot;adminId&quot;) || &quot;&quot;,<br>    meeting_id: meetingId || &quot;&quot;,<br>   }),<br>   headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br>  });<br>  const resJson = await res.json();<br>  setAdminBool(resJson.admin);<br> };<br><br> useEffect(() =&gt; {<br>  isAdmin();<br> }, []);<br><br> return (<br>  &lt;div<br>   style={{<br>    height: &quot;100vh&quot;,<br>    width: &quot;100vw&quot;,<br>    display: &quot;flex&quot;,<br>    justifyContent: &quot;center&quot;,<br>    alignItems: &quot;center&quot;,<br>    color: &quot;white&quot;,<br>   }}<br>  &gt;<br>   {isAdminBool == null ? (<br>    &lt;&gt;Loading...&lt;/&gt;<br>   ) : (<br>    &lt;&gt;<br>     {isAdminBool || (isAdminBool === false &amp;&amp; isRefImgUploaded) ? (<br>      &lt;Meet isAdminBool={isAdminBool} /&gt;<br>     ) : (<br>      &lt;ImageUploader setImgUploadedStatus={setImgUploadedStatus} /&gt;<br>     )}<br>    &lt;/&gt;<br>   )}<br>  &lt;/div&gt;<br> );<br>};<br><br>export default Stage;</pre><p>Now, let’s delve into the Meet component that renders on route /meeting/:meetingId.</p><p>When the admin clicks on the link provided on the / route, he gets redirected to the meeting page, where we add the user to the meeting as a participant with group_call_host preset. 🤙</p><p>Since this user created the meeting and was redirected to the meet page, we will assign him the admin role. Now the link from the address bar can be shared with the candidates.</p><p>When a candidate opens the shared link, they become a regular user. And for every regular user, the component emits screenshots of the users’ videos to our directed to our Python server. 🐍</p><pre>/* eslint-disable */<br><br>import { useState, useEffect, useRef } from &quot;react&quot;;<br>import { DyteMeeting, provideDyteDesignSystem } from &quot;@dytesdk/react-ui-kit&quot;;<br>import { useDyteClient } from &quot;@dytesdk/react-web-core&quot;;<br>import Proctor from &quot;./Proctor&quot;;<br>import Heading from &quot;./Heading&quot;;<br>import { SendImageToBackendMiddleware, joinMeeting } from &quot;./utils&quot;;<br>// Constants<br>const SERVER_URL = process.env.SERVER_URL || &quot;http://localhost:8000&quot;;<br>let LAST_BACKEND_PING_TIME = 0;<br>const DETECT_FACES_ENDPOINT = `${SERVER_URL}/same_faces`;<br>const TIME_BETWEEN_BACKEND_PINGS = 10000;<br><br>const Meet = () =&gt; {<br> const meetingEl = useRef();<br> const [meeting, initMeeting] = useDyteClient();<br> const [userToken, setUserToken] = useState();<br> const [isAdminBool, setAdminBool] = useState(null);<br> const meetingId = window.location.pathname.split(&quot;/&quot;)[2];<br><br> function SendImageToBackendMiddleware() {<br>  return async (canvas, ctx) =&gt; {<br>   const currentTime = Date.now();<br>   if (currentTime - LAST_BACKEND_PING_TIME &gt; TIME_BETWEEN_BACKEND_PINGS) {<br>    LAST_BACKEND_PING_TIME = currentTime;<br>    const imgBase64String = canvas.toDataURL(&quot;image/png&quot;);<br>    const response = await fetch(DETECT_FACES_ENDPOINT, {<br>     method: &quot;POST&quot;,<br>     headers: {<br>      &quot;Content-Type&quot;: &quot;application/json&quot;,<br>     },<br>     body: JSON.stringify({<br>      base64_img: imgBase64String,<br>      participant_id: meeting?.self.id,<br>      participant_name: meeting?.self.name,<br>      ref_image_url: window.localStorage.getItem(&quot;refImgUrl&quot;),<br>      meeting_id: meetingId,<br>     }),<br>    });<br>   }<br>  };<br> }<br><br> const isAdmin = async (id) =&gt; {<br>  const res = await fetch(`${SERVER_URL}/is_admin`, {<br>   method: &quot;POST&quot;,<br>   body: JSON.stringify({<br>    admin_id: window.localStorage.getItem(&quot;adminId&quot;) || &quot;&quot;,<br>    meeting_id: meetingId || &quot;&quot;,<br>   }),<br>   headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br>  });<br>  const resJson = await res.json();<br>  setAdminBool(resJson.admin);<br> };<br><br> const joinMeetingId = async () =&gt; {<br>  if (meetingId) {<br>   const authToken = await joinMeeting(meetingId);<br>   await initMeeting({<br>    authToken,<br>   });<br>   setUserToken(authToken);<br>  }<br> };<br><br> useEffect(() =&gt; {<br>  if (meetingId &amp;&amp; !userToken) joinMeetingId();<br>  isAdmin();<br> }, []);<br><br> useEffect(() =&gt; {<br>  if (userToken) {<br>   provideDyteDesignSystem(meetingEl.current, {<br>    theme: &quot;dark&quot;,<br>   });<br>  }<br> }, [userToken]);<br><br> useEffect(() =&gt; {<br>  if (isAdminBool === false &amp;&amp; meeting?.self) {<br>   meeting.self.addVideoMiddleware(SendImageToBackendMiddleware);<br>  }<br> }, [meeting?.self]);<br><br> return (<br>  &lt;div style={{ height: &quot;96vh&quot;, width: &quot;100vw&quot;, display: &quot;flex&quot; }}&gt;<br>   {userToken &amp;&amp; (<br>    &lt;&gt;<br>     {isAdminBool &amp;&amp; (<br>      &lt;div<br>       style={{<br>        width: &quot;40vw&quot;,<br>        height: &quot;100vh&quot;,<br>        overflowY: &quot;scroll&quot;,<br>        backgroundColor: &quot;black&quot;,<br>        borderRight: &quot;solid 0.5px gray&quot;,<br>       }}<br>      &gt;<br>       &lt;Heading text={&quot;Proctoring Information&quot;} /&gt;<br>       &lt;Proctor meeting={meeting} /&gt;<br>      &lt;/div&gt;<br>     )}<br>     {isAdminBool ? (<br>      &lt;div style={{ width: &quot;60vw&quot;, height: &quot;96vh&quot; }}&gt;<br>       &lt;Heading text={&quot;Proctoring Admin Interface&quot;} /&gt;<br>       &lt;DyteMeeting mode=&quot;fill&quot; meeting={meeting} ref={meetingEl} /&gt;<br>      &lt;/div&gt;<br>     ) : (<br>      &lt;div style={{ width: &quot;100vw&quot;, height: &quot;96vh&quot; }}&gt;<br>       &lt;Heading text={&quot;Proctoring Candidate Interface&quot;} /&gt;<br>       &lt;DyteMeeting mode=&quot;fill&quot; meeting={meeting} ref={meetingEl} /&gt;<br>      &lt;/div&gt;<br>     )}<br>    &lt;/&gt;<br>   )}<br>  &lt;/div&gt;<br> );<br>};<br><br>export default Meet;</pre><p>Let’s briefly go through some of the functions:</p><ul><li>isAdmin talks to the Python server to identify whether the current client is the admin.</li><li>joinMeeting adds the current client to the meeting.</li><li>SendImageToBackendMiddleware sends screenshots of candidates&#39; videos to the Python server.</li></ul><h3>Proctor component</h3><p>The proctor component gets activated only for admins. The proctor component, with the help of adminId fetches the list of suspicious candidates and renders it in a chat-like format. Create a file frontend/src/Proctor.jsx</p><pre>import { useEffect, useState } from &quot;react&quot;;<br>import { getCandidateStatus } from &quot;./utils&quot;;<br><br>const Proctor = () =&gt; {<br> const [candidateStatuses, updateCandidateStatusState] = useState([]);<br> const [error, setError] = useState(&quot;&quot;);<br><br> const updateCandidateStatus = async () =&gt; {<br>  try {<br>   const res = await getCandidateStatus();<br>   updateCandidateStatusState(res);<br>  } catch (e) {<br>   setError(&quot;User don&#39;t have admin privileges.&quot;);<br>  }<br> };<br><br> useEffect(() =&gt; {<br>  updateCandidateStatus();<br> }, []);<br><br> useEffect(() =&gt; {<br>  if (candidateStatuses?.map) {<br>   const id = setInterval(() =&gt; {<br>    updateCandidateStatus();<br>   }, 30000);<br>   return () =&gt; {<br>    clearInterval(id);<br>   };<br>  }<br> }, [candidateStatuses]);<br><br> return (<br>  &lt;&gt;<br>   &lt;div style={{ padding: &quot;0px 20px&quot; }}&gt;<br>    {candidateStatuses?.map &amp;&amp; candidateStatuses ? (<br>     candidateStatuses.map((status) =&gt; (<br>      &lt;div<br>       style={{<br>        display: &quot;flex&quot;,<br>        justifyContent: &quot;start&quot;,<br>        margin: &quot;50px 20px&quot;,<br>       }}<br>      &gt;<br>       &lt;div style={{ marginRight: &quot;20px&quot; }}&gt;<br>        &lt;img<br>         src=&quot;https://images.yourstory.com/cs/images/companies/Dyte-1608553297314.jpg&quot;<br>         style={{<br>          borderRadius: &quot;50px&quot;,<br>          height: &quot;60px&quot;,<br>          border: &quot;1px double lightblue&quot;,<br>         }}<br>        /&gt;<br>       &lt;/div&gt;<br>       &lt;div<br>        style={{<br>         textAlign: &quot;center&quot;,<br>         padding: &quot;20px&quot;,<br>         backgroundColor: &quot;#2160fd&quot;,<br>         fontSize: &quot;large&quot;,<br>         fontWeight: &quot;400&quot;,<br>         borderRadius: &quot;10px 10px 10px 10px&quot;,<br>         width: &quot;80%&quot;,<br>        }}<br>       &gt;<br>        &lt;div<br>         style={{<br>          color: &quot;white&quot;,<br>          padding: &quot;20px 0px&quot;,<br>          textAlign: &quot;left&quot;,<br>         }}<br>        &gt;<br>         {status[4].split(&quot;&lt;&gt;&quot;).map((text) =&gt; (<br>          &lt;div&gt;{text}&lt;/div&gt;<br>         ))}<br>         &lt;div&gt;Timestamp: {new Date(status[0]).toLocaleString()}&lt;/div&gt;<br>        &lt;/div&gt;<br>        &lt;img<br>         style={{ borderRadius: &quot;10px&quot;, width: &quot;100%&quot; }}<br>         src={status[3]}<br>        /&gt;<br>       &lt;/div&gt;<br>      &lt;/div&gt;<br>     ))<br>    ) : (<br>     &lt;div style={{ color: &quot;white&quot; }}&gt;<br>      Wait or check if you have admin privileges to access the proctoring<br>      dashboard.<br>     &lt;/div&gt;<br>    )}<br>   &lt;/div&gt;<br>  &lt;/&gt;<br> );<br>};<br><br>export default Proctor;</pre><h3>Utility functions</h3><p>Now we will add utility functions.</p><p>Create a file frontend/src/utils.js.</p><pre>const joinMeeting = async (id) =&gt; {<br> const res = await fetch(`http://localhost:8000/meetings/${id}/participants`, {<br>  method: &quot;POST&quot;,<br>  body: JSON.stringify({<br>   name: &quot;new user&quot;,<br>   preset_name: &quot;group_call_host&quot;,<br>   meeting_id: meetingId,<br>  }),<br>  headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br> });<br> const resJson = await res.json();<br> console.log(resJson.detail);<br> const data = JSON.parse(resJson.detail);<br> return data.data.token;<br>};<br><br>const getCandidateStatus = async () =&gt; {<br> const response = await fetch(&quot;http://localhost:8000/suspicious_list&quot;, {<br>  method: &quot;POST&quot;,<br>  headers: {<br>   &quot;Content-Type&quot;: &quot;application/json&quot;,<br>  },<br>  body: JSON.stringify({<br>   meeting_id: window.location.pathname.split(&quot;/&quot;)[2],<br>   admin_id: window.localStorage.getItem(&quot;adminId&quot;) || &quot;undefined&quot;,<br>  }),<br> });<br> const res = await response.json();<br> if (res.details) return undefined;<br> console.log(res);<br> return res;<br>};<br><br>export { joinMeeting, getCandidateStatus };</pre><p>To start the React app on the local server, we can run the following command:</p><pre>yarn start</pre><p>Now, upon visiting <a href="http://localhost:3000/">http://localhost:3000/</a>, we should be able to see the Dyte meeting in our browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IDsgSoUwpbwCZWnM.png" /></figure><h3>Step 6: Adding the face detection logic to the frontend</h3><p>Since now we have a nice backend server to detect faces and a great UI 🎉, we can add the face detection logic to our frontend. For this, we will first add some constants to our previously edited frontend/src/App.jsx file:</p><p>We will be using the above constants in the SendImageToBackendMiddleware function, which we will add to our App component just after the useDyteClient hook. 🪝</p><p>The SendImageToBackendMiddleware is a <a href="https://dyte.io/blog/streams-blog/">Dyte Video</a> <a href="https://docs.dyte.io/web-core/local-user/extras#using-middlewares">Middleware</a>. Middlewares are add-ons that we can use to easily add effects and filters to your audio and video streams.</p><p>Here, we use the middleware functionality to get the canvas object of the participant’s webcam feed, convert it to a base64 encoded image, and send it to our backend server.</p><p>That was all the code we needed to add basic proctoring functionality to our Dyte meeting. 👍</p><p>The app sends a screenshot of the participant’s webcam feed to the backend server every 30 seconds. If the backend detects the face isn’t similar to the reference image, it sends a warning notification to the proctor. ⚠️</p><p>The backend also logs the participant’s ID and the time of the detection in the terminal. This can be used to keep track of the participants who may have cheated during the meeting for later review.</p><h3>Step 7: Trying out our live proctoring system</h3><p>Ta-da! 🎩✨ It’s time to put our live proctoring system to the test and see it in action!</p><ul><li>First, let’s look at the candidate’s view after uploading the reference image. The candidate can see that the proctor is in the meeting but cannot see the Proctoring Panel. 🧑‍💻</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i5Q35qiaWdyOTALD.png" /></figure><ul><li>Now, let’s swap the candidates.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zeWyN-WLRKOPkPWL.png" /></figure><ul><li>In the proctor’s view, we can see the details (proctoring information) along with proof when the candidate got swapped in the candidate’s webcam view. 🙌</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IQGWGARa_SDBtXpA.png" /></figure><p>🧑‍💻 You can try out the live proctoring system <a href="https://dyte-proc.netlify.app/">here</a>. And here’s <a href="https://github.com/dyte-io/proctored-interviews-detecting-participant-switching">the link</a> to the repository for you to take a look at the whole codebase.</p><h3>Conclusion</h3><p>Celebrate! 🎉✨ We’ve built a powerful live proctoring system with Dyte, ensuring integrity and fairness in online exams and interviews. But that’s not all! We can now create our own customized online classroom or meeting platform.</p><p>We can now use this live proctoring system to proctor our online exams and interviews. ✍️</p><p>The possibilities are endless with Dyte, go ahead and try bringing your ideas to life by visiting <a href="https://dyte.io/">dyte.io</a>! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7173cce6969c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dyte/building-a-live-proctoring-system-to-detect-participant-switching-7173cce6969c">Building a Live Proctoring System to Detect Participant Switching</a> was originally published in <a href="https://medium.com/dyte">Dyte</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Live Proctoring System to Detect Multiple Speakers]]></title>
            <link>https://medium.com/dyte/building-a-live-proctoring-system-to-detect-multiple-speakers-d0ee0092a92f?source=rss----9477120405bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/d0ee0092a92f</guid>
            <category><![CDATA[video-sdk]]></category>
            <category><![CDATA[proctoring-software]]></category>
            <category><![CDATA[video-api]]></category>
            <dc:creator><![CDATA[Ravindra Rathor]]></dc:creator>
            <pubDate>Tue, 02 Jan 2024 11:46:16 GMT</pubDate>
            <atom:updated>2024-01-02T11:46:16.400Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iU8h9cXwP4otnlYIlmU5lQ.png" /></figure><h3>Introduction</h3><p>In the early days of online education, as the need arose to maintain the integrity of exams, proctoring emerged as a trusted monitoring method to prevent students from engaging in dishonest behavior. As the digital landscape expanded and online courses became more prevalent, the traditional ways of proctoring started to show their limitations.</p><p>Consider online exams, where having a physical proctor for every student becomes impractical. This gave rise to the adoption of live automatic proctoring.</p><p>This modern approach harnesses the power of a student’s webcam and microphone to oversee the exam process. By incorporating computer vision and machine learning, it efficiently detects cheating attempts.</p><p>In this tutorial, we’ll create a live proctoring system using <a href="https://dyte.io/">Dyte</a> APIs, enabling an admin to detect the presence of multiple speakers in a candidate’s background throughout the exam in real-time. ✨</p><h3>High-Level Design of the application</h3><p>Our aim is to notify the proctor if we hear multiple voices from the candidate’s background.</p><p>The proctor would get the candidate’s details right in his meeting sidebar. 🔊✨</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cghr05EXGO6eSms5.png" /></figure><ul><li>In this project, we will use React with <a href="https://dyte.io/blog/custom-ui-kit-sdk/">Dyte UI kit</a> and <a href="https://www.npmjs.com/package/@dytesdk/react-web-core">Dyte React Web Core</a> packages for the frontend.</li><li>For the backend, we will use <a href="https://fastapi.tiangolo.com/lo/">FastApi</a> (Python 3).</li><li>We will also use database as a service by <a href="https://www.elephantsql.com/">ElephantSQL</a> (PostgreSQL).</li><li>Lastly, we will be using <a href="https://deepgram.com/">Deepgram</a> Audio Diarization APIs to get speaker details from a sample audio clip.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2bf8eYGeercJSjvD.png" /></figure><h3>Folder Structure</h3><p>After completing the tutorial, the folder structure will look like this. 👇</p><pre>dyte-proctoring/<br>├── frontend<br>│   ├── README.md<br>│   ├── package.json<br>│   ├── public<br>│   ├── src<br>│   │   ├── App.css<br>│   │   ├── App.jsx<br>│   │   ├── Heading.jsx<br>│   │   ├── ImageInput.jsx<br>│   │   ├── Meet.jsx<br>│   │   ├── Proctor.jsx<br>│   │   ├── index.css<br>│   │   ├── index.jsx<br>│   │   ├── logo.svg<br>│   │   ├── react-app-env.d.ts<br>│   │   ├── reportWebVitals.ts<br>│   │   ├── setupTests.ts<br>│   │   ├── stage.jsx<br>│   │   └── utils.js<br>│   ├── tsconfig.json<br>├── venv<br>├── app.py<br>├── audio_diarization.py<br>└── requirements.txt</pre><h3>Step 0: Configurations and Setup</h3><p>🧑‍💻 Before building our live proctoring system, we must set up a Dyte account.</p><p>We can create a free account by clicking the “Start Building” button on Dyte.io and signing up using Google or GitHub. 🎉</p><p>Once signed up, we can access our Dyte API keys from the “API Keys” tab in the left sidebar. We will keep these keys secure as we will use them later. 🔑🤫</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/0*ADjVB-zdQ4-71DD0.gif" /></figure><p>For our live proctoring system, we will use <a href="https://react.dev/">React</a> for the frontend and <a href="https://fastapi.tiangolo.com/lo/">FastAPI</a> for building the Backend and APIs.</p><p>We will begin by creating a new directory for our project, called dyte-proctoring, and navigating into it using the following commands:</p><pre>mkdir dyte-proctoring<br>cd dyte-proctoring</pre><p>NOTE</p><p>We will require to create accounts on the following platforms:</p><ul><li>ElephantSQL: Here is a <a href="https://developers.deepgram.com/docs/getting-started-with-pre-recorded-audio">step-by-step guide</a> to create a db on ElephantSQL.</li><li>Deepgram: Here is a <a href="https://developers.deepgram.com/docs">step-by-step guide</a> to start using Deepgram.</li></ul><p>Now back to the tutorial.</p><h3>Step 1: Setting up the Frontend</h3><p>Let’s start setting up our frontend project using React and Dyte! ✨</p><p>We will create a boilerplate React app using create-react-app. We can do this with the following command:</p><pre>yarn create react-app frontend</pre><p>This will initialize a new React app in the frontend directory. 📁</p><p>Then, we will go ahead and install the dyte react-web-core, dyte react-ui-kit and react-router packages in this project using the following command 👇</p><pre>yarn add @dytesdk/react-web-core @dytesdk/react-ui-kit react-router react-router-dom</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/0*SA-tm6946CM6Q7vc.gif" /></figure><h3>Step 2: Setting up the backend</h3><p>Let’s get started with setting up our FastAPI backend now. 🙌</p><p>We will go back to the root directory of our project and initiate our project here itself for the ease of hosting:</p><pre>cd ..</pre><p>First of all, we will go ahead and create our requirements.txt file in the root directory itself with the following content 👇</p><p>requirements.txt</p><pre>cmake<br>fastapi<br>uvicorn<br>face_recognition<br>numpy<br>python-multipart<br>psycopg2-binary<br>httpx<br>python-dotenv<br>pydantic<br>requests</pre><p>After this, we will go ahead and create our virtual environment with venv and install the dependencies.</p><pre>python -m venv venv<br>source venv/bin/activate # for linux/mac<br>venv\Scripts\activate.bat # for windows<br>pip install -r requirements.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LlQ5NC8cMrbZeueo.gif" /></figure><p>We will also create an environment variable file .env, for storing our credentials.</p><p>.env</p><pre>DYTE_ORG_ID=&lt;ID&gt;<br>DYTE_API_KEY=&lt;KEY&gt;<br>IMGUR_CLIENT_ID=&lt;ID&gt;<br>DB_USER=&lt;ID&gt;<br>DB_PASSWORD=&lt;PASSWORD&gt;<br>DB_HOST=&lt;HOST&gt;<br>DEEPGRAM_API_KEY=&lt;API_KEY&gt;</pre><h3>Step 3: Setting up our backend application</h3><p>Now, we will create a new file named app.py and add our 🐘 ElephantSQL PostgreSQL database connection and code for our APIs, including face detection logic.</p><p>In this file, we would need to create the following routes:</p><p>GET / - Root route</p><p>POST /is_admin/ - Check if the user is an admin</p><p>POST /multiple_voices_list/ - This route retrieves a list of participants who are detected as suspicious</p><p>POST /multiple_voices/ - Detect if there are multiple voices in the audio stream</p><p>POST /meetings - Create a new meeting</p><p>POST /meetings/{meetingId}/participants - This route is responsible for adding a participant to a specific meeting identified by meetingId</p><p>So let’s get started 👇</p><p>app.py</p><pre>import base64<br>import io<br>import logging<br>import random<br>import requests<br><br>import uvicorn<br>from fastapi import FastAPI, Form, UploadFile, File<br>from fastapi.middleware.cors import CORSMiddleware<br>from pydantic import BaseModel<br>import psycopg2<br>from utils import upload_audio<br><br>import os<br>import base64<br>from fastapi import FastAPI, HTTPException<br>from dotenv import load_dotenv<br>from httpx import AsyncClient<br>import httpx<br>import uuid<br><br>from audio_diarization import speakers_count<br><br>load_dotenv()<br><br>DYTE_API_KEY = os.getenv(&quot;DYTE_API_KEY&quot;)<br>DYTE_ORG_ID = os.getenv(&quot;DYTE_ORG_ID&quot;)<br><br>API_HASH = base64.b64encode(f&quot;{DYTE_ORG_ID}:{DYTE_API_KEY}&quot;.encode(&#39;utf-8&#39;)).decode(&#39;utf-8&#39;)<br><br>timeout = httpx.Timeout(10.0, read=None)<br>DYTE_API = AsyncClient(base_url=&#39;https://api.cluster.dyte.in/v2&#39;, headers={&#39;Authorization&#39;: f&quot;Basic {API_HASH}&quot;}, timeout=timeout)<br><br>logger = logging.getLogger(__name__)<br>logging.basicConfig(level=logging.INFO)<br><br>fh = logging.FileHandler(&quot;app.log&quot;)<br>fh.setLevel(logging.DEBUG)<br>formatter = logging.Formatter(&quot;%(asctime)s - %(name)s - %(levelname)s - %(message)s&quot;)<br>fh.setFormatter(formatter)<br>logger.addHandler(fh)<br><br><br>class ProctorPayload(BaseModel):<br>    meeting_id: str<br>    admin_id: str<br><br>class AdminProp(BaseModel):<br>    meeting_id: str<br>    admin_id: str<br><br>class Meeting(BaseModel):<br>    title: str<br><br>class Participant(BaseModel):<br>    name: str<br>    preset_name: str<br>    meeting_id: str<br><br>origins = [<br>    # allow all<br>    &quot;*&quot;,<br>]<br><br>app = FastAPI()<br><br># enable cors<br>app.add_middleware(<br>    CORSMiddleware,<br>    allow_origins=origins,<br>    allow_credentials=True,<br>    allow_methods=[&quot;*&quot;],  # allow all<br>    allow_headers=[&quot;*&quot;],  # allow all<br>)<br><br>def connect_to_db():<br>    conn = psycopg2.connect(<br>            dbname=os.getenv(&#39;DB_USER&#39;),<br>            user=os.getenv(&#39;DB_USER&#39;),<br>            password=os.getenv(&#39;DB_PASSWORD&#39;),<br>            host=os.getenv(&#39;DB_HOST&#39;),<br>            port=5432<br>    )<br>    return conn<br><br>@app.get(&quot;/&quot;)<br>async def root():<br>    return {&quot;message&quot;: &quot;Hello World&quot;}<br><br>@app.post(&quot;/is_admin/&quot;)<br>async def multiple_faces_list(admin: AdminProp):<br>    conn = connect_to_db()<br>    cur = conn.cursor()<br>    cur.execute(&quot;SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s&quot;, (admin.meeting_id, admin.admin_id,))<br><br>    count = cur.fetchone()[0]<br><br>    if(count &gt; 0):<br>        return { &quot;admin&quot;: True }<br>    else:<br>        return { &quot;admin&quot;: False }<br><br>@app.post(&quot;/multiple_voices_list/&quot;)<br>async def multiple_faces_list(meeting: ProctorPayload):<br>    conn = connect_to_db()<br>    cur = conn.cursor()<br>    cur.execute(&quot;SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s&quot;, (meeting.meeting_id, meeting.admin_id,))<br><br>    count = cur.fetchone()[0]<br><br>    if(count &gt; 0):<br>        cur.execute(&quot;CREATE TABLE IF NOT EXISTS meeting_audio_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), audio_url VARCHAR(255), verdict VARCHAR(255))&quot;)<br>        cur.execute(&quot;SELECT * FROM meeting_audio_proc_details WHERE meeting_id = %s ORDER BY ts DESC&quot;, (meeting.meeting_id,))<br>        rows = cur.fetchall()<br>        conn.commit()<br>        cur.close()<br>        conn.close()<br>        return rows<br>    else:<br>        conn.commit()<br>        cur.close()<br>        conn.close()<br>        raise HTTPException(status_code=401, detail=&quot;Participant dose not has admin role&quot;)<br><br>@app.post(&quot;/multiple_voices/&quot;)<br>async def multiple_voices(file: UploadFile = File(...), meeting_id: str = Form(...), participant_id: str = Form(...), participant_name: str = Form(...)):<br>    contents = file.file.read()<br>    filename = file.filename.split(&#39;.&#39;)[0] + str(random.randint(1, 100)) + &#39;.mp3&#39;<br>    with open(filename, &#39;wb&#39;) as f:<br>        f.write(contents)<br>    file.file.close()<br><br>    conn = connect_to_db()<br>    cur = conn.cursor()<br><br>    cur.execute(&quot;CREATE TABLE IF NOT EXISTS meeting_audio_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), audio_url VARCHAR(255), verdict VARCHAR(255))&quot;)<br><br>    try:<br>        count = speakers_count(filename)<br>    except Exception as e:<br>        logger.error(e)<br><br>    if count &gt; 1:<br>        logger.info(<br>            f&quot;Detected different voices for participant {participant_id}&quot;<br>        )<br><br>        verdict = f&quot;Participant Name: {participant_name} &lt;&gt; Anomaly: Different Voices Detected &lt;&gt; Participant ID: {participant_id}&quot;<br>        cur.execute(&quot;SELECT count(1) FROM meeting_audio_proc_details WHERE meeting_id=%s AND participant_id=%s AND ts &gt;= (current_timestamp - INTERVAL &#39;10 minutes&#39;)&quot;, (meeting_id, participant_id))<br>        count = cur.fetchone()[0]<br><br>        if count == 0:<br>            # if there is a requirement of storing audio files<br>            upload_resp = await upload_audio(img_data)<br>            cur.execute(&quot;INSERT INTO meeting_audio_proc_details (ts, meeting_id, participant_id, audio_url, verdict) VALUES (current_timestamp, %s, %s, %s, %s)&quot;,<br>                (meeting_id, participant_id, upload_resp, verdict)<br>            )<br><br>        conn.commit()<br>        cur.close()<br>        conn.close()<br><br>        if count == 0:<br>            return { &quot;id&quot;: participant_id, &quot;different_voices_detected&quot;: True, &quot;url&quot;: upload_resp }<br>        return { &quot;id&quot;: participant_id, &quot;different_voices_detected&quot;: True, &quot;url&quot;: &quot;not uploaded&quot; }<br><br>    return {&quot;id&quot;: participant_id, &quot;different_voices_detected&quot;: False}<br><br>@app.post(&quot;/meetings&quot;)<br>async def create_meeting(meeting: Meeting):<br>    response = await DYTE_API.post(&#39;/meetings&#39;, json=meeting.dict())<br>    if response.status_code &gt;= 300:<br>        raise HTTPException(status_code=response.status_code, detail=response.text)<br>    admin_id = &#39;&#39;.join(random.choices(&#39;abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789&#39;, k=32))<br>    resp_json = response.json()<br>    resp_json[&#39;admin_id&#39;] = admin_id<br>    meeting_id = resp_json[&#39;data&#39;][&#39;id&#39;]<br><br>    conn = connect_to_db()<br>    cur = conn.cursor()<br>    cur.execute(&quot;INSERT INTO meeting_host_info (ts, meeting_id, admin_id) VALUES (CURRENT_TIMESTAMP, %s, %s)&quot;, (meeting_id, admin_id))<br>    conn.commit()<br>    cur.close()<br>    conn.close()<br><br>    return resp_json<br><br><br>@app.post(&quot;/meetings/{meetingId}/participants&quot;)<br>async def add_participant(meetingId: str, participant: Participant):<br>    client_specific_id = f&quot;react-samples::{participant.name.replace(&#39; &#39;, &#39;-&#39;)}-{str(uuid.uuid4())[0:7]}&quot;<br>    payload = participant.dict()<br>    payload.update({&quot;client_specific_id&quot;: client_specific_id})<br>    del payload[&#39;meeting_id&#39;]<br>    resp = await DYTE_API.post(f&#39;/meetings/{meetingId}/participants&#39;, json=payload)<br>    if resp.status_code &gt; 200:<br>        raise HTTPException(status_code=resp.status_code, detail=resp.text)<br>    return resp.text<br><br>if __name__ == &quot;__main__&quot;:<br>    uvicorn.run(&quot;app:app&quot;, host=&quot;localhost&quot;, port=8000, log_level=&quot;debug&quot;, reload=True)</pre><p>This code defines a ⚡️ FastAPI application with an endpoint /multiple_voices, which takes in an audio file in binary format and returns a boolean value indicating if there are multiple voices in the audio submitted.</p><p>It uses the <a href="https://deepgram.com/ageitgey">audio_diarization api by Deepgram</a> API service to detect multiple voices in the audio received.</p><p>audio_diarization.py</p><pre>import os<br>from httpx import AsyncClient<br>import requests<br>from dotenv import load_dotenv<br><br>load_dotenv()<br><br><br>DEEPGRAM_API_KEY = os.getenv(&quot;DEEPGRAM_API_KEY&quot;)<br>DEEPGRAM_API = AsyncClient(base_url=&#39;https://api.deepgram.com/v1&#39;, headers={&#39;Authorization&#39;: f&quot;Token {DEEPGRAM_API_KEY}&quot;, &#39;Content-Type&#39;: &quot;audio/mp3&quot;})<br><br>def speakers_count(audio_file_path):<br>    speaker_set = set()<br>    audio = open(audio_file_path, &#39;rb&#39;)<br>    response = requests.post(&#39;https://api.deepgram.com/v1/listen?diarize=true&#39;, data=audio, headers={&#39;Authorization&#39;: f&quot;Token {DEEPGRAM_API_KEY}&quot;, &#39;Content-Type&#39;: &quot;audio/mp3&quot;})<br>    resp_json = response.json()<br>    statements = resp_json[&#39;results&#39;][&#39;channels&#39;][0][&#39;alternatives&#39;]<br>    for statement in statements:<br>        words = statement[&#39;words&#39;]<br>        for word in words:<br>            speaker_set.add(word[&#39;speaker&#39;])<br>    return len(speaker_set)</pre><p>The code also makes use of the Dyte API for meeting-related operations. 📹</p><p>We can start the backend server simply by using the following:</p><pre>python app.py</pre><p>This Python server helps us create and join meetings, detect different faces, and get the list of suspicious candidates. 🕵️</p><p>Here, when we hit the /multiple_voices endpoint with an audio file, the multiple_voices_detected key of the response would be set to True if there are multiple voices in the audio file received, else it will be set to False.</p><p>We can call this from our frontend with the participant’s mic.</p><p>With this sorted, let’s return to our React application and create our UI. ✨</p><h3>Step 4: Setting up the Meeting UI</h3><p>First, let us add our CSS file, create a new file frontend/src/App.css and paste the following code.</p><pre>.App {<br>	text-align: center;<br>}<br><br>.App-logo {<br>	height: 40vmin;<br>	pointer-events: none;<br>}<br><br>@media (prefers-reduced-motion: no-preference) {<br>	.App-logo {<br>		animation: App-logo-spin infinite 20s linear;<br>	}<br>}<br><br>.App-header {<br>	background-color: #282c34;<br>	min-height: 100vh;<br>	display: flex;<br>	flex-direction: column;<br>	align-items: center;<br>	justify-content: center;<br>	font-size: calc(10px + 2vmin);<br>	color: white;<br>}<br><br>.App-link {<br>	color: #61dafb;<br>}<br><br>.heading-proctor {<br>	font-size: x-large;<br>	font-weight: bolder;<br>	color: #fff;<br>}<br><br>@keyframes App-logo-spin {<br>	from {<br>		transform: rotate(0deg);<br>	}<br>	to {<br>		transform: rotate(360deg);<br>	}<br>}<br><br>.file-input label {<br>	margin-top: 20px;<br>	display: block;<br>	position: relative;<br>	width: 200px;<br>	height: 50px;<br>	border-radius: 10px;<br>	background-color: #2160fd;<br>	display: flex;<br>	align-items: center;<br>	justify-content: center;<br>	flex-direction: row;<br>	color: #fff;<br>	font-weight: bold;<br>	cursor: pointer;<br>	transition: transform 0.2s ease-out;<br>}<br><br>.file {<br>	opacity: 0;<br>	width: 0.1px;<br>	height: 0.1px;<br>	position: absolute;<br>}</pre><p>Next, we will add the initial Dyte Meeting component to our app. We can do this by replacing the contents of frontend/src/App.jsx with the following code:</p><pre>import { useEffect, useState } from &quot;react&quot;;<br>import Home from &quot;./Home&quot;;<br>import { BrowserRouter, Routes, Route, Link } from &quot;react-router-dom&quot;;<br>import &quot;./App.css&quot;;<br>import Stage from &quot;./Stage&quot;;<br><br>const SERVER_URL = process.env.REACT_APP_SERVER_URL || &quot;http://localhost:8000&quot;;<br><br>function App() {<br> const [meetingId, setMeetingId] = useState();<br><br> const createMeeting = async () =&gt; {<br>  const res = await fetch(`${SERVER_URL}/meetings`, {<br>   method: &quot;POST&quot;,<br>   body: JSON.stringify({ title: &quot;Joint Entrance Examination&quot; }),<br>   headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br>  });<br>  const resJson = await res.json();<br>  window.localStorage.setItem(&quot;adminId&quot;, resJson.admin_id);<br>  setMeetingId(resJson.data.id);<br> };<br><br> useEffect(() =&gt; {<br>  window.localStorage.removeItem(&quot;refImgUrl&quot;);<br>  const id = window.location.pathname.split(&quot;/&quot;)[2];<br>  if (!!!id) {<br>   createMeeting();<br>  }<br> }, []);<br><br> return (<br>  &lt;BrowserRouter&gt;<br>   &lt;Routes&gt;<br>    &lt;Route path=&quot;/&quot; element={&lt;Home meetingId={meetingId} /&gt;}&gt;&lt;/Route&gt;<br>    &lt;Route path=&quot;/meeting/:meetingId&quot; element={&lt;Stage /&gt;}&gt;&lt;/Route&gt;<br>   &lt;/Routes&gt;<br>  &lt;/BrowserRouter&gt;<br> );<br>}<br><br>export default App;</pre><p>This component will create a Dyte meeting link and an adminId for the admin. We will store the adminId secretly in localstorage. The adminId will be used later for accessing any sensitive data.</p><h3>Home component</h3><p>The home component renders the / route. Create a file as frontend/src/Home.jsx.</p><pre>import { Link } from &quot;react-router-dom&quot;;<br>function Home({ meetingId }) {<br>	return (<br>		&lt;div<br>			style={{<br>				height: &quot;100vh&quot;,<br>				width: &quot;100vw&quot;,<br>				fontSize: &quot;x-large&quot;,<br>				display: &quot;flex&quot;,<br>				justifyContent: &quot;center&quot;,<br>				alignItems: &quot;center&quot;,<br>			}}<br>		&gt;<br>			{meetingId &amp;&amp; !window.location.pathname.split(&quot;/&quot;)[2] &amp;&amp; (<br>				&lt;Link to={`/meeting/${meetingId}`}&gt;Create and Join Meeting&lt;/Link&gt;<br>			)}<br>		&lt;/div&gt;<br>	);<br>}<br><br>export default Home;</pre><h3>Heading component</h3><p>Now we will create a file as frontend/src/Heading.jsx.</p><pre>const Heading = ({ text }) =&gt; {<br>	return (<br>		&lt;div<br>			className=&quot;heading-proctor&quot;<br>			style={{<br>				padding: &quot;10px&quot;,<br>				textAlign: &quot;center&quot;,<br>				backgroundColor: &quot;#000&quot;,<br>				borderBottom: &quot;solid 0.5px gray&quot;,<br>			}}<br>		&gt;<br>			{text}<br>		&lt;/div&gt;<br>	);<br>};<br><br>export default Heading;</pre><p>Let’s create a staging area for participants joining the meeting. Admin bypasses the staging area, but the candidate will be asked to upload a reference image of himself on this page.</p><p>For this, we will create another file, frontend/src/Stage.jsx</p><pre>import { useState, useEffect } from &quot;react&quot;;<br>import Meet from &quot;./Meet&quot;;<br><br>const SERVER_URL = process.env.REACT_APP_SERVER_URL || &quot;http://localhost:8000&quot;;<br><br>const Stage = () =&gt; {<br> const [isAdminBool, setAdminBool] = useState(null);<br> const meetingId = window.location.pathname.split(&quot;/&quot;)[2];<br><br> const isAdmin = async (id) =&gt; {<br>  const res = await fetch(`${SERVER_URL}/is_admin`, {<br>   method: &quot;POST&quot;,<br>   body: JSON.stringify({<br>    admin_id: window.localStorage.getItem(&quot;adminId&quot;) || &quot;&quot;,<br>    meeting_id: meetingId || &quot;&quot;,<br>   }),<br>   headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br>  });<br>  const resJson = await res.json();<br>  setAdminBool(resJson.admin);<br> };<br><br> useEffect(() =&gt; {<br>  isAdmin();<br> }, []);<br><br> return (<br>  &lt;div<br>   style={{<br>    height: &quot;100vh&quot;,<br>    width: &quot;100vw&quot;,<br>    display: &quot;flex&quot;,<br>    justifyContent: &quot;center&quot;,<br>    alignItems: &quot;center&quot;,<br>    color: &quot;white&quot;,<br>   }}<br>  &gt;<br>   {isAdminBool == null ? (<br>    &lt;&gt;Loading...&lt;/&gt;<br>   ) : (<br>    &lt;&gt;<br>     &lt;Meet isAdminBool={isAdminBool} /&gt;<br>    &lt;/&gt;<br>   )}<br>  &lt;/div&gt;<br> );<br>};<br><br>export default Stage;</pre><p>Now, let’s delve into the Meet component that renders on route /meeting/:meetingId.</p><p>When the admin clicks on the link provided on the / route, he gets redirected to the meeting page, where we add the user to the meeting as a participant with audio_proc_preset preset. 🤙</p><p>Since this user created the meeting and was redirected to the meet page, we will assign him the admin role. Now the link from the address bar can be shared with the candidates.</p><p>When a candidate opens the shared link, they become a regular user. And for every regular user, the component emits screenshots of the users’ videos to our directed to our Python server. 🐍</p><pre>/* eslint-disable */<br><br>import { useState, useEffect, useRef } from &quot;react&quot;;<br>import { DyteMeeting, provideDyteDesignSystem } from &quot;@dytesdk/react-ui-kit&quot;;<br>import { useDyteClient } from &quot;@dytesdk/react-web-core&quot;;<br>import Proctor from &quot;./Proctor&quot;;<br>import Heading from &quot;./Heading&quot;;<br>import { joinMeeting } from &quot;./utils&quot;;<br>import lamejs from &quot;lamejstmp&quot;;<br><br>// Constants<br>const SERVER_URL = process.env.REACT_APP_SERVER_URL || &quot;http://localhost:8000&quot;;<br>let LAST_BACKEND_PING_TIME = 0;<br>const TIME_BETWEEN_BACKEND_PINGS = 60000;<br><br>function convertFloat32ToInt16(buffer) {<br> var l = buffer.length;<br> var buf = new Int16Array(l);<br> while (l--) {<br>  buf[l] = Math.min(1, buffer[l]) * 0x7fff;<br> }<br> return buf;<br>}<br><br>const Meet = () =&gt; {<br> const meetingEl = useRef();<br> const [meeting, initMeeting] = useDyteClient();<br> const [userToken, setUserToken] = useState();<br> const [isAdminBool, setAdminBool] = useState(null);<br> const meetingId = window.location.pathname.split(&quot;/&quot;)[2];<br><br> async function audioToMp3Middleware(audioContext) {<br>  const processor = audioContext.createScriptProcessor(1024, 1, 1);<br>  const encoder = new lamejs.Mp3Encoder(1, 44100, 128);<br>  const bufferSize = 512;<br>  let mp3Data = [];<br><br>  processor.onaudioprocess = (e) =&gt; {<br>   const inputData = e.inputBuffer.getChannelData(0);<br>   const mp3buf = encoder.encodeBuffer(convertFloat32ToInt16(inputData));<br>   if (mp3buf.length &gt; 0) {<br>    mp3Data.push(mp3buf);<br>   }<br>   const currentTime = Date.now();<br>   if (<br>    mp3Data.length &gt;= bufferSize &amp;&amp;<br>    currentTime - LAST_BACKEND_PING_TIME &gt; TIME_BETWEEN_BACKEND_PINGS<br>   ) {<br>    LAST_BACKEND_PING_TIME = currentTime;<br>    const file = new File(mp3Data, &quot;audio.mp3&quot;);<br>    const form = new FormData();<br>    form.append(&quot;file&quot;, file);<br>    form.append(&quot;meeting_id&quot;, meetingId);<br>    form.append(&quot;participant_id&quot;, meeting?.self.id);<br>    form.append(&quot;participant_name&quot;, meeting?.self.name);<br><br>    fetch(`${SERVER_URL}/multiple_voices`, {<br>     method: &quot;POST&quot;,<br>     body: form,<br>    });<br><br>    mp3Data = [];<br>   }<br>  };<br><br>  return processor;<br> }<br><br> const isAdmin = async (id) =&gt; {<br>  const res = await fetch(`${SERVER_URL}/is_admin`, {<br>   method: &quot;POST&quot;,<br>   body: JSON.stringify({<br>    admin_id: window.localStorage.getItem(&quot;adminId&quot;) || &quot;&quot;,<br>    meeting_id: meetingId || &quot;&quot;,<br>   }),<br>   headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br>  });<br>  const resJson = await res.json();<br>  setAdminBool(resJson.admin);<br> };<br><br> const joinMeetingId = async () =&gt; {<br>  if (meetingId) {<br>   const authToken = await joinMeeting(meetingId);<br>   await initMeeting({<br>    authToken,<br>   });<br>   setUserToken(authToken);<br>  }<br> };<br><br> useEffect(() =&gt; {<br>  if (meetingId &amp;&amp; !userToken) joinMeetingId();<br>  isAdmin();<br> }, []);<br><br> useEffect(() =&gt; {<br>  if (userToken) {<br>   provideDyteDesignSystem(meetingEl.current, {<br>    theme: &quot;dark&quot;,<br>   });<br>  }<br> }, [userToken]);<br><br> useEffect(() =&gt; {<br>  if (isAdminBool === false &amp;&amp; meeting?.self) {<br>   meeting.self.addAudioMiddleware(audioToMp3Middleware);<br>  }<br><br>  return () =&gt; {<br>   if (isAdminBool === false &amp;&amp; meeting?.self) {<br>    meeting.self.removeAudioMiddleware(audioToMp3Middleware);<br>   }<br>  };<br> }, [meeting?.self, isAdminBool]);<br><br> return (<br>  &lt;div style={{ height: &quot;96vh&quot;, width: &quot;100vw&quot;, display: &quot;flex&quot; }}&gt;<br>   {userToken &amp;&amp; (<br>    &lt;&gt;<br>     {isAdminBool &amp;&amp; (<br>      &lt;div<br>       style={{<br>        width: &quot;40vw&quot;,<br>        height: &quot;100vh&quot;,<br>        overflowY: &quot;scroll&quot;,<br>        backgroundColor: &quot;black&quot;,<br>        borderRight: &quot;solid 0.5px gray&quot;,<br>       }}<br>      &gt;<br>       &lt;Heading text={&quot;Proctoring Information&quot;} /&gt;<br>       &lt;Proctor /&gt;<br>      &lt;/div&gt;<br>     )}<br>     {isAdminBool ? (<br>      &lt;div style={{ width: &quot;60vw&quot;, height: &quot;96vh&quot; }}&gt;<br>       &lt;Heading text={&quot;Proctoring Admin Interface&quot;} /&gt;<br>       &lt;DyteMeeting mode=&quot;fill&quot; meeting={meeting} ref={meetingEl} /&gt;<br>      &lt;/div&gt;<br>     ) : (<br>      &lt;div style={{ width: &quot;100vw&quot;, height: &quot;96vh&quot; }}&gt;<br>       &lt;Heading text={&quot;Proctoring Candidate Interface&quot;} /&gt;<br>       &lt;DyteMeeting mode=&quot;fill&quot; meeting={meeting} ref={meetingEl} /&gt;<br>      &lt;/div&gt;<br>     )}<br>    &lt;/&gt;<br>   )}<br>  &lt;/div&gt;<br> );<br>};<br><br>export default Meet;</pre><p>Let’s briefly go through some of the functions:</p><ul><li>isAdmin talks to the Python server to identify whether the current client is the admin.</li><li>joinMeeting adds the current client to the meeting.</li><li>audioToMp3Middleware sends audio samples of candidates to the Python server.</li></ul><h3>Proctor component</h3><p>The proctor component gets activated only for admins. The proctor component, with the help of adminId fetches the list of suspicious candidates and renders it in a chat-like format.</p><p>Create a file frontend/src/Proctor.jsx.</p><pre>import { useEffect, useState } from &quot;react&quot;;<br>import { getCandidateStatus } from &quot;./utils&quot;;<br><br>const Proctor = () =&gt; {<br> const [candidateStatuses, updateCandidateStatusState] = useState([]);<br> const [error, setError] = useState(&quot;&quot;);<br><br> const updateCandidateStatus = async () =&gt; {<br>  try {<br>   const res = await getCandidateStatus();<br>   updateCandidateStatusState(res);<br>  } catch (e) {<br>   setError(&quot;User don&#39;t have admin privileges.&quot;);<br>  }<br> };<br><br> useEffect(() =&gt; {<br>  if (candidateStatuses?.map) {<br>   const id = setInterval(() =&gt; {<br>    updateCandidateStatus();<br>   }, 30000);<br>   return () =&gt; {<br>    clearInterval(id);<br>   };<br>  }<br> }, [candidateStatuses]);<br><br> return (<br>  &lt;&gt;<br>   &lt;div style={{ padding: &quot;0px 20px&quot; }}&gt;<br>    {candidateStatuses?.map &amp;&amp; candidateStatuses ? (<br>     candidateStatuses.map((status) =&gt; (<br>      &lt;div<br>       style={{<br>        display: &quot;flex&quot;,<br>        justifyContent: &quot;start&quot;,<br>        margin: &quot;50px 20px&quot;,<br>       }}<br>      &gt;<br>       &lt;div style={{ marginRight: &quot;20px&quot; }}&gt;<br>        &lt;img<br>         src=&quot;https://images.yourstory.com/cs/images/companies/Dyte-1608553297314.jpg&quot;<br>         style={{<br>          borderRadius: &quot;50px&quot;,<br>          height: &quot;60px&quot;,<br>          border: &quot;1px double lightblue&quot;,<br>         }}<br>        /&gt;<br>       &lt;/div&gt;<br>       &lt;div<br>        style={{<br>         textAlign: &quot;center&quot;,<br>         padding: &quot;20px&quot;,<br>         backgroundColor: &quot;#2160fd&quot;,<br>         fontSize: &quot;large&quot;,<br>         fontWeight: &quot;400&quot;,<br>         borderRadius: &quot;10px 10px 10px 10px&quot;,<br>         width: &quot;80%&quot;,<br>        }}<br>       &gt;<br>        &lt;div<br>         style={{<br>          color: &quot;white&quot;,<br>          padding: &quot;20px 0px&quot;,<br>          textAlign: &quot;left&quot;,<br>         }}<br>        &gt;<br>         {status[4].split(&quot;&lt;&gt;&quot;).map((text) =&gt; (<br>          &lt;div&gt;{text}&lt;/div&gt;<br>         ))}<br>         &lt;div&gt;Timestamp: {new Date(status[0]).toLocaleString()}&lt;/div&gt;<br>        &lt;/div&gt;<br>       &lt;/div&gt;<br>      &lt;/div&gt;<br>     ))<br>    ) : (<br>     &lt;div style={{ color: &quot;white&quot; }}&gt;<br>      Wait or check if you have admin privileges to access the proctoring<br>      dashboard.<br>     &lt;/div&gt;<br>    )}<br>   &lt;/div&gt;<br>  &lt;/&gt;<br> );<br>};<br><br>export default Proctor;</pre><h3>Utility functions</h3><p>Now we will add utility functions.</p><p>Create a file frontend/src/utils.js.</p><pre>const SERVER_URL = process.env.REACT_APP_SERVER_URL || &quot;http://localhost:8000&quot;;<br><br>const joinMeeting = async (id) =&gt; {<br> const res = await fetch(`${SERVER_URL}/meetings/${id}/participants`, {<br>  method: &quot;POST&quot;,<br>  body: JSON.stringify({<br>   name: &quot;new user&quot;,<br>   preset_name: &quot;audio_proc_preset&quot;,<br>   meeting_id: id,<br>  }),<br>  headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br> });<br> const resJson = await res.json();<br> const data = JSON.parse(resJson.detail);<br> return data.data.token;<br>};<br><br>const getCandidateStatus = async () =&gt; {<br> const response = await fetch(`${SERVER_URL}/multiple_voices_list`, {<br>  method: &quot;POST&quot;,<br>  headers: {<br>   &quot;Content-Type&quot;: &quot;application/json&quot;,<br>  },<br>  body: JSON.stringify({<br>   meeting_id: window.location.pathname.split(&quot;/&quot;)[2],<br>   admin_id: window.localStorage.getItem(&quot;adminId&quot;) || &quot;undefined&quot;,<br>  }),<br> });<br> const res = await response.json();<br> if (res.details) return undefined;<br> return res;<br>};<br><br>const FloatArray2Int16 = (floatbuffer) =&gt; {<br> var int16Buffer = new Int16Array(floatbuffer.length);<br> for (var i = 0, len = floatbuffer.length; i &lt; len; i++) {<br>  if (floatbuffer[i] &lt; 0) {<br>   int16Buffer[i] = 0x8000 * floatbuffer[i];<br>  } else {<br>   int16Buffer[i] = 0x7fff * floatbuffer[i];<br>  }<br> }<br> return int16Buffer;<br>};<br><br>export { joinMeeting, getCandidateStatus, FloatArray2Int16 };</pre><p>To start the React app on the local server, we can run the following command:</p><pre>yarn start</pre><p>Now, upon visiting <a href="http://localhost:3000/">http://localhost:3000/</a>, we should be able to see the Dyte meeting in our browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9mfOlz7drjMUMuUj.png" /></figure><h3>Step 5: Adding multiple voice detection logic to the Frontend</h3><p>Since now we have a nice backend server to detect multiple voices and a great UI, we can add the multiple voice detection logic to our frontend. For this, we will first add some constants to our previously edited frontend/src/App.jsx file:</p><p>We will be using the above constants in the audioToMp3Middleware function, which we will add to our Meet component.</p><p>The audioToMp3Middleware is a <a href="https://dyte.io/blog/streams-blog/">Dyte Audio</a> <a href="https://docs.dyte.io/web-core/local-user/extras#using-middlewares">Middleware</a>. Middlewares are add-ons that we can use to add effects and filters to your audio and video streams with ease.</p><p>Here, we are using the middleware functionality to get the audio sample from the participant’s mic and send it to our backend server. We are also ensuring that the backend is pinged only once every 60 seconds to avoid unnecessary load on the server.</p><p>That was all the code we needed to add basic proctoring functionality to our Dyte meeting. 👍</p><p>The app sends an audio sample from the participant’s mic to the backend server every 60 seconds, and if the backend detects multiple voices in the audio sample, it sends a warning notification to the proctor. ⚠️</p><p>The backend also logs the participant’s ID and the time of the detection in the terminal. This can be used to keep track of the participants who may have cheated during the meeting for later review.</p><h3>Step 6: Trying out our project</h3><p>Ta-da! 🎩✨ It’s time to put our proctoring system to the test and see it in action!</p><ul><li>First, let’s look at the candidate’s view after uploading the reference image. The candidate can see that the proctor is in the meeting but cannot see the Proctoring Panel. 🧑‍💻</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b1OmhO4F8z4DIRe8.png" /></figure><ul><li>In the proctor’s view, we can see the details (proctoring information) along with proof when two or more people can be heard talking in the candidate’s background. 🙌</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zAOQCEGKzIir9FIw.png" /></figure><p>Here’s <a href="https://github.com/dyte-io/proctored-interviews-detecting-multiple-speakers">the link</a> to the repository for you to take a look at the whole codebase. You can also try out the project <a href="https://dyte-audio.netlify.app/">here</a>. 🔗</p><h3>Conclusion</h3><p>Celebrate! 🎉✨ We’ve built a powerful live proctoring system with Dyte, ensuring integrity and fairness in online exams and interviews. But that’s not all! We can now create our own customized online classroom or meeting platform.</p><p>We can now use this live proctoring system to proctor our online exams and interviews. ✍️</p><p>The possibilities are endless with Dyte; go ahead and try to bring your ideas to life by visiting <a href="https://dyte.io/">dyte.io</a>! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0ee0092a92f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dyte/building-a-live-proctoring-system-to-detect-multiple-speakers-d0ee0092a92f">Building a Live Proctoring System to Detect Multiple Speakers</a> was originally published in <a href="https://medium.com/dyte">Dyte</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Live Proctoring System With React and Dyte]]></title>
            <link>https://medium.com/dyte/building-a-live-proctoring-system-with-react-and-dyte-4039521b5d6f?source=rss----9477120405bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/4039521b5d6f</guid>
            <category><![CDATA[video-api]]></category>
            <category><![CDATA[proctoring-solution]]></category>
            <category><![CDATA[video-sdk]]></category>
            <dc:creator><![CDATA[Ravindra Rathor]]></dc:creator>
            <pubDate>Tue, 02 Jan 2024 11:43:16 GMT</pubDate>
            <atom:updated>2024-01-02T11:43:16.314Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VsZ3JOWt6NWvqmv5Htpukw.png" /></figure><h3>Introduction</h3><p>Proctoring is a method of monitoring students during an examination to prevent them from dishonest conduct.</p><p>In the case of online exams, it is not possible to have a proctor for each student. 🕵️</p><p>This is where live automatic proctoring comes into the picture. It is a method of monitoring students during an online exam using a webcam and a microphone.</p><p>With the help of computer vision and machine learning, one can detect if a student is trying to cheat during an online exam.</p><p>✨ In this tutorial, we will build a live proctoring system using <a href="https://dyte.io/">Dyte</a> APIs that allow an admin to monitor if there are multiple people in the video frame of a candidate in real-time and send them a warning message.</p><p>Let’s begin with a pinch of humor! 😁</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/0*3BivJ9XN6gQV6_WU.gif" /></figure><h3>High-Level Design of the application</h3><p>We want to notify the proctor if anyone other than the candidate is seen on the webcam.</p><p>The proctor would get the candidate’s details and a photograph from their webcam as proof in the meeting sidebar. 📸✨</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K1M86EDKPEl9huYg.png" /></figure><ul><li>In this project, we will use React with <a href="https://dyte.io/blog/custom-ui-kit-sdk/">Dyte UI kit</a> and <a href="https://www.npmjs.com/package/@dytesdk/react-web-core">Dyte React Web Core</a> packages for the frontend.</li><li>For the backend, we will use <a href="https://fastapi.tiangolo.com/lo/">FastApi</a> (Python 3).</li><li>We will also use Database as a service by <a href="https://www.elephantsql.com/">ElephantSQL</a> (PostgreSQL).</li><li>Lastly, we will use <a href="https://apidocs.imgur.com/">Imgur</a> for storing screenshots. And the image metadata will then be saved to the same database.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Plie9Kjk9RG0RMZe.png" /></figure><h3>Folder Structure</h3><p>After completing the tutorial, the folder structure will look like this. 👇</p><pre>dyte-proctoring/<br>├── frontend<br>│   ├── README.md<br>│   ├── package.json<br>│   ├── public<br>│   ├── src<br>│   │   ├── App.css<br>│   │   ├── App.jsx<br>│   │   ├── Heading.jsx<br>│   │   ├── Meet.jsx<br>│   │   ├── Proctor.jsx<br>│   │   ├── index.css<br>│   │   ├── index.jsx<br>│   │   ├── logo.svg<br>│   │   ├── react-app-env.d.ts<br>│   │   ├── reportWebVitals.ts<br>│   │   ├── setupTests.ts<br>│   │   └── utils.js<br>│   ├── tsconfig.json<br>├── venv<br>├── app.py<br>├── imgur.py<br>└── requirements.txt</pre><p>Step 0: Configurations and setup<br>🧑‍💻 Before we start building our live proctoring system, we will need to set up a Dyte account.</p><p>We can create a free account by clicking the “Start Building” button on Dyte.io and signing up using Google or GitHub 🎉.</p><p>Once signed up, we can access our Dyte API keys from the “API Keys” tab in the left sidebar. We will keep these keys secure as we will use them later.🔑🤫</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/0*v_2pKBl1JnWed52r.gif" /></figure><p>For our live proctoring system, we will use <a href="https://react.dev/">React</a> for the frontend and <a href="https://fastapi.tiangolo.com/lo/">FastAPI</a> for building the Backend and APIs.</p><p>We will begin by creating a new directory for our project, called dyte-proctoring and navigating into it using the following commands:</p><pre>mkdir dyte-proctoring<br>cd dyte-proctoring</pre><p>Please note:</p><p>We will also require accounts on the following platforms:</p><ul><li>Imgur: Create an account on Imgur and create an API key. Here is a <a href="https://apidocs.imgur.com/">step-by-step guide</a></li><li>ElephantSQL: Here is a <a href="https://www.elephantsql.com/docs/index.html">step-by-step guide</a> to creating a db on ElephantSQL.</li></ul><p>Now back to the tutorial.</p><h3>Step 1: Setting up the frontend</h3><p>Let’s start setting up our frontend project using React and Dyte! ✨</p><p>We will create a boilerplate React app using create-react-app. We can do this with the following command:</p><pre>yarn create react-app frontend</pre><p>This will initialize a new React app in the frontend directory. 📁</p><p>Then, we will go ahead and install the dyte react-web-core, dyte react-ui-kit and react-router packages in this project using the following command 👇</p><pre>yarn add @dytesdk/react-web-core @dytesdk/react-ui-kit react-router react-router-dom</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/0*qTXptPKi-eT1FYFY.gif" /></figure><h3>Step 2: Setting up the backend</h3><p>Let’s get started with setting up our FastAPI backend now. 🙌</p><p>We will go back to the root directory of our project and initiate our project here itself for the ease of hosting:</p><pre>cd ..</pre><p>First of all, we will go ahead and create our requirements.txt file in the root directory itself with the following content. 👇</p><p>requirements.txt</p><pre>cmake<br>fastapi<br>uvicorn<br>face_recognition<br>numpy<br>python-multipart<br>psycopg2-binary<br>httpx<br>python-dotenv<br>pydantic</pre><p>After this, we will go ahead and create our virtual environment with venv and install the dependencies.</p><pre>python -m venv venv<br>source venv/bin/activate # for linux/mac<br>venv\Scripts\activate.bat # for windows<br>pip install -r requirements.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EVLlUN7SmAeNr7Sl.gif" /></figure><p>We will also create an environment variable file .env, for storing our credentials.</p><p>.env</p><pre>DYTE_ORG_ID=&lt;ID&gt;<br>DYTE_API_KEY=&lt;KEY&gt;<br>IMGUR_CLIENT_ID=&lt;ID&gt;<br>DB_USER=&lt;ID&gt;<br>DB_PASSWORD=&lt;PASSWORD&gt;<br>DB_HOST=&lt;HOST&gt;</pre><h3>Step 3: Setting up image upload with Imgur</h3><p>Let’s create a new file named imgur.py and add the following code. This will help us upload our screenshots to Imgur. 🌅</p><p>Here we are connecting the Imgur API using the CLIENT_ID that we can get from our <a href="https://apidocs.imgur.com/#authorization-and-oauth">Imgur API Dashboard</a> and using it to upload the suspicious candidate’s image and get back the link to it. 🔗👇</p><pre>import base64<br>from fastapi import FastAPI, UploadFile, HTTPException<br>from httpx import AsyncClient<br>from dotenv import load_dotenv<br><br>load_dotenv()<br><br>app = FastAPI()<br>IMGUR_CLIENT_ID = os.getenv(&quot;IMGUR_CLIENT_ID&quot;)<br><br>async def upload_image(img_data):<br>    headers = {<br>        &quot;Authorization&quot;: f&quot;Client-ID {IMGUR_CLIENT_ID}&quot;<br>    }<br>    data = {<br>        &quot;image&quot;: img_data<br>    }<br><br>    async with AsyncClient() as client:<br>        response = await client.post(&quot;https://api.imgur.com/3/image&quot;, headers=headers, data=data)<br><br>    if response.status_code != 200:<br>        raise HTTPException(status_code=500, detail=&quot;Could not upload image.&quot;)<br><br>    print(response.json())<br>    return response.json()[&quot;data&quot;][&quot;link&quot;]</pre><h3>Step 4: Setting up our backend application</h3><p>Now, we will create a new file named app.py and add our 🐘 ElephantSQL PostgreSQL database connection and code for our APIs, including face detection logic.</p><p>In this file, we would need to create the following routes:</p><p>GET / - Root route</p><p>POST /is_admin/ - Check if the user is an admin</p><p>POST /multiple_faces_list/ - This route retrieves a list of participants with multiple faces detected in their images.</p><p>POST /detect_faces/ - Detect multiple faces in participant images</p><p>POST /meetings - Create a new meeting</p><p>POST /meetings/{meetingId}/participants - This route is responsible for adding a participant to a specific meeting identified by meetingId.</p><p>So, let’s get started. 👇</p><p>app.py</p><pre>import base64<br>import io<br>import logging<br>import random<br>import uvicorn<br>from fastapi import FastAPI<br>from fastapi.middleware.cors import CORSMiddleware<br>from pydantic import BaseModel<br>from imgur import upload_image<br>import face_recognition<br>import psycopg2<br><br>import os<br>import base64<br>from fastapi import FastAPI, HTTPException<br>from pydantic import BaseModel<br>from dotenv import load_dotenv<br>from httpx import AsyncClient<br>import uuid<br><br>load_dotenv()<br><br>DYTE_API_KEY = os.getenv(&quot;DYTE_API_KEY&quot;)<br>DYTE_ORG_ID = os.getenv(&quot;DYTE_ORG_ID&quot;)<br><br>API_HASH = base64.b64encode(f&quot;{DYTE_ORG_ID}:{DYTE_API_KEY}&quot;.encode(&#39;utf-8&#39;)).decode(&#39;utf-8&#39;)<br><br>DYTE_API = AsyncClient(base_url=&#39;https://api.cluster.dyte.in/v2&#39;, headers={&#39;Authorization&#39;: f&quot;Basic {API_HASH}&quot;})<br><br>logger = logging.getLogger(__name__)<br>logging.basicConfig(level=logging.INFO)<br><br>fh = logging.FileHandler(&quot;app.log&quot;)<br>fh.setLevel(logging.DEBUG)<br>formatter = logging.Formatter(&quot;%(asctime)s - %(name)s - %(levelname)s - %(message)s&quot;)<br>fh.setFormatter(formatter)<br>logger.addHandler(fh)<br><br><br>class ParticipantScreen(BaseModel):<br>    base64_img: str<br>    participant_id: str<br>    meeting_id: str<br>    participant_name: str<br><br>class ProctorPayload(BaseModel):<br>    meeting_id: str<br>    admin_id: str<br><br>class AdminProp(BaseModel):<br>    meeting_id: str<br>    admin_id: str<br><br>class Meeting(BaseModel):<br>    title: str<br><br>class Participant(BaseModel):<br>    name: str<br>    preset_name: str<br>    meeting_id: str<br><br>origins = [<br>    # allow all<br>    &quot;*&quot;,<br>]<br><br>app = FastAPI()<br><br># enable cors<br>app.add_middleware(<br>    CORSMiddleware,<br>    allow_origins=origins,<br>    allow_credentials=True,<br>    allow_methods=[&quot;*&quot;],  # allow all<br>    allow_headers=[&quot;*&quot;],  # allow all<br>)<br><br>def connect_to_db():<br>    conn = psycopg2.connect(<br>            dbname=os.getenv(&#39;DB_USER&#39;),<br>            user=os.getenv(&#39;DB_USER&#39;),<br>            password=os.getenv(&#39;DB_PASSWORD&#39;),<br>            host=os.getenv(&#39;DB_HOST&#39;),<br>            port=5432<br>    )<br>    return conn<br><br>@app.get(&quot;/&quot;)<br>async def root():<br>    return {&quot;message&quot;: &quot;Hello World&quot;}<br><br>@app.post(&quot;/is_admin/&quot;)<br>async def multiple_faces_list(admin: AdminProp):<br>    conn = connect_to_db()<br>    cur = conn.cursor()<br>    cur.execute(&quot;SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s&quot;, (admin.meeting_id, admin.admin_id,))<br><br>    count = cur.fetchone()[0]<br><br>    if(count &gt; 0):<br>        return { &quot;admin&quot;: True }<br>    else:<br>        return { &quot;admin&quot;: False }<br><br>@app.post(&quot;/multiple_faces_list/&quot;)<br>async def multiple_faces_list(meeting: ProctorPayload):<br>    conn = connect_to_db()<br>    cur = conn.cursor()<br>    cur.execute(&quot;SELECT count(1) FROM meeting_host_info WHERE meeting_id = %s AND admin_id = %s&quot;, (meeting.meeting_id, meeting.admin_id,))<br><br>    count = cur.fetchone()[0]<br><br>    if(count &gt; 0):<br>        cur.execute(&quot;SELECT * FROM meeting_proc_details WHERE meeting_id = %s ORDER BY ts DESC&quot;, (meeting.meeting_id,))<br>        rows = cur.fetchall()<br>        conn.commit()<br>        cur.close()<br>        conn.close()<br>        return rows<br>    else:<br>        conn.commit()<br>        cur.close()<br>        conn.close()<br>        raise HTTPException(status_code=401, detail=&quot;Participant dose not has admin role&quot;)<br><br>@app.post(&quot;/detect_faces/&quot;)<br>async def detect_faces(participant: ParticipantScreen):<br>    img_data = participant.base64_img.split(&quot;,&quot;)[1]<br>    img_data_dc = base64.b64decode(participant.base64_img.split(&quot;,&quot;)[1])<br><br>    file_obj = io.BytesIO(img_data_dc)<br>    img = face_recognition.load_image_file(file_obj)<br><br>    face_locations = face_recognition.face_locations(img)<br><br>    if len(face_locations) &gt; 1:<br>        logger.info(<br>            f&quot;Detected more than one face for participant {participant.participant_id}&quot;<br>        )<br><br>        conn = connect_to_db()<br>        cur = conn.cursor()<br><br>        cur.execute(&quot;CREATE TABLE IF NOT EXISTS meeting_proc_details (ts TIMESTAMP, meeting_id VARCHAR(255), participant_id VARCHAR(255), img_url VARCHAR(255), verdict VARCHAR(255))&quot;)<br><br>        verdict = f&quot;Participant Name: {participant.participant_name} &lt;&gt; Anomaly: Multiple Faces Detected &lt;&gt; Participant ID: {participant.participant_id}&quot;<br>        cur.execute(&quot;SELECT count(1) FROM meeting_proc_details WHERE meeting_id=%s AND participant_id=%s AND ts &gt;= (current_timestamp - INTERVAL &#39;10 minutes&#39;)&quot;, (participant.meeting_id, participant.participant_id))<br>        count = cur.fetchone()[0]<br><br>        if count == 0:<br>            upload_resp = await upload_image(img_data)<br>            cur.execute(&quot;INSERT INTO meeting_proc_details (ts, meeting_id, participant_id, img_url, verdict) VALUES (current_timestamp, %s, %s, %s, %s)&quot;,<br>                (participant.meeting_id, participant.participant_id, upload_resp, verdict)<br>            )<br><br>        conn.commit()<br>        cur.close()<br>        conn.close()<br><br>        if count == 0:<br>            return { &quot;id&quot;: participant.participant_id, &quot;multiple_detected&quot;: True, &quot;url&quot;: upload_resp }<br>        return { &quot;id&quot;: participant.participant_id, &quot;multiple_detected&quot;: True, &quot;url&quot;: &quot;not uploaded&quot; }<br><br>    return {&quot;id&quot;: participant.participant_id, &quot;multiple_detected&quot;: False}<br><br><br>@app.post(&quot;/meetings&quot;)<br>async def create_meeting(meeting: Meeting):<br>    response = await DYTE_API.post(&#39;/meetings&#39;, json=meeting.dict())<br>    if response.status_code &gt;= 300:<br>        raise HTTPException(status_code=response.status_code, detail=response.text)<br>    admin_id = &#39;&#39;.join(random.choices(&#39;abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789&#39;, k=32))<br>    resp_json = response.json()<br>    resp_json[&#39;admin_id&#39;] = admin_id<br>    meeting_id = resp_json[&#39;data&#39;][&#39;id&#39;]<br><br>    conn = connect_to_db()<br>    cur = conn.cursor()<br>    cur.execute(&quot;INSERT INTO meeting_host_info (ts, meeting_id, admin_id) VALUES (CURRENT_TIMESTAMP, %s, %s)&quot;, (meeting_id, admin_id))<br>    conn.commit()<br>    cur.close()<br>    conn.close()<br><br>    return resp_json<br><br><br>@app.post(&quot;/meetings/{meetingId}/participants&quot;)<br>async def add_participant(meetingId: str, participant: Participant):<br>    client_specific_id = f&quot;react-samples::{participant.name.replace(&#39; &#39;, &#39;-&#39;)}-{str(uuid.uuid4())[0:7]}&quot;<br>    payload = participant.dict()<br>    payload.update({&quot;client_specific_id&quot;: client_specific_id})<br>    del payload[&#39;meeting_id&#39;]<br>    resp = await DYTE_API.post(f&#39;/meetings/{meetingId}/participants&#39;, json=payload)<br>    if resp.status_code &gt; 200:<br>        raise HTTPException(status_code=resp.status_code, detail=resp.text)<br>    return resp.text<br><br>if __name__ == &quot;__main__&quot;:<br>    uvicorn.run(&quot;app:app&quot;, host=&quot;localhost&quot;, port=8000, log_level=&quot;debug&quot;, reload=True)</pre><p>This code defines a ⚡️ FastAPI application with an endpoint /detect_faces which takes in a base64 encoded image and returns a boolean value indicating if there is more than one face in the picture.</p><p>It uses the <a href="https://github.com/ageitgey/face_recognition">face_recognition</a> library to detect faces in the image received.</p><p>The code also makes use of the Dyte API for meeting-related operations. 📹</p><p>We can start the backend server simply by using the following command 🧑‍💻:</p><pre>python app.py</pre><p>This Python server helps us create and join meetings, detect multiple faces, and get the list of suspicious candidates. 🕵️</p><p>When we hit the /detect_faces endpoint with an image file encoded as a base64 string, the multiple_detected key of the response would be set to True if there is more than one face in the image, else it will be set to False.</p><p>We can call this from our frontend with the participant’s webcam feed to detect if there is more than one face in the frame.</p><p>With this sorted, let’s return to our React application and create our UI. ✨</p><h3>Step 5: Setting up the meeting UI</h3><p>First, we’ll add our CSS file. Create a new file frontend/src/App.css and paste the following code.</p><pre>.App {<br> text-align: center;<br>}<br><br>.App-logo {<br> height: 40vmin;<br> pointer-events: none;<br>}<br><br>@media (prefers-reduced-motion: no-preference) {<br> .App-logo {<br>  animation: App-logo-spin infinite 20s linear;<br> }<br>}<br><br>.App-header {<br> background-color: #282c34;<br> min-height: 100vh;<br> display: flex;<br> flex-direction: column;<br> align-items: center;<br> justify-content: center;<br> font-size: calc(10px + 2vmin);<br> color: white;<br>}<br><br>.App-link {<br> color: #61dafb;<br>}<br><br>.heading-proctor {<br> font-size: x-large;<br> font-weight: bolder;<br> color: #fff;<br>}<br><br>@keyframes App-logo-spin {<br> from {<br>  transform: rotate(0deg);<br> }<br> to {<br>  transform: rotate(360deg);<br> }<br>}</pre><p>Next, we will add the initial Dyte Meeting component to our app. We can do this by replacing the contents of frontend/src/App.jsx with the following code:</p><pre>import { useEffect, useState } from &quot;react&quot;;<br>import Meet from &quot;./Meet&quot;;<br>import Home from &quot;./Home&quot;;<br>import { BrowserRouter, Routes, Route } from &quot;react-router-dom&quot;;<br>import &quot;./App.css&quot;;<br><br>function App() {<br> const [meetingId, setMeetingId] = useState();<br><br> const createMeeting = async () =&gt; {<br>  const res = await fetch(&quot;http://localhost:8000/meetings&quot;, {<br>   method: &quot;POST&quot;,<br>   body: JSON.stringify({ title: &quot;Joint Entrance Examination&quot; }),<br>   headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br>  });<br>  const resJson = await res.json();<br>  console.log(resJson);<br>  window.localStorage.setItem(&quot;adminId&quot;, resJson.admin_id);<br>  setMeetingId(resJson.data.id);<br> };<br><br> useEffect(() =&gt; {<br>  const id = window.location.pathname.split(&quot;/&quot;)[2];<br>  if (!!!id) {<br>   createMeeting();<br>  }<br> }, []);<br><br> return (<br>  &lt;BrowserRouter&gt;<br>   &lt;Routes&gt;<br>    &lt;Route path=&quot;/&quot; element={&lt;Home meetingId={meetingId} /&gt;}&gt;&lt;/Route&gt;<br>    &lt;Route path=&quot;/meeting/:meetingId&quot; element={&lt;Meet /&gt;}&gt;&lt;/Route&gt;<br>   &lt;/Routes&gt;<br>  &lt;/BrowserRouter&gt;<br> );<br>}<br><br>export default App;</pre><p>This component will create a Dyte Meeting link and an adminId for the admin. We will store the adminId secretly in localstorage. The adminId will be used later for accessing any sensitive data.</p><h3>Home component</h3><p>The home component renders the / route. Create a file as frontend/src/Home.jsx.</p><pre>import { Link } from &quot;react-router-dom&quot;;<br>function Home({ meetingId }) {<br>	return (<br>		&lt;div<br>			style={{<br>				height: &quot;100vh&quot;,<br>				width: &quot;100vw&quot;,<br>				fontSize: &quot;x-large&quot;,<br>				display: &quot;flex&quot;,<br>				justifyContent: &quot;center&quot;,<br>				alignItems: &quot;center&quot;,<br>			}}<br>		&gt;<br>			{meetingId &amp;&amp; !window.location.pathname.split(&quot;/&quot;)[2] &amp;&amp; (<br>				&lt;Link to={`/meeting/${meetingId}`}&gt;Create and Join Meeting&lt;/Link&gt;<br>			)}<br>		&lt;/div&gt;<br>	);<br>}<br><br>export default Home;</pre><h3>Heading component</h3><p>We will now create a file as frontend/src/Heading.jsx.</p><pre>const Heading = ({ text }) =&gt; {<br>	return (<br>		&lt;div<br>			className=&quot;heading-proctor&quot;<br>			style={{<br>				padding: &quot;10px&quot;,<br>				textAlign: &quot;center&quot;,<br>				backgroundColor: &quot;#000&quot;,<br>				borderBottom: &quot;solid 0.5px gray&quot;,<br>			}}<br>		&gt;<br>			{text}<br>		&lt;/div&gt;<br>	);<br>};<br><br>export default Heading;</pre><h3>Utility functions</h3><p>It’s time to add a few utility functions for our application to use later. 🧑‍💻</p><p>We will create a file frontend/src/utils.js with the following functions in it. 👇</p><pre>const joinMeeting = async (id) =&gt; {<br> const res = await fetch(`http://localhost:8000/meetings/${id}/participants`, {<br>  method: &quot;POST&quot;,<br>  body: JSON.stringify({<br>   name: &quot;new user&quot;,<br>   preset_name: &quot;group_call_host&quot;,<br>   meeting_id: meetingId,<br>  }),<br>  headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br> });<br> const resJson = await res.json();<br> console.log(resJson.detail);<br> const data = JSON.parse(resJson.detail);<br> return data.data.token;<br>};<br><br>const getCandidateStatus = async () =&gt; {<br> const response = await fetch(&quot;http://localhost:8000/multiple_faces_list&quot;, {<br>  method: &quot;POST&quot;,<br>  headers: {<br>   &quot;Content-Type&quot;: &quot;application/json&quot;,<br>  },<br>  body: JSON.stringify({<br>   meeting_id: window.location.pathname.split(&quot;/&quot;)[2],<br>   admin_id: window.localStorage.getItem(&quot;adminId&quot;) || &quot;undefined&quot;,<br>  }),<br> });<br> const res = await response.json();<br> if (res.details) return undefined;<br> console.log(res);<br> return res;<br>};<br><br>export { joinMeeting, getCandidateStatus };</pre><p>Delving into the Meet component that renders on route /meeting/:meetingId.</p><p>When the admin clicks on the link provided on the / route, he gets redirected to the meeting page, where we add the user to the meeting as a participant with group_call_host preset. 🤙</p><p>Since this user created the meeting and was redirected to the meet page, we will assign him an admin role. The link from the address bar can be shared with candidates.</p><p>When a candidate opens the shared link, they become a regular user. And for every regular user, the component emits screenshots of the users’ videos directed to our Python server. 🐍</p><pre>/* eslint-disable */<br><br>import { useState, useEffect, useRef } from &quot;react&quot;;<br>import { DyteMeeting, provideDyteDesignSystem } from &quot;@dytesdk/react-ui-kit&quot;;<br>import { useDyteClient } from &quot;@dytesdk/react-web-core&quot;;<br>import Proctor from &quot;./Proctor&quot;;<br>import Heading from &quot;./Heading&quot;;<br>import { SendImageToBackendMiddleware, joinMeeting } from &quot;./utils&quot;;<br>// Constants<br>const SERVER_URL = process.env.SERVER_URL || &quot;http://localhost:8000&quot;;<br>let LAST_BACKEND_PING_TIME = 0;<br>const DETECT_FACES_ENDPOINT = `${SERVER_URL}/detect_faces`;<br>const TIME_BETWEEN_BACKEND_PINGS = 5000;<br><br>const Meet = () =&gt; {<br> const meetingEl = useRef();<br> const [meeting, initMeeting] = useDyteClient();<br> const [userToken, setUserToken] = useState();<br> const [isAdminBool, setAdminBool] = useState(null);<br> const meetingId = window.location.pathname.split(&quot;/&quot;)[2];<br><br> function SendImageToBackendMiddleware() {<br>  return async (canvas, ctx) =&gt; {<br>   const currentTime = Date.now();<br>   if (currentTime - LAST_BACKEND_PING_TIME &gt; TIME_BETWEEN_BACKEND_PINGS) {<br>    LAST_BACKEND_PING_TIME = currentTime;<br>    const imgBase64String = canvas.toDataURL(&quot;image/png&quot;);<br>    const response = await fetch(DETECT_FACES_ENDPOINT, {<br>     method: &quot;POST&quot;,<br>     headers: {<br>      &quot;Content-Type&quot;: &quot;application/json&quot;,<br>     },<br>     body: JSON.stringify({<br>      base64_img: imgBase64String,<br>      participant_id: meeting?.self.id,<br>      participant_name: meeting?.self.name,<br>      meeting_id: meetingId,<br>     }),<br>    });<br>    const res = await response.json();<br>    if (res[&quot;multiple_detected&quot;]) {<br>     console.log(&quot;Warning: Multiple faces detected!&quot;);<br>     sendNotification({<br>      id: &quot;multiple_faces_detected&quot;,<br>      message: &quot;Warning: Multiple faces detected!&quot;,<br>     });<br>    }<br>   }<br>  };<br> }<br><br> const isAdmin = async (id) =&gt; {<br>  const res = await fetch(`${SERVER_URL}/is_admin`, {<br>   method: &quot;POST&quot;,<br>   body: JSON.stringify({<br>    admin_id: window.localStorage.getItem(&quot;adminId&quot;) || &quot;&quot;,<br>    meeting_id: meetingId || &quot;&quot;,<br>   }),<br>   headers: { &quot;Content-Type&quot;: &quot;application/json&quot; },<br>  });<br>  const resJson = await res.json();<br>  setAdminBool(resJson.admin);<br> };<br><br> const joinMeetingId = async () =&gt; {<br>  if (meetingId) {<br>   const authToken = await joinMeeting(meetingId);<br>   await initMeeting({<br>    authToken,<br>   });<br>   setUserToken(authToken);<br>  }<br> };<br><br> useEffect(() =&gt; {<br>  if (meetingId &amp;&amp; !userToken) joinMeetingId();<br>  isAdmin();<br> }, []);<br><br> useEffect(() =&gt; {<br>  if (userToken) {<br>   provideDyteDesignSystem(meetingEl.current, {<br>    theme: &quot;dark&quot;,<br>   });<br>  }<br> }, [userToken]);<br><br> useEffect(() =&gt; {<br>  if (isAdminBool === false &amp;&amp; meeting?.self) {<br>   meeting.self.addVideoMiddleware(SendImageToBackendMiddleware);<br>  }<br> }, [meeting?.self]);<br><br> return (<br>  &lt;div style={{ height: &quot;96vh&quot;, width: &quot;100vw&quot;, display: &quot;flex&quot; }}&gt;<br>   {userToken &amp;&amp; (<br>    &lt;&gt;<br>     {isAdminBool &amp;&amp; (<br>      &lt;div<br>       style={{<br>        width: &quot;40vw&quot;,<br>        height: &quot;100vh&quot;,<br>        overflowY: &quot;scroll&quot;,<br>        backgroundColor: &quot;black&quot;,<br>        borderRight: &quot;solid 0.5px gray&quot;,<br>       }}<br>      &gt;<br>       &lt;Heading text={&quot;Proctoring Information&quot;} /&gt;<br>       &lt;Proctor meeting={meeting} /&gt;<br>      &lt;/div&gt;<br>     )}<br>     {isAdminBool ? (<br>      &lt;div style={{ width: &quot;60vw&quot;, height: &quot;96vh&quot; }}&gt;<br>       &lt;Heading text={&quot;Proctoring Admin Interface&quot;} /&gt;<br>       &lt;DyteMeeting mode=&quot;fill&quot; meeting={meeting} ref={meetingEl} /&gt;<br>      &lt;/div&gt;<br>     ) : (<br>      &lt;div style={{ width: &quot;100vw&quot;, height: &quot;96vh&quot; }}&gt;<br>       &lt;Heading text={&quot;Proctoring Candidate Interface&quot;} /&gt;<br>       &lt;DyteMeeting mode=&quot;fill&quot; meeting={meeting} ref={meetingEl} /&gt;<br>      &lt;/div&gt;<br>     )}<br>    &lt;/&gt;<br>   )}<br>  &lt;/div&gt;<br> );<br>};<br><br>export default Meet;</pre><p>Let’s briefly go through some of the functions:</p><ul><li>isAdmin talks to the Python server to identify whether the current client is an admin.</li><li>joinMeeting adds the current client to the meeting.</li><li>SendImageToBackendMiddleware sends screenshots of candidates&#39; videos to the Python server.</li></ul><h3>Proctor component</h3><p>The proctor component gets activated only for admins and, with the help of adminId, fetches the suspicious candidates&#39; list and renders it in a chat-like format. Create a file frontend/src/Proctor.jsx</p><pre>import { useEffect, useState } from &quot;react&quot;;<br>import { getCandidateStatus } from &quot;./utils&quot;;<br><br>const Proctor = () =&gt; {<br> const [candidateStatuses, updateCandidateStatusState] = useState([]);<br> const [error, setError] = useState(&quot;&quot;);<br><br> const updateCandidateStatus = async () =&gt; {<br>  try {<br>   const res = await getCandidateStatus();<br>   updateCandidateStatusState(res);<br>  } catch (e) {<br>   setError(&quot;User don&#39;t have admin privileges.&quot;);<br>  }<br> };<br><br> useEffect(() =&gt; {<br>  updateCandidateStatus();<br> }, []);<br><br> useEffect(() =&gt; {<br>  if (candidateStatuses?.map) {<br>   const id = setInterval(() =&gt; {<br>    updateCandidateStatus();<br>   }, 2000);<br>   return () =&gt; {<br>    clearInterval(id);<br>   };<br>  }<br> }, [candidateStatuses]);<br><br> return (<br>  &lt;&gt;<br>   &lt;div style={{ padding: &quot;0px 20px&quot; }}&gt;<br>    {candidateStatuses?.map &amp;&amp; candidateStatuses ? (<br>     candidateStatuses.map((status) =&gt; (<br>      &lt;div<br>       style={{<br>        display: &quot;flex&quot;,<br>        justifyContent: &quot;start&quot;,<br>        margin: &quot;50px 20px&quot;,<br>       }}<br>      &gt;<br>       &lt;div style={{ marginRight: &quot;20px&quot; }}&gt;<br>        &lt;img<br>         src=&quot;https://images.yourstory.com/cs/images/companies/Dyte-1608553297314.jpg&quot;<br>         style={{<br>          borderRadius: &quot;50px&quot;,<br>          height: &quot;60px&quot;,<br>          border: &quot;1px double lightblue&quot;,<br>         }}<br>        /&gt;<br>       &lt;/div&gt;<br>       &lt;div<br>        style={{<br>         textAlign: &quot;center&quot;,<br>         padding: &quot;20px&quot;,<br>         backgroundColor: &quot;#2160fd&quot;,<br>         fontSize: &quot;large&quot;,<br>         fontWeight: &quot;400&quot;,<br>         borderRadius: &quot;10px 10px 10px 10px&quot;,<br>         width: &quot;80%&quot;,<br>        }}<br>       &gt;<br>        &lt;div<br>         style={{<br>          color: &quot;white&quot;,<br>          padding: &quot;20px 0px&quot;,<br>          textAlign: &quot;left&quot;,<br>         }}<br>        &gt;<br>         {status[4].split(&quot;&lt;&gt;&quot;).map((text) =&gt; (<br>          &lt;div&gt;{text}&lt;/div&gt;<br>         ))}<br>         &lt;div&gt;Timestamp: {new Date(status[0]).toLocaleString()}&lt;/div&gt;<br>        &lt;/div&gt;<br>        &lt;img<br>         style={{ borderRadius: &quot;10px&quot;, width: &quot;100%&quot; }}<br>         src={status[3]}<br>        /&gt;<br>       &lt;/div&gt;<br>      &lt;/div&gt;<br>     ))<br>    ) : (<br>     &lt;div style={{ color: &quot;white&quot; }}&gt;<br>      Wait or check if you have admin privileges to access the proctoring<br>      dashboard.<br>     &lt;/div&gt;<br>    )}<br>   &lt;/div&gt;<br>  &lt;/&gt;<br> );<br>};<br><br>export default Proctor;</pre><p>To start the React app on the local server, we can run the following command:</p><pre>yarn start</pre><p>If we visit <a href="http://localhost:3000/">http://localhost:3000/</a>, we should see the Dyte meeting in our browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bFyrYcNJMDjerJm-.png" /></figure><h3>Step 6: Adding the face detection logic to the frontend</h3><p>Since now we have a nice backend server to detect faces and a great UI 🎉, we can add the face detection logic to our frontend. For this, we will first add some constants to our previously edited frontend/src/App.jsx file:</p><p>We will be using the above constants in the SendImageToBackendMiddleware function, which we will add to our App component, just after the useDyteClient hook. 🪝</p><p>The SendImageToBackendMiddleware is a <a href="https://dyte.io/blog/streams-blog/">Dyte Video</a> <a href="https://docs.dyte.io/web-core/local-user/extras#using-middlewares">Middleware</a>. Middlewares are add-ons that you can use to easily add effects and filters to your audio and video streams.</p><p>Here, we use the middleware functionality to get the canvas object of the participant’s webcam feed, convert it to a base64 encoded image, and send it to our backend server. We also ensure that the backend is pinged only once every 30 seconds to avoid unnecessary load on the server.</p><p>We then use the sendNotification function to <a href="https://docs.dyte.io/react-ui-kit/reference#sendnotification">send a notification</a> to the participant if the backend returns True for the multiple_detected key of the response.</p><p>That was all the code we needed to add basic live proctoring functionality to our Dyte meeting. 👍</p><p>The app sends a screenshot of the participant’s webcam feed to the backend server every 30 seconds, and if the backend detects more than one face in the image, it sends a warning notification to the projector. ⚠️</p><p>The backend also logs the participant’s ID and the time of the detection in the terminal. This can be used to keep track of the participants who may have cheated during the meeting for later review.</p><h3>Step 7: Trying out our live proctoring system</h3><p>Ta-da! 🎩✨ It’s time to put our live proctoring system to the test and see it in action!</p><ul><li>First, let us look at the candidate’s view; the candidate can see that the proctor is in the meeting but cannot see the Proctoring Panel. 🧑‍💻</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eQriQ8Sh4k4B4Fac.png" /></figure><ul><li>In the proctor’s view, we can see the details (live proctoring information) with proof when two people are in the candidate’s webcam view. 🙌</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O2HzuuMS37X7FoQt.png" /></figure><p>🧑‍💻 You can try out the live proctoring system <a href="https://dyte-multi-faces.netlify.app/">here</a>. And here’s <a href="https://github.com/dyte-io/proctored-live-interviews">the link</a> to the repository for you to take a look at the whole codebase.</p><h3>Conclusion</h3><p>Celebrate! 🎉✨ We’ve built a powerful live proctoring system with Dyte, ensuring integrity and fairness in online exams and interviews. But that’s not all! We can now create our own customized online classroom or meeting platform.</p><p>We can now use this system to proctor our online exams and interviews. ✍️</p><p>The possibilities are endless with Dyte, go ahead and try bringing your own ideas to life by visiting <a href="https://dyte.io/">dyte.io</a>! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4039521b5d6f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dyte/building-a-live-proctoring-system-with-react-and-dyte-4039521b5d6f">Building a Live Proctoring System With React and Dyte</a> was originally published in <a href="https://medium.com/dyte">Dyte</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Live Auction Platform With React and Dyte]]></title>
            <link>https://medium.com/dyte/building-a-live-auction-platform-with-react-and-dyte-c2eb69c539ef?source=rss----9477120405bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/c2eb69c539ef</guid>
            <category><![CDATA[video-sdk]]></category>
            <category><![CDATA[video-calling-software]]></category>
            <category><![CDATA[video-calling]]></category>
            <dc:creator><![CDATA[Ishita Kabra]]></dc:creator>
            <pubDate>Fri, 17 Nov 2023 08:59:31 GMT</pubDate>
            <atom:updated>2023-11-17T08:59:31.849Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*ydgtcpttUII1kcX4_OsSGA.png" /></figure><p>In today’s digital landscape, the art of buying and selling has taken a remarkable leap forward with the advent of real-time online marketplaces and auctions. A live auction is an interactive bidding process in real-time, facilitated through digital platforms. Participants place live bids on goods or services within a defined timeframe, and the highest bid at the end of the auction wins the item or service being auctioned.</p><p>We want to help people create engaging live experiences with Dyte, and live auctions just happened to be on the market. By blending technology with real-time bidding, in this blog, we’ll delve into the essentials and take you through the application flow of building and running your own live auction platform before the fall of the hammer.</p><p>Let’s discover the products up for sale before the bidding starts!</p><h3>Why build a live auction platform?</h3><ul><li>Convenient: Participate in live auctions from anywhere, at any time.</li><li>Real-time bidding: Experience the thrill of bidding against other enthusiasts in real-time.</li><li>Wide range of items: Discover diverse items, from collectibles to artwork and more.</li><li>Seamless integration: The live auction app provides users with a smooth and uninterrupted auction experience with Dyte’s reliable audio/video conferencing and customizable UI.</li></ul><h3>Before you start</h3><ul><li>Basic knowledge of <a href="https://react.dev/">React.js</a> is required to build this application.</li><li>Please ensure that <a href="https://nodejs.org/">Node.js</a> is installed on your machine. We will use it to run our application.</li><li>Lastly, you will need the API Key and organization ID from the <a href="https://dev.dyte.io/">developer portal</a> to authenticate yourself when you call Dyte’s REST APIs.</li></ul><h3>Building the live bidding platform</h3><h3>Installation</h3><p>You need to install Dyte’s React UI Kit and Core packages to get started. You can do so by using npm or Yarn.</p><pre>npm install @dytesdk/react-ui-kit @dytesdk/react-web-core</pre><h3>Getting started</h3><p>We will first fetch the organization ID and API Key from the <a href="https://dev.dyte.io/">developer portal</a>. Then, we’ll create an account and navigate to the API Keys page.</p><p>Please ensure that you do not upload your API Key anywhere.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YVQJcHcHNIi-_WJq.png" /></figure><p>Next, we will create a meeting using the following <a href="https://www.notion.so/demo-docs-dyte-io-6f55d86117c84c1b8bcba43f97917da4?pvs=21">Rest API</a>. Here is a sample response.</p><pre>{<br>  &quot;success&quot;: true,<br>  &quot;data&quot;: {<br>    &quot;id&quot;: &quot;497f6eca-6276-4993-bfeb-53cbbbbaxxxx&quot;,<br>    &quot;name&quot;: &quot;string&quot;,<br>    &quot;picture&quot;: &quot;&lt;http://example.com&gt;&quot;,<br>    &quot;custom_participant_id&quot;: &quot;string&quot;,<br>    &quot;preset_name&quot;: &quot;string&quot;,<br>    &quot;created_at&quot;: &quot;2019-08-24T14:15:22Z&quot;,<br>    &quot;updated_at&quot;: &quot;2019-08-24T14:15:22Z&quot;,<br>    &quot;token&quot;: &quot;string&quot;<br>  }<br>}</pre><p>We use the ID from the payload to generate an auth token using the <a href="https://docs.dyte.io/api#/operations/add_participant">Add Participants API</a>. This auth token is used to initialize a Dyte client. Let’s start setting up the project.</p><h3>Build your custom UI</h3><p>Create a file src/App.tsx. We will use the DyteMeeting hook to initialize a new Dyte meeting. The DyteProvider is used to pass the meeting object to all child components inside this application. We will also set up event listeners for joining and leaving the room.</p><pre>import { useEffect, useState } from &#39;react&#39;;<br>import { DyteProvider, useDyteClient } from &#39;@dytesdk/react-web-core&#39;;<br>import { LoadingScreen } from &#39;./pages&#39;;<br>import { Meeting, SetupScreen } from &#39;./pages&#39;;<br><br>function App() {<br>  const [meeting, initMeeting] = useDyteClient();<br>  const [roomJoined, setRoomJoined] = useState&lt;boolean&gt;(false);<br><br>  useEffect(() =&gt; {<br>    const searchParams = new URL(window.location.href).searchParams;<br>    const authToken = searchParams.get(&#39;authToken&#39;);<br><br>    if (!authToken) {<br>      alert(<br>        &quot;An authToken wasn&#39;t passed, please pass an authToken in the URL query to join a meeting.&quot;<br>      );<br>      return;<br>    }<br><br>    initMeeting({<br>      authToken,<br>      defaults: {<br>        audio: false,<br>        video: false,<br>      },<br>    });<br>  }, []);<br><br>  useEffect(() =&gt; {<br>    if (!meeting) return;<br><br>    const roomJoinedListener = () =&gt; {<br>      setRoomJoined(true);<br>    };<br>    const roomLeftListener = () =&gt; {<br>      setRoomJoined(false);<br>    };<br>    meeting.self.on(&#39;roomJoined&#39;, roomJoinedListener);<br>    meeting.self.on(&#39;roomLeft&#39;, roomLeftListener);<br><br>    return () =&gt; {<br>      meeting.self.removeListener(&#39;roomJoined&#39;, roomJoinedListener);<br>      meeting.self.removeListener(&#39;roomLeft&#39;, roomLeftListener);<br>    }<br><br>  }, [meeting])<br><br>  return (<br>    &lt;DyteProvider value={meeting} fallback={&lt;LoadingScreen /&gt;}&gt;<br>      {<br>        !roomJoined ? &lt;SetupScreen /&gt; : &lt;Meeting /&gt;<br>      }<br>    &lt;/DyteProvider&gt;<br>  )<br>}</pre><p>Now, we will build a setup screen; this is the first page that the user will see.</p><p>Create a file src/pages/setupScreen/setupScreen.tsx. We will update the user&#39;s display name and join the Dyte meeting from this page.</p><pre>import { useEffect, useState } from &#39;react&#39;<br>import &#39;./setupScreen.css&#39;<br>import { useDyteMeeting } from &#39;@dytesdk/react-web-core&#39;;<br>import {<br>  DyteAudioVisualizer,<br>  DyteAvatar,<br>  DyteCameraToggle,<br>  DyteMicToggle,<br>  DyteNameTag,<br>  DyteParticipantTile,<br>} from &#39;@dytesdk/react-ui-kit&#39;;<br><br>const SetupScreen = () =&gt; {<br>  const { meeting } = useDyteMeeting();<br>  const [isHost, setIsHost] = useState&lt;boolean&gt;(false);<br>  const [name, setName] = useState&lt;string&gt;(&#39;&#39;);<br><br>  useEffect(() =&gt; {<br>    if (!meeting) return;<br>    const preset = meeting.self.presetName;<br>    const name = meeting.self.name;<br>    setName(name);<br><br>    if (preset.includes(&#39;host&#39;)) {<br>      setIsHost(true);<br>    }<br>  }, [meeting])<br><br>  const joinMeeting = () =&gt; {<br>    meeting?.self.setName(name);<br>    meeting.joinRoom();<br>  }<br><br>  return (<br>    &lt;div className=&#39;setup-screen&#39;&gt;<br>      &lt;div className=&quot;setup-media&quot;&gt;<br>        &lt;div className=&quot;video-container&quot;&gt;<br>          &lt;DyteParticipantTile meeting={meeting} participant={meeting.self}&gt;<br>            &lt;DyteAvatar size=&quot;md&quot; participant={meeting.self}/&gt;<br>            &lt;DyteNameTag meeting={meeting} participant={meeting.self}&gt;<br>              &lt;DyteAudioVisualizer size=&#39;sm&#39; slot=&quot;start&quot; participant={meeting.self} /&gt;<br>            &lt;/DyteNameTag&gt;<br>            &lt;div className=&#39;setup-media-controls&#39;&gt;<br>              &lt;DyteMicToggle size=&quot;sm&quot; meeting={meeting}/&gt;<br>              &amp;ensp;<br>              &lt;DyteCameraToggle size=&quot;sm&quot; meeting={meeting}/&gt;<br>            &lt;/div&gt;<br>          &lt;/DyteParticipantTile&gt;<br>        &lt;/div&gt;<br>      &lt;/div&gt;<br>      &lt;div className=&quot;setup-information&quot;&gt;<br>        &lt;div className=&quot;setup-content&quot;&gt;<br>          &lt;h2&gt;Welcome! {name}&lt;/h2&gt;<br>          &lt;p&gt;{isHost ? &#39;You are joining as a Host&#39; : &#39;You are joining as a bidder&#39;}&lt;/p&gt;<br>          &lt;input disabled={!meeting.self.permissions.canEditDisplayName ?? false} className=&#39;setup-name&#39; value={name} onChange={(e) =&gt; {<br>            setName(e.target.value)<br>          }} /&gt;<br>          &lt;button className=&#39;setup-join&#39; onClick={joinMeeting}&gt;<br>            Join Meeting<br>          &lt;/button&gt;<br>        &lt;/div&gt;<br>      &lt;/div&gt;<br>    &lt;/div&gt;<br>  )<br>}<br><br>export default SetupScreen</pre><p>Now that we have the basic setup let’s build the live auction platform.</p><p>Create a file src/pages/meeting/Meeting.tsx.</p><p>Our live auction app will implement the following functionality:</p><ul><li>Give hosts an option to start/stop the auction.</li><li>Give hosts an option to navigate between different auction products.</li><li>Allow users to make real-time bids for each product.</li><li>Show the highest bid to all users.</li></ul><pre>import { useEffect, useState } from &#39;react&#39;<br>import &#39;./meeting.css&#39;<br>import {<br>  DyteCameraToggle,<br>  DyteChatToggle,<br>  DyteGrid,<br>  DyteHeader,<br>  DyteLeaveButton,<br>  DyteMicToggle,<br>  DyteNotifications,<br>  DyteParticipantsAudio,<br>  DyteSidebar,<br>  sendNotification,<br>} from &#39;@dytesdk/react-ui-kit&#39;<br>import { useDyteMeeting } from &#39;@dytesdk/react-web-core&#39;;<br>import { AuctionControlBar, Icon } from &#39;../../components&#39;;<br>import { bidItems } from &#39;../../constants&#39;;<br><br>interface Bid {<br>  bid: number;<br>  user: string;<br>}<br><br>const Meeting = () =&gt; {<br>  const { meeting } = useDyteMeeting();<br><br>  const [item, setItem] = useState(0);<br>  const [isHost, setIsHost] = useState&lt;boolean&gt;(false);<br>  const [showPopup, setShowPopup] = useState&lt;boolean&gt;(true);<br>  const [auctionStarted, setAuctionStarted] = useState&lt;boolean&gt;(false);<br>  const [activeSidebar, setActiveSidebar] = useState&lt;boolean&gt;(false);<br>  const [highestBid, setHighestBid] = useState&lt;Bid&gt;({ bid: 100, user: &#39;default&#39; });<br><br>  const handlePrev = () =&gt; {<br>    if (item - 1 &lt; 0) return;<br>    setItem(item - 1)<br>    meeting.participants.broadcastMessage(&#39;item-changed&#39;, { item: item - 1 })<br>  }<br>  const handleNext = () =&gt; {<br>    if ( item + 1 &gt;= bidItems.length) return;<br>    setItem(item + 1)<br>    meeting.participants.broadcastMessage(&#39;item-changed&#39;, { item: item + 1 })<br>  }<br><br>  useEffect(() =&gt; {<br>    setHighestBid({<br>      bid: bidItems[item].startingBid,<br>      user: &#39;default&#39;<br>    })<br>  }, [item])<br><br>  useEffect(() =&gt; {<br>    if (!meeting) return;<br><br>    const preset = meeting.self.presetName;<br>    if (preset.includes(&#39;host&#39;)) {<br>      setIsHost(true);<br>    }<br><br>    const handleBroadcastedMessage = ({ type, payload }: { type: string, payload: any }) =&gt; {<br>      switch(type) {<br>        case &#39;auction-toggle&#39;: {<br>          setAuctionStarted(payload.started);<br>          break;<br>        }<br>        case &#39;item-changed&#39;: {<br>          setItem(payload.item);<br>          break;<br>        }<br>        case &#39;new-bid&#39;: {<br>          sendNotification({<br>            id: &#39;new-bid&#39;,<br>            message: `${payload.user} just made a bid of $ ${payload.bid}!`,<br>            duration: 2000,<br>          })<br>          if (parseFloat(payload.bid) &gt; highestBid.bid) setHighestBid(payload)<br>          break;<br>        }<br>        default:<br>          break;<br>      }<br>    }<br>    meeting.participants.on(&#39;broadcastedMessage&#39;, handleBroadcastedMessage);<br><br>    const handleDyteStateUpdate = ({detail}: any) =&gt; {<br>        if (detail.activeSidebar) {<br>         setActiveSidebar(true);<br>        } else {<br>          setActiveSidebar(false);<br>        }<br>    }<br><br>    document.body.addEventListener(&#39;dyteStateUpdate&#39;, handleDyteStateUpdate);<br><br>    return () =&gt; {<br>      document.body.removeEventListener(&#39;dyteStateUpdate&#39;, handleDyteStateUpdate);<br>      meeting.participants.removeListener(&#39;broadcastedMessage&#39;, handleBroadcastedMessage);<br>    }<br>  }, [meeting])<br><br>  useEffect(() =&gt; {<br>    const participantJoinedListener = () =&gt; {<br>      if (!auctionStarted) return;<br>      setTimeout(() =&gt; {<br>        meeting.participants.broadcastMessage(&#39;auction-toggle&#39;, {<br>          started: auctionStarted<br>        })<br>      }, 500)<br>    <br>    }<br>    meeting.participants.joined.on(&#39;participantJoined&#39;, participantJoinedListener);<br>    return () =&gt; {<br>      meeting.participants.joined.removeListener(&#39;participantJoined&#39;, participantJoinedListener);<br>    }<br>  }, [meeting, auctionStarted])<br><br>  const toggleAuction = () =&gt; {<br>    if (!isHost) return;<br>    meeting.participants.broadcastMessage(&#39;auction-toggle&#39;, {<br>      started: !auctionStarted<br>    })<br>    if (!auctionStarted) {<br>      meeting.self.pin();<br>    } else {<br>      meeting.self.unpin();<br>    }<br>    setAuctionStarted(!auctionStarted);<br>  }<br><br>  return (<br>    &lt;div className=&#39;meeting-container&#39;&gt;<br>      &lt;DyteParticipantsAudio meeting={meeting} /&gt;<br>      &lt;DyteNotifications meeting={meeting} /&gt;<br><br>      &lt;DyteHeader meeting={meeting} size=&#39;lg&#39;&gt;<br>        &lt;div className=&quot;meeting-header&quot;&gt;<br>          {<br>            auctionStarted &amp;&amp; (<br>              &lt;div className=&quot;show-auction-popup&quot; onClick={() =&gt; setShowPopup(() =&gt; !showPopup)}&gt;<br>                &lt;Icon size=&#39;sm&#39; icon={showPopup ? &#39;close&#39; : &#39;next&#39;} /&gt;<br>              &lt;/div&gt;<br>            )<br>          }<br>        &lt;/div&gt;<br>      &lt;/DyteHeader&gt;<br><br>      &lt;div className=&#39;meeting-grid&#39;&gt;<br>        {<br>          auctionStarted &amp;&amp; (<br>            &lt;div className={`auction-container ${!showPopup ? &#39;hide-auction-popup&#39; : &#39;&#39;}`}&gt;<br>              &lt;img className=&#39;auction-img&#39; src={bidItems[item].link} /&gt;<br>              &lt;div className=&#39;auction-desc&#39;&gt;<br>                {bidItems[item].description}<br>              &lt;/div&gt;<br>              &lt;AuctionControlBar<br>                item={item}<br>                highestBid={highestBid}<br>                handleNext={handleNext}<br>                handlePrev={handlePrev}<br>                isHost={isHost}<br>              /&gt;<br>          &lt;/div&gt;<br>          )<br>        }<br>        &lt;DyteGrid layout=&#39;column&#39; meeting={meeting} style={{ height: &#39;100%&#39; }}/&gt;<br>        {activeSidebar &amp;&amp; &lt;DyteSidebar meeting={meeting} /&gt;}<br>      &lt;/div&gt;<br><br>      &lt;div className=&#39;meeting-controlbar&#39;&gt;<br>        &lt;DyteMicToggle size=&#39;md&#39; meeting={meeting} /&gt;<br>        &lt;DyteCameraToggle size=&#39;md&#39;  meeting={meeting} /&gt;<br>        &lt;DyteLeaveButton size=&#39;md&#39; /&gt;<br>        &lt;DyteChatToggle size=&#39;md&#39; meeting={meeting} /&gt;<br>        {<br>          isHost &amp;&amp; (<br>            &lt;button className=&#39;auction-toggle-button&#39; onClick={toggleAuction}&gt;<br>              &lt;Icon size=&#39;lg&#39; icon=&#39;auction&#39; /&gt;<br>              {auctionStarted ? &#39;Stop&#39; : &#39;Start&#39;} Auction<br>            &lt;/button&gt;<br>          )<br>        }<br>      &lt;/div&gt;<br>    &lt;/div&gt;<br>  )<br>}<br><br>export default Meeting</pre><p>Et voila! Our live auction app is ready.</p><p>You can play around with this live bidding app to get you going with some real-time bidding at — <a href="https://dyte-live-bidding.vercel.app/">https://dyte-live-bidding.vercel.app/</a>! Dyte going once, going twice…</p><p>You can check out the complete code for the project here.</p><p>Are you sold yet, or do we need to say more?</p><p>Conclusion<br>With this, we have built our live auction platform. All you have to do is pull out your gavel and shout at the top of your lungs. We look forward to you outbidding us!</p><p>If you have any thoughts or feedback, please reach out to us on LinkedIn and Twitter. Stay tuned for more related blog posts in the future!</p><p>Get better insights on leveraging Dyte’s technology and discover how it can revolutionize your app’s communication capabilities with its SDKs. Head over to dyte.io to learn how to start quickly on your 10,000 free minutes, which renew every month. You can reach us at <a href="/service/mailto:support@dyte.io">support@dyte.io</a> or ask our developer community if you have any questions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2eb69c539ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dyte/building-a-live-auction-platform-with-react-and-dyte-c2eb69c539ef">Building a Live Auction Platform With React and Dyte</a> was originally published in <a href="https://medium.com/dyte">Dyte</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Relic Alerts by Terraform]]></title>
            <link>https://medium.com/dyte/new-relic-alerts-by-terraform-ad7ac7eff1d4?source=rss----9477120405bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/ad7ac7eff1d4</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[new-relic]]></category>
            <dc:creator><![CDATA[Milap Singh]]></dc:creator>
            <pubDate>Fri, 13 Oct 2023 12:43:34 GMT</pubDate>
            <atom:updated>2023-10-13T12:43:36.096Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pidGgbAdqe7mAC7K.png" /></figure><p>At Dyte, we have recognized the potential of Terraform in streamlining our alert setup process. By adopting Terraform, we have empowered our engineering teams to set alerts for their respective services without relying on the SRE/DevOps team.</p><p>Setting up alerts on New Relic can be tedious and repetitive, requiring manual effort. But with the advent of Terraform, New Relic has started supporting the creation of alerts by Terraform.</p><p>Adopting Terraform has also enabled us to better manage our New Relic alerts, with greater control and flexibility in setting up thresholds and conditions. This has led to a more efficient and streamlined process, allowing our teams to focus on other essential tasks. Overall, it has been a game-changer for us, helping us to automate and optimize our alert setup process.</p><h3>What was our problem in setting New Relic alerts for services?</h3><p>Our infrastructure consists of <a href="https://dyte.io/blog/multi-region-resiliency/">multiple clusters</a> offering a range of services. However, we are encountering an issue when setting NRQL (New Relic query language) alerts with different thresholds while minimizing the use of repeated codes. Fortunately, we have discovered that Terraform provides New Relic providers ( <a href="https://registry.terraform.io/providers/newrelic/newrelic/latest">Terraform providers</a>), which can help address this concern significantly. By leveraging Terraform’s New Relic capabilities, we can easily configure our NRQL alerts with different thresholds across all of our clusters without having to write the same code over and over again.</p><h3>How have we done it?</h3><p>NRQL queries will allow us to filter and aggregate data from our applications, infrastructure, and services, enabling us to create targeted alerts specific to our needs. In provider.tf, we&#39;ll first set up the provider.</p><pre>terraform { required_version = &quot;&gt;= 0.12&quot; required_providers { newrelic = { source = &quot;newrelic/newrelic&quot; version = &quot;~&gt; 3.11.0&quot; } } }</pre><p>We used simple local variables with looping to simplify setting up New Relic alerts for our various services. Each variable object was given specific values based on the corresponding service. This ensures that the alerting system is tailored to each service’s specific needs, improving our systems’ overall performance and stability.</p><p>Overall, this approach has made it much easier for our engineering teams to set up their own alerts without relying on SRE/DevOps teams.</p><pre>locals { alerts = { cluster-1 = { cluster_name = &quot;cluster-1&quot; policy_id = &quot;policy_id&quot; threshold = 5, threshold_duration = 300, aggregation_window = 60, description = &quot;Description about alerts&quot;, operator = &quot;above&quot; }, cluster-2 = { cluster_name = &quot;production-us&quot; policy_id = data.newrelic_alert_policy.production_infra_alert_policy.id threshold = 5, threshold_duration = 300, aggregation_window = 60, description = &quot;Log link: https://one.newrelic.com/logger?account=xxxxx&amp;duration=1800000&amp;state=xxxxxxxxxxxxx&quot;, operator = &quot;above&quot; } }</pre><p>We are creating a resource by looping this local variable.</p><pre>resource &quot;newrelic_nrql_alert_condition&quot; &quot;alert&quot; { for_each = local.alerts account_id = var.newrelic_account_id policy_id = each.value.policy_id type = &quot;static&quot; name = &quot;${each.value.cluster_name}-error-percentage&quot; description = each.value.description enabled = true violation_time_limit_seconds = 1800 aggregation_window = each.value.aggregation_window aggregation_method = &quot;event_flow&quot; aggregation_delay = 1 open_violation_on_expiration = false close_violations_on_expiration = false nrql { query = &quot;SELECT percentage(count(*),where message like &#39;%error%&#39; AND message NOT LIKE &#39;%ignore%&#39;) FROM Log WHERE cluster_name = &#39;${each.value.cluster_name}&#39;&quot; } critical { operator = each.value.operator threshold = each.value.threshold threshold_duration = each.value.threshold_duration threshold_occurrences = &quot;ALL&quot; } }</pre><h3>How are we using heredoc(EOT) for the NRQL query?</h3><p>When working with complex queries, reading and understanding the code can be difficult when it is all in one line. To help with this issue, we can use heredocs (the EOT is the delimiting indicator, for more detail, visit this <a href="https://en.wikipedia.org/wiki/Here_document">URL</a>) to define and write multi-line queries. This is especially useful when creating NRQL queries with multiple conditions, as it allows us to separate each condition and make the query easier to comprehend.</p><p><strong>The Syntax for NRQL heredocs query</strong></p><pre>query = &lt;&lt;EOT SELECT count(*) FROM Log WHERE cluster_name = &#39;${local.cluster}&#39; AND `container_name` = &#39;Name&#39; AND level=&#39;ERROR&#39; AND NOT ( `message` LIKE &#39;%condtions name 1 %&#39; OR `message` LIKE &#39;%condtions name 1 %&#39; ) EOT</pre><p>We used heredoc for our NRQL query to make it clearer and readable for human eyes. As a code reviewer, you should be able to identify the query change easily; if the query is in one line, it’s hard to compare.</p><p>A problem we faced with the heredoc syntax &lt;&lt;EOT...EOT is that with this, your query formed on the New Relic console is with leading spaces, which is, again, not so readable. Then we introduced &lt;&lt;-EOT...EOT, which removes all leading spaces from the query.</p><h3>How to enrich alerts with the information we are receiving?</h3><p>To ensure that our live-support team can quickly and accurately identify the issue and affected service, we have conducted numerous iterations to define the alerts we receive. By doing so, we have focused on providing rich and detailed information within each alert, including contextual details, relevant error codes, and any other pertinent data that could assist the support team in resolving the issue.</p><p>We started to define New Relic alerts in the description part of the alert in the IaC (Infrastructure as Code) code itself with information. You can see these values with the alert tag.</p><ul><li>{{tags.podName}}</li><li>{{tags.cluster_name}}</li><li>Log link for the particular application</li></ul><p>Visit <a href="https://docs.newrelic.com/docs/alerts-applied-intelligence/new-relic-alerts/advanced-alerts/advanced-techniques/alert-custom-violation-descriptions/">New Relic</a> for more detail.</p><h3>How is the CI/CD flow working?</h3><p>The core problem with standard infra provisioning is that other teams are blocked on the DevOps team for their tasks or features. As DevOps, we like to automate things and resolve all kinds of blockers. And to solve this particular problem, we implemented CI/CD by leveraging GitHub actions.</p><p>With automating the whole CI/CD pipeline for infra, we have unblocked all other teams to code the resources as they need and simply raise the PR on our infra GitHub repo. After raising the PR, there are some steps to provision the infra:</p><ol><li>Developers can raise PR to &lt;main&gt;, which triggers GitHub Action for terraform plan; terraform plan is added as a comment on PR itself.</li><li>A DevOps team member will review and verify the changes.</li><li>If all the changes seem desired, the DevOps team can pass the “apply” comment, which further triggers GitHub action for Terraform apply.</li></ol><p><strong>Note:</strong> We are using Terraform Cloud to store our Terraform state files.</p><h3>Conclusion</h3><p>Setting alerts manually is really hectic and repetitive task for DevOps/SRE teams in any organization. In this blog, we explained how we can leverage Terraform to create NRQL alerts with the help of code in an automated manner. We have also covered heredocs to make NRQL queries more readable, and easy to understand. In the last part, we illustrated the CI/CD part of IaC, and how easily other dev teams can add alerts for their applications without depending upon SRE/DevOps team.</p><p>I hope you found this post resourceful. If you have any thoughts or feedback, please get in touch with me on <a href="https://www.linkedin.com/in/milap-singh-09740930/">LinkedIn</a> and <a href="https://twitter.com/imilapsingh">Twitter</a>. Stay tuned for more related blog posts in the future!</p><p><em>Originally published at </em><a href="https://dyte.io/blog/new-relic-alerts-terraform/"><em>https://dyte.io</em></a><em> on August 16, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad7ac7eff1d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dyte/new-relic-alerts-by-terraform-ad7ac7eff1d4">New Relic Alerts by Terraform</a> was originally published in <a href="https://medium.com/dyte">Dyte</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrate From Twilio Video Using Dyte’s Twilio Shim]]></title>
            <link>https://medium.com/dyte/migrate-from-twilio-video-using-dytes-twilio-shim-31e54989692f?source=rss----9477120405bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/31e54989692f</guid>
            <category><![CDATA[dyte]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[webrtc]]></category>
            <category><![CDATA[twilio]]></category>
            <dc:creator><![CDATA[Rohan Mukherjee]]></dc:creator>
            <pubDate>Fri, 13 Oct 2023 12:42:16 GMT</pubDate>
            <atom:updated>2023-10-13T12:42:15.920Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*LMzUHh4MeFY5k058H3DCBw.png" /></figure><p>Our business has been fortunate enough to have garnered the interest of several customers. These customers have expressed their satisfaction with our products and services, which we take great pride in providing.</p><p>Dyte has been gaining popularity among customers who require reliable cloud communication services. Many of these customers, who were initially using Twilio Video, have been making the switch to Dyte, possibly owing to our call quality, reliability, expeditious customer support, and pricing.</p><p>This guide presents an approach to help users smoothly migrate from Twilio Video to Dyte with the help of Shims.</p><h3>Thinking of migrating from Twilio Video?</h3><p>Since you are reading this page, I understand that you are probably considering a switch from Twilio Video to another video provider. In recent times, a lot of our customers migrated from Twilio Video to Dyte. They made this choice due to Dyte’s compelling advantages, which include:</p><ul><li>Easy to use APIs with high-level abstractions and inbuilt access control — fewer bugs and less maintenance required</li><li>UI components library — responsive components for React, Angular, WebComponents, React Native, Flutter, Native Android, and iOS. Build real-time applications faster.</li><li>Better media performance — with Dyte’s global mesh network of media servers, calls spanning multiple geographies see a significantly better experience.</li><li>Better support — dedicated Slack support, code reviews, and technical assistance 24/7.</li></ul><p>You can view a comprehensive feature <a href="https://dyte.io/twilio-video-competitor">comparison between Dyte and Twilio Video</a>.</p><p>Once you’ve reviewed the reasons behind migrating from Twilio Video, it becomes evident that Dyte will likely meet all your requirements!</p><p>However, transitioning from one SDK to another can be a demanding process, considering the time and effort involved. Will migrating from Twilio Video to Dyte entail a similar level of effort? The answer is — No.</p><p>Enter Dyte’s Twilio Shim, which simplifies the migration.</p><h3>Introducing our Shim library</h3><p>Even though Dyte can be easily integrated (don’t believe me? Check out our <a href="https://docs.dyte.io/">docs</a>) into your codebase, some developers may find the process cumbersome and time-consuming as they have to get rid of their old code and refactor it to fit a new SDK.</p><p>To make the lives of these developers easier, <strong>we created a shim library</strong> — <a href="https://www.npmjs.com/package/@dytesdk/twilio-shim">@dytesdk/twilio-shim</a>.</p><p>You may be wondering, what is a shim? According to <a href="https://en.wikipedia.org/wiki/Shim_(computing)">Wikipedia</a>:</p><blockquote><em>In computer programming, a shim is a library that transparently intercepts API calls and changes the arguments passed, handles the operation itself or redirects the operation elsewhere. Shims can be used to support an old API in a newer environment, or a new API in an older environment.</em></blockquote><p>Basically, we created a wrapper around Dyte’s SDK with a Twilio Video-compatible external API as a drop-in <a href="https://dyte.io/twilio-video-competitor">replacement for the Twilio Video SDK</a>. You can use the same classes, methods, and other APIs that you used with Twilio Video, but behind the scenes, the shim will perform some magic and translate it to Dyte API calls.</p><p>For instance, if you were to connect to a room using Twilio Video and login to the console whenever a new participant joined the room, this is what your code would look like:</p><pre>const { connect } = require(&#39;twilio-video&#39;);<br><br>const authToken = &#39;&lt;AUTH_TOKEN_FROM_TWILIO_API&gt;&#39;;<br><br>const room = await connect(authToken, {audio: false, video: true});<br>console.log(`Successfully joined a Room: ${room.name}`);<br>console.log(room);<br>room.on(&#39;participantConnected&#39;, participant =&gt; {<br>    console.log(`A remote Participant connected: ${participant}`);<br>});</pre><p>The following code snippet demonstrates how you would do it with the shim:</p><pre>const { connect } = require(&#39;@dytesdk/twilio-shim&#39;);<br><br>const authToken = &#39;&lt;AUTH_TOKEN_FROM_DYTE_API&gt;&#39;;<br><br>const room = await connect(authToken, {audio: false, video: true});<br>console.log(`Successfully joined a Room: ${room.name}`);<br>console.log(room);<br>room.on(&#39;participantConnected&#39;, participant =&gt; {<br>    console.log(`A remote Participant connected: ${participant}`);<br>});</pre><p>Can you spot the difference?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/0*y7NukOy5da3CV6WO.png" /></figure><p>As mentioned previously, the external API is almost identical to Twilio Video. However, intended actions, such as joining a room, are performed internally using Dyte’s <a href="https://docs.dyte.io/web-core">web-core</a> SDK.</p><p>You can check out more examples of using our Twilio shim <a href="https://www.npmjs.com/package/@dytesdk/twilio-shim">here</a>.</p><p>Here is the complete API compatibility list of methods and classes we currently translate</p><p><a href="https://docs.dyte.io/guides/migration/twilio/compatibility-shim-twilio">https://docs.dyte.io/guides/migration/twilio/compatibility-shim-twilio</a></p><p>Also, another important note is that this library should just be used to test out the integration, and clients should plan to eventually move to our SDK natively to utilize Dyte’s capabilities fully</p><h3>That’s all, folks!</h3><p>That was a quick and easy way to switch from Twilio Video — just like integrating Dyte into your project. We are working on creating migration guides for other commonly used real-time communication APIs, such as live streaming and AI-powered chat, among others. If you would like us to create a migration guide from your current video provider, you can contact us on our <a href="https://community.dyte.io/">community Discord</a>.</p><p><em>If you haven’t heard about Dyte yet, head over to </em><a href="https://dyte.io/"><em>dyte.io</em></a><em> to learn how we are revolutionizing communication through our </em><a href="https://dyte.io/video-sdk"><em>SDKs</em></a><em> and libraries and how you can </em><a href="https://accounts.dyte.in/auth/register"><em>get started</em></a><em> quickly on your 10,000 free minutes, which renew every month. You can reach us at </em><a href="/service/mailto:support@dyte.io"><em>support@dyte.io</em></a><em> or ask our </em><a href="https://community.dyte.io/"><em>developer community</em></a><em> if you have any questions.</em></p><p><em>Originally published at </em><a href="https://dyte.io/blog/migrate-twilio-video-dyte-shim/"><em>https://dyte.io</em></a><em> on August 8, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31e54989692f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dyte/migrate-from-twilio-video-using-dytes-twilio-shim-31e54989692f">Migrate From Twilio Video Using Dyte’s Twilio Shim</a> was originally published in <a href="https://medium.com/dyte">Dyte</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Packaging Libraries in iOS: A Comprehensive Guide]]></title>
            <link>https://medium.com/dyte/packaging-libraries-in-ios-a-comprehensive-guide-4436dea9fb70?source=rss----9477120405bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/4436dea9fb70</guid>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[mobile]]></category>
            <category><![CDATA[ios-app-development]]></category>
            <category><![CDATA[ios]]></category>
            <dc:creator><![CDATA[Shaunak Jagtap]]></dc:creator>
            <pubDate>Wed, 11 Oct 2023 08:44:06 GMT</pubDate>
            <atom:updated>2023-10-11T08:44:06.336Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*cxrrAZZFTI0vPDSR7dGm-Q.png" /></figure><p>Software Development Kits (SDKs) are the lifeblood of development, offering a treasure of pre-packaged libraries, tools, and resources that empower developers to craft rich and feature-packed applications. However, for those new to distributing SDKs in iOS, the path can be filled with uncertainty.</p><p>We have shipped 100s of iOS SDK builds in the form of our WebRTC clients for customers looking to integrate real time communications within their applications. With that experience, this technical blog post is tailored for iOS developers and aims to illuminate the nuances of SDK distribution, including key concepts, best practices, and tools. With this knowledge, developers can unlock the full potential of their SDKs and ensure seamless integration experiences.</p><h3>Static vs. dynamic libraries and frameworks in iOS</h3><p>Seasoned iOS developers often grapple with crucial decisions regarding system frameworks, packaging their code, and integrating third-party components. Among these decisions, choosing between static and dynamic libraries or frameworks is pivotal, making profound implications for application performance and resource management.</p><h3>Implications on app size and launch time</h3><p>The choice between static and dynamic libraries or frameworks carries significant weight regarding your app’s binary size and launch time.</p><p>Here’s a concise summary of how static and dynamic linking impacts various facets of your application:</p><p>While statically linked modules proffer a smaller app size and accelerated loading times, dynamic linking has its own set of compelling use cases:</p><ul><li><strong>Multiple static modules depending on the same module:</strong> When your app comprises multiple static modules that lean on a common module, you might encounter warnings about duplicate symbols at runtime. Transitioning the shared module to a dynamic one can effectively mitigate this issue.</li><li><strong>iOS increased app launch time with many dynamic libraries/frameworks:</strong> Loading numerous third-party dynamic libraries or frameworks on iOS can lead to prolonged app launch times. Vigilant monitoring and optimisation efforts are essential to upholding app launch performance.</li></ul><h3>Static/dynamic with different integration techniques</h3><p>To make well-informed decisions about linking, a profound understanding of how various dependency managers handle linking behaviour is imperative:</p><ul><li><strong>Own targets or projects linked directly:</strong> Exert precise control over the linking behaviour of targets within your repository or external repositories by fine-tuning Build Settings. A simple adjustment to the Build Setting allows you to toggle between static library and dynamic library.</li><li><strong>CocoaPods:</strong> By default, CocoaPods constructs and links dependencies as static libraries. The introduction of <strong>use_frameworks!</strong> in your Podfile enables the construction of dynamic frameworks. Moreover, <strong>:linkage =&gt; :static</strong> can be employed to shape dependencies as static frameworks.</li><li><strong>Swift Package Manager (SPM):</strong> SPM, by default, fabricates dependencies as static libraries, offering minimal control over linking behaviour. However, if you are the custodian of a package, you can specify <strong>type: .dynamic</strong> within your <strong>Package.swift</strong> file to fashion a dynamic package.</li><li><strong>Carthage:</strong> In its default configuration, Carthage leans towards using dynamic frameworks for dependencies. Nevertheless, you retain the flexibility to configure it to construct and link them statically when circumstances dictate such an arrangement.</li></ul><h3>Understanding different formats</h3><p>In the realm of iOS development, many formats for packaging libraries and resources exist. Each format serves distinct purposes, and as an expert iOS developer, comprehending their nuances is indispensable. Here’s an exploration of some essential formats:</p><h3>xcframework</h3><p><strong>XCFramework</strong> is a relatively recent addition to Apple’s arsenal of formats. It is a versatile container meticulously designed for packaging frameworks for diverse platforms and architectures into a single, harmonious bundle. Embracing XCFrameworks streamlines the distribution of binary frameworks, ensuring harmonious coexistence across various Apple devices and processor architectures. Adopting XCFrameworks bestows developers with the gift of streamlined development, reduced integration complexities, and enhanced application performance.</p><h3>Framework</h3><p>The classic <strong>framework</strong> format remains a stalwart choice for packaging code and resources in iOS development. Frameworks offer a structured habitat for your codebase, gracefully accommodating header files, binaries, and resources. They advocate the virtues of encapsulation and modularity, simplifying the process of integration and maintenance of your code. Frameworks wear the dual hats of either static or dynamic, contingent upon whether the code binds at compile time (static) or runtime (dynamic).</p><h3>.a (Static Library) and .o (Object File)</h3><p><strong>.a files</strong>, heralded as static libraries, house compiled code snugly woven into the fabric of your application at compile time. These libraries accompany your app’s binary, bestowing it with a petite binary size and promising accelerated startup times. <strong>.o files</strong>, or object files, inhabit the realm of intermediate compilation units, wielding the potential to unite and give birth to static libraries or dynamically linked frameworks. Distinguishing between the use cases of static libraries and object files is pivotal for optimising your app’s performance and memory footprint.</p><h3>.dylib (Dynamic Library)</h3><p>The world of <strong>.dylib files</strong> beckons dynamic libraries, extending an invitation for their ad hoc arrival at runtime as your app springs into existence. Dynamic libraries usher in a measure of flexibility in code sharing but might nudge your app’s binary size northward. Dynamic libraries traditionally find their calling in system frameworks and shared system components. Handling them with care and vigilance is a requisite, for any slip-up in configuration or inclusion can usher in the Spector of runtime crashes.</p><h3>Universal Framework</h3><p>Universal frameworks, the “Jack of all trades” among formats, don the cloak of versatility. They are a specialised framework format engineered to accommodate multiple architectures and platforms under one sprawling roof. This format performs an invaluable service, simplifying the distribution of cross-platform libraries. Developers can present a single binary, an embodiment of unity, capable of functioning seamlessly across an array of iOS devices and processor architectures.</p><h3>Swift Package Manager</h3><p>As a seasoned iOS developer, you’re no stranger to the capabilities wielded by Swift Package Manager (SPM) when it comes to dependency management. Within the labyrinthine corridors of SPM, two pivotal concepts demand your attention: <strong>.binaryTarget</strong>, , and the strategic utilization of <strong>linkerSettings</strong>.</p><h3>.binaryTarget</h3><p>Introduced as a breath of fresh air, <strong>.binaryTarget</strong> assumes the mantle of a feature within Swift Package Manager designed explicitly to streamline the integration of binary dependencies. Binary dependencies are pre-compiled libraries or frameworks presented by third-party sources, emerging as swift and efficient companions for integration. Here&#39;s the lowdown:</p><ul><li><strong>Efficient integration</strong>: With the declaration of a <strong>.binaryTarget</strong>, Swift Package Manager orchestrates retrieving a pre-compiled binary from a designated source, whether a Git repository or a URL. This streamlined approach expedites integration, ushering inconvenience.</li><li><strong>Platform-agnostic brilliance</strong>: Binary targets shine as platform-agnostic stars, casting their glow across diverse platforms and architectures, including iOS, macOS, and beyond. This trait is especially beneficial for those engaged in cross-platform development endeavors.</li><li><strong>Version control vigilance</strong>: Binary targets, operating in the realm of pre-compiled artifacts, dwell outside the confines of direct version control within your package. Instead, the version or tag of the binary dependency takes center stage within your <strong>Package.swift</strong> file.</li><li><strong>Swift harmony</strong>: Ensure the selected binary target aligns with your project’s Swift version for harmony and compatibility. A mismatch in Swift versions can sow the seeds of discord.</li></ul><h3>.target</h3><p>In stark contrast to the pragmatic elegance of <strong>.binaryTarget</strong>, emerges as the go-to directive for source-based dependencies. Source-based packages carry their source code, ready to undergo the rites of compilation upon integration into your project. Vital insights into this concept include:</p><ul><li><strong>Source code integration</strong>: Invocation of a commands Swift Package Manager to embark on a journey of source code retrieval, cloning the package’s source code and forging it into your project. This process offers the boon of customization, enabling you to mold or modify the package as per your requirements.</li><li><strong>Version control ascendancy</strong>: Source-based packages ascend to prominence as the champions of direct version control within your project’s repository. This translates into the power to wield influence over the package’s code, affording the liberty to make modifications as needed.</li><li><strong>Swift compatibility</strong>: As with binary targets, extending a cordial handshake of compatibility between your project’s Swift version and the chosen source-based package is paramount. Avoidance of mismatched Swift versions can be your shield against compatibility conundrums.</li><li><strong>The web of dependency</strong>: Source-based packages often weave a web of dependencies, crafting a sprawling tapestry of interconnected packages. Swift Package Manager rises to the challenge, orchestrating the management of this intricate dependency graph, ensuring that all required packages partake in the grand symphony of integration.</li></ul><p><strong>linkerSettings</strong>, an entity of significance within the grand configuration of Swift Package Manager, entrusts you with the reins of control over linker flags and settings that govern the orchestration of your targets. Its importance is not to be underestimated:</p><ul><li><strong>Fine-grained command</strong>: <strong>linkerSettings</strong> stands as your herald, bearing the mandate of fine-grained control over the linking process. It paves the way for the specification of linker flags, search paths, and sundry settings, exerting a profound influence on how the package melds with your project.</li><li><strong>Integration tailoring</strong>: Instances may arise where a package exhibits a penchant for specific linker flags or settings. Enter <strong>linkerSettings</strong>; this savior allows you to fashion an integration that gracefully accommodates the package&#39;s unique requirements.</li><li><strong>Mitigating linking conflicts</strong>: In the integration, where multiple packages jostle for space, conflicts on the frontiers of linking can rear their heads. <strong>linkerSettings</strong> steps in as the peacemaker, offering the means to navigate these conflicts with poise and elegance, ensuring a harmonious integration experience.</li><li><strong>Compatibility crusade</strong>: To preserve the sanctity of your project, heed the call of compatibility. Scrutinize the compatibility of linker flags and settings with your project’s Swift version and platform. Mismatched configurations have the power to disrupt the tranquil flow of runtime.</li></ul><h3>The role of checksums in SPM</h3><p>Checksums, those guardians of integrity and security, play a pivotal role within the domain of Swift Package Manager (SPM), specifically in the realm of package management. They serve as sentinels tasked with verifying the authenticity and integrity of external dependencies before ushering them into your project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P4xfKIwHWgRBCRyx.png" /></figure><ol><li><strong>Package resolution</strong>: Your project’s <strong>Package.swift</strong> file hosts declarations of dependencies. When summoned, the Swift Package Manager embarks on a mission to retrieve the package manifest, an essential dossier housing information about the package and its version.</li><li><strong>Download and check</strong>: The saga continues with downloading the package’s source code or binary artifact. Simultaneously, Swift Package Manager, armed with diligence, fetches the checksum linked to the package’s version from a trustworthy source, often the package repository.</li><li><strong>Verification</strong>: As the download completes, Swift Package Manager commences an expedition into checksum calculation. The calculated checksum stands face-to-face with its repository-forged counterpart, and only harmony, represented by a perfect match, rings true. A matching checksum signifies that the downloaded package mirrors the precise form and content expected by the package repository.</li></ol><h3>Mergeable libraries (New in Town!)</h3><p>Apple announced Mergeable libraries in WWDC23, the unsung heroes sometimes cloaked in the monikers of “umbrella frameworks” or “universals,” which is the fusion of multiple frameworks or libraries into a singular, cohesive framework. This gives a leaner, meaner, and more efficient application binary. Let’s shine a light on the critical facets of mergeable libraries:</p><h3>1. Reduction in binary size</h3><p>The most prominent jewel in the crown of mergeable libraries is the substantial reduction in the size of your application binary. By amalgamating multiple frameworks into a singular entity, developers surgically excise redundant code and resources, rendering the binary trim and sleek. This trimness finds its true value in mobile applications, where app size directly influences download times and device storage.</p><h3>2. Streamlined maintenance</h3><p>Mergeable libraries don the mantle of the custodian of dependencies, simplifying the labyrinthine maintenance process. Instead of juggling several individual libraries, each with its own versioning and update cycle, developers are entrusted with the guardianship of a solitary, consolidated library. This harmonisation streamlines the update process and curtails the risk of version conflicts and compatibility conundrums.</p><h3>3. Improved load/launch times</h3><p>Diminished binary sizes usher in improved app launch times. With fewer resources to load into memory, the app leaps to life more swiftly, embellishing the user experience with the gift of alacrity. Reduced load times are particularly invaluable in scenarios where instant access to the application is not just a luxury but an expectation.</p><h3>4. Cross-platform compatibility</h3><p>Mergeable libraries possess the unique capability of accommodating multiple platforms and architectures, transforming them into the darlings of cross-platform development. In the hands of adept developers, a single library can extend its benevolent embrace across iOS, macOS, watchOS, tvOS, and many other platforms, fostering an ecosystem of harmonious coexistence.</p><p>In the vast landscape of iOS development, two linkers shine brightly: Dyld and its evolutionary offspring, Dyld3. These dynamic linkers perform the critical role of managing the loading and linking of libraries during an app’s launch. For expert iOS developers, a deep understanding of their inner workings is essential.</p><ol><li><strong>Startup performance</strong>: Dyld, a stalwart of iOS, has been meticulously designed to deliver efficient startup performance. It employs an arsenal of optimisations to minimise the time required for loading and linking libraries when an app takes its first breath. This efficiency is paramount in ensuring a seamless user experience.</li><li><strong>Lazy binding</strong>: Dyld employs the ingenious strategy of lazy binding, which defers symbol resolution until the precise moment when a symbol is first utilized. This mechanism trims the startup overhead by avoiding unnecessary work during the initial stages of the app launch.</li><li><strong>Shared caches</strong>: In its quest to enhance startup performance, Dyld harnesses the power of shared caches. These caches store pre-processed libraries, allowing multiple applications to share them. This shared resource optimises resource utilisation and further expedites the launch process.</li></ol><h3>Dyld</h3><h3>Dyld3: The evolutionary leap</h3><p>As iOS and macOS continued to evolve, the demands on dynamic linking also grew. This prompted the emergence of Dyld3, representing a substantial leap forward in dynamic linking technology. Dyld3 introduced several key advancements aimed at optimising app performance and resource management.</p><ol><li><strong>Reduced memory overhead</strong>: Dyld3 was engineered with a focus on minimising memory overhead. It employs a more efficient data structure for managing loaded libraries, precious in resource-constrained environments such as mobile devices.</li><li><strong>Parallel loading</strong>: Dyld3 introduces the paradigm of parallel loading, enabling it to load multiple libraries concurrently. This parallelism takes full advantage of multi-core processors, resulting in faster app launch times.</li><li><strong>On-demand loading</strong>: Dyld3 adopts the strategy of on-demand loading, loading only the portions of libraries that are required at runtime. This “just-in-time” approach conserves memory and accelerates startup times.</li><li><strong>Improved symbol binding</strong>: Dystrongly emphasizess on enhancing symbol binding performance, ensuring that symbols are resolved efficiently as an app runs. This is crucial for maintaining smooth app performance during execution.</li></ol><h3>Key advancements in Dyld3:</h3><p>Creating and distributing an iOS SDK demands careful consideration of various elements. To embark on this journey, you must make critical decisions regarding the type of library (static or dynamic) that best suits your needs.</p><p>Dependency management tools like CocoaPods and Swift Package Manager (SPM) are crucial in linking and integrating your SDK into other projects. Understanding library formats, such as frameworks and xcframeworks, is essential for packaging your code effectively. Don’t forget the importance of checksums in ensuring the security of your SDK.</p><p>Additionally, exploring the advantages of mergeable libraries can help reduce app size and simplify maintenance. Lastly, delve into the world of dynamic linkers like Dyld and Dyld3 to optimise app startup performance and memory usage. By mastering these components, you’ll be well-prepared to create and distribute iOS SDKs that enhance the efficiency and reliability of your development projects.</p><p><em>Originally published at </em><a href="https://dyte.io/blog/packaging-libraries-ios/"><em>https://dyte.io</em></a><em> on September 20, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4436dea9fb70" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dyte/packaging-libraries-in-ios-a-comprehensive-guide-4436dea9fb70">Packaging Libraries in iOS: A Comprehensive Guide</a> was originally published in <a href="https://medium.com/dyte">Dyte</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HLS in Depth]]></title>
            <link>https://medium.com/dyte/hls-in-depth-fa29b3bb84da?source=rss----9477120405bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/fa29b3bb84da</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[video-call]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[live-streaming]]></category>
            <category><![CDATA[hls]]></category>
            <dc:creator><![CDATA[Fenil Jain]]></dc:creator>
            <pubDate>Thu, 06 Jul 2023 17:47:57 GMT</pubDate>
            <atom:updated>2023-07-06T17:47:57.252Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ppel1XUIy7zJFVWhCP83aw.png" /></figure><h3>What is HLS?</h3><p>HLS (HTTP Live Streaming) is a live streaming protocol that leverages current widespread HTTP technologies to deliver live video + audio experiences to large audiences.</p><p>Initially developed by Apple in 2009, it has gained wide adoption across devices ranging from desktops and mobile phones to smart TVs. The iPhone drove its initial adoption, and having a significant market share in the mobile space, its supporting HLS by default was an excellent booster.</p><h3>Understanding HLS working</h3><p>Let’s understand how HLS works on a high level. Looking from a client’s (any web player’s) perspective, these are the steps it follows to understand and play media:</p><ul><li>It makes a GET request to a server to fetch a master playlist.</li><li>The client chooses the best bitrate from the list considering the user’s network conditions.</li><li>Next, it makes a GET request to fetch the playlist corresponding to that bitrate.</li><li>It understands the playlist, starts fetching different media segments, and starts playing.</li><li>It repeats the last two mentioned steps in a loop.</li></ul><p>This was a rough plan for how clients work with HLS. Now let’s see what these playlists/segments actually are in more detail. To generate this data, I am going to take the help of the swiss-army knife of media-related stuff: ffmpeg</p><p>Let’s create a sample video using a simple command:</p><pre>ffmpeg -f lavfi -i testsrc -t 30 -pix_fmt yuv420p testsrc.mp4</pre><p>This creates a sample video of 30 seconds in duration.</p><p>Now we’ll create an HLS playlist from the following command:</p><pre>ffmpeg -i testsrc.mp4 -c:v copy -c:a copy -f hls -hls_segment_filename data%02d.ts index.m3u8</pre><p>This will create a playlist in the same dir, with files:</p><p>Puzzled about what they are? Let’s understand what the command did and what these files mean.</p><p>We’ll first break down <strong>the command.</strong></p><p>We are telling ffmpeg by -i to take MP4 video as input, note this can also take RTMP input directly; one just needs to say something like:</p><pre>ffmpeg -listen 1 -i rtmp://127.0.0.1:1938/live -c:v copy -c:a copy -f hls -hls_segment_filename data%02d.ts index.m3u8</pre><p>This is another exciting aspect we will discuss further in this article.</p><p>Next, understanding -c:v copy and -c:a copy. This is a way of saying copy audio/video data of encoding, etc., from source input to output as it is. This is where you could do encoding in different formats too. For example, if you want more than one quality in your HLS playlist.</p><p>-f tells ffmpeg what our output format will be, which in this case is HLS.</p><p>-hls_segment_filename tells ffmpeg what format we want our segment filenames to be in.</p><p>And finally, the last argument is the name of our master playlist.</p><p>Now, moving on to <strong>the files</strong>:</p><p>Opening the master playlist in an editor, it looks like this:</p><pre>#EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:0 #EXTINF:10.000000, data00.ts #EXTINF:10.000000, data01.ts #EXTINF:10.000000, data02.ts #EXT-X-ENDLIST</pre><p>Let’s understand these <strong>tags</strong>:</p><ul><li>EXTM3U: M3U(MP3 URL or Moving Picture Experts Group Audio Layer 3 Uniform Resource Locator in full) is a multimedia playlist format name, and this line indicates that we are using extended M3U format, according to HLS spec: &quot;It MUST be the first line of every Media Playlist and every Master Playlist.&quot;</li><li>EXT-X-VERSION: This tag indicates the compatibility version of our playlist file, there has been more than one iteration on the playlist format, and there are older clients who do not understand newer versions, so this becomes compulsory to send for any server-generating playlist format version greater than 1. It being present multiple times should lead to clients rejecting the whole playlist.</li><li>These were the Basic Tags as specified by the spec.</li><li>We will now move on to something called Media Playlist Tags. The first among them is EXT-X-TARGETDURATION. It is a required flag again telling the client what will be the longest segment length it can expect. Its value is in seconds.</li><li>EXT-X-MEDIA-SEQUENCE: This tag mentions the number of the first media segments. It is unnecessary and, by default, considered 0 when not present. Its only restriction is that it should appear before the first media segment.</li><li>EXTINF: This tag comes under Media Segment Tags and indicates the duration and metadata for each media segment. Its format, according to spec, is: &lt;duration&gt;, [&lt;title&gt;]. Duration is required to be specified in floating point numbers when using any format above version 3. The title is mostly metadata for humans and optional, ffmpeg seems to have skipped it by default.</li><li>Each following line after EXTINF specifies where to find the media segment. In our case, they are in the same folder, so they are directly listed, but they can also be in subfolders like 480p, which would mean their name would be: 480p/data01.ts. Note that data01 is also a format given by us in the above command, and one can change it very easily.</li><li>EXT-X-ENDLIST: I guess the most difficult-to-understand tag? One would definitely not be able to look and tell what it does. Jokes aside, except for indicating that the list has ended, an exciting property is that it can appear anywhere in the playlist file. If manually building these files, you may want to be careful about placing it in the right place.</li></ul><p>For keen readers, they may have noticed that no multiple bitrates are listed here. Well, I wanted to explain basic structure first with a simple example.</p><p>To understand ABR, let’s take an already hosted URL, something like: <a href="https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel.ism/.m3u8">https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel.ism/.m3u8</a></p><p>OTTVerse maintains a nice list of test HLS URLs <a href="https://ottverse.com/free-hls-m3u8-test-urls/">here</a>.</p><p>On downloading the playlist, it looks something like this:</p><pre>#EXTM3U #EXT-X-VERSION:1 ## Created with Unified Streaming Platform (version=1.11.20-26889) # variants #EXT-X-STREAM-INF:BANDWIDTH=493000,CODECS=&quot;mp4a.40.2,avc1.66.30&quot;,RESOLUTION=224x100,FRAME-RATE=24 tears-of-steel-audio_eng=64008-video_eng=401000.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=932000,CODECS=&quot;mp4a.40.2,avc1.66.30&quot;,RESOLUTION=448x200,FRAME-RATE=24 tears-of-steel-audio_eng=128002-video_eng=751000.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=1197000,CODECS=&quot;mp4a.40.2,avc1.77.31&quot;,RESOLUTION=784x350,FRAME-RATE=24 tears-of-steel-audio_eng=128002-video_eng=1001000.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=1727000,CODECS=&quot;mp4a.40.2,avc1.100.40&quot;,RESOLUTION=1680x750,FRAME-RATE=24,VIDEO-RANGE=SDR tears-of-steel-audio_eng=128002-video_eng=1501000.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=2468000,CODECS=&quot;mp4a.40.2,avc1.100.40&quot;,RESOLUTION=1680x750,FRAME-RATE=24,VIDEO-RANGE=SDR tears-of-steel-audio_eng=128002-video_eng=2200000.m3u8 # variants #EXT-X-STREAM-INF:BANDWIDTH=68000,CODECS=&quot;mp4a.40.2&quot; tears-of-steel-audio_eng=64008.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=136000,CODECS=&quot;mp4a.40.2&quot; tears-of-steel-audio_eng=128002.m3u8</pre><p>Over here, we see a new tag EXT-X-STREAM-INF. This tag gives more info about all the variants available for a given stream. The tag is to be followed by an arbitrary number of attributes.</p><ul><li>BANDWIDTH: It is a compulsory attribute whose value tells us how many bits per second will be sent with this variant.</li><li>CODECS: It is another mandatory attribute specifying our codecs for video and audio, respectively.</li><li>RESOLUTION: It is an optional attribute mentioning the variant&#39;s pixel resolution.</li><li>VIDEO-RANGE: We can ignore this tag cause it is not specified in spec, though at the time of writing this article, there&#39;s an <a href="https://github.com/video-dev/hls.js/issues/2489">open issue</a> on hls.js to add support for it.</li><li>SUBTITLES: It is an optional attribute specified as a quoted string. It must match the value of the GROUP-ID attribute on a EXT-X-MEDIA tag. This tag should also have SUBTITLES as its TYPE attribute.</li><li>CLOSED-CAPTIONS: It is an optional attribute, mostly the same as SUBTITLES. Two things that differentiate it are:</li><li>Its attribute value can be a quoted string or NONE.</li><li>Tag value on EXT-X-MEDIA for closed captions should be CLOSED-CAPTIONS.</li></ul><p>The line following these stream-inf tags specifies where to find the corresponding variant stream, a playlist similar to what we saw earlier.</p><h3>RTMP ingestion</h3><p>Earlier, we saw an RTMP input ffmpeg command. It helped us understand another property of live HLS playlists. Revisiting the command, it looked like this:</p><pre>ffmpeg -listen 1 -i rtmp://127.0.0.1:1938/live -c:v copy -c:a copy -f hls -hls_segment_filename data%02d.ts index.m3u8</pre><p>The command should give an output similar to what we have below, which seems stuck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CCdyCw0hPVEjPHpM.png" /></figure><p>Now, let’s satisfy your dream of becoming a streamer! One can use any RTMP source, even ffmpeg, but in our demo, we will use OBS to send data to our ffmpeg server listening for RTMP input. I have created a simple screen share capture input in it and then navigated to Stream section in Settings, here you will see a screen similar to this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yMgEvRAZaO8LDKfa.png" /></figure><p>Let’s fill in our Service as Custom and add the Server rtmp://12.0.0.1:1938/live with an empty Stream Key. Now click on apply and press OK.</p><p>We will land at the default view of OBS here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FHkmJ3dkn0ssaTjI.png" /></figure><p>When we click on Start Streaming, we will stream to our local RTMP-to-HLS server. You will notice the folder in which you ran the ffmpeg command. There are new files with names:</p><ul><li>index.m3u8</li><li>data files with template data{0-9}{0-9}.ts</li></ul><p>Let’s check index.m3u8 in one instant, I captured this as:</p><pre>#EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION:4 #EXT-X-MEDIA-SEQUENCE:3 #EXTINF:4.167000, data03.ts #EXTINF:4.167000, data04.ts #EXTINF:4.166000, data05.ts #EXTINF:4.167000, data06.ts #EXTINF:4.167000, data07.ts</pre><p>Notice how there’s no end playlist tag? That’s because this is a live playlist that keeps on changing over time with new segments. For example, after some time, my local playlist looks something like this:</p><pre>#EXTM3U #EXT-X-VERSION:3 #EXT-X-TARGETDURATION:4 #EXT-X-MEDIA-SEQUENCE:9 #EXTINF:4.167000, data09.ts #EXTINF:4.167000, data10.ts #EXTINF:4.166000, data11.ts #EXTINF:4.167000, data12.ts #EXTINF:0.640667, data13.ts #EXT-X-ENDLIST</pre><p>Old segments are replaced with new segments to load. This helps one deliver the latest data when live streaming an event.</p><p>This covers playlists to a reasonable extent. Now let’s talk about segment files a bit.</p><h3>Container formats</h3><p>Media segments use container formats to store actual encoded video/audio data. Discussing container formats in detail would be out of the scope of this particular article. A good starting point to know more about it might be <a href="https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Containers">this</a>.</p><p>HLS initially only supported MPEG-2 TS containers. This decision was different from container formats used by other HTTP-based protocols. For example, DASH has always used fMP4 (fragmented MP4). Though finally, Apple announced support for fMP4 in HLS, and now it is officially supported by spec.</p><h3>Media segment size</h3><p>The media segment size is not defined. People usually started with ~10 seconds segment size, but another part of the spec caused this number to change. According to the spec, one must fetch and ready at least three segments to start playing. This would mean for 10 seconds segments, it would require 30 seconds of buffer to be loaded before the client could begin playing. One can change it as and when one wants, and this is a common tuning parameter one employs while reducing latency.</p><h3>Component-wise breakdown</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jBEtE4mnTjXwyeR9.png" /></figure><ul><li><strong>Ingest:</strong> Ingest (relatively less relevant) marks the point where our system takes input from the user, this can be done in various formats with different protocols, but platforms like Twitch and Youtube have popularized RTMP. However, this seems to be changing with the introduction of WebRTC in OBS, making it possible to use WHIP to ingest content much faster at lower latencies.</li><li><strong>Processing:</strong> Media Processing includes transcoding, transmuxing, etc., depending on the ingest format used. Input here is converted into MPEG-TS or fMP4 segments indexed in the playlist and made ready to be consumed.</li><li><strong>Delivery:</strong> Delivery is an essential aspect of ensuring HLS is scalable. A web server hosts the playlists and segments, which are then sent to the end user through a CDN. A CDN caches these GET requests and scales itself to return static data on its side, making protocol highly scalable when asked for it at much higher traffic times.</li><li><strong>Playback:</strong> A smooth playback is vital for a pleasant user-end experience. ABR must be adequately implemented on the player side so that the user experiences minimal experience dips during the stream.</li></ul><h3>Features provided by HLS</h3><p><strong>Distributes over common format HTTP making it easily reachable everywhere</strong></p><ul><li>Using HTTP, HLS can leverage advantages from underlying TCP implementation, making retransmissions handled out of the box.</li><li>Much of the internet relies on HTTP, making it automatically deliverable on all consumer devices.</li><li>This becomes challenging for UDP-based protocols like RTP, which sometimes need stuff like NAT traversals to work properly.</li></ul><p><strong>Easy CDN-level caching</strong></p><ul><li>Scaling HLS becomes super easy because caching HTTP is widely supported, and offloading it to good CDNs can reduce the load on the source server to a minimum while ensuring delivering the same audio + video data to all the consumers.</li></ul><p><strong>Adaptive Bitrate Streaming (ABR)</strong></p><ul><li>HLS supports ABR, so users with a poor internet connection can quickly shift to lower bitrates and continue enjoying the stream.</li></ul><p><strong>Inbuilt ad insertion support</strong></p><ul><li>HLS provides easy-to-use tags to insert ads during a stream dynamically.</li></ul><p><strong>Closed captions</strong></p><ul><li>It also supports embedded closed captions and even DRM content.</li></ul><p><strong>Using fMP4</strong></p><ul><li>Using fragmented MP4 helps reduce encoding and delivery costs and increases compatibility with MPEG-DASH.</li></ul><h3>Disadvantages of HLS</h3><ul><li>HLS is widely known for its painstakingly higher latencies. One of the simple ways to make it slightly better is by fine-tuning segment size and then changing the ingest protocol to lower latency protocols like RTMP. But still, for crisp live experiences, vanilla HLS is not enough. The computing world has offered community LL-HLS and Apple LL-HLS as the solutions. Both have their way of approaching the problem and can promise some nice latencies.</li><li>HLS spec specifies that the client needs to at least load three segments before starting. So now, imagine you set the segment size as 10 seconds. You already see 30 seconds of latency in the stream from the start. This also makes tuning segment size very important.</li></ul><h3>Final Thoughts</h3><p>Having open standards helps adoption and growth while making it much easier to interop between open and closed software/hardware stacks. HLS drives the same in live streaming space — it is widely adopted, simple, yet robust. It did not try to reinvent the wheel from scratch, instead embraced current widely accepted protocols and built on top of it.</p><p><em>Get better insights on leveraging Dyte’s technology and discover how it can revolutionize your app’s communication capabilities with its </em><a href="https://dyte.io/video-sdk"><em>SDKs</em></a><em>.</em></p><p>If this article tickled the builder or the hacker in you, then check out <a href="https://dyte.io/live-streaming-sdk">Dyte’s live streaming SDK</a>. We provide easy-to-use SDKs to deliver interactive live streams at scale with much lower latency than traditional HLS. Feel at home with the same friendly developer experience and extensibility. Dyte’s live streaming SDK is <a href="https://dyte.io/ph">launching soon on Product Hunt</a>!</p><p><em>Originally published at </em><a href="https://dyte.io/blog/hls-in-depth/"><em>https://dyte.io</em></a><em> on July 5, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa29b3bb84da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dyte/hls-in-depth-fa29b3bb84da">HLS in Depth</a> was originally published in <a href="https://medium.com/dyte">Dyte</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adding AR Filters to Video Calls Using DeepAR and Dyte]]></title>
            <link>https://medium.com/dyte/adding-ar-filters-to-video-calls-using-deepar-and-dyte-13badf17ef65?source=rss----9477120405bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/13badf17ef65</guid>
            <category><![CDATA[filters]]></category>
            <category><![CDATA[ar]]></category>
            <category><![CDATA[video-calling-app]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[video-call]]></category>
            <dc:creator><![CDATA[Ishita Kabra]]></dc:creator>
            <pubDate>Wed, 05 Jul 2023 11:36:49 GMT</pubDate>
            <atom:updated>2023-07-05T11:36:49.679Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H5KkxaJaS7c_lKwS3cJMqw.png" /></figure><p>Unless you live under a rock, you would have used social media platforms such as TikTok, Instagram, Facebook, or Snapchat. When posting a photo or video on these platforms, you usually have the option to enhance the photo or video using various filters.</p><p>You can transform yourself in an image or video with a different hair color or wear a cap even though you didn’t take the photo or record a video wearing one.</p><p>Social media platforms use Augmented Reality (AR) filters to make all these photo or video enhancements possible. AR filters are digital effects that can be applied to images or videos in real time, using computer vision technology to track the movements and facial features of someone in an image or video.</p><p>The AR filters are mainly used for entertainment, branding, and marketing because they offer a fun and engaging way to enhance visual content. These filters can also create immersive experiences, such as virtual try-ons for clothing, and showcase products uniquely and interactively.</p><p>In this article, you will learn how to build a video calling app with AR filters using <a href="https://dyte.io/video-sdk">Dyte video SDK</a>, DeepAR, and web components UI kit. Dyte is a developer-friendly platform that offers robust SDKs to build live experiences within an application.</p><h3>Prerequisites</h3><p>To follow through with this article, you will need the following software installed on your computer:</p><ul><li>A text editor: I will use Visual Studio Code as a text editor for this article. To install VS Code, navigate to the <a href="https://code.visualstudio.com/download">VS Code download page</a> and download it for your operating system.</li></ul><h3>Create and set up a Dyte account</h3><p><strong>Step 1:</strong> Navigate to <a href="https://dyte.io/">dyte.io</a> and click the ‘Start building’ button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vcd_mkjZ23JVHR0s.png" /></figure><p><strong>Step 2:</strong> Sign up for a free Dyte account on the next page using your Google or GitHub account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B_sZjRnGqXbSNDDx.png" /></figure><p><strong>Step 3:</strong> After creating a Dyte account, you will be redirected to the ‘API Keys’ page, with an Organization ID and API key provided. Keep your API key secure, and don’t share it with anyone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tUbzA2uVMqEaRfbU.png" /></figure><h3>Create Presets</h3><p>Dyte includes a bunch of pre-configured presets. You can simply use the default preset if you don’t wish to create one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ueMEaSemw2_-mX6n.png" /></figure><h3>Create a Dyte Meeting</h3><p><strong>Step 1:</strong> Navigate to the <a href="https://docs.dyte.io/api?v=v2#/operations/create_meeting">Create a Meeting page.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*33PCGv1kbU1CqRIN.png" /></figure><p><strong>Step 2:</strong> On the right side of the page, fill the Username input field with Organization ID and the Password input field with your API Key provided on the API Keys page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/0*jCdvX3rYgoJL_UR1.png" /></figure><p><strong>Step 3:</strong> In the Body section, give the meeting a title of “AR Filters.” Then click the Send API Request button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/0*KbxCEcAf311Gto9-.png" /></figure><p><strong>Step 4:</strong> Navigate to the Meetings page in the <a href="https://accounts.dyte.io/auth/login">Dyte Developer Portal</a>, and you should see that a meeting titled AR Filters was created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*STuB_ZJAepfq1eHY.png" /></figure><h3>Add a participant to the meeting</h3><p><strong>Step 1:</strong> Navigate to the <a href="https://docs.dyte.io/api?v=v2#/operations/add_participant">Add a Participant page</a> and fill out the Organization ID and API Key (you can get these from the <a href="https://accounts.dyte.io/auth/login">Dyte Developer Portal</a>) input fields as shown in the step above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i-mlNCvndqh_7WBL.png" /></figure><p><strong>Step 2:</strong> In the Parameters section, add the Meeting ID of the AR Filters meeting we created earlier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*X_tc-jYMdbG0DS_z.png" /></figure><p><strong>Step 3:</strong> In the Body section, fill out the options as shown below. Then click the ‘Send API Request’ button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/0*CfGSPm2-noOIbGAY.png" /></figure><p><strong>Step 4:</strong> Obtain the token. This token is unique for each participant in the meeting, and you would ideally want to fetch it for each participant by calling Dyte&#39;s APIs in your own backend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/0*XBQu_SdRygm7lfMv.png" /></figure><h3>Creating your Dyte Meeting</h3><p><strong>Step 1: </strong>Create a folder called ‘ARFilters’ on your computer. Then open the folder in a text editor and add to it an index.html file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q4-2p8o2-XL4hsXz.png" /></figure><p><strong>Step 2: </strong>Install the Dyte SDK.</p><pre>&lt;head&gt; &lt;script type=&quot;module&quot;&gt; import { defineCustomElements } from &#39;&lt;https://cdn.jsdelivr.net/npm/@dytesdk/ui-kit@1.51.1/loader/index.es2017.js&gt;&#39;; defineCustomElements(); &lt;/script&gt; &lt;!-- Import Web Core via CDN too --&gt; &lt;script src=&quot;&lt;https://cdn.dyte.in/core/dyte-1.14.0.js&gt;&quot;&gt;&lt;/script&gt; &lt;/head&gt;</pre><p><strong>Step 3: </strong>In your index.html build a basic meeting UI with two custom buttons to control the AR filters.</p><pre>&lt;body&gt; &lt;div id=&quot;app&quot;&gt; &lt;dyte-dialog-manager class=&quot;dyte&quot;&gt;&lt;/dyte-dialog-manager&gt; &lt;dyte-participants-audio class=&quot;dyte&quot;&gt;&lt;/dyte-participants-audio&gt; &lt;dyte-notifications class=&quot;dyte&quot;&gt;&lt;/dyte-notifications&gt; &lt;dyte-header class=&quot;dyte&quot;&gt;&lt;/dyte-header&gt; &lt;div class=&quot;dyte-grid&quot;&gt; &lt;dyte-grid class=&quot;dyte&quot;&gt;&lt;/dyte-grid&gt; &lt;dyte-sidebar class=&quot;dyte&quot; default-section=&quot;none&quot;&gt; &lt;/dyte-sidebar&gt; &lt;/div&gt; &lt;div class=&quot;dyte-controlbar&quot;&gt; &lt;div class=&quot;controlbar-section&quot;&gt; &lt;dyte-settings-toggle class=&quot;dyte&quot;&gt;&lt;/dyte-settings-toggle&gt; &lt;/div&gt; &lt;div class=&quot;controlbar-section&quot;&gt; &lt;dyte-mic-toggle class=&quot;dyte&quot;&gt;&lt;/dyte-mic-toggle&gt; &lt;dyte-camera-toggle class=&quot;dyte&quot;&gt;&lt;/dyte-camera-toggle&gt; &lt;dyte-leave-button class=&quot;dyte&quot;&gt;&lt;/dyte-leave-button&gt; &lt;dyte-controlbar-button label=&quot;Add Filter&quot; size=&quot;lg&quot; id=&quot;arFilter&quot; style=&quot;background-color: black; color: white&quot;&gt; &lt;/dyte-controlbar-button&gt; &lt;dyte-controlbar-button label=&quot;Change Filter&quot; size=&quot;lg&quot; id=&quot;switchFilter&quot; style=&quot;background-color: black; color: white&quot;&gt; &lt;/dyte-controlbar-button&gt; &lt;/div&gt; &lt;div class=&quot;controlbar-section&quot;&gt; &lt;dyte-chat-toggle class=&quot;dyte&quot;&gt;&lt;/dyte-chat-toggle&gt; &lt;dyte-participants-toggle class=&quot;dyte&quot;&gt;&lt;/dyte-participants-toggle&gt; &lt;dyte-polls-toggle class=&quot;dyte&quot;&gt;&lt;/dyte-polls-toggle&gt; &lt;/div&gt; &lt;/div&gt; &lt;/div&gt; &lt;script type=&quot;module&quot; src=&quot;/main.js&quot;&gt;&lt;/script&gt; &lt;/body&gt;</pre><p>We use custom components offered by Dyte to create a default meeting UI.</p><ul><li><strong>Dyte Header:</strong> This component loads all the basic meeting details like the title, participant count, clock, and much more.</li><li><strong>Dyte Grid:</strong> This component renders all the video streams from all the participants in the meeting.</li><li><strong>Dyte Sidebar:</strong> This component renders chat, polls, participants list, and more.</li><li><strong>Dyte Controlbar:</strong> This is where we add controls for our meeting. A user can toggle their camera and mic, adjust settings or manage the AR filters from this control bar.</li></ul><p>We also insert a script that links to main.js, an external file that will contain all our business logic.</p><p><strong>Step 4: </strong>Write the business logic</p><p>We initialize a basic Dyte Meeting using our authToken and pass the meeting object to all our components.</p><pre>import icons from &#39;./icons.json&#39;; const init = async () =&gt; { // Initialize Dyte Meeting const meeting = await DyteClient.init({ authToken: import.meta.env.VITE_AUTH_TOKEN, defaults: { audio: false, video: false, }, }); // Passing the `meeting` prop to all our components const els = document.getElementsByClassName(&#39;dyte&#39;); els[0].config = meeting.self.config; for (const el of els) { el.meeting = meeting; } // Adding state management for the sidebar component document.body.addEventListener(&#39;dyteStateUpdate&#39;, ({ detail }) =&gt; { document.querySelector(&#39;dyte-sidebar&#39;).style.display = detail.activeSidebar ? &#39;flex&#39; : &#39;none&#39;; document.querySelector(&#39;dyte-sidebar&#39;).states = { sidebar: detail.activeSidebar ? detail.sidebar : &#39;none&#39; }; }); // Adding icons to our custom buttons const AddFilter = document.getElementById(&quot;arFilter&quot;); const SwitchFilter = document.getElementById(&quot;switchFilter&quot;); AddFilter.icon = icons.addFilter; SwitchFilter.icon = icons.switchFilter; }; init();</pre><p><strong>Step 5: </strong>Adding CSS to our meeting.</p><p>Create a style.css; this file will hold all our styles for the meeting.</p><pre>body, html { height: 100vh; width: 100vw; margin: 0; padding: 0; display: flex; flex-direction: column; background: rgb(8, 8, 8); } #app { display: flex; flex-direction: column; font-family: Arial, Helvetica, sans-serif; flex-grow: 1; color: white; } .ended-screen { align-items: center; justify-content: center; } .dyte-grid { flex-grow: 1; display: flex; flex-direction: row; } dyte-grid { flex-grow: 1; } dyte-sidebar { margin: 10px 20px; border-radius: 10px; } .dyte-controlbar { display: flex; flex-direction: row; align-items: center; justify-content: space-between; min-height: 20px; margin:10px 0; width: 100%; } .controlbar-section { display: flex; flex-direction: row; align-items: center; }</pre><p><strong>Step 6:</strong> Open the HTML file on a browser, and you will be prompted to allow access to your microphone and camera and you should be able to see the meeting interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PSAnuP5mrPzCgBrw.png" /></figure><h3>Setting up a DeepAR account</h3><p><strong>Step 1:</strong> Navigate to the <a href="https://developer.deepar.ai/">DeepAR developer home page</a> and click the ‘Get Started’ button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*n6FwGJK4DPPNFqTe.png" /></figure><p><strong>Step 2:</strong> Sign up for a free DeepAR account using your Google or GitHub account on the next page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zf2D-mVETJtpI1TF.png" /></figure><h3>Create a DeepAR project</h3><p><strong>Step 1:</strong> After you’ve signed up and logged in to your DeepAR account, you will be redirected to the projects page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*utqOOFb1kl5mmXme.png" /></figure><p><strong>Step 2:</strong> Click the ‘Create A Project’ button. Name the project AR Filters, agree to the terms and conditions, then click the continue button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/0*lD1d1oVAOriSN7sn.png" /></figure><p><strong>Step 3:</strong> After creating the project, you will be redirected to add an app. In this case, go to the Web App section and select Add App option. You will be prompted to add a domain name. Add your domain name and click the continue button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_triySoKn5mnJ1zO.png" /></figure><p><strong>Step 4: </strong>You will receive DeepAR App Key after creating the web app. The key will initialize the DeepAR filter as a license key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GnWrDNPbz6qcy9Pj.png" /></figure><h3>Adding AR Filters to your video calling app</h3><p>Now that you have obtained your License key and have a basic Dyte Meeting up and running. It is time to add fun AR filters to your meeting.</p><p><strong>Step 1: </strong>Install DeepAR</p><p>You can import DeepAR into your project using CDN.</p><pre>&lt;script src=&quot;&lt;https://cdn.jsdelivr.net/npm/deepar/js/deepar.js&gt;&quot;&gt;&lt;/script&gt;</pre><p><strong>Step 2: </strong>Initialize the DeepAR module</p><p>Add some filters to our project repositories and provide a path to those filters.</p><pre>// Initialize DeepAR const filters = [ &quot;./effects/lion&quot;, &quot;./effects/flowers&quot;, &quot;./effects/dalmatian&quot;, &quot;./effects/background_segmentation&quot;, &quot;./effects/background_blur&quot;, &quot;./effects/aviators&quot;, ];</pre><p>Next, we create a canvas element and initialize the DeepAR module.</p><pre>const deepARCanvas = document.createElement(&quot;canvas&quot;); deepARCanvas.width = 680; deepARCanvas.height = 480; const deepAR = await deepar.initialize({ licenseKey: import.meta.env.VITE_DEEP_AR_TOKEN, canvas: deepARCanvas, effect: filters[0], additionalOptions: { cameraConfig: { disableDefaultCamera: true, }, }, });</pre><p><strong>Step 3: </strong>Add AR filters to your Dyte video call</p><p>Here, we are creating two functions, one to toggle the AR filter on/off and the other to switch between filters.</p><p>We add middleware to Dyte using its AddVideoMiddleware API, which takes in a middleware function.</p><p>To switch between filters, we use DeepAR’s switchEffect method.</p><pre>let count = 0; let filterIndex = 0; const AddFilter = document.getElementById(&quot;arFilter&quot;); const SwitchFilter = document.getElementById(&quot;switchFilter&quot;); AddFilter.icon = icons.addFilter; SwitchFilter.icon = icons.switchFilter; AddFilter.addEventListener(&quot;click&quot;, toggleAR); SwitchFilter.addEventListener(&quot;click&quot;, filterChangeHandler); function toggleAR() { count++; if (count % 2 == 0) { AddFilter.label = &#39;Add Filter&#39; meeting.self.removeVideoMiddleware(AddTheme); } else { AddFilter.label = &#39;Remove Filter&#39; meeting.self.addVideoMiddleware(AddTheme); } } async function filterChangeHandler() { filterIndex = (filterIndex + 1) % filters.length; await deepAR.switchEffect(filters[filterIndex]); }</pre><p>Now, let’s create our video middleware function using DeepAR.</p><pre>async function AddTheme() { let lastProcessedImage = null; const intermediatoryCanvas = document.createElement(&quot;canvas&quot;); intermediatoryCanvas.width = 640; intermediatoryCanvas.height = 480; const intermediatoryCanvasCtx = intermediatoryCanvas.getContext(&quot;2d&quot;); return async (canvas, ctx) =&gt; { intermediatoryCanvasCtx.drawImage(canvas, 0, 0); if (lastProcessedImage) { ctx.drawImage( lastProcessedImage, 0, 0, lastProcessedImage.width, lastProcessedImage.height, 0, 0, canvas.width, canvas.height ); } await deepAR.processImage(intermediatoryCanvas); const image = new Image(); image.id = &quot;picture&quot;; image.src = await deepAR.takeScreenshot(); lastProcessedImage = image; }; }</pre><p>With that done, we have built a video calling app with AR filters using <a href="https://dyte.io/video-sdk">Dyte video SDK</a> and DeepAR! Get your AR filters on to look something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dcSJQ3OD5RWcQ3H8.png" /></figure><p>You can try out the live demo of the sample application <a href="https://dyte-io.github.io/video-calling-with-ar-filters/">here</a>, and the entire source code for the project is available <a href="https://github.com/dyte-io/video-calling-with-ar-filters">here</a>.</p><h3>Conclusion</h3><p>In conclusion, we have walked you through how to build a video calling app with AR filters using the Dyte video and DeepAR SDKs. With Dyte’s customizable UI Kit, you can easily manage controls, add or remove buttons, create custom controls, and do much more with just a few lines of code.</p><p>If this article tickled the builder in you, then check out <a href="https://dyte.io/live-streaming-sdk">Dyte’s live streaming SDK</a>. We provide easy-to-use SDKs to deliver interactive live streams at scale with much lower latency than traditional HLS. Feel at home with the same friendly developer experience and extensibility. Dyte’s live streaming SDK is <a href="https://dyte.io/ph">launching soon on Product Hunt</a>!</p><p><em>Get better insights on leveraging Dyte’s technology and discover how it can revolutionize your app’s communication capabilities with its </em><a href="https://dyte.io/video-sdk"><em>SDKs</em></a><em>. You can reach us at </em><a href="https://mailto:support@dyte.io/"><em>support@dyte.io</em></a><em> or ask our </em><a href="https://community.dyte.io/"><em>developer community</em></a><em> if you have any questions.</em></p><p><em>Originally published at </em><a href="https://dyte.io/blog/video-calling-app-ar-filters-dyte-deepar/"><em>https://dyte.io</em></a><em> on June 13, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13badf17ef65" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dyte/adding-ar-filters-to-video-calls-using-deepar-and-dyte-13badf17ef65">Adding AR Filters to Video Calls Using DeepAR and Dyte</a> was originally published in <a href="https://medium.com/dyte">Dyte</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding package.json II: Scripts]]></title>
            <link>https://medium.com/dyte/understanding-package-json-ii-scripts-b60431e11380?source=rss----9477120405bc---4</link>
            <guid isPermaLink="false">https://medium.com/p/b60431e11380</guid>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[npm]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Rohan Mukherjee]]></dc:creator>
            <pubDate>Wed, 05 Jul 2023 11:32:40 GMT</pubDate>
            <atom:updated>2023-07-05T11:32:40.447Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*AowPF4HeEuS6koPElBiqhg.jpeg" /><figcaption>Understanding package.json scripts</figcaption></figure><h3>Introduction to package.json scripts</h3><p>Welcome to the world of Javascript development, where building, testing, and deploying applications can be complex and time-consuming. As developers, we strive to automate these tasks as much as possible, and that’s where <a href="https://docs.npmjs.com/cli/v9/using-npm/scripts">npm scripts</a> come in. npm scripts are powerful and flexible tools that allow us to define custom commands and automate repetitive tasks with ease. They can help us save time, reduce errors, and improve collaboration within our teams.</p><p>In this blog post, we’ll explore the basics of npm scripts and how they can simplify your Node.js development workflow. We&#39;ll cover the syntax and structure of npm scripts, how to run them, and some common use cases. Whether you&#39;re a seasoned Node.js developer or just getting started, this guide will help you unlock the full potential of npm scripts and streamline your development process.</p><p>This article is a continuation of another article I recently published — <a href="https://dyte.io/blog/package-json/">Discovering package.json</a>, make sure to check it out as well!</p><h3>Why use npm scripts?</h3><p>npm scripts are used in every Node.js project. Certain workflows and commands are defined in the scripts section of the package.json file. npm (and other package managers) provide commands that can run these scripts without any additional dependencies or build requirements.</p><ol><li>Simplify build and development workflows: npm scripts allow developers to define custom commands for tasks like building, testing, and deploying applications. This makes it easy to automate repetitive tasks and simplify complex workflows, improving overall efficiency and productivity.</li><li>No need for additional build tools or dependencies: npm scripts are built into Node.js, so there’s no need for additional build tools or dependencies. This reduces the complexity of the development environment and makes it easier to share code across different platforms.</li><li>Easy to use and customize: npm scripts are easy to use and can be customized to suit the specific needs of each project. They provide a simple and flexible way to define custom commands and automate tasks without requiring extensive knowledge of build tools or complicated configuration files.</li><li>Document and share workflows: npm scripts can be used to document and share development workflows within a team. This helps ensure that everyone is on the same page and that the development process is consistent and reproducible.</li><li>Provide pre-and post-scripts: npm scripts can define pre- and post-scripts that run before or after a specific command. This allows developers to run any necessary setup or cleanup tasks before or after a command is executed, improving the overall reliability of the development process.</li></ol><h3>Basic npm scripts</h3><p>In your package.json file, you can add a scripts field to define your scripts. Inside this object, you can enter key-value pairs for your npm scripts, where the key is the script&#39;s name and the value is the shell command. The commands are typically passed to <a href="https://linux.die.net/man/1/sh">sh</a>, so you can run commands like ls too in your npm scripts.</p><p>The basic syntax for defining a script in the scripts field is:</p><pre>&quot;script-name&quot;: &quot;command&quot;</pre><p>Here, script-name is the name you want to give your script, and command is the shell command you want to execute when the script is run. For example, to define a script that runs tests using the mocha testing framework, you could add the following to your scripts field:</p><pre>&quot;test&quot;: &quot;mocha&quot;</pre><p>To run this script, you would execute the following command in your terminal:</p><pre>npm run test</pre><p>You can also define scripts that run multiple commands or use variables. For example, you could define a script that builds your application and then starts a development server like this:</p><pre>&quot;start&quot;: &quot;npm run build &amp;&amp; node server.js&quot;</pre><p>In this case, the start script runs two commands: npm run build (which could be defined to compile your source code) and node server.js (which starts the development server).</p><p>The start script is a special script, like test, etc. You can run npm start to run the start script instead of npm run start. Similarly, for running a script called test, you can just run npm test or even npm t instead of npm run test.</p><h3>Tricks and tips in scripts</h3><p>Let’s say you have the following package.json. You might not be aware of some neat tips and tricks in using these scripts.</p><pre>{<br>  &quot;name&quot;: &quot;dyte&quot;,<br>  &quot;version&quot;: &quot;1.0.0&quot;,<br>  &quot;description&quot;:&quot;  &quot;,<br>  &quot;main&quot;: &quot;index.js&quot;,<br>  &quot;scripts&quot;: {<br>    &quot;build&quot;: &quot;echo VERSION=$npm_package_version&quot;,<br>    &quot;test&quot;: &quot;echo &#39;Running tests&#39;&quot;,<br>    &quot;build-and-test&quot;: &quot;npm run build &amp;&amp; npm run test&quot;,<br>    &quot;build-and-test-parallel&quot;: &quot;npm run build &amp; npm run test&quot;<br>  },<br>  &quot;author&quot;: &quot;Rohan Mukherjee &lt;rohan@dyte.io&gt; (&lt;https://dyte.io&gt;)&quot;,<br>  &quot;license&quot;: &quot;ISC&quot;<br>}</pre><h3>Using features provided</h3><p>When you run npm run build, you can see that the $npm_package_version variable is replaced with the version of the package specified in the package.json.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/0*l5Igm-t4UXIWrJJ4.png" /></figure><p>Similarly, you can use other variables like $npm_package_name in your script to extract values from the package.json file.</p><p>If you want to execute a script you’re unsure about the existence of, you can use the -if-present flag so that the command doesn&#39;t exit with a non-zero status code.</p><pre>npm run build --if-present</pre><p>If you’re using workspaces and want to run a script in a different workspace, you can specify the workspace name using --workspace or -w.</p><p>Let’s say that you’ve already defined a lint script as follows:</p><pre>&quot;lint&quot;: &quot;eslint ./src --ext .ts --ext .js&quot;</pre><p>Upon running the lint script, eslint will list all the linting issues from the src directory of your project. You want to define a script to fix all auto-fixable linting issues. You&#39;d typically have to define a command like</p><pre>&quot;lint:fix&quot;: &quot;eslint ./src --ext .ts --ext .js --fix&quot;</pre><p>Now, if you want to include the extension for .jsx files also in your lint script, you&#39;d have to make the change in both places, and it&#39;s easy to miss changing one of these places.</p><p>Since the lint:fix command is very similar to the lint command, and only requires an additional flag, you can reference the lint command in the lint:fix command to reuse it. For instance, you can run</p><pre>&quot;lint:fix&quot;: &quot;npm run lint -- --fix&quot;</pre><p>This will run the lint command and pass an additional argument --fix to it. Anything you add after -- will be passed as a string to the npm command specified before.</p><h3>Using sh tricks in scripts</h3><p>Since the scripts are run using sh (on all Unix-based systems), you can use some neat sh tricks to run multiple scripts together. On Windows systems, the scripts are run using the default shell - cmd (Command Prompt). However, you can change the shell used on Windows to something like git bash using the following command.</p><pre>npm config set script-shell &quot;C:\\\\Program Files\\\\git\\\\bin\\\\bash.exe&quot;</pre><p>You can also run 2 scripts using the &amp;&amp; operator in a script. For instance, if you run the build-and-test script, it will execute the test npm command only after the build script completes successfully (exits with status code 0).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*JuLOjNwweASPKHJu.png" /></figure><p>You can also run these scripts in parallel, using a since &amp;. The build-and-test-parallel command runs the scripts together and is not guaranteed which process will complete first. In this example, the test script finished before the build script.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/0*ry8VZ9YLqWsfmhtu.png" /></figure><p>These operators are however not specific to npm scripts. You can run them in sh or bash directly as well. It&#39;s also possible to run a command upon the failure of another. For instance, if you want to run a command upon the failure of another, you can use the || operator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/0*FuIdz9ZiT9yZfM0C.png" /></figure><p>In the above screenshot, one command exits with the code 1 (which indicates that the command failed), so the Goodbye script runs post that.</p><p>In fact, you can chain commands using &amp;&amp; and || to decide what to execute upon success and what to execute upon failure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/0*BgS9j-TO1vj8e-3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/0*_00u7CTNxj-BTV2V.png" /></figure><h3>Lifecycle Scripts</h3><p>lifecycle scripts are a set of predefined script names that are executed automatically by npm at specific stages during the lifecycle of a package. These scripts are defined in the package.json file of a Node.js project and can be used to perform various tasks such as installing dependencies, running tests, building the project, and deploying the application.</p><p>The npm lifecycle consists of several stages that are executed in a specific order:</p><ul><li>“prepublish”: This script runs before the package is packed and published, and is used to prepare the package for distribution.</li><li>“prepare”: This script runs both during local development and when the package is installed as a dependency of another package. It is used to prepare the package for use, such as by building or compiling the code.</li><li>“preinstall”: This script runs before the package is installed and is used to perform any necessary setup tasks before dependencies are installed.</li><li>“postinstall”: This script runs after the package is installed and is used to perform any necessary setup tasks after dependencies are installed.</li><li>“preuninstall”: This script runs before the package is uninstalled and is used to perform any necessary cleanup tasks before dependencies are removed.</li><li>“postuninstall”: This script runs after the package is uninstalled and is used to perform any necessary cleanup tasks after dependencies are removed.</li><li>“preversion”: This script runs before the version of the package is updated and is used to perform any necessary tasks before the version is changed.</li><li>“postversion”: This script runs after the version of the package is updated and is used to perform any necessary tasks after the version is changed.</li></ul><p>To define a custom pre- or post-script, you need to add a property to the scripts object in your package.json file. The property name should be in the format of &quot;pre&quot; or &quot;post&quot; followed by the name of the script you want to run. For example, to define a preinstall script, you can add the following to your package.json file:</p><pre>{<br>  &quot;name&quot;: &quot;my-package&quot;,<br>  &quot;version&quot;: &quot;1.0.0&quot;,<br>  &quot;scripts&quot;: {<br>    &quot;preinstall&quot;: &quot;echo Running pre-install script&quot;<br>  }<br>}</pre><p>In this example, the pre-install script will run before the npm install command is executed. You can replace the echo command with any shell command or node script you want to run.</p><p>Similarly, you can define a post-install script by adding a property named “post-install” to the scripts object:</p><pre>{<br>  &quot;name&quot;: &quot;my-package&quot;,<br>  &quot;version&quot;: &quot;1.0.0&quot;,<br>  &quot;scripts&quot;: {<br>    &quot;postinstall&quot;: &quot;echo Running post-install script&quot;<br>  }<br>}</pre><p>This script will run after the npm install command is executed.</p><p>You can define pre- and post-scripts for other npm commands as well, such as “pretest”, “posttest”, “prepublish”, “postpublish”, etc. Just prefix the script name with “pre” or “post” accordingly.</p><p>Note that the pre and post-scripts are executed in the order they are defined in the package.json file, and you can define multiple pre and post-scripts for the same command.</p><blockquote>Disclaimer: Use “postinstall” scripts carefully, as supply chain security software will fire up alerts upon detecting postinstall scripts. In npm packages, postinstall scripts can be used as a means to install malware into the installer’s system, hence it’s advisable not to use it in an npm package. However, it’s okay to use it as long as you remove it from your package.json when publishing the package.</blockquote><h3>Best Practices</h3><p>Here are some best practices for writing npm scripts:</p><ol><li>Keep scripts simple and readable: It is important to keep your scripts simple and easy to read. Use meaningful script names and avoid writing long scripts with many commands. Break down complex tasks into smaller sub-tasks and write separate scripts for them.</li><li>Use variables: Use environment variables to store commonly used values, such as directories or file paths, and use them in your scripts to avoid hardcoding.</li><li>Use package.json to declare dependencies: Declare dependencies in your package.json file instead of installing them inside your scripts. This will ensure that your dependencies are installed before your script runs.</li><li>Avoid platform-specific commands: Avoid using platform-specific commands in your scripts. Use cross-platform tools like Node.js or Bash to ensure that your scripts work on different platforms. For instance, if you want your npm script to remove a certain directory using the rm -rf command, this would work perfectly on a Linux or Mac machine but would error out on Windows. To avoid this, you can use a cross-platform package such as rimraf.</li><li>Keep scripts independent: Keep your scripts independent of each other to avoid dependency issues. If you need to run one script after another, use a task runner like <a href="https://gulpjs.com/docs/en/getting-started/quick-start/">Gulp</a> or <a href="https://gruntjs.com/">Grunt</a> to define tasks and their dependencies.</li><li>Use exit codes: Use exit codes to indicate the success or failure of your scripts. Return a non-zero exit code to indicate that the script failed and zero to indicate that the script succeeded.</li><li>Use pre- and post-scripts: Use pre- and post-scripts to execute common tasks before and after your scripts run. For example, use pre-scripts to ensure that your dependencies are up-to-date and post-scripts to clean up any temporary files.</li></ol><p>Take a look at <a href="https://github.com/dyte-in/docs/blob/main/package.json#L5-L22">this</a> package.json file in our <a href="https://github.com/dyte-in/docs/blob/main/package.json#L5-L22">docs repository</a> to get a good idea about how npm scripts are used in repositories. You can also check out how lifecycle scripts can be used efficiently in this example: <a href="https://github.com/roerohan/react-vnc/blob/main/package.json#L50-L63">react-vnc</a>.</p><h3>Summary</h3><p>In conclusion, npm scripts are a powerful tool for automating tasks and simplifying the development workflow in modern web development. With just a few lines of code, you can define scripts to run tests, build your project, start your development server, and much more.</p><p>To make the most of npm scripts, it is essential to follow some best practices such as keeping scripts simple and readable, using variables, avoiding platform-specific commands, documenting your scripts, and using pre- and post-scripts.</p><p>By using package.json npm scripts, you can streamline your development workflow and save time and effort. You can also make your code more reliable and maintainable by automating repetitive tasks and ensuring that your code is error-free.</p><p>Overall, npm scripts are a powerful and essential tool for any modern Node.js developer. With the right knowledge and best practices, you can harness the full potential of npm scripts to create more efficient, reliable, and scalable projects.</p><p>I hope you found this post informative and engaging. If you have any thoughts or feedback, feel free to reach out to me on <a href="https://twitter.com/roerohan">Twitter</a> or <a href="https://www.linkedin.com/in/roerohan/">LinkedIn</a> 😄. Stay tuned for more related blog posts in the future!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b60431e11380" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dyte/understanding-package-json-ii-scripts-b60431e11380">Understanding package.json II: Scripts</a> was originally published in <a href="https://medium.com/dyte">Dyte</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>