Обработка потока WebRTC RTC на сервере node js с помощью ffmpeg

Я пытаюсь создать приложение для видеочата, в котором я создаю RTCPeerConnection, создаю предложение и сохраняю SDP в файле. Я хочу, чтобы SDP, который я отправил на сервер, был отправлен на RTMP-сервер, например Nginx RTMP. Вот предложение SDP, которое я получаю

v=0
o=- 3688975056307578818 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=msid-semantic: WMS AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq
m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Xtid
a=ice-pwd:jE3iBRpWqFaIN3UJVOAh0G/1
a=ice-options:trickle
a=fingerprint:sha-256 48:E4:36:A6:24:66:F6:40:0F:93:9C:AB:C9:93:DF:C7:0F:D1:21:F5:9E:F7:FA:A8:58:84:1F:68:A1:61:B6:0F
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq 284f316d-6c5d-4283-af4d-86d44803807d
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:112 telephone-event/32000
a=rtpmap:113 telephone-event/16000
a=rtpmap:126 telephone-event/8000
a=ssrc:2317617486 cname:fpKjO3/hiHYYBw7w
a=ssrc:2317617486 msid:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq 284f316d-6c5d-4283-af4d-86d44803807d
a=ssrc:2317617486 mslabel:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq
a=ssrc:2317617486 label:284f316d-6c5d-4283-af4d-86d44803807d
m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102 122 127 121 125 107 108 109 124 120 123 119 114 115 116
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Xtid
a=ice-pwd:jE3iBRpWqFaIN3UJVOAh0G/1
a=ice-options:trickle
a=fingerprint:sha-256 48:E4:36:A6:24:66:F6:40:0F:93:9C:AB:C9:93:DF:C7:0F:D1:21:F5:9E:F7:FA:A8:58:84:1F:68:A1:61:B6:0F
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07
a=extmap:9 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendrecv
a=msid:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq 7106bf9f-3f79-4f24-959b-b82b603a7acc
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 VP9/90000
a=rtcp-fb:100 goog-remb
a=rtcp-fb:100 transport-cc
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=fmtp:100 profile-id=2
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:102 H264/90000
a=rtcp-fb:102 goog-remb
a=rtcp-fb:102 transport-cc
a=rtcp-fb:102 ccm fir
a=rtcp-fb:102 nack
a=rtcp-fb:102 nack pli
a=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:122 rtx/90000
a=fmtp:122 apt=102
a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f
a=rtpmap:121 rtx/90000
a=fmtp:121 apt=127
a=rtpmap:125 H264/90000
a=rtcp-fb:125 goog-remb
a=rtcp-fb:125 transport-cc
a=rtcp-fb:125 ccm fir
a=rtcp-fb:125 nack
a=rtcp-fb:125 nack pli
a=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=125
a=rtpmap:108 H264/90000
a=rtcp-fb:108 goog-remb
a=rtcp-fb:108 transport-cc
a=rtcp-fb:108 ccm fir
a=rtcp-fb:108 nack
a=rtcp-fb:108 nack pli
a=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f
a=rtpmap:109 rtx/90000
a=fmtp:109 apt=108
a=rtpmap:124 H264/90000
a=rtcp-fb:124 goog-remb
a=rtcp-fb:124 transport-cc
a=rtcp-fb:124 ccm fir
a=rtcp-fb:124 nack
a=rtcp-fb:124 nack pli
a=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d0032
a=rtpmap:120 rtx/90000
a=fmtp:120 apt=124
a=rtpmap:123 H264/90000
a=rtcp-fb:123 goog-remb
a=rtcp-fb:123 transport-cc
a=rtcp-fb:123 ccm fir
a=rtcp-fb:123 nack
a=rtcp-fb:123 nack pli
a=fmtp:123 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=640032
a=rtpmap:119 rtx/90000
a=fmtp:119 apt=123
a=rtpmap:114 red/90000
a=rtpmap:115 rtx/90000
a=fmtp:115 apt=114
a=rtpmap:116 ulpfec/90000
a=ssrc-group:FID 967462980 1884395933
a=ssrc:967462980 cname:fpKjO3/hiHYYBw7w
a=ssrc:967462980 msid:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq 7106bf9f-3f79-4f24-959b-b82b603a7acc
a=ssrc:967462980 mslabel:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq
a=ssrc:967462980 label:7106bf9f-3f79-4f24-959b-b82b603a7acc
a=ssrc:1884395933 cname:fpKjO3/hiHYYBw7w
a=ssrc:1884395933 msid:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq 7106bf9f-3f79-4f24-959b-b82b603a7acc
a=ssrc:1884395933 mslabel:AqaeHB0L8drTd5r0qQnzCSeYVf4bHFaVqfrq
a=ssrc:1884395933 label:7106bf9f-3f79-4f24-959b-b82b603a7acc
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:Xtid
a=ice-pwd:jE3iBRpWqFaIN3UJVOAh0G/1
a=ice-options:trickle
a=fingerprint:sha-256 48:E4:36:A6:24:66:F6:40:0F:93:9C:AB:C9:93:DF:C7:0F:D1:21:F5:9E:F7:FA:A8:58:84:1F:68:A1:61:B6:0F
a=setup:actpass
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

это команда ffmpeg

ffmpeg -protocol_whitelist rtp,udp,file -loglevel trace -analyzeduration 300M -probesize 300M -i test.sdp -c:v copy -c:a aac -ar 16k -ac 1 -preset ultrafast -tune zerolatency rtmp://127.0.0.1/live/1234

Также пробую это

ffmpeg -loglevel debug -protocol_whitelist file,crypto,udp,rtp -re -vcodec libvpx -acodec opus -i test.sdp -vcodec libx264 -acodec aac -y output.mp4

FFMpeg выдает вот такую ​​ошибку

ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.17)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2_2 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --disable-libjack --disable-indev=jack
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Splitting the commandline.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'.
Reading option '-protocol_whitelist' ... matched as AVOption 'protocol_whitelist' with argument 'file,crypto,udp,rtp'.
Reading option '-re' ... matched as option 're' (read input at native frame rate) with argument '1'.
Reading option '-vcodec' ... matched as option 'vcodec' (force video codec ('copy' to copy stream)) with argument 'libvpx'.
Reading option '-acodec' ... matched as option 'acodec' (force audio codec ('copy' to copy stream)) with argument 'opus'.
Reading option '-i' ... matched as input url with argument 'test.sdp'.
Reading option '-vcodec' ... matched as option 'vcodec' (force video codec ('copy' to copy stream)) with argument 'libx264'.
Reading option '-acodec' ... matched as option 'acodec' (force audio codec ('copy' to copy stream)) with argument 'aac'.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option 'output.mp4' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument debug.
Applying option y (overwrite output files) with argument 1.
Successfully parsed a group of options.
Parsing a group of options: input url test.sdp.
Applying option re (read input at native frame rate) with argument 1.
Applying option vcodec (force video codec ('copy' to copy stream)) with argument libvpx.
Applying option acodec (force audio codec ('copy' to copy stream)) with argument opus.
Successfully parsed a group of options.
Opening an input file: test.sdp.
[NULL @ 0x7f949000ac00] Opening 'test.sdp' for reading
[sdp @ 0x7f949000ac00] Format sdp probed with size=2048 and score=50
[sdp @ 0x7f949000ac00] audio codec set to: opus
[sdp @ 0x7f949000ac00] audio samplerate set to: 48000
[sdp @ 0x7f949000ac00] audio channels set to: 2
[sdp @ 0x7f949000ac00] audio codec set to: opus
[sdp @ 0x7f949000ac00] audio samplerate set to: 16000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: opus
[sdp @ 0x7f949000ac00] audio samplerate set to: 32000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: adpcm_g722
[sdp @ 0x7f949000ac00] audio samplerate set to: 8000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: pcm_mulaw
[sdp @ 0x7f949000ac00] audio samplerate set to: 8000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: pcm_alaw
[sdp @ 0x7f949000ac00] audio samplerate set to: 8000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: pcm_alaw
[sdp @ 0x7f949000ac00] audio samplerate set to: 32000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: pcm_alaw
[sdp @ 0x7f949000ac00] audio samplerate set to: 16000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: (null)
[sdp @ 0x7f949000ac00] audio samplerate set to: 8000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: (null)
[sdp @ 0x7f949000ac00] audio samplerate set to: 48000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: (null)
[sdp @ 0x7f949000ac00] audio samplerate set to: 32000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: (null)
[sdp @ 0x7f949000ac00] audio samplerate set to: 16000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] audio codec set to: (null)
[sdp @ 0x7f949000ac00] audio samplerate set to: 8000
[sdp @ 0x7f949000ac00] audio channels set to: 1
[sdp @ 0x7f949000ac00] video codec set to: vp8
    Last message repeated 20 times
[udp @ 0x7f948fc07200] bind failed: Address already in use
[AVIOContext @ 0x7f948fe1c4c0] Statistics: 5985 bytes read, 0 seeks
test.sdp: Invalid data found when processing input

Может ли кто-нибудь указать, что я делаю неправильно здесь? или я на правильном пути? Пожалуйста помоги!


person Dave B    schedule 22.07.2020    source источник


Ответы (1)


К сожалению, это не сработает (сегодня), ffmpeg может добавить поддержку WebRTC в будущем!

  • WebRTC требует обмена предложением/ответом. Таким образом, ffmpeg должен генерировать возврат SDP (ответ).
  • ffmpeg не имеет реализации ICE.
  • ffmpeg не умеет делать DTLS -> SRTP

Чтобы сделать WebRTC -> RTMP, вы можете использовать это, который выполняет WebRTC, а затем отправляет в ffmpeg как [это] (https://github.com/pion/example-webrtc-applications/blob/master/twitch/main.go#L139-L140). Вы можете оптимизировать это много. Единственная причина, по которой он делает mkv через стандартный ввод, заключается в том, что он переносим на Windows. Вы можете сделать несколько каналов и отправить H264/Opus напрямую и сэкономить много кода, если вы используете только Linux.

Вы также можете использовать GStreamer и сделать webrtc -> rtmpsink !

person Sean DuBois    schedule 22.07.2020
comment
Это большая помощь @Sean. Спасибо! Я попробую. - person Dave B; 23.07.2020