forked from loeade/invidious
Compare commits
No commits in common. "82c962f3b820aa6c312feec04440e00419165f96" and "971155db9d8f9a056fb3c4c6008fa0313f59380e" have entirely different histories.
82c962f3b8
...
971155db9d
18 changed files with 136 additions and 247 deletions
39
.github/workflows/ci.yml
vendored
39
.github/workflows/ci.yml
vendored
|
@ -38,11 +38,10 @@ jobs:
|
||||||
matrix:
|
matrix:
|
||||||
stable: [true]
|
stable: [true]
|
||||||
crystal:
|
crystal:
|
||||||
|
- 1.9.2
|
||||||
- 1.10.1
|
- 1.10.1
|
||||||
- 1.11.2
|
- 1.11.2
|
||||||
- 1.12.1
|
- 1.12.1
|
||||||
- 1.13.2
|
|
||||||
- 1.14.0
|
|
||||||
include:
|
include:
|
||||||
- crystal: nightly
|
- crystal: nightly
|
||||||
stable: false
|
stable: false
|
||||||
|
@ -52,11 +51,6 @@ jobs:
|
||||||
with:
|
with:
|
||||||
submodules: true
|
submodules: true
|
||||||
|
|
||||||
- name: Install required APT packages
|
|
||||||
run: |
|
|
||||||
sudo apt install -y libsqlite3-dev
|
|
||||||
shell: bash
|
|
||||||
|
|
||||||
- name: Install Crystal
|
- name: Install Crystal
|
||||||
uses: crystal-lang/install-crystal@v1.8.0
|
uses: crystal-lang/install-crystal@v1.8.0
|
||||||
with:
|
with:
|
||||||
|
@ -65,9 +59,7 @@ jobs:
|
||||||
- name: Cache Shards
|
- name: Cache Shards
|
||||||
uses: actions/cache@v3
|
uses: actions/cache@v3
|
||||||
with:
|
with:
|
||||||
path: |
|
path: ./lib
|
||||||
./lib
|
|
||||||
./bin
|
|
||||||
key: shards-${{ hashFiles('shard.lock') }}
|
key: shards-${{ hashFiles('shard.lock') }}
|
||||||
|
|
||||||
- name: Install Shards
|
- name: Install Shards
|
||||||
|
@ -79,6 +71,14 @@ jobs:
|
||||||
- name: Run tests
|
- name: Run tests
|
||||||
run: crystal spec
|
run: crystal spec
|
||||||
|
|
||||||
|
- name: Run lint
|
||||||
|
run: |
|
||||||
|
if ! crystal tool format --check; then
|
||||||
|
crystal tool format
|
||||||
|
git diff
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
- name: Build
|
- name: Build
|
||||||
run: crystal build --warnings all --error-on-warnings --error-trace src/invidious.cr
|
run: crystal build --warnings all --error-on-warnings --error-trace src/invidious.cr
|
||||||
|
|
||||||
|
@ -124,12 +124,8 @@ jobs:
|
||||||
- name: Test Docker
|
- name: Test Docker
|
||||||
run: while curl -Isf http://localhost:3000; do sleep 1; done
|
run: while curl -Isf http://localhost:3000; do sleep 1; done
|
||||||
|
|
||||||
lint:
|
ameba_lint:
|
||||||
|
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
|
|
||||||
continue-on-error: true
|
|
||||||
|
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
with:
|
with:
|
||||||
|
@ -149,18 +145,7 @@ jobs:
|
||||||
key: shards-${{ hashFiles('shard.lock') }}
|
key: shards-${{ hashFiles('shard.lock') }}
|
||||||
|
|
||||||
- name: Install Shards
|
- name: Install Shards
|
||||||
run: |
|
run: shards install
|
||||||
if ! shards check; then
|
|
||||||
shards install
|
|
||||||
fi
|
|
||||||
|
|
||||||
- name: Check Crystal formatter compliance
|
|
||||||
run: |
|
|
||||||
if ! crystal tool format --check; then
|
|
||||||
crystal tool format
|
|
||||||
git diff
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
- name: Run Ameba linter
|
- name: Run Ameba linter
|
||||||
run: bin/ameba
|
run: bin/ameba
|
||||||
|
|
12
CHANGELOG.md
12
CHANGELOG.md
|
@ -5,12 +5,6 @@
|
||||||
|
|
||||||
### Full list of pull requests merged since the last release (newest first)
|
### Full list of pull requests merged since the last release (newest first)
|
||||||
|
|
||||||
* Add "Filipino (auto-generated)" to the list of caption languages ([#4995], by @SamantazFox)
|
|
||||||
* Makefile: Add MT option to enable the 'preview_mt' flag ([#4993], by @SamantazFox)
|
|
||||||
* SigHelper: Reconnect to signature helper ([#4991], thanks @Fijxu)
|
|
||||||
* Fix player menus hiding onHover ready ([#4750], thanks @giacomocerquone)
|
|
||||||
* Use connection pools when requesting images from YouTube ([#4326], thanks @syeopite)
|
|
||||||
* Add support for using Invidious through a HTTP Proxy ([#4270], thanks @syeopite)
|
|
||||||
* Search: Fix 'youtu.be' URLs in sanitizer ([#4894], by @SamantazFox)
|
* Search: Fix 'youtu.be' URLs in sanitizer ([#4894], by @SamantazFox)
|
||||||
* Ameba: Disable Style/RedundantNext rule ([#4888], thanks @syeopite)
|
* Ameba: Disable Style/RedundantNext rule ([#4888], thanks @syeopite)
|
||||||
* Playlists: Fix 'invalid byte sequence' error when subscribing ([#4887], thanks @DmitrySandalov)
|
* Playlists: Fix 'invalid byte sequence' error when subscribing ([#4887], thanks @DmitrySandalov)
|
||||||
|
@ -28,10 +22,7 @@
|
||||||
|
|
||||||
[#4122]: https://github.com/iv-org/invidious/pull/4122
|
[#4122]: https://github.com/iv-org/invidious/pull/4122
|
||||||
[#4193]: https://github.com/iv-org/invidious/pull/4193
|
[#4193]: https://github.com/iv-org/invidious/pull/4193
|
||||||
[#4270]: https://github.com/iv-org/invidious/pull/4270
|
|
||||||
[#4326]: https://github.com/iv-org/invidious/pull/4326
|
|
||||||
[#4652]: https://github.com/iv-org/invidious/pull/4652
|
[#4652]: https://github.com/iv-org/invidious/pull/4652
|
||||||
[#4750]: https://github.com/iv-org/invidious/pull/4750
|
|
||||||
[#4850]: https://github.com/iv-org/invidious/pull/4850
|
[#4850]: https://github.com/iv-org/invidious/pull/4850
|
||||||
[#4862]: https://github.com/iv-org/invidious/pull/4862
|
[#4862]: https://github.com/iv-org/invidious/pull/4862
|
||||||
[#4863]: https://github.com/iv-org/invidious/pull/4863
|
[#4863]: https://github.com/iv-org/invidious/pull/4863
|
||||||
|
@ -42,9 +33,6 @@
|
||||||
[#4928]: https://github.com/iv-org/invidious/pull/4928
|
[#4928]: https://github.com/iv-org/invidious/pull/4928
|
||||||
[#4930]: https://github.com/iv-org/invidious/pull/4930
|
[#4930]: https://github.com/iv-org/invidious/pull/4930
|
||||||
[#4942]: https://github.com/iv-org/invidious/pull/4942
|
[#4942]: https://github.com/iv-org/invidious/pull/4942
|
||||||
[#4991]: https://github.com/iv-org/invidious/pull/4991
|
|
||||||
[#4993]: https://github.com/iv-org/invidious/pull/4993
|
|
||||||
[#4995]: https://github.com/iv-org/invidious/pull/4995
|
|
||||||
|
|
||||||
|
|
||||||
## v2.20240825.2 (2024-08-26)
|
## v2.20240825.2 (2024-08-26)
|
||||||
|
|
9
Makefile
9
Makefile
|
@ -7,11 +7,6 @@ STATIC := 0
|
||||||
|
|
||||||
NO_DBG_SYMBOLS := 0
|
NO_DBG_SYMBOLS := 0
|
||||||
|
|
||||||
# Enable multi-threading.
|
|
||||||
# Warning: Experimental feature!!
|
|
||||||
# invidious is not stable when MT is enabled.
|
|
||||||
MT := 0
|
|
||||||
|
|
||||||
|
|
||||||
FLAGS ?=
|
FLAGS ?=
|
||||||
|
|
||||||
|
@ -24,10 +19,6 @@ ifeq ($(STATIC), 1)
|
||||||
FLAGS += --static
|
FLAGS += --static
|
||||||
endif
|
endif
|
||||||
|
|
||||||
ifeq ($(MT), 1)
|
|
||||||
FLAGS += -Dpreview_mt
|
|
||||||
endif
|
|
||||||
|
|
||||||
|
|
||||||
ifeq ($(NO_DBG_SYMBOLS), 1)
|
ifeq ($(NO_DBG_SYMBOLS), 1)
|
||||||
FLAGS += --no-debug
|
FLAGS += --no-debug
|
||||||
|
|
|
@ -68,7 +68,6 @@
|
||||||
|
|
||||||
.video-js.player-style-youtube .vjs-menu-button-popup .vjs-menu {
|
.video-js.player-style-youtube .vjs-menu-button-popup .vjs-menu {
|
||||||
margin-bottom: 2em;
|
margin-bottom: 2em;
|
||||||
padding-top: 2em
|
|
||||||
}
|
}
|
||||||
|
|
||||||
.video-js.player-style-youtube .vjs-progress-control .vjs-progress-holder, .video-js.player-style-youtube .vjs-progress-control {height: 5px;
|
.video-js.player-style-youtube .vjs-progress-control .vjs-progress-holder, .video-js.player-style-youtube .vjs-progress-control {height: 5px;
|
||||||
|
|
|
@ -173,17 +173,6 @@ https_only: false
|
||||||
##
|
##
|
||||||
#force_resolve:
|
#force_resolve:
|
||||||
|
|
||||||
##
|
|
||||||
## Configuration for using a HTTP proxy
|
|
||||||
##
|
|
||||||
## If unset, then no HTTP proxy will be used.
|
|
||||||
##
|
|
||||||
http_proxy:
|
|
||||||
user:
|
|
||||||
password:
|
|
||||||
host:
|
|
||||||
port:
|
|
||||||
|
|
||||||
|
|
||||||
##
|
##
|
||||||
## Use Innertube's transcripts API instead of timedtext for closed captions
|
## Use Innertube's transcripts API instead of timedtext for closed captions
|
||||||
|
|
|
@ -286,7 +286,6 @@
|
||||||
"Esperanto": "Esperanto",
|
"Esperanto": "Esperanto",
|
||||||
"Estonian": "Estonian",
|
"Estonian": "Estonian",
|
||||||
"Filipino": "Filipino",
|
"Filipino": "Filipino",
|
||||||
"Filipino (auto-generated)": "Filipino (auto-generated)",
|
|
||||||
"Finnish": "Finnish",
|
"Finnish": "Finnish",
|
||||||
"French": "French",
|
"French": "French",
|
||||||
"French (auto-generated)": "French (auto-generated)",
|
"French (auto-generated)": "French (auto-generated)",
|
||||||
|
|
2
mocks
2
mocks
|
@ -1 +1 @@
|
||||||
Subproject commit b55d58dea94f7144ff0205857dfa70ec14eaa872
|
Subproject commit 11ec372f72747c09d48ffef04843f72be67d5b54
|
|
@ -10,7 +10,7 @@ shards:
|
||||||
|
|
||||||
backtracer:
|
backtracer:
|
||||||
git: https://github.com/sija/backtracer.cr.git
|
git: https://github.com/sija/backtracer.cr.git
|
||||||
version: 1.2.2
|
version: 1.2.1
|
||||||
|
|
||||||
db:
|
db:
|
||||||
git: https://github.com/crystal-lang/crystal-db.git
|
git: https://github.com/crystal-lang/crystal-db.git
|
||||||
|
@ -20,10 +20,6 @@ shards:
|
||||||
git: https://github.com/crystal-loot/exception_page.git
|
git: https://github.com/crystal-loot/exception_page.git
|
||||||
version: 0.2.2
|
version: 0.2.2
|
||||||
|
|
||||||
http_proxy:
|
|
||||||
git: https://github.com/mamantoha/http_proxy.git
|
|
||||||
version: 0.10.3
|
|
||||||
|
|
||||||
kemal:
|
kemal:
|
||||||
git: https://github.com/kemalcr/kemal.git
|
git: https://github.com/kemalcr/kemal.git
|
||||||
version: 1.1.2
|
version: 1.1.2
|
||||||
|
@ -46,7 +42,7 @@ shards:
|
||||||
|
|
||||||
spectator:
|
spectator:
|
||||||
git: https://github.com/icy-arctic-fox/spectator.git
|
git: https://github.com/icy-arctic-fox/spectator.git
|
||||||
version: 0.10.6
|
version: 0.10.4
|
||||||
|
|
||||||
sqlite3:
|
sqlite3:
|
||||||
git: https://github.com/crystal-lang/crystal-sqlite3.git
|
git: https://github.com/crystal-lang/crystal-sqlite3.git
|
||||||
|
|
|
@ -28,9 +28,6 @@ dependencies:
|
||||||
athena-negotiation:
|
athena-negotiation:
|
||||||
github: athena-framework/negotiation
|
github: athena-framework/negotiation
|
||||||
version: ~> 0.1.1
|
version: ~> 0.1.1
|
||||||
http_proxy:
|
|
||||||
github: mamantoha/http_proxy
|
|
||||||
version: ~> 0.10.3
|
|
||||||
|
|
||||||
development_dependencies:
|
development_dependencies:
|
||||||
spectator:
|
spectator:
|
||||||
|
|
|
@ -17,8 +17,8 @@ Spectator.describe "parse_video_info" do
|
||||||
# Basic video infos
|
# Basic video infos
|
||||||
|
|
||||||
expect(info["title"].as_s).to eq("I Gave My 100,000,000th Subscriber An Island")
|
expect(info["title"].as_s).to eq("I Gave My 100,000,000th Subscriber An Island")
|
||||||
expect(info["views"].as_i).to eq(220_226_287)
|
expect(info["views"].as_i).to eq(126_573_823)
|
||||||
expect(info["likes"].as_i).to eq(6_870_691)
|
expect(info["likes"].as_i).to eq(5_157_654)
|
||||||
|
|
||||||
# For some reason the video length from VideoDetails and the
|
# For some reason the video length from VideoDetails and the
|
||||||
# one from microformat differs by 1s...
|
# one from microformat differs by 1s...
|
||||||
|
@ -48,12 +48,12 @@ Spectator.describe "parse_video_info" do
|
||||||
|
|
||||||
expect(info["relatedVideos"].as_a.size).to eq(20)
|
expect(info["relatedVideos"].as_a.size).to eq(20)
|
||||||
|
|
||||||
expect(info["relatedVideos"][0]["id"]).to eq("krsBRQbOPQ4")
|
expect(info["relatedVideos"][0]["id"]).to eq("Hwybp38GnZw")
|
||||||
expect(info["relatedVideos"][0]["title"]).to eq("$1 vs $250,000,000 Private Island!")
|
expect(info["relatedVideos"][0]["title"]).to eq("I Built Willy Wonka's Chocolate Factory!")
|
||||||
expect(info["relatedVideos"][0]["author"]).to eq("MrBeast")
|
expect(info["relatedVideos"][0]["author"]).to eq("MrBeast")
|
||||||
expect(info["relatedVideos"][0]["ucid"]).to eq("UCX6OQ3DkcsbYNE6H8uQQuVA")
|
expect(info["relatedVideos"][0]["ucid"]).to eq("UCX6OQ3DkcsbYNE6H8uQQuVA")
|
||||||
expect(info["relatedVideos"][0]["view_count"]).to eq("230617484")
|
expect(info["relatedVideos"][0]["view_count"]).to eq("179877630")
|
||||||
expect(info["relatedVideos"][0]["short_view_count"]).to eq("230M")
|
expect(info["relatedVideos"][0]["short_view_count"]).to eq("179M")
|
||||||
expect(info["relatedVideos"][0]["author_verified"]).to eq("true")
|
expect(info["relatedVideos"][0]["author_verified"]).to eq("true")
|
||||||
|
|
||||||
# Description
|
# Description
|
||||||
|
@ -76,11 +76,11 @@ Spectator.describe "parse_video_info" do
|
||||||
expect(info["ucid"].as_s).to eq("UCX6OQ3DkcsbYNE6H8uQQuVA")
|
expect(info["ucid"].as_s).to eq("UCX6OQ3DkcsbYNE6H8uQQuVA")
|
||||||
|
|
||||||
expect(info["authorThumbnail"].as_s).to eq(
|
expect(info["authorThumbnail"].as_s).to eq(
|
||||||
"https://yt3.ggpht.com/fxGKYucJAVme-Yz4fsdCroCFCrANWqw0ql4GYuvx8Uq4l_euNJHgE-w9MTkLQA805vWCi-kE0g=s48-c-k-c0x00ffffff-no-rj"
|
"https://yt3.ggpht.com/ytc/AL5GRJVuqw82ERvHzsmBxL7avr1dpBtsVIXcEzBPZaloFg=s48-c-k-c0x00ffffff-no-rj"
|
||||||
)
|
)
|
||||||
|
|
||||||
expect(info["authorVerified"].as_bool).to be_true
|
expect(info["authorVerified"].as_bool).to be_true
|
||||||
expect(info["subCountText"].as_s).to eq("320M")
|
expect(info["subCountText"].as_s).to eq("143M")
|
||||||
end
|
end
|
||||||
|
|
||||||
it "parses a regular video with no descrition/comments" do
|
it "parses a regular video with no descrition/comments" do
|
||||||
|
@ -99,8 +99,8 @@ Spectator.describe "parse_video_info" do
|
||||||
# Basic video infos
|
# Basic video infos
|
||||||
|
|
||||||
expect(info["title"].as_s).to eq("Chris Rea - Auberge")
|
expect(info["title"].as_s).to eq("Chris Rea - Auberge")
|
||||||
expect(info["views"].as_i).to eq(14_324_584)
|
expect(info["views"].as_i).to eq(10_943_126)
|
||||||
expect(info["likes"].as_i).to eq(35_870)
|
expect(info["likes"].as_i).to eq(0)
|
||||||
expect(info["lengthSeconds"].as_i).to eq(283_i64)
|
expect(info["lengthSeconds"].as_i).to eq(283_i64)
|
||||||
expect(info["published"].as_s).to eq("2012-05-21T00:00:00Z")
|
expect(info["published"].as_s).to eq("2012-05-21T00:00:00Z")
|
||||||
|
|
||||||
|
@ -132,14 +132,14 @@ Spectator.describe "parse_video_info" do
|
||||||
|
|
||||||
# Related videos
|
# Related videos
|
||||||
|
|
||||||
expect(info["relatedVideos"].as_a.size).to eq(20)
|
expect(info["relatedVideos"].as_a.size).to eq(19)
|
||||||
|
|
||||||
expect(info["relatedVideos"][0]["id"]).to eq("gUUdQfnshJ4")
|
expect(info["relatedVideos"][0]["id"]).to eq("Ww3KeZ2_Yv4")
|
||||||
expect(info["relatedVideos"][0]["title"]).to eq("Chris Rea - The Road To Hell 1989 Full Version")
|
expect(info["relatedVideos"][0]["title"]).to eq("Chris Rea")
|
||||||
expect(info["relatedVideos"][0]["author"]).to eq("NEA ZIXNH")
|
expect(info["relatedVideos"][0]["author"]).to eq("PanMusic")
|
||||||
expect(info["relatedVideos"][0]["ucid"]).to eq("UCYMEOGcvav3gCgImK2J07CQ")
|
expect(info["relatedVideos"][0]["ucid"]).to eq("UCsKAPSuh1iNbLWUga_igPyA")
|
||||||
expect(info["relatedVideos"][0]["view_count"]).to eq("53298661")
|
expect(info["relatedVideos"][0]["view_count"]).to eq("31581")
|
||||||
expect(info["relatedVideos"][0]["short_view_count"]).to eq("53M")
|
expect(info["relatedVideos"][0]["short_view_count"]).to eq("31K")
|
||||||
expect(info["relatedVideos"][0]["author_verified"]).to eq("false")
|
expect(info["relatedVideos"][0]["author_verified"]).to eq("false")
|
||||||
|
|
||||||
# Description
|
# Description
|
||||||
|
@ -156,13 +156,11 @@ Spectator.describe "parse_video_info" do
|
||||||
|
|
||||||
# Author infos
|
# Author infos
|
||||||
|
|
||||||
expect(info["author"].as_s).to eq("ChrisReaVideos")
|
expect(info["author"].as_s).to eq("ChrisReaOfficial")
|
||||||
expect(info["ucid"].as_s).to eq("UC_5q6nWPbD30-y6oiWF_oNA")
|
expect(info["ucid"].as_s).to eq("UC_5q6nWPbD30-y6oiWF_oNA")
|
||||||
|
|
||||||
expect(info["authorThumbnail"].as_s).to eq(
|
expect(info["authorThumbnail"].as_s).to be_empty
|
||||||
"https://yt3.ggpht.com/ytc/AIdro_n71nsegpKfjeRKwn1JJmK5IVMh_7j5m_h3_1KnUUg=s48-c-k-c0x00ffffff-no-rj"
|
|
||||||
)
|
|
||||||
expect(info["authorVerified"].as_bool).to be_false
|
expect(info["authorVerified"].as_bool).to be_false
|
||||||
expect(info["subCountText"].as_s).to eq("3.11K")
|
expect(info["subCountText"].as_s).to eq("-")
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
|
@ -23,7 +23,6 @@ require "kilt"
|
||||||
require "./ext/kemal_content_for.cr"
|
require "./ext/kemal_content_for.cr"
|
||||||
require "./ext/kemal_static_file_handler.cr"
|
require "./ext/kemal_static_file_handler.cr"
|
||||||
|
|
||||||
require "http_proxy"
|
|
||||||
require "athena-negotiation"
|
require "athena-negotiation"
|
||||||
require "openssl/hmac"
|
require "openssl/hmac"
|
||||||
require "option_parser"
|
require "option_parser"
|
||||||
|
@ -93,10 +92,6 @@ SOFTWARE = {
|
||||||
|
|
||||||
YT_POOL = YoutubeConnectionPool.new(YT_URL, capacity: CONFIG.pool_size)
|
YT_POOL = YoutubeConnectionPool.new(YT_URL, capacity: CONFIG.pool_size)
|
||||||
|
|
||||||
# Image request pool
|
|
||||||
|
|
||||||
GGPHT_POOL = YoutubeConnectionPool.new(URI.parse("https://yt3.ggpht.com"), capacity: CONFIG.pool_size)
|
|
||||||
|
|
||||||
# CLI
|
# CLI
|
||||||
Kemal.config.extra_options do |parser|
|
Kemal.config.extra_options do |parser|
|
||||||
parser.banner = "Usage: invidious [arguments]"
|
parser.banner = "Usage: invidious [arguments]"
|
||||||
|
|
|
@ -55,15 +55,6 @@ struct ConfigPreferences
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
struct HTTPProxyConfig
|
|
||||||
include YAML::Serializable
|
|
||||||
|
|
||||||
property user : String
|
|
||||||
property password : String
|
|
||||||
property host : String
|
|
||||||
property port : Int32
|
|
||||||
end
|
|
||||||
|
|
||||||
class Config
|
class Config
|
||||||
include YAML::Serializable
|
include YAML::Serializable
|
||||||
|
|
||||||
|
@ -138,8 +129,6 @@ class Config
|
||||||
property host_binding : String = "0.0.0.0"
|
property host_binding : String = "0.0.0.0"
|
||||||
# Pool size for HTTP requests to youtube.com and ytimg.com (each domain has a separate pool of `pool_size`)
|
# Pool size for HTTP requests to youtube.com and ytimg.com (each domain has a separate pool of `pool_size`)
|
||||||
property pool_size : Int32 = 100
|
property pool_size : Int32 = 100
|
||||||
# HTTP Proxy configuration
|
|
||||||
property http_proxy : HTTPProxyConfig? = nil
|
|
||||||
|
|
||||||
# Use Innertube's transcripts API instead of timedtext for closed captions
|
# Use Innertube's transcripts API instead of timedtext for closed captions
|
||||||
property use_innertube_for_captions : Bool = false
|
property use_innertube_for_captions : Bool = false
|
||||||
|
|
|
@ -18,40 +18,6 @@ end
|
||||||
class HTTP::Client
|
class HTTP::Client
|
||||||
property family : Socket::Family = Socket::Family::UNSPEC
|
property family : Socket::Family = Socket::Family::UNSPEC
|
||||||
|
|
||||||
# Override stdlib to automatically initialize proxy if configured
|
|
||||||
#
|
|
||||||
# Accurate as of crystal 1.12.1
|
|
||||||
|
|
||||||
def initialize(@host : String, port = nil, tls : TLSContext = nil)
|
|
||||||
check_host_only(@host)
|
|
||||||
|
|
||||||
{% if flag?(:without_openssl) %}
|
|
||||||
if tls
|
|
||||||
raise "HTTP::Client TLS is disabled because `-D without_openssl` was passed at compile time"
|
|
||||||
end
|
|
||||||
@tls = nil
|
|
||||||
{% else %}
|
|
||||||
@tls = case tls
|
|
||||||
when true
|
|
||||||
OpenSSL::SSL::Context::Client.new
|
|
||||||
when OpenSSL::SSL::Context::Client
|
|
||||||
tls
|
|
||||||
when false, nil
|
|
||||||
nil
|
|
||||||
end
|
|
||||||
{% end %}
|
|
||||||
|
|
||||||
@port = (port || (@tls ? 443 : 80)).to_i
|
|
||||||
|
|
||||||
self.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
|
|
||||||
end
|
|
||||||
|
|
||||||
def initialize(@io : IO, @host = "", @port = 80)
|
|
||||||
@reconnect = false
|
|
||||||
|
|
||||||
self.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
|
|
||||||
end
|
|
||||||
|
|
||||||
private def io
|
private def io
|
||||||
io = @io
|
io = @io
|
||||||
return io if io
|
return io if io
|
||||||
|
|
|
@ -175,9 +175,8 @@ module Invidious::SigHelper
|
||||||
@queue = {} of TransactionID => Transaction
|
@queue = {} of TransactionID => Transaction
|
||||||
|
|
||||||
@conn : Connection
|
@conn : Connection
|
||||||
@uri_or_path : String
|
|
||||||
|
|
||||||
def initialize(@uri_or_path)
|
def initialize(uri_or_path)
|
||||||
@conn = Connection.new(uri_or_path)
|
@conn = Connection.new(uri_or_path)
|
||||||
listen
|
listen
|
||||||
end
|
end
|
||||||
|
@ -187,26 +186,10 @@ module Invidious::SigHelper
|
||||||
|
|
||||||
LOGGER.debug("SigHelper: Multiplexor listening")
|
LOGGER.debug("SigHelper: Multiplexor listening")
|
||||||
|
|
||||||
|
# TODO: reopen socket if unexpectedly closed
|
||||||
spawn do
|
spawn do
|
||||||
loop do
|
loop do
|
||||||
begin
|
receive_data
|
||||||
receive_data
|
|
||||||
rescue ex
|
|
||||||
LOGGER.info("SigHelper: Connection to helper died with '#{ex.message}' trying to reconnect...")
|
|
||||||
# We close the socket because for some reason is not closed.
|
|
||||||
@conn.close
|
|
||||||
loop do
|
|
||||||
begin
|
|
||||||
@conn = Connection.new(@uri_or_path)
|
|
||||||
LOGGER.info("SigHelper: Reconnected to SigHelper!")
|
|
||||||
rescue ex
|
|
||||||
LOGGER.debug("SigHelper: Reconnection to helper unsuccessful with error '#{ex.message}'. Retrying")
|
|
||||||
sleep 500.milliseconds
|
|
||||||
next
|
|
||||||
end
|
|
||||||
break if !@conn.closed?
|
|
||||||
end
|
|
||||||
end
|
|
||||||
Fiber.yield
|
Fiber.yield
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
|
@ -11,9 +11,29 @@ module Invidious::Routes::Images
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
# We're encapsulating this into a proc in order to easily reuse this
|
||||||
|
# portion of the code for each request block below.
|
||||||
|
request_proc = ->(response : HTTP::Client::Response) {
|
||||||
|
env.response.status_code = response.status_code
|
||||||
|
response.headers.each do |key, value|
|
||||||
|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
|
||||||
|
env.response.headers[key] = value
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
env.response.headers["Access-Control-Allow-Origin"] = "*"
|
||||||
|
|
||||||
|
if response.status_code >= 300
|
||||||
|
env.response.headers.delete("Transfer-Encoding")
|
||||||
|
return
|
||||||
|
end
|
||||||
|
|
||||||
|
proxy_file(response, env)
|
||||||
|
}
|
||||||
|
|
||||||
begin
|
begin
|
||||||
GGPHT_POOL.client &.get(url, headers) do |resp|
|
HTTP::Client.get("https://yt3.ggpht.com#{url}") do |resp|
|
||||||
return self.proxy_image(env, resp)
|
return request_proc.call(resp)
|
||||||
end
|
end
|
||||||
rescue ex
|
rescue ex
|
||||||
end
|
end
|
||||||
|
@ -41,10 +61,27 @@ module Invidious::Routes::Images
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
request_proc = ->(response : HTTP::Client::Response) {
|
||||||
|
env.response.status_code = response.status_code
|
||||||
|
response.headers.each do |key, value|
|
||||||
|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
|
||||||
|
env.response.headers[key] = value
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
env.response.headers["Connection"] = "close"
|
||||||
|
env.response.headers["Access-Control-Allow-Origin"] = "*"
|
||||||
|
|
||||||
|
if response.status_code >= 300
|
||||||
|
return env.response.headers.delete("Transfer-Encoding")
|
||||||
|
end
|
||||||
|
|
||||||
|
proxy_file(response, env)
|
||||||
|
}
|
||||||
|
|
||||||
begin
|
begin
|
||||||
get_ytimg_pool(authority).client &.get(url, headers) do |resp|
|
HTTP::Client.get("https://#{authority}.ytimg.com#{url}") do |resp|
|
||||||
env.response.headers["Connection"] = "close"
|
return request_proc.call(resp)
|
||||||
return self.proxy_image(env, resp)
|
|
||||||
end
|
end
|
||||||
rescue ex
|
rescue ex
|
||||||
end
|
end
|
||||||
|
@ -64,9 +101,26 @@ module Invidious::Routes::Images
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
request_proc = ->(response : HTTP::Client::Response) {
|
||||||
|
env.response.status_code = response.status_code
|
||||||
|
response.headers.each do |key, value|
|
||||||
|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
|
||||||
|
env.response.headers[key] = value
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
env.response.headers["Access-Control-Allow-Origin"] = "*"
|
||||||
|
|
||||||
|
if response.status_code >= 300 && response.status_code != 404
|
||||||
|
return env.response.headers.delete("Transfer-Encoding")
|
||||||
|
end
|
||||||
|
|
||||||
|
proxy_file(response, env)
|
||||||
|
}
|
||||||
|
|
||||||
begin
|
begin
|
||||||
get_ytimg_pool("i9").client &.get(url, headers) do |resp|
|
HTTP::Client.get("https://i9.ytimg.com#{url}") do |resp|
|
||||||
return self.proxy_image(env, resp)
|
return request_proc.call(resp)
|
||||||
end
|
end
|
||||||
rescue ex
|
rescue ex
|
||||||
end
|
end
|
||||||
|
@ -111,7 +165,8 @@ module Invidious::Routes::Images
|
||||||
if name == "maxres.jpg"
|
if name == "maxres.jpg"
|
||||||
build_thumbnails(id).each do |thumb|
|
build_thumbnails(id).each do |thumb|
|
||||||
thumbnail_resource_path = "/vi/#{id}/#{thumb[:url]}.jpg"
|
thumbnail_resource_path = "/vi/#{id}/#{thumb[:url]}.jpg"
|
||||||
if get_ytimg_pool("i9").client &.head(thumbnail_resource_path, headers).status_code == 200
|
# This can likely be optimized into a (small) pool sometime in the future.
|
||||||
|
if HTTP::Client.head("https://i.ytimg.com#{thumbnail_resource_path}").status_code == 200
|
||||||
name = thumb[:url] + ".jpg"
|
name = thumb[:url] + ".jpg"
|
||||||
break
|
break
|
||||||
end
|
end
|
||||||
|
@ -126,28 +181,29 @@ module Invidious::Routes::Images
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
request_proc = ->(response : HTTP::Client::Response) {
|
||||||
|
env.response.status_code = response.status_code
|
||||||
|
response.headers.each do |key, value|
|
||||||
|
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
|
||||||
|
env.response.headers[key] = value
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
env.response.headers["Access-Control-Allow-Origin"] = "*"
|
||||||
|
|
||||||
|
if response.status_code >= 300 && response.status_code != 404
|
||||||
|
return env.response.headers.delete("Transfer-Encoding")
|
||||||
|
end
|
||||||
|
|
||||||
|
proxy_file(response, env)
|
||||||
|
}
|
||||||
|
|
||||||
begin
|
begin
|
||||||
get_ytimg_pool("i").client &.get(url, headers) do |resp|
|
# This can likely be optimized into a (small) pool sometime in the future.
|
||||||
return self.proxy_image(env, resp)
|
HTTP::Client.get("https://i.ytimg.com#{url}") do |resp|
|
||||||
|
return request_proc.call(resp)
|
||||||
end
|
end
|
||||||
rescue ex
|
rescue ex
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
private def self.proxy_image(env, response)
|
|
||||||
env.response.status_code = response.status_code
|
|
||||||
response.headers.each do |key, value|
|
|
||||||
if !RESPONSE_HEADERS_BLACKLIST.includes?(key.downcase)
|
|
||||||
env.response.headers[key] = value
|
|
||||||
end
|
|
||||||
end
|
|
||||||
|
|
||||||
env.response.headers["Access-Control-Allow-Origin"] = "*"
|
|
||||||
|
|
||||||
if response.status_code >= 300
|
|
||||||
return env.response.headers.delete("Transfer-Encoding")
|
|
||||||
end
|
|
||||||
|
|
||||||
return proxy_file(response, env)
|
|
||||||
end
|
|
||||||
end
|
end
|
||||||
|
|
|
@ -123,7 +123,6 @@ module Invidious::Videos
|
||||||
"Esperanto",
|
"Esperanto",
|
||||||
"Estonian",
|
"Estonian",
|
||||||
"Filipino",
|
"Filipino",
|
||||||
"Filipino (auto-generated)",
|
|
||||||
"Finnish",
|
"Finnish",
|
||||||
"French",
|
"French",
|
||||||
"French (auto-generated)",
|
"French (auto-generated)",
|
||||||
|
|
|
@ -1,6 +1,17 @@
|
||||||
# Mapping of subdomain => YoutubeConnectionPool
|
def add_yt_headers(request)
|
||||||
# This is needed as we may need to access arbitrary subdomains of ytimg
|
request.headers.delete("User-Agent") if request.headers["User-Agent"] == "Crystal"
|
||||||
private YTIMG_POOLS = {} of String => YoutubeConnectionPool
|
request.headers["User-Agent"] ||= "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
|
||||||
|
|
||||||
|
request.headers["Accept-Charset"] ||= "ISO-8859-1,utf-8;q=0.7,*;q=0.7"
|
||||||
|
request.headers["Accept"] ||= "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
|
||||||
|
request.headers["Accept-Language"] ||= "en-us,en;q=0.5"
|
||||||
|
|
||||||
|
# Preserve original cookies and add new YT consent cookie for EU servers
|
||||||
|
request.headers["Cookie"] = "#{request.headers["cookie"]?}; CONSENT=PENDING+#{Random.rand(100..999)}"
|
||||||
|
if !CONFIG.cookies.empty?
|
||||||
|
request.headers["Cookie"] = "#{(CONFIG.cookies.map { |c| "#{c.name}=#{c.value}" }).join("; ")}; #{request.headers["cookie"]?}"
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
struct YoutubeConnectionPool
|
struct YoutubeConnectionPool
|
||||||
property! url : URI
|
property! url : URI
|
||||||
|
@ -15,16 +26,12 @@ struct YoutubeConnectionPool
|
||||||
|
|
||||||
def client(&)
|
def client(&)
|
||||||
conn = pool.checkout
|
conn = pool.checkout
|
||||||
# Proxy needs to be reinstated every time we get a client from the pool
|
|
||||||
conn.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
|
|
||||||
|
|
||||||
begin
|
begin
|
||||||
response = yield conn
|
response = yield conn
|
||||||
rescue ex
|
rescue ex
|
||||||
conn.close
|
conn.close
|
||||||
|
|
||||||
conn = HTTP::Client.new(url)
|
conn = HTTP::Client.new(url)
|
||||||
conn.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
|
|
||||||
conn.family = CONFIG.force_resolve
|
conn.family = CONFIG.force_resolve
|
||||||
conn.family = Socket::Family::INET if conn.family == Socket::Family::UNSPEC
|
conn.family = Socket::Family::INET if conn.family == Socket::Family::UNSPEC
|
||||||
conn.before_request { |r| add_yt_headers(r) } if url.host == "www.youtube.com"
|
conn.before_request { |r| add_yt_headers(r) } if url.host == "www.youtube.com"
|
||||||
|
@ -47,21 +54,6 @@ struct YoutubeConnectionPool
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
def add_yt_headers(request)
|
|
||||||
request.headers.delete("User-Agent") if request.headers["User-Agent"] == "Crystal"
|
|
||||||
request.headers["User-Agent"] ||= "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
|
|
||||||
|
|
||||||
request.headers["Accept-Charset"] ||= "ISO-8859-1,utf-8;q=0.7,*;q=0.7"
|
|
||||||
request.headers["Accept"] ||= "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
|
|
||||||
request.headers["Accept-Language"] ||= "en-us,en;q=0.5"
|
|
||||||
|
|
||||||
# Preserve original cookies and add new YT consent cookie for EU servers
|
|
||||||
request.headers["Cookie"] = "#{request.headers["cookie"]?}; CONSENT=PENDING+#{Random.rand(100..999)}"
|
|
||||||
if !CONFIG.cookies.empty?
|
|
||||||
request.headers["Cookie"] = "#{(CONFIG.cookies.map { |c| "#{c.name}=#{c.value}" }).join("; ")}; #{request.headers["cookie"]?}"
|
|
||||||
end
|
|
||||||
end
|
|
||||||
|
|
||||||
def make_client(url : URI, region = nil, force_resolve : Bool = false)
|
def make_client(url : URI, region = nil, force_resolve : Bool = false)
|
||||||
client = HTTP::Client.new(url)
|
client = HTTP::Client.new(url)
|
||||||
|
|
||||||
|
@ -85,31 +77,3 @@ def make_client(url : URI, region = nil, force_resolve : Bool = false, &)
|
||||||
client.close
|
client.close
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
def make_configured_http_proxy_client
|
|
||||||
# This method is only called when configuration for an HTTP proxy are set
|
|
||||||
config_proxy = CONFIG.http_proxy.not_nil!
|
|
||||||
|
|
||||||
return HTTP::Proxy::Client.new(
|
|
||||||
config_proxy.host,
|
|
||||||
config_proxy.port,
|
|
||||||
|
|
||||||
username: config_proxy.user,
|
|
||||||
password: config_proxy.password,
|
|
||||||
)
|
|
||||||
end
|
|
||||||
|
|
||||||
# Fetches a HTTP pool for the specified subdomain of ytimg.com
|
|
||||||
#
|
|
||||||
# Creates a new one when the specified pool for the subdomain does not exist
|
|
||||||
def get_ytimg_pool(subdomain)
|
|
||||||
if pool = YTIMG_POOLS[subdomain]?
|
|
||||||
return pool
|
|
||||||
else
|
|
||||||
LOGGER.info("ytimg_pool: Creating a new HTTP pool for \"https://#{subdomain}.ytimg.com\"")
|
|
||||||
pool = YoutubeConnectionPool.new(URI.parse("https://#{subdomain}.ytimg.com"), capacity: CONFIG.pool_size)
|
|
||||||
YTIMG_POOLS[subdomain] = pool
|
|
||||||
|
|
||||||
return pool
|
|
||||||
end
|
|
||||||
end
|
|
||||||
|
|
|
@ -638,11 +638,6 @@ module YoutubeAPI
|
||||||
# Send the POST request
|
# Send the POST request
|
||||||
body = YT_POOL.client() do |client|
|
body = YT_POOL.client() do |client|
|
||||||
client.post(url, headers: headers, body: data.to_json) do |response|
|
client.post(url, headers: headers, body: data.to_json) do |response|
|
||||||
if response.status_code != 200
|
|
||||||
raise InfoException.new("Error: non 200 status code. Youtube API returned \
|
|
||||||
status code #{response.status_code}. See <a href=\"https://docs.invidious.io/youtube-errors-explained/\"> \
|
|
||||||
https://docs.invidious.io/youtube-errors-explained/</a> for troubleshooting.")
|
|
||||||
end
|
|
||||||
self._decompress(response.body_io, response.headers["Content-Encoding"]?)
|
self._decompress(response.body_io, response.headers["Content-Encoding"]?)
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
Loading…
Add table
Add a link
Reference in a new issue