1
0
Fork 0
forked from loeade/invidious

Compare commits

..

No commits in common. "2fa96fc31a22b0ec76d64e0d76b9f7f865453e87" and "147f732e40218699730414841d991e5d2d62ddb8" have entirely different histories.

15 changed files with 134 additions and 348 deletions

View file

@ -13,11 +13,14 @@ jobs:
- uses: actions/stale@v8 - uses: actions/stale@v8
with: with:
repo-token: ${{ secrets.GITHUB_TOKEN }} repo-token: ${{ secrets.GITHUB_TOKEN }}
days-before-stale: 730 days-before-stale: 365
days-before-pr-stale: -1 days-before-pr-stale: 90
days-before-close: 60 days-before-close: 30
exempt-pr-labels: blocked,exempt-stale
stale-issue-message: 'This issue has been automatically marked as stale and will be closed in 30 days because it has not had recent activity and is much likely outdated. If you think this issue is still relevant and applicable, you just have to post a comment and it will be unmarked.' stale-issue-message: 'This issue has been automatically marked as stale and will be closed in 30 days because it has not had recent activity and is much likely outdated. If you think this issue is still relevant and applicable, you just have to post a comment and it will be unmarked.'
stale-pr-message: 'This pull request has been automatically marked as stale and will be closed in 30 days because it has not had recent activity and is much likely abandoned or outdated. If you think this pull request is still relevant and applicable, you just have to post a comment and it will be unmarked.'
stale-issue-label: "stale" stale-issue-label: "stale"
stale-pr-label: "stale"
ascending: true ascending: true
# Exempt the following types of issues from being staled # Never mark feature requests/enhancements as stale
exempt-issue-labels: "feature-request,enhancement,discussion,exempt-stale" exempt-issue-labels: "feature-request,enhancement,exempt-stale"

View file

@ -5,14 +5,6 @@
### Full list of pull requests merged since the last release (newest first) ### Full list of pull requests merged since the last release (newest first)
* Stale bot updates ([#5060], thanks @syeopite)
* Channels: Fix "Youtube API returned error 400" ([#5059], by @SamantazFox)
* Channels: Fix for live videos ([#5027], thanks @iBicha)
* Locales: Add Bulgarian, Welsh and Lombard to the list ([#5046], by @SamantazFox)
* Shards: Update database dependencies ([#5034], by @SamantazFox)
* Logger: Add color support for different log levels ([#4931], thanks @Fijxu)
* Fix named arg syntax when passing force_resolve ([#4754], thanks @syeopite)
* Use make_client instead of calling HTTP::Client ([#4709], thanks @syeopite)
* Add "Filipino (auto-generated)" to the list of caption languages ([#4995], by @SamantazFox) * Add "Filipino (auto-generated)" to the list of caption languages ([#4995], by @SamantazFox)
* Makefile: Add MT option to enable the 'preview_mt' flag ([#4993], by @SamantazFox) * Makefile: Add MT option to enable the 'preview_mt' flag ([#4993], by @SamantazFox)
* SigHelper: Reconnect to signature helper ([#4991], thanks @Fijxu) * SigHelper: Reconnect to signature helper ([#4991], thanks @Fijxu)
@ -39,9 +31,7 @@
[#4270]: https://github.com/iv-org/invidious/pull/4270 [#4270]: https://github.com/iv-org/invidious/pull/4270
[#4326]: https://github.com/iv-org/invidious/pull/4326 [#4326]: https://github.com/iv-org/invidious/pull/4326
[#4652]: https://github.com/iv-org/invidious/pull/4652 [#4652]: https://github.com/iv-org/invidious/pull/4652
[#4709]: https://github.com/iv-org/invidious/pull/4709
[#4750]: https://github.com/iv-org/invidious/pull/4750 [#4750]: https://github.com/iv-org/invidious/pull/4750
[#4754]: https://github.com/iv-org/invidious/pull/4754
[#4850]: https://github.com/iv-org/invidious/pull/4850 [#4850]: https://github.com/iv-org/invidious/pull/4850
[#4862]: https://github.com/iv-org/invidious/pull/4862 [#4862]: https://github.com/iv-org/invidious/pull/4862
[#4863]: https://github.com/iv-org/invidious/pull/4863 [#4863]: https://github.com/iv-org/invidious/pull/4863
@ -51,16 +41,10 @@
[#4923]: https://github.com/iv-org/invidious/pull/4923 [#4923]: https://github.com/iv-org/invidious/pull/4923
[#4928]: https://github.com/iv-org/invidious/pull/4928 [#4928]: https://github.com/iv-org/invidious/pull/4928
[#4930]: https://github.com/iv-org/invidious/pull/4930 [#4930]: https://github.com/iv-org/invidious/pull/4930
[#4931]: https://github.com/iv-org/invidious/pull/4931
[#4942]: https://github.com/iv-org/invidious/pull/4942 [#4942]: https://github.com/iv-org/invidious/pull/4942
[#4991]: https://github.com/iv-org/invidious/pull/4991 [#4991]: https://github.com/iv-org/invidious/pull/4991
[#4993]: https://github.com/iv-org/invidious/pull/4993 [#4993]: https://github.com/iv-org/invidious/pull/4993
[#4995]: https://github.com/iv-org/invidious/pull/4995 [#4995]: https://github.com/iv-org/invidious/pull/4995
[#5027]: https://github.com/iv-org/invidious/pull/5027
[#5034]: https://github.com/iv-org/invidious/pull/5034
[#5046]: https://github.com/iv-org/invidious/pull/5046
[#5059]: https://github.com/iv-org/invidious/pull/5059
[#5060]: https://github.com/iv-org/invidious/pull/5060
## v2.20240825.2 (2024-08-26) ## v2.20240825.2 (2024-08-26)

View file

@ -233,17 +233,6 @@ http_proxy:
## ##
#log_level: Info #log_level: Info
##
## Enables colors in logs. Useful for debugging purposes
## This is overridden if "-k" or "--colorize"
## are passed on the command line.
## Colors are also disabled if the environment variable
## NO_COLOR is present and has any value
##
## Accepted values: true, false
## Default: true
##
#colorize_logs: false
# ----------------------------- # -----------------------------
# Features # Features

View file

@ -14,7 +14,7 @@ shards:
db: db:
git: https://github.com/crystal-lang/crystal-db.git git: https://github.com/crystal-lang/crystal-db.git
version: 0.13.1 version: 0.10.1
exception_page: exception_page:
git: https://github.com/crystal-loot/exception_page.git git: https://github.com/crystal-loot/exception_page.git
@ -34,7 +34,7 @@ shards:
pg: pg:
git: https://github.com/will/crystal-pg.git git: https://github.com/will/crystal-pg.git
version: 0.28.0 version: 0.24.0
protodec: protodec:
git: https://github.com/iv-org/protodec.git git: https://github.com/iv-org/protodec.git
@ -50,5 +50,5 @@ shards:
sqlite3: sqlite3:
git: https://github.com/crystal-lang/crystal-sqlite3.git git: https://github.com/crystal-lang/crystal-sqlite3.git
version: 0.21.0 version: 0.18.0

View file

@ -12,10 +12,10 @@ targets:
dependencies: dependencies:
pg: pg:
github: will/crystal-pg github: will/crystal-pg
version: ~> 0.28.0 version: ~> 0.24.0
sqlite3: sqlite3:
github: crystal-lang/crystal-sqlite3 github: crystal-lang/crystal-sqlite3
version: ~> 0.21.0 version: ~> 0.18.0
kemal: kemal:
github: kemalcr/kemal github: kemalcr/kemal
version: ~> 1.1.2 version: ~> 1.1.2

View file

@ -122,9 +122,6 @@ Kemal.config.extra_options do |parser|
parser.on("-l LEVEL", "--log-level=LEVEL", "Log level, one of #{LogLevel.values} (default: #{CONFIG.log_level})") do |log_level| parser.on("-l LEVEL", "--log-level=LEVEL", "Log level, one of #{LogLevel.values} (default: #{CONFIG.log_level})") do |log_level|
CONFIG.log_level = LogLevel.parse(log_level) CONFIG.log_level = LogLevel.parse(log_level)
end end
parser.on("-k", "--colorize", "Colorize logs") do
CONFIG.colorize_logs = true
end
parser.on("-v", "--version", "Print version") do parser.on("-v", "--version", "Print version") do
puts SOFTWARE.to_pretty_json puts SOFTWARE.to_pretty_json
exit exit
@ -141,7 +138,7 @@ if CONFIG.output.upcase != "STDOUT"
FileUtils.mkdir_p(File.dirname(CONFIG.output)) FileUtils.mkdir_p(File.dirname(CONFIG.output))
end end
OUTPUT = CONFIG.output.upcase == "STDOUT" ? STDOUT : File.open(CONFIG.output, mode: "a") OUTPUT = CONFIG.output.upcase == "STDOUT" ? STDOUT : File.open(CONFIG.output, mode: "a")
LOGGER = Invidious::LogHandler.new(OUTPUT, CONFIG.log_level, CONFIG.colorize_logs) LOGGER = Invidious::LogHandler.new(OUTPUT, CONFIG.log_level)
# Check table integrity # Check table integrity
Invidious::Database.check_integrity(CONFIG) Invidious::Database.check_integrity(CONFIG)

View file

@ -1,3 +1,78 @@
def produce_channel_content_continuation(ucid, content_type, page = 1, auto_generated = nil, sort_by = "newest", v2 = false)
object_inner_2 = {
"2:0:embedded" => {
"1:0:varint" => 0_i64,
},
"5:varint" => 50_i64,
"6:varint" => 1_i64,
"7:varint" => (page * 30).to_i64,
"9:varint" => 1_i64,
"10:varint" => 0_i64,
}
object_inner_2_encoded = object_inner_2
.try { |i| Protodec::Any.cast_json(i) }
.try { |i| Protodec::Any.from_json(i) }
.try { |i| Base64.urlsafe_encode(i) }
.try { |i| URI.encode_www_form(i) }
content_type_numerical =
case content_type
when "videos" then 15
when "livestreams" then 14
else 15 # Fallback to "videos"
end
sort_by_numerical =
case sort_by
when "newest" then 1_i64
when "popular" then 2_i64
when "oldest" then 4_i64
else 1_i64 # Fallback to "newest"
end
object_inner_1 = {
"110:embedded" => {
"3:embedded" => {
"#{content_type_numerical}:embedded" => {
"1:embedded" => {
"1:string" => object_inner_2_encoded,
},
"2:embedded" => {
"1:string" => "00000000-0000-0000-0000-000000000000",
},
"3:varint" => sort_by_numerical,
},
},
},
}
object_inner_1_encoded = object_inner_1
.try { |i| Protodec::Any.cast_json(i) }
.try { |i| Protodec::Any.from_json(i) }
.try { |i| Base64.urlsafe_encode(i) }
.try { |i| URI.encode_www_form(i) }
object = {
"80226972:embedded" => {
"2:string" => ucid,
"3:string" => object_inner_1_encoded,
"35:string" => "browse-feed#{ucid}videos102",
},
}
continuation = object.try { |i| Protodec::Any.cast_json(i) }
.try { |i| Protodec::Any.from_json(i) }
.try { |i| Base64.urlsafe_encode(i) }
.try { |i| URI.encode_www_form(i) }
return continuation
end
def make_initial_content_ctoken(ucid, content_type, sort_by) : String
return produce_channel_content_continuation(ucid, content_type, sort_by: sort_by)
end
module Invidious::Channel::Tabs module Invidious::Channel::Tabs
extend self extend self
@ -26,7 +101,7 @@ module Invidious::Channel::Tabs
end end
def get_videos(author : String, ucid : String, *, continuation : String? = nil, sort_by = "newest") def get_videos(author : String, ucid : String, *, continuation : String? = nil, sort_by = "newest")
continuation ||= make_initial_videos_ctoken(ucid, sort_by) continuation ||= make_initial_content_ctoken(ucid, "videos", sort_by)
initial_data = YoutubeAPI.browse(continuation: continuation) initial_data = YoutubeAPI.browse(continuation: continuation)
return extract_items(initial_data, author, ucid) return extract_items(initial_data, author, ucid)
@ -55,10 +130,14 @@ module Invidious::Channel::Tabs
# Shorts # Shorts
# ------------------- # -------------------
def get_shorts(channel : AboutChannel, *, continuation : String? = nil, sort_by = "newest") def get_shorts(channel : AboutChannel, continuation : String? = nil)
continuation ||= make_initial_shorts_ctoken(channel.ucid, sort_by) if continuation.nil?
# EgZzaG9ydHPyBgUKA5oBAA%3D%3D is the protobuf object to load "shorts"
# TODO: try to extract the continuation tokens that allows other sorting options
initial_data = YoutubeAPI.browse(channel.ucid, params: "EgZzaG9ydHPyBgUKA5oBAA%3D%3D")
else
initial_data = YoutubeAPI.browse(continuation: continuation) initial_data = YoutubeAPI.browse(continuation: continuation)
end
return extract_items(initial_data, channel.author, channel.ucid) return extract_items(initial_data, channel.author, channel.ucid)
end end
@ -66,8 +145,9 @@ module Invidious::Channel::Tabs
# Livestreams # Livestreams
# ------------------- # -------------------
def get_livestreams(channel : AboutChannel, *, continuation : String? = nil, sort_by = "newest") def get_livestreams(channel : AboutChannel, continuation : String? = nil, sort_by = "newest")
continuation ||= make_initial_livestreams_ctoken(channel.ucid, sort_by) continuation ||= make_initial_content_ctoken(channel.ucid, "livestreams", sort_by)
initial_data = YoutubeAPI.browse(continuation: continuation) initial_data = YoutubeAPI.browse(continuation: continuation)
return extract_items(initial_data, channel.author, channel.ucid) return extract_items(initial_data, channel.author, channel.ucid)
@ -91,102 +171,4 @@ module Invidious::Channel::Tabs
return items, next_continuation return items, next_continuation
end end
# -------------------
# C-tokens
# -------------------
private def sort_options_videos_short(sort_by : String)
case sort_by
when "newest" then return 4_i64
when "popular" then return 2_i64
when "oldest" then return 5_i64
else return 4_i64 # Fallback to "newest"
end
end
# Generate the initial "continuation token" to get the first page of the
# "videos" tab. The following page requires the ctoken provided in that
# first page, and so on.
private def make_initial_videos_ctoken(ucid : String, sort_by = "newest")
object = {
"15:embedded" => {
"2:embedded" => {
"1:string" => "00000000-0000-0000-0000-000000000000",
},
"4:varint" => sort_options_videos_short(sort_by),
},
}
return channel_ctoken_wrap(ucid, object)
end
# Generate the initial "continuation token" to get the first page of the
# "shorts" tab. The following page requires the ctoken provided in that
# first page, and so on.
private def make_initial_shorts_ctoken(ucid : String, sort_by = "newest")
object = {
"10:embedded" => {
"2:embedded" => {
"1:string" => "00000000-0000-0000-0000-000000000000",
},
"4:varint" => sort_options_videos_short(sort_by),
},
}
return channel_ctoken_wrap(ucid, object)
end
# Generate the initial "continuation token" to get the first page of the
# "livestreams" tab. The following page requires the ctoken provided in that
# first page, and so on.
private def make_initial_livestreams_ctoken(ucid : String, sort_by = "newest")
sort_by_numerical =
case sort_by
when "newest" then 12_i64
when "popular" then 14_i64
when "oldest" then 13_i64
else 12_i64 # Fallback to "newest"
end
object = {
"14:embedded" => {
"2:embedded" => {
"1:string" => "00000000-0000-0000-0000-000000000000",
},
"5:varint" => sort_by_numerical,
},
}
return channel_ctoken_wrap(ucid, object)
end
# The protobuf structure common between videos/shorts/livestreams
private def channel_ctoken_wrap(ucid : String, object)
object_inner = {
"110:embedded" => {
"3:embedded" => object,
},
}
object_inner_encoded = object_inner
.try { |i| Protodec::Any.cast_json(i) }
.try { |i| Protodec::Any.from_json(i) }
.try { |i| Base64.urlsafe_encode(i) }
.try { |i| URI.encode_www_form(i) }
object = {
"80226972:embedded" => {
"2:string" => ucid,
"3:string" => object_inner_encoded,
},
}
continuation = object.try { |i| Protodec::Any.cast_json(i) }
.try { |i| Protodec::Any.from_json(i) }
.try { |i| Base64.urlsafe_encode(i) }
.try { |i| URI.encode_www_form(i) }
return continuation
end
end end

View file

@ -78,8 +78,6 @@ class Config
property output : String = "STDOUT" property output : String = "STDOUT"
# Default log level, valid YAML values are ints and strings, see src/invidious/helpers/logger.cr # Default log level, valid YAML values are ints and strings, see src/invidious/helpers/logger.cr
property log_level : LogLevel = LogLevel::Info property log_level : LogLevel = LogLevel::Info
# Enables colors in logs. Useful for debugging purposes
property colorize_logs : Bool = false
# Database configuration with separate parameters (username, hostname, etc) # Database configuration with separate parameters (username, hostname, etc)
property db : DBConfig? = nil property db : DBConfig? = nil

View file

@ -1,22 +1,8 @@
# Languages requiring a better level of translation (at least 20%)
# to be added to the list below:
#
# "af" => "", # Afrikaans
# "az" => "", # Azerbaijani
# "be" => "", # Belarusian
# "bn_BD" => "", # Bengali (Bangladesh)
# "ia" => "", # Interlingua
# "or" => "", # Odia
# "tk" => "", # Turkmen
# "tok => "", # Toki Pona
#
LOCALES_LIST = { LOCALES_LIST = {
"ar" => "العربية", # Arabic "ar" => "العربية", # Arabic
"bg" => "български", # Bulgarian
"bn" => "বাংলা", # Bengali "bn" => "বাংলা", # Bengali
"ca" => "Català", # Catalan "ca" => "Català", # Catalan
"cs" => "Čeština", # Czech "cs" => "Čeština", # Czech
"cy" => "Cymraeg", # Welsh
"da" => "Dansk", # Danish "da" => "Dansk", # Danish
"de" => "Deutsch", # German "de" => "Deutsch", # German
"el" => "Ελληνικά", # Greek "el" => "Ελληνικά", # Greek
@ -37,7 +23,6 @@ LOCALES_LIST = {
"it" => "Italiano", # Italian "it" => "Italiano", # Italian
"ja" => "日本語", # Japanese "ja" => "日本語", # Japanese
"ko" => "한국어", # Korean "ko" => "한국어", # Korean
"lmo" => "Lombard", # Lombard
"lt" => "Lietuvių", # Lithuanian "lt" => "Lietuvių", # Lithuanian
"nb-NO" => "Norsk bokmål", # Norwegian Bokmål "nb-NO" => "Norsk bokmål", # Norwegian Bokmål
"nl" => "Nederlands", # Dutch "nl" => "Nederlands", # Dutch

View file

@ -1,5 +1,3 @@
require "colorize"
enum LogLevel enum LogLevel
All = 0 All = 0
Trace = 1 Trace = 1
@ -12,9 +10,7 @@ enum LogLevel
end end
class Invidious::LogHandler < Kemal::BaseLogHandler class Invidious::LogHandler < Kemal::BaseLogHandler
def initialize(@io : IO = STDOUT, @level = LogLevel::Debug, use_color : Bool = true) def initialize(@io : IO = STDOUT, @level = LogLevel::Debug)
Colorize.enabled = use_color
Colorize.on_tty_only!
end end
def call(context : HTTP::Server::Context) def call(context : HTTP::Server::Context)
@ -43,22 +39,10 @@ class Invidious::LogHandler < Kemal::BaseLogHandler
@io.flush @io.flush
end end
def color(level)
case level
when LogLevel::Trace then :cyan
when LogLevel::Debug then :green
when LogLevel::Info then :white
when LogLevel::Warn then :yellow
when LogLevel::Error then :red
when LogLevel::Fatal then :magenta
else :default
end
end
{% for level in %w(trace debug info warn error fatal) %} {% for level in %w(trace debug info warn error fatal) %}
def {{level.id}}(message : String) def {{level.id}}(message : String)
if LogLevel::{{level.id.capitalize}} >= @level if LogLevel::{{level.id.capitalize}} >= @level
puts("#{Time.utc} [{{level.id}}] #{message}".colorize(color(LogLevel::{{level.id.capitalize}}))) puts("#{Time.utc} [{{level.id}}] #{message}")
end end
end end
{% end %} {% end %}

View file

@ -31,7 +31,9 @@ module Invidious::Routes::API::V1::Search
query = env.params.query["q"]? || "" query = env.params.query["q"]? || ""
begin begin
client = make_client(URI.parse("https://suggestqueries-clients6.youtube.com"), force_youtube_headers: true) client = HTTP::Client.new("suggestqueries-clients6.youtube.com")
client.before_request { |r| add_yt_headers(r) }
url = "/complete/search?client=youtube&hl=en&gl=#{region}&q=#{URI.encode_www_form(query)}&gs_ri=youtube&ds=yt" url = "/complete/search?client=youtube&hl=en&gl=#{region}&q=#{URI.encode_www_form(query)}&gs_ri=youtube&ds=yt"
response = client.get(url).body response = client.get(url).body

View file

@ -20,11 +20,10 @@ module Invidious::Routes::Channels
sort_by = env.params.query["sort_by"]?.try &.downcase sort_by = env.params.query["sort_by"]?.try &.downcase
if channel.auto_generated if channel.auto_generated
sort_by ||= "last"
sort_options = {"last", "oldest", "newest"} sort_options = {"last", "oldest", "newest"}
items, next_continuation = fetch_channel_playlists( items, next_continuation = fetch_channel_playlists(
channel.ucid, channel.author, continuation, sort_by channel.ucid, channel.author, continuation, (sort_by || "last")
) )
items.uniq! do |item| items.uniq! do |item|
@ -50,11 +49,9 @@ module Invidious::Routes::Channels
end end
next_continuation = nil next_continuation = nil
else else
sort_by ||= "newest"
sort_options = {"newest", "oldest", "popular"} sort_options = {"newest", "oldest", "popular"}
items, next_continuation = Channel::Tabs.get_videos(
items, next_continuation = Channel::Tabs.get_60_videos( channel, continuation: continuation, sort_by: (sort_by || "newest")
channel, continuation: continuation, sort_by: sort_by
) )
end end
end end
@ -85,12 +82,13 @@ module Invidious::Routes::Channels
end end
next_continuation = nil next_continuation = nil
else else
sort_by = env.params.query["sort_by"]?.try &.downcase || "newest" # TODO: support sort option for shorts
sort_options = {"newest", "oldest", "popular"} sort_by = ""
sort_options = [] of String
# Fetch items and continuation token # Fetch items and continuation token
items, next_continuation = Channel::Tabs.get_shorts( items, next_continuation = Channel::Tabs.get_shorts(
channel, continuation: continuation, sort_by: sort_by channel, continuation: continuation
) )
end end

View file

@ -42,7 +42,7 @@ module Invidious::Routes::VideoPlayback
headers["Range"] = "bytes=#{range_for_head}" headers["Range"] = "bytes=#{range_for_head}"
end end
client = make_client(URI.parse(host), region, force_resolve: true) client = make_client(URI.parse(host), region, force_resolve = true)
response = HTTP::Client::Response.new(500) response = HTTP::Client::Response.new(500)
error = "" error = ""
5.times do 5.times do
@ -57,7 +57,7 @@ module Invidious::Routes::VideoPlayback
if new_host != host if new_host != host
host = new_host host = new_host
client.close client.close
client = make_client(URI.parse(new_host), region, force_resolve: true) client = make_client(URI.parse(new_host), region, force_resolve = true)
end end
url = "#{location.request_target}&host=#{location.host}#{region ? "&region=#{region}" : ""}" url = "#{location.request_target}&host=#{location.host}#{region ? "&region=#{region}" : ""}"
@ -71,7 +71,7 @@ module Invidious::Routes::VideoPlayback
fvip = "3" fvip = "3"
host = "https://r#{fvip}---#{mn}.googlevideo.com" host = "https://r#{fvip}---#{mn}.googlevideo.com"
client = make_client(URI.parse(host), region, force_resolve: true) client = make_client(URI.parse(host), region, force_resolve = true)
rescue ex rescue ex
error = ex.message error = ex.message
end end
@ -196,7 +196,7 @@ module Invidious::Routes::VideoPlayback
break break
else else
client.close client.close
client = make_client(URI.parse(host), region, force_resolve: true) client = make_client(URI.parse(host), region, force_resolve = true)
end end
end end

View file

@ -22,8 +22,12 @@ struct YoutubeConnectionPool
response = yield conn response = yield conn
rescue ex rescue ex
conn.close conn.close
conn = make_client(url, force_resolve: true)
conn = HTTP::Client.new(url)
conn.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
conn.family = CONFIG.force_resolve
conn.family = Socket::Family::INET if conn.family == Socket::Family::UNSPEC
conn.before_request { |r| add_yt_headers(r) } if url.host == "www.youtube.com"
response = yield conn response = yield conn
ensure ensure
pool.release(conn) pool.release(conn)
@ -33,15 +37,12 @@ struct YoutubeConnectionPool
end end
private def build_pool private def build_pool
options = DB::Pool::Options.new( DB::Pool(HTTP::Client).new(initial_pool_size: 0, max_pool_size: capacity, max_idle_pool_size: capacity, checkout_timeout: timeout) do
initial_pool_size: 0, conn = HTTP::Client.new(url)
max_pool_size: capacity, conn.family = CONFIG.force_resolve
max_idle_pool_size: capacity, conn.family = Socket::Family::INET if conn.family == Socket::Family::UNSPEC
checkout_timeout: timeout conn.before_request { |r| add_yt_headers(r) } if url.host == "www.youtube.com"
) conn
DB::Pool(HTTP::Client).new(options) do
next make_client(url, force_resolve: true)
end end
end end
end end
@ -61,17 +62,15 @@ def add_yt_headers(request)
end end
end end
def make_client(url : URI, region = nil, force_resolve : Bool = false, force_youtube_headers : Bool = false) def make_client(url : URI, region = nil, force_resolve : Bool = false)
client = HTTP::Client.new(url) client = HTTP::Client.new(url)
client.proxy = make_configured_http_proxy_client() if CONFIG.http_proxy
# Force the usage of a specific configured IP Family # Force the usage of a specific configured IP Family
if force_resolve if force_resolve
client.family = CONFIG.force_resolve client.family = CONFIG.force_resolve
client.family = Socket::Family::INET if client.family == Socket::Family::UNSPEC
end end
client.before_request { |r| add_yt_headers(r) } if url.host.try &.ends_with?("youtube.com") || force_youtube_headers client.before_request { |r| add_yt_headers(r) } if url.host == "www.youtube.com"
client.read_timeout = 10.seconds client.read_timeout = 10.seconds
client.connect_timeout = 10.seconds client.connect_timeout = 10.seconds
@ -79,7 +78,7 @@ def make_client(url : URI, region = nil, force_resolve : Bool = false, force_you
end end
def make_client(url : URI, region = nil, force_resolve : Bool = false, &) def make_client(url : URI, region = nil, force_resolve : Bool = false, &)
client = make_client(url, region, force_resolve: force_resolve) client = make_client(url, region, force_resolve)
begin begin
yield client yield client
ensure ensure

View file

@ -21,7 +21,6 @@ private ITEM_PARSERS = {
Parsers::ItemSectionRendererParser, Parsers::ItemSectionRendererParser,
Parsers::ContinuationItemRendererParser, Parsers::ContinuationItemRendererParser,
Parsers::HashtagRendererParser, Parsers::HashtagRendererParser,
Parsers::LockupViewModelParser,
} }
private alias InitialData = Hash(String, JSON::Any) private alias InitialData = Hash(String, JSON::Any)
@ -468,9 +467,9 @@ private module Parsers
# Parses an InnerTube richItemRenderer into a SearchVideo. # Parses an InnerTube richItemRenderer into a SearchVideo.
# Returns nil when the given object isn't a RichItemRenderer # Returns nil when the given object isn't a RichItemRenderer
# #
# A richItemRenderer seems to be a simple wrapper for a various other types, # A richItemRenderer seems to be a simple wrapper for a videoRenderer, used
# used on the hashtags result page and the channel podcast tab. It is located # by the result page for hashtags and for the podcast tab on channels.
# itself inside a richGridRenderer container. # It is located inside a continuationItems container for hashtags.
# #
module RichItemRendererParser module RichItemRendererParser
def self.process(item : JSON::Any, author_fallback : AuthorFallback) def self.process(item : JSON::Any, author_fallback : AuthorFallback)
@ -483,8 +482,6 @@ private module Parsers
child = VideoRendererParser.process(item_contents, author_fallback) child = VideoRendererParser.process(item_contents, author_fallback)
child ||= ReelItemRendererParser.process(item_contents, author_fallback) child ||= ReelItemRendererParser.process(item_contents, author_fallback)
child ||= PlaylistRendererParser.process(item_contents, author_fallback) child ||= PlaylistRendererParser.process(item_contents, author_fallback)
child ||= LockupViewModelParser.process(item_contents, author_fallback)
child ||= ShortsLockupViewModelParser.process(item_contents, author_fallback)
return child return child
end end
@ -499,9 +496,6 @@ private module Parsers
# reelItemRenderer items are used in the new (2022) channel layout, # reelItemRenderer items are used in the new (2022) channel layout,
# in the "shorts" tab. # in the "shorts" tab.
# #
# NOTE: As of 10/2024, it might have been fully replaced by shortsLockupViewModel
# TODO: Confirm that hypothesis
#
module ReelItemRendererParser module ReelItemRendererParser
def self.process(item : JSON::Any, author_fallback : AuthorFallback) def self.process(item : JSON::Any, author_fallback : AuthorFallback)
if item_contents = item["reelItemRenderer"]? if item_contents = item["reelItemRenderer"]?
@ -588,135 +582,6 @@ private module Parsers
end end
end end
# Parses an InnerTube lockupViewModel into a SearchPlaylist.
# Returns nil when the given object is not a lockupViewModel.
#
# This structure is present since November 2024 on the "podcasts" and
# "playlists" tabs of the channel page. It is usually encapsulated in either
# a richItemRenderer or a richGridRenderer.
#
module LockupViewModelParser
def self.process(item : JSON::Any, author_fallback : AuthorFallback)
if item_contents = item["lockupViewModel"]?
return self.parse(item_contents, author_fallback)
end
end
private def self.parse(item_contents, author_fallback)
playlist_id = item_contents["contentId"].as_s
thumbnail_view_model = item_contents.dig(
"contentImage", "collectionThumbnailViewModel",
"primaryThumbnail", "thumbnailViewModel"
)
thumbnail = thumbnail_view_model.dig("image", "sources", 0, "url").as_s
# This complicated sequences tries to extract the following data structure:
# "overlays": [{
# "thumbnailOverlayBadgeViewModel": {
# "thumbnailBadges": [{
# "thumbnailBadgeViewModel": {
# "text": "430 episodes",
# "badgeStyle": "THUMBNAIL_OVERLAY_BADGE_STYLE_DEFAULT"
# }
# }]
# }
# }]
#
# NOTE: this simplistic `.to_i` conversion might not work on larger
# playlists and hasn't been tested.
video_count = thumbnail_view_model.dig("overlays").as_a
.compact_map(&.dig?("thumbnailOverlayBadgeViewModel", "thumbnailBadges").try &.as_a)
.flatten
.find(nil, &.dig?("thumbnailBadgeViewModel", "text").try { |node|
{"episodes", "videos"}.any? { |str| node.as_s.ends_with?(str) }
})
.try &.dig("thumbnailBadgeViewModel", "text").as_s.to_i(strict: false)
metadata = item_contents.dig("metadata", "lockupMetadataViewModel")
title = metadata.dig("title", "content").as_s
# TODO: Retrieve "updated" info from metadata parts
# rows = metadata.dig("metadata", "contentMetadataViewModel", "metadataRows").as_a
# parts_text = rows.map(&.dig?("metadataParts", "text", "content").try &.as_s)
# One of these parts should contain a string like: "Updated 2 days ago"
# TODO: Maybe add a button to access the first video of the playlist?
# item_contents.dig("rendererContext", "commandContext", "onTap", "innertubeCommand", "watchEndpoint")
# Available fields: "videoId", "playlistId", "params"
return SearchPlaylist.new({
title: title,
id: playlist_id,
author: author_fallback.name,
ucid: author_fallback.id,
video_count: video_count || -1,
videos: [] of SearchPlaylistVideo,
thumbnail: thumbnail,
author_verified: false,
})
end
def self.parser_name
return {{@type.name}}
end
end
# Parses an InnerTube shortsLockupViewModel into a SearchVideo.
# Returns nil when the given object is not a shortsLockupViewModel.
#
# This structure is present since around October 2024 on the "shorts" tab of
# the channel page and likely replaces the reelItemRenderer structure. It is
# usually (always?) encapsulated in a richItemRenderer.
#
module ShortsLockupViewModelParser
def self.process(item : JSON::Any, author_fallback : AuthorFallback)
if item_contents = item["shortsLockupViewModel"]?
return self.parse(item_contents, author_fallback)
end
end
private def self.parse(item_contents, author_fallback)
# TODO: Maybe add support for "oardefault.jpg" thumbnails?
# thumbnail = item_contents.dig("thumbnail", "sources", 0, "url").as_s
# Gives: https://i.ytimg.com/vi/{video_id}/oardefault.jpg?...
video_id = item_contents.dig(
"onTap", "innertubeCommand", "reelWatchEndpoint", "videoId"
).as_s
title = item_contents.dig("overlayMetadata", "primaryText", "content").as_s
view_count = short_text_to_number(
item_contents.dig("overlayMetadata", "secondaryText", "content").as_s
)
# Approximate to one minute, as "shorts" generally don't exceed that.
# NOTE: The actual duration is not provided by Youtube anymore.
# TODO: Maybe use -1 as an error value and handle that on the frontend?
duration = 60_i32
SearchVideo.new({
title: title,
id: video_id,
author: author_fallback.name,
ucid: author_fallback.id,
published: Time.unix(0),
views: view_count,
description_html: "",
length_seconds: duration,
premiere_timestamp: Time.unix(0),
author_verified: false,
badges: VideoBadges::None,
})
end
def self.parser_name
return {{@type.name}}
end
end
# Parses an InnerTube continuationItemRenderer into a Continuation. # Parses an InnerTube continuationItemRenderer into a Continuation.
# Returns nil when the given object isn't a continuationItemRenderer. # Returns nil when the given object isn't a continuationItemRenderer.
# #