> didn't realize Kagi had no aspirations to build their own general purpose index
Kagi employee here. We're actively working on building our own indexes beyond the limited ones we have now, not just a general index but also purpose built indexes for things like programming, etc.
I did not intend to spread misinformation here, and would like to hear more about the general-purpose index Kagi is working on. I had based my comment on several Kagi pages, but mostly https://help.kagi.com/kagi/search-details/search-sources.htm..., which mentions Teclis as Kagi's own index, but https://teclis.com/ makes it pretty clear that it's a "small web"-focused tool:
> Teclis is an attempt to surface the less known web, the web of creativity and self expression, the more humane web.
> Teclis includes its own crawl as well as results from Kagi Small Web index and results with permission from Marginalia Search.
> Teclis works best with broad queries such as 'machine learning', 'vegan diet', 'religion' etc..
Is there another crawler doing the general-purpose stuff?
What are the challenges of doing that when so much of the internet has turned itself into SEO slop to fit Google's algorithms?
I imagine there is still a whole load of stuff out there on the internet that Google would never surface because it doesn't have enough adsense or whatever. Are you finding that?
That's what SlopStop is for :) developing methodologies that scale for detecting slop.
I mean it sounds like that already has a lot of overlap with our Small Web indexing efforts, so that part of our indexing efforts could be an extension of that. A lot if this is still in development though so I can't speak on specifics just yet.
Anubis is easy, just use a whitelisted user agent or a headless browser if some sites disable that - you need one to index web app abominations anyway. Cloudflare and Google reCaptcha are bigger problems.
If a search engine starts censoring by whatever means, I'll not be using it, neither free or paid no matter how good it is (and by definition it can't be). Shills can provide a list of countries they don't want to see, hopefully some crooked search engine will satisfy their desire for censorship.
No search engines are uncensored, not even Yandex. Partly because they have to exist in a country somewhere, and partly because really nobody wants to index CSAM.
Yes of course and if you think that's all that western search engines censor then you can keep using Google and Bing - but some of us actually do want the full picture.
There's not just one war on, there are many. Should Kagi exclude all the results from American companies because of the actions of the US government in Iran?
This is so funny, as though wanting to boycott specific entities is some kind of absurd notion, and as though saying "Sure, what ELSE don't you like?" is some kind of proof that it's an absurdity
It kind of is though. Someone else will say "why are you sourcing results from an Israeli company?" and another will say "why are you sourcing results from a Chinese company?" and another one yet will say "why are you sourcing results from the US?".
Why are the ethics of working with Yandex or Baidu any better or worse than the ethics of working with Google or Microsoft? Except that they're not western.
The logical answer is that a person like this wants a very strong firewall, so ethically impure bits don't cross into their LAN.
Maybe not that ridiculous, since one can guess that the underlying thoughts are more about geostrategic concerns than favorite color of the day.
The United States (I guess that's also the premise here, I'm not USA citizen myself) has notable rivalries with several countries, including China, Russia, Iran, North Korea, and Venezuela. These nations are often considered adversaries due to various geopolitical tensions and conflicts.
I guess that Kahi is doing nothing illegal, so if people have that kind of question, it feels legitimate to reply with a demand of what is the extend this patriotism stance is going beyond the judicial requirements.
Sure, rapture the leader with only a hundred civil left dead and this is it, everybody in the country now is hearth supporter of those who launched that raid.
Western companies also bought strategically important Nazi Germany industrial products in the 1930s because they were considered superior. Commercial convenience and technical quality are not moral or geopolitical absolution. Would you buy cheap quality gold if the Nazis were selling it knowing what it supports?
What's the point in stopping paying money to Russia if Kagi is incorporated in Palo Alto, so it's paying money and will continue to do so to a country causing no less troubles than Russia?
Two wrongs don’t make a right. Much of the HN audience lives in the US, giving money to the US is unavoidable in life. But in return we do have the democratic ability to try to alter the behavior of the country.
It still makes sense to avoid giving money to other bad actors who are acting in direct opposition to your home country, and whom you have no control over, when you can.
Then one should ask for Kagi to move to a country that doesn't use taxes to invade other countries, and stop supporting them, instead of just asking to "stop paying money to Russia". Because fixing two wrongs is better than just fixing one, right? And I want to believe that living in the US doesn't make you agree with your country's foreign policy (which doesn't seem to change significantly even with your democratic ability to try to alter it) specially if you're worried about Russia.
Ru-speaking audience is ~2 times bigger than Russia, why would they cut this? Only because of SJWs like you?
I'd much prefer one search engine that searches well on two languages, instead of using different engines for each language. Ru-net has huge amount of usefulness in it, cutting it out is like cutting a finger. Fun fact, before the RU-UKR war, most Ukranians contributed to the Runet, so that would cut their heritage too.
I'm frankly a bit surprised that it's even legal for a US company like Kagi to do business with Yandex, considering it's sanctioned: https://sanctionssearch.ofac.treas.gov/Details.aspx?id=18711. Though in fairness, I don't know enough about how exactly sanction laws work so it might be legally okay even if I find it morally questionable.
Yes, Yandex was effectively seized by Ru govt "friends" and turned into propaganda tool. And that's very unfortunate. But I'm not talking about getting the propaganda, only about search index.
About money - don't think this income is even remotely comparable with oil/gas incomes, which EU passes to RU.
Please don't start political debate. I do not like censorship of any kind, hence my initial response. I want to have available information in full.
Please don't pretend I'm turning this into a "political debate". Your position is, "Kagi ought to keep doing business with Russia". That in itself is highly political. No side of this issue is apolitical.
My position was more about access to information, that business side for me is secondary, as unavoidable evil. If kagi will find a ru-index with decent quality, I'll be more than happier. Right now yandex has the best index, but given the decay of ru tech sector now, especially now when it's oligarch-managed, that might not be for long.
Yandex has quite a few international entities, which are probably not direct subsidiaries, which in turn probably helps with sanctions. Yandex Cloud seems to be sold by a UAE company internationally: https://yandex.cloud/en/about#impressum
Not sure what you mean with "completely disables it". I have watch history disabled and still see shorts in search results or subscriptions results https://www.youtube.com/feed/subscriptions
If you click on one though and go to swipe (unless it's in your subscriptions) it doesn't allow you to "scroll". You can watch them when necessary but it's impossible to get sucked into an infinite feed.
If you turn off watch history it completely disables shorts as a whole (with no recommendations on the homepage as a side effect, but one I'm willing to live with). https://news.ycombinator.com/item?id=42795204
The no recommendations at all sure feels like malicious compliance with California privacy law.
Even while pretending they've not recorded your viewing history they could still make recommendations from your subscriptions or give you the same glurg that they give viewers they know nothing about... but instead they break the site.
It's still better than having shorts on the screen.
The word "want" is the key there -- they have zero interest in what you 'want' to watch, they have every interest in what will compel you to watch for the longest time! Maybe a certain person wants to watch a few 2-minute cute cat videos, and subscribe to those exclusively. But research showed Google that those people's watch minutes per day can be tripled if you fill their homepage with "Trump did WHAT?" videos (or whatever effectively baits their rage, makes them more afraid, or stokes some addiction or anxiety).
Short term yes, but long term it turns people away from YouTube.
A year ago, I had a serious YouTube habit, once I replaced my trash Jellyfin server with a Plex server I can listen to my music collection on my phone anywhere… so no more music from YouTube. I got tired of asmongold and all his imitator gaming YouTubers, fell out of the habit of watching Ukraine warbloggers, etc. I saw other people who got into toxic rabbit holes in YouTube so bad that they decided to physically destroy their computers…
Gambling has been around forever. Hyper aggressive slot machines do nothing to dissuade addicts, and dark patterns on the web are the same. They are trying to build addiction, and addiction doesn't care that something hurts to do, you need it.
The few of us who go "ew" and recoil are vastly outnumbered by the billions who just watch.
Every complaint about ads on youtube is someone who can't even be bothered to download an adblocker before Chrome killed it. It was one click, but that didn't dissuade the vast majority of eyeballs.
>Short term yes, but long term it turns people away from YouTube
for some people, like me, for example, it turns them away even in the short term, and also in the permanent term, so to speak ha ha, not only in/after the long term.
because, you know, we know our rights and likes. and we wrong and dislike people who disrespect them! :) choice of rhyming words used for effect, but the point is also true.
If they give you want you want you might just enjoy it and leave satisfied. They don't want you to leave, what you want is largely immaterial except as an input to the machine designed to brainwash you into staying.
I love how passive aggressive the home page becomes: it momentarily displays a grid of thumbnails, then erases them and says, "Your watch history is off. You can change your setting at any time to get the latest videos tailored to you" with a button to do that.
I finally broke wasting hours on YouTube shorts (and youtube in general) by turning off the watch history on my account [1]. It completely removes all videos from the "homepage" (including shorts from the sidebar). There are still shorts in the subscriptions page, but I think this is an acceptable tradeoff. YouTube for me now has just become who I'm subscribed to, which is a much more pleasant experience - there's an "end" where I'm finally caught up and can move on to doing something else. This is also for my entire account, so it's not something I can just disable from my browser bar or that won't work on mobile. I don't need to remember to set it up on a new device either.
There is a great browser extension for Chrome/Firefox called "Unhook" which allows you to selectively remove parts of the YouTube UI you find distracting. Personally I have shorts and recommended videos turned off.
I've always found "doom-scrolling" fascinating because, for all of my addiction-prone traits and ADHD-granted hyperfocus, I never seem to get sucked into it. I've opened TikTok a few times for some random video I've searched and continued scrolling the next few videos out of some UX-driven guidance...then completely lost interest after 4 or 5.
Funny/Memey videos with low content value are entertaining, here and there. A rapid succession of them does nothing to the reward center of my brain. Or worse, the video would clearly be better as a longer form video and now I'm just frustrated (this is more common with YouTube Shorts).
That being said, I probably have YouTube normal-long form content running in the background 4-8hours out of the day.
Kagi employee here. We're actively working on building our own indexes beyond the limited ones we have now, not just a general index but also purpose built indexes for things like programming, etc.
reply