I didn't try original BERT at all because I didn't get good results from any LLMs on small document excerpts, so I assumed that a substantial context was necessary for good results. Traditional BERT only accepts up to 512 tokens, while ModernBERT goes up to 8192. I ended up using a 2048 token limit.
Would you happen to know of any resources for how to distill a ModernBERT model out of a larger one? I'm interested in doing exactly what you did, but I don't know how to start.
I was trying to identify "evergreen" and "time-sensitive" kinds of writing -- basically, I wanted to figure out if web pages captured in 2016 would still have content that's interesting to read today or if the passage of time would have rendered them irrelevant.
Here's the training code that I used to fine-tune ModernBERT from the ~5000 pages I had labeled with Llama 3.3. It should be a good starting point if you have your own fine-tuning task like this. If you can get away with a smaller context than I used here, it will be much faster and the batches can be larger (requires experimentation).
We were also allowed to borrow and re-shrinkwrap games at the Game store I worked in, in the UK, in 2000. Seemed like official company policy to give us better product knowledge!
I have to disagree. It repeats the same pseudoscientific presentation of correlation, as having to do with the semantics of the data.
Generically, correlation has nothing to do with whether data are related, dependent or independent.
It only is an indication of this *if you already believe* they are dependent. There has to be a prior semantic model of what the data means (what it is a measure of, how reliable its a measure of it, etc.) before correlation measures anything at all.
The only reason we would suppose correlated data to be related, is just that we design experiments to already contain possible dependencies. We do not, as a habit, measure say, the fall of rain and the beat of a song playing. But these would, given suitable measurements, be correlated.
This becomes absolutely vital to understand when experiments do take this "hoover up everything, arbitrarily" approach. As often they do in the social and psychological sciences.
In the UK the tax incentives for Electric cars may be skewing demand towards new Vs secondhand EVs.
I can lease a new EV via my employer's salary sacrifice scheme. I can pay my lease payments from my pre-tax income. There is an additional tax due on cars leased this way in the UK called Benefit-in-Kind tax (BIK). The rate of this tax is fairly high for petrol/diesel cars but for EVs is currently near zero (based on 2% of the car's value).
The problem is that most of the major lease firms that operate these programs for employers only offer new vehicles. Ideally I would like a nearly-new EV. I have escalated and apparently our lease provider (Tusker) are looking at rolling this out in the first half of this year. I currently know of only one other lease firm that offers this option. I suspect is in the interest of lease firms to prop up the value of the used EV market, but this depends also on their margins on new vehicles. I wonder if it would make sense for the tax incentives for used Vs new EVs to be rejigged to avoid incentivising unnecessary new car production?
I appear to have a yeast intolerance that stops me drinking some (but not all) beers. I can drink Guinness though! I didn't realise it used a distinct strain of brewers yeast. If only breweries listed the yeast they used in the ingredients, I could narrow down which one(s) are problematic...
Pretty much every beer uses a distinct "strain" of yeast so that information is unlikely to be useful and it's probably full of noise[0]. As per this paper one thing makes Guinness interesting is that it's strain is fairly divergent.
[0] There are strain banks like white labs that perform lots of analysis on the various beer strains.
> Pretty much every beer uses a distinct "strain" of yeast
I am a professional brewer and this is hilariously incorrect.
_Some_ beers have their own strain of yeast, but banking a yeast strain at a yeast lab is prohibitively expensive for most breweries.
The vast majority of breweries will have maybe 2 strains in house, an ale and a lager yeast. Sometimes they'll throw in a strain for NEIPA / Hazy beers as well. These strains are not specific to that brewery however, just a strain they like and will reuse for several pitches until starting fresh with a new batch from the lab.
To put it in White Labs terms, WLP001 aka Chico yeast is used in the overwhelming majority of IPAs. As referenced elsewhere in this thread almost all stouts are brewed with Guinness yeast aka WLP004. For the Hazy, WLP008 or WLP066. For the lager, WLP830 or WLP840 typically.
This is not to say that plenty of breweries don't have and maintain a proprietary yeast, especially at the macro level, but to say "pretty much every beer" has it's own yeast is incredibly off the mark.
I assumed they meant "pretty much every beer _type_" has a different strain, which you pretty well proved in your reply. That might just be an assumption, though.
About 15 years ago, I developed an unfortunate reaction to drinking beer: heartburn that made me wish I was dead. Kinda ruined Mexican and BBQ dinners, both of which are not just tasty, but cultural events here in Texas. The condition got progressively worse, but I still stepped up for the abuse from time to time,until I tried a thick, chewy porter or stout that did NOT trigger the dreaded heartburn. I still have the heartburn (I manage it with an unusual mineral supplement), but can almost always enjoy a good, dark, porter or stout (the darker, the better), which has improved life quite a bit!
If you don’t mind sharing, what are the symptoms you experience when you have a yeast you can’t tolerate? I ask because I believe I might have a yeast intolerance of my own.
Relatively fast onset gastro symptoms. Used to be fine with any beer but at about 21 started noticing the problem with a lot of largers. Asahi, Tsing Tao, seemed less of a issue. Not formula diagnosed.
I don't know specifically about your symptoms, but one thing it could be is the carbonation.
All draught Guinness (including Guinness Draught cans) is not carbonated with just CO2, but with nitrogen and CO2 in a 70-30% (thereabouts) mix.
Additionally, when pouring a typically carbonated beer incorrectly (or not pouring it into a glass), a larger amount of CO2 remains in solution, leading to gastro bloat that can feel more uncomfortable than one would expect. You can experience bad pours even at bars, depending on who's behind the tap.
Lagers will have higher carbonation levels than most stouts, so that could be a factor.
I'm not your parent but similar issue. I doubt it's the CO2 if he's anything like me. I can drink coke all day long (well sugar free one anyway - but I don't drink pop except in rare cases)
Beer? Not much and it's worse the more days in a row I drink it. So 3 beers over 3 days total is worse than 3 beers one evening and then nothing.
And the alcohol doesn't matter. Same issue with alcohol free versions of the same beer. Hefeweizen like Erdinger which has lots of yeast on the bottom of the bottle is worse.
It's like the yeasty parts of the beer just take over my gut and wreak havoc.
I got things under control by both not drinking beer and switching to hard stuff if I'm after the alcohol part or just completely different drinks if I'm after the taste part as well as finding a probiotic that works for me.
Nota bene: for me I also got headaches with it all. Like really bad migraines and even with certain foods. The probiotics made it all go away as long as I don't overpower them with beer or Pizza for that matter. Ordered in Pizza, if I eat too much guarantees headache a day to two later.
Some yeast produce more msg than others? The way it accumulates for over time seems quite different than the gp that indicated fast onset. That sounds more like they have an active allergy. Perhaps to sulphites?
Interesting. I have what are basically eosinophilic esophagitis symptoms (feels like a food impaction) that seems to be acutely triggered by unfiltered wheat beers but I’m not bothered by rice beers. I had also homed in on yeast as a possible culprit.
Many beers are filtered to remove the yeast since it's not really desirable once fermentation has completed. BUT - and a big "but" - there are styles like hefeweizen and other unfiltered beers that deliberately leave the yeast in for flavour.
Really interesting point about yeast being left in beers. I always assumed yeast was removed at one point in all beer due to its role in fermentation. The idea that styles like hefeweizen intentionally keep the yeast for added flavour surprised me. On a similar note, as someone who only drinks non-alcoholic beer, I've noticed that while most use yeast, some opt for 'simulated fermentation' without it. Yet, to my taste, there isn't much difference between the two. This makes me curious about what specific flavour characteristics yeast is supposed to add. What should one look out for?
If it doesn't taste like hops or grain, it's a yeast flavor. At different temperatures it contributes different flavors. Banana and clove flavors are hallmarks of a hefe yeast, and are produced at relatively higher fermenting temperatures. The chemical categories are esters (the fruit flavors) and phenols (the spice flavors). Many esters are reminiscent of fruits because fruits in nature begin to ferment with the same yeast.
Beers are also sometimes fermented with organisms other than Saccharomyces. These organisms contribute a wide variety of flavors, generally funky or sour.
Many beers are unfiltered because the filtering process can remove other, desirable flavors (mostly hops). Filtering used to be very rare in American craft beer for this reason.
So a Hefeweizen translated literally would be a "yeast wheat".
Very tasty beer though. And I have the same issue. Still my favorite beer: Erdinger Weizen. Preferably the dark one. But it has a ton of yeast on the bottom. And I get gastro issues very quickly when I start drinking them again. Non alcoholic versions don't help.
Wheat beers are my favourite, but agree they cause gastro issues fast, particularly the Erdinger Alkoholfrei.
I've discovered that Schneider Weisse 0.5% and, to some degree, Rothaus Hefe Weizen 0.5% are less likely to provoke gastro issues while closely mirroring the taste of their alcoholic equivalents.
If you like lagers and don't drink alcohol, I've found those from Fee Damm (Estrella)[1] and Luckysaint[2] to not cause any issues either, particularly the 'Tostada Amber' from Free Damm and the larger from Luckysaint (Their Hazy IPA is great too!).
Many (most?) craft brewers also skip this step. "Desirable" is in the eye of the beholder, and crucially, it saves effort/time/money to leave it in. You can even market it as "bottle conditioned," if you allow the final bit of fermentation to do the carbonation. (i.e. add a bit of sugar at the bottling step, rather than bottling carbonated beer.
Is it an intolerance to the fungus itself or to one of their metabolic products, like a particular complex sugar or fusel oil, that end up being metabolized by your own intestinal flora to your detriment?
Those hazy-style IPAs absolutely wreak havoc on my stomach. I think the unfiltered yeast that does it. I also do drink Guinness without issues. Mostly it's hoppy IPA type beers that dont agree with me. More filtered traditional beers are ok.
A lot of those go from tank to serving so fast now that they're still fermenting actively when served. They're also brimming with hop matter, organic alcohols, and odd yeast byproducts that get produced when certain yeasts eat a lot of certain hop compounds. Some people don't tolerate that mixture very well.
"that they're still fermenting actively when served"
That was true of beer brewed hundreds of years ago so it would have some carbonation and its sourness would be tolerable but it is not true of any modern brewery. Unless they want exploding cans.
Even though hazies are best drank fresh they go through all the normal fermentation as any other ale. Otherwise they would be rough, estery, butter bombs from not giving the yeast time to clean themselves up. Many breweries just screw them up as it's one of the harder styles of beers to do properly.
If the hazy your drinking has yeast sediment in the bottom of the glass/can it was done wrong.
I could name half a dozen breweries that have had exploding can issues in the past few years. My wife used to work for one of them in NYC.
> Otherwise they would be rough, estery, butter bombs from not giving the yeast time to clean themselves up
Yeah, that sounds like a lot of trash hasizes people are cranking out these days! Except for "butter bombs", you have to work pretty hard to produce diacetyl. I've done some homebrew club experiments and you can do almost everything wrong and not get that textbook buttery flavor.
> If the hazy your drinking has yeast sediment in the bottom of the glass/can it was done wrong.
Yeah, I agree it's just not uncommon in my experience.
How did you find you have a yeast intolerance? Was there a test you took? I've had some GI issues that have also limited my consumption of beer despite having previously brewed beer along with many other things I can no longer eat.
If you took a test that determined it was a yeast issue disregard but I'm curious if you've tried things like hop water (carbonated water with hops from beer). If you've had any reactions to that when it's just hops and water.
Does bread give you issues? What about sourdough bread vs traditional bread yeast bread?
At least for me, all it took to discover an intolerance to certain yeasts was visiting a tasting room for yeasts (https://www.whitelabs.com/) and sniffing a beer containing the specific type of yeast (Note: all the beers I was tasting were variating just the brewing yeast).
The reaction caused a massive nasal congestion and a runny nose that essentially ended my ability to taste or enjoy further tastings for the day.
I've also seen this reaction occur across very similar beers: Weihenstephaner Hefeweissbier non-organic is totally fine, but the organic one will cause the nasal reaction.
How much beer can you drink before you have issues?
I used to get really sick around 16-24oz in. Then, one day someone explained to me that corn syrup has a lot of the reagent still in it, and I noticed that when I didn't consume drinks with corn syrup, I could consume more beer.
That's funny. I went for an open day in the UK Computer Science dept in 1999. It was an exciting department but one of the things that put me off was the internet was so slow in the lab it was almost unusable. Perhaps they were having a bad day...
UKC was on Janet so it shouldn’t have been so terrible compared to other institutions. One of the biggest issues back then was people running remote X sessions on shared departmental machines so everything would grind to a halt at the start of lessons (unless you knew how to use Blackbox etc). And all the best machines were reserved for multimedia students.
I've often wondered this. Would be interesting to see how useful that 3dfx IP ended up being? When I was a teenager I tried to get my parents to open a brokerage account so I could buy some 3dfx stock as I was so excited by what they were doing.
