This codec would have been great 27 years ago! I suspect it wouldn't have gotten past the patent lawyers in 1999 though. It seems cute today but the entire codec space was a patent minefield, the N64 mp3 decoder was implemented twice (once based on the reference codec, once clean) with Nintendo/Rare then paying RAD game tools to license their decoder... of which none of their code was used but they had a license from the mp3 patent holder (Fraunhofer) which then shielded Nintendo from lawsuits. Fraunhofer wanted something like $250k to license directly which wasn't going to fly.
I'll also note the frame rate issues in Conker were mainly down to raw RDP fillrate issues, which was for sure the weakest part of the N64 hardware, it could draw beautiful triangles (for the time) but oh so so slowly. There was a decent amount of speech during gameplay using the RSP mp3 codec. The cartridge size was still an issue, 64Mb got a good amount of mono speech but was still very much at a premium.
Maybe in audio, but video still seems like an absolute nightmare. Look at the absolute silliness with like having to pay $.99 to play h265 video in Windows.
Real example, I recorded the world series using an ATSC antenna, and I re-encoded a 90 second sample using a bunch of different settings on a 32-thread znver5:
(NB: The same CRF results in a higher quality video at slower presets, this was for archival so we're using the some of the slowest presets).
(1) and (2) are completely indistinguishable from the original unless I go frame by frame looking for transient motion artifacts. (3) is only distinguishable in one shot where a dark background was washed out. Not only does AV1 produce results just as good as H265, it encodes quite a bit faster for the same size/quality.
That said, I've watched so much h264-encoded video over the past decade I think I've been trained to perceive the subtle visual "snow noise" as a sign of a high quality video: I find how AV1 blends backgrounds more visually objectionable than the dithering in H265, even though I think it's objectively higher fidelity. Similar to how I find high frame rate video "fake" after having watched NTSC TV for the first half of my life.
These were the CRFs necessary to get the same visual quality with libx265 at faster presets and the resulting sizes:
Almost nobody actually pays for this, since every OEM includes a volume license that's baked into the price tag. VLC includes FFMpeg so it just works ootb.
Video codecs have historically been like this, to some extent or another. It's really just a manner of if the manufacturer has absorbed it into the price tag or pushed it to the consumer.
No one likes paying for a codec license anyways, but since patents are at play, they kind of have to.
i've heard the rambus dram latency was also a big deal but does that have any effect on the fillrate as well? or is that mostly a bottlneck affecting the cpu?
and memory bandwidth. Our three weakest parts are our raw RDP fillrate, our tiny texture buffer, our paltry memory bottleneck...and compiling in debug mode... Our four... no... *Amongst our weakest parts... are such elements as... I'll come in again.
Don't shit on the N64 from too high. It introduced 1998 (and 2026) real-time rendering concepts at a time when two years was an eternity in PC hardware.
It's why modern programmers can make it do things that would have given John Carmack sleepless nights if they had accomplished them back in 1996.
In case anyone else got to the demo thinking the quality is awful -- the vocals are incomprehensible and keep cutting out -- it turns out it's like that in the original track:
Tom’s Diner by Suzanne Vega would've been a neat choice, as it was the track used to assess and refine the MP3 compression algorithm during it's development.
I wonder if this would also be a great fit for the MAS3507D! Supposedly you can replace the program with your own in a 1-kiloword RAM (the default MP3 decoder is in ROM, and probably larger). If you can fit this codec in 1024 MIPS instructions, perhaps you can do the same in 1024 of these DSP instructions…
Too bad there is seemingly zero public documentation on the format of these programs, though (and no emulator or similar).
No it's not, but it's splitting into pro-AI and anti-AI segments.
Think about it, if someone embedded GPT-2 into a demo in 2019 the same scene would've found it hilarious. It's not the AI, it's how we use AI and everyone being sick of it.
Pardon my ignorance but one thing I'm curious about, if the decoder fills almost the entire 4k RSP instruction memory (IMEM) - wouldn't that be something games need for graphics? If so, then is this still usable for say a game?
Like the DMEM, you can replace the IMEM at any time. In this case, when you're done with processing a block of music, you can DMA the code for 3d transforms or anything else to the RSP.
Libdragon handles this transparently for you with a queue[1].
At this point, they're coming out faster than I can play them. I'm in the middle of the 2nd Episode of Duke 3D right now and it's quite nice. The original 64 version was always fun, but mid-level saves are really a game changer.
There's an explosion of ALL ports lately. It's crazy.
Ocarina of Time only got released today officially but it's already playable on ARM Macs. It's not even a port per se but executing the Switch ARM code natively. And of course it's also playable on emulators regardless of the platform
libdragon certainly isn't new; I distinctly recall talking to dragonminded (the original author) about her efforts to reverse-engineer the RSP code that came with the n64 devkit back in 2008 or so and her subsequent efforts to reverse the RSP. I can totally believe the increased rate of games though
Didn't Top Gear Overdrive (1998) use MP3 tracks? It had several tracks by Grindstone that would play during races. The audio quality isn't that great but lyrics are mostly comprehendible.
Wipeout 64 had some music by Fluke and Propellerheads but I'm pretty sure it used some tracker format to re-create the tracks from short recordings and honestly, they sound awful, not just because the audio quality but because the repetitiveness and lack of progression.
I'll also note the frame rate issues in Conker were mainly down to raw RDP fillrate issues, which was for sure the weakest part of the N64 hardware, it could draw beautiful triangles (for the time) but oh so so slowly. There was a decent amount of speech during gameplay using the RSP mp3 codec. The cartridge size was still an issue, 64Mb got a good amount of mono speech but was still very much at a premium.
Maybe in audio, but video still seems like an absolute nightmare. Look at the absolute silliness with like having to pay $.99 to play h265 video in Windows.
Real example, I recorded the world series using an ATSC antenna, and I re-encoded a 90 second sample using a bunch of different settings on a 32-thread znver5:
(NB: The same CRF results in a higher quality video at slower presets, this was for archival so we're using the some of the slowest presets).(1) and (2) are completely indistinguishable from the original unless I go frame by frame looking for transient motion artifacts. (3) is only distinguishable in one shot where a dark background was washed out. Not only does AV1 produce results just as good as H265, it encodes quite a bit faster for the same size/quality.
That said, I've watched so much h264-encoded video over the past decade I think I've been trained to perceive the subtle visual "snow noise" as a sign of a high quality video: I find how AV1 blends backgrounds more visually objectionable than the dithering in H265, even though I think it's objectively higher fidelity. Similar to how I find high frame rate video "fake" after having watched NTSC TV for the first half of my life.
These were the CRFs necessary to get the same visual quality with libx265 at faster presets and the resulting sizes:
Of course there is always VLC
0: https://apps.microsoft.com/detail/9n4wgh0z6vhq
what?
Video codecs have historically been like this, to some extent or another. It's really just a manner of if the manufacturer has absorbed it into the price tag or pushed it to the consumer.
No one likes paying for a codec license anyways, but since patents are at play, they kind of have to.
I thought weakest part was the size of the texture buffer.
Nobody expects the N64! Our two weakest parts are our raw RDP fillrate, and our tiny Texture buffer.
If the RDP had been clocked 4x faster and had 4x the texture size then it would have been incredible… it makes beautiful pixels!
It's why modern programmers can make it do things that would have given John Carmack sleepless nights if they had accomplished them back in 1996.
https://www.youtube.com/watch?v=i1gVxKhdGPs&t=57s
not the music taste itself, but perhaps he should have gone for something more known that would be easy for A/B tests.
Too bad there is seemingly zero public documentation on the format of these programs, though (and no emulator or similar).
The demoscene and homebrew scene is dying as we speak.
Think about it, if someone embedded GPT-2 into a demo in 2019 the same scene would've found it hilarious. It's not the AI, it's how we use AI and everyone being sick of it.
Pardon my ignorance but one thing I'm curious about, if the decoder fills almost the entire 4k RSP instruction memory (IMEM) - wouldn't that be something games need for graphics? If so, then is this still usable for say a game?
Libdragon handles this transparently for you with a queue[1].
[1] https://libdragon.dev/ref/rspq_8h.html
Some I'm tracking:
- Duke Nukem 3D (ie, the fully-functional PC version) https://github.com/BigChat88/N64DukeNukem3DPort
- Portal 64 Still alive https://github.com/mwpenny/portal64-still-alive
- Resident Evil 64 (ie, not RE2, but the first one) https://www.youtube.com/watch?v=_xRiwy8Z6S0
- Silent Hill (still an early WIP) https://www.youtube.com/watch?v=v7fM06jmHso
I haven't personally tested, but these look interesting:
- Simpsons Arcade https://www.youtube.com/watch?v=zH9JYQVzKDE
- X-men Arcade https://www.youtube.com/watch?v=ah3nB8MP4CM
- TMNT Arcade Collection https://www.youtube.com/watch?v=CiR5-ZpLtyI
At this point, they're coming out faster than I can play them. I'm in the middle of the 2nd Episode of Duke 3D right now and it's quite nice. The original 64 version was always fun, but mid-level saves are really a game changer.
There's an explosion of ALL ports lately. It's crazy.
Ocarina of Time only got released today officially but it's already playable on ARM Macs. It's not even a port per se but executing the Switch ARM code natively. And of course it's also playable on emulators regardless of the platform
https://www.youtube.com/watch?v=kyXsWd7DpEs
Wipeout 64 had some music by Fluke and Propellerheads but I'm pretty sure it used some tracker format to re-create the tracks from short recordings and honestly, they sound awful, not just because the audio quality but because the repetitiveness and lack of progression.