Fatdog64 811 is released
Fatdog64 811 was released on 10 September 2020.Release notes here, announcement here.
Get it from the usual mirrors: ibiblio.org, aarnet, uoc.gr, or nluug.nl.
Comments - Edit - Delete
RetroPie
I've always enjoyed retro gaming. In Fatdog we have always had dosbox (emulator for DOS games) for as long as I can remember, and scummvm (for Sierra games) came a little bit later. I've recently added a few more emulators: mednafen (multi-emulators for GBA, SNES and others), desmume (NDS), duckstation and pcsxr (PS1) as well as pcsx2 (PS2). I threw ZX-Tune as well, to play the music from those old games.But very recently I've been introduced to RetroPie, a distro for Raspberry Pi (raspi) which is dedicated to turning your raspi to a retro gaming machine. Since I have a dusty Pi3 laying around doing nothing (which I was supposed to use for testing Mick's Raspbian Buster but never got my lazy bum off to actually do it - sorry Mick!), I reckon, why not give it a try. If it doesn't work all I wasted is a couple of minutes downloading ~800MB image and couple of minutes "dd"-ing it.
As it turned out, it worked the first time around. Once the SD card as prepared, I put it inside the raspi, hooked the raspi to my TV, and then turned the raspi on. I was instantly asked to configure my controller (I didn't have one, but no problem, retropie accepts keyboard too). I configured the optional wifi setup (I didn't have to do it, but I wanted to try its samba feature). Then all I needed to do was to install the games (which I can install via USB, SFTP, or Samba). No stupid questions, no hassle, no config file. It's all ready to use. Most of the popular emulators were already included, and those that don't, are literally available for installation, a few clicks away.
With retropie on it, the raspi has been magically turned into a retro gaming machine. In the beginning I didn't believe that the raspi had enough muster to pull up a decent emulation, but I was pleasantly proven wrong. Most emulators worked well. Some had a few stutters every once a while but it was nothing to fret about.
The display quality was good, too. Retro games were notorious for displaying low-res pixelated images on today's high-res TV, but the emulators included in retropi had a few tricks up their sleeve to make the images sharper. I'm not surprised about that (after all I compiled some of those for Fatdog64 too) but what surprised me that the little raspi could pull off the enhanced graphics too. Well again not all emulators and not all games, but as I said, nothing to fret about.
Now, I have FatdogArm, Fatdog's variants that runs on ARM machines, the raspi included. If I really want to, I could, in theory, build and package all these myself too, producing a custom FatdogArm build that does what retropie does. All the software used in retropie is FOSS software. The standard release of FatdogArm was built to run as a "desktop-replacement" OS, but of course at its very core it was designed to be modded to produce a build that met specific needs. Making a retro gaming machine would be one of those.
But of course, why bother? The folks who makes retropie does a very good job, and the result is a very polished retro gaming software. Unless you try very very hard, you won't even know that beneath all the pretty faces, it runs Raspian Buster Linux. Rather than spending thankless hours building software on FatdogArm so it can become retropie wannabe, I may as well use retropie as is and start gaming with my kids
Anyway. In case you want to try retropie and don't have raspi, there is no need to fret and there is no need to fork out extra dollars too. While originally designed for raspi (all varians from pi 0/1/2/3/4), today's retropie supports other platforms: Odroid C1/C2, Odroid XU3/XU4, as well as standard PC! Yes, standard PC. You can use your old laptop, old desktop, NUC if you have one, etc basically any PC. The details are all on their website, if you're interested, go ahead and check it out.
Meanwhile, I've got a few games I need to catch up.
Disclaimer: I am not affiliated with retropie or any of its developers. In fact I wasn't aware that it exists until last week. I'm writing this to share with fellow retro gamers who aren't aware of retropie.
Comments - Edit - Delete
spcplayer patch for libao
I've been intro retro gaming lately, and one of the interesting aspects of the retro gaming is their music; in particular, the music in their original format.One of those formats is SPC. There are plenty of players for Windows (mostly plugins to popular music players), but standalone players for Linux are few and far between.
There is this player called AudioOverload but it is closed-source and output only to /dev/dsp. It does have a nice interface and supports many different video game formats, not only SPC.
The older version of gstreamer (0.10 branch) has a decoder in their 'bad plugins' collection called "GstNsfDec" which apparently can be used to decode SPC file (I haven't tested) - but do I really want to install the entire train of old gstreamer libs just to do that?
Then I found this vspcplay is an SDL-based player which is really nice and full featured; unfortunately the emulation isn't accurate; it sounds different from what I heard in the game and it eats 100% of one of my CPU core when running it.
Finally I found this one: SPC-PLAYER, a CLI player for SPC but unfortunately like AudioOverload it also outputs to /dev/dsp (and other OSS devices).
The problem with outputting to /dev/dsp is that whatever program is outputting the sound takes an exclusive use of the soundcard. I cannot play anything if I have my web brower open and one of its tab is pointing to youtube (even if nothing is being played), because the browser keep the sound access open and this prevents other programs to access /dev/dsp.
The only way to make work nicely is to use ALSA, which comes with built-in mixer (dmix) allowing multiple programs to use the sound card at the same time.
So I've written a simple patch for SPC-PLAYER to use libao, a very nice sound output library that enables us to output to a variety of output devices (ALSA, OSS< Pulse, etc) with exactly the same, simple API. Nice.
Well, if you're interested, you can find the patch in my patches page.
Note that SPC-PLAYER compiles in 32-bit only due to its extensive usage of x86 assembly in its APU emulation.
__________________________
EDIT: and just after I spent the effort to create the patch, I found ZXTune which is a full-fledged multi-format chiptune player which includes support for SPC and many other formats! You know, you can't guess by its name alone (ZXTune - it plays only sounds for ZX Spectrum? Come on ...). It's open source and binary is provided, too! Works out of the box and supports the formats I'd love to play. The program comes in GUI flavours (Qt) and CLI, with only minimal dependencies needed (the rest of the libs needed were compiled statically with the binary).
So go ahead and use that instead.
------------------------
EDIT: After finding ZX-Tune, JakeSFR informed me that deadbeef music player can also play SPC files.
Interesting. I looked into that, and found that deadbeen uses a library called "game music player", also known as libgme, to do that. And I found out ZXTune uses the same library too!
The library also has a simple "sample player" which uses SDL for output and hence is not affected by the "OSS problem".
You can get libgme from here.
That probably explains why nobody bothers to patch SPC-PLAYER, but that's okay. At least now I know how to use libao for my future projects
Comments - Edit - Delete
New forum for Fatdog
Fatdog has, for years, since its inception, always been piggybacking on Puppy Linux forum. Firstly because of its obvious roots; also because we simply didn't have enough man power to run and police an independent forum ourselves.As the previous blog entry indicates, however, John de Murga, the owner of the Puppy Linux forum, sadly passed away, and the original Puppy Linux forum went down (for a time).
Behind the scenes people are frantically working to bring it up, but there are also efforts to setup a replacement forums, including one by rockedge.
By now, both efforts have been fruitful: the old forum has been resurrected and the rockedge forum has flourished with old and new members (re-)joining it.
The stewards of Puppy Linux has decided that the old forum will continue in read-only mode as archives, while rockedge forum will take over as the new Puppy Linux forum.
Rockedge has graciously offered a section of the forum specifically for Fatdog, for which we are very grateful.
After some internal discussions, we have decided to take the offer and continue the tradition of piggybacking on Puppy Linux forum. All of Fatdog team members have now re-joined there as well.
For your reference:
---------
The new forum address: https://forum.puppylinux.com (Fatdog posts will be in the Fatdog section)
The old (archived) forum (in case you need to review/check old postings for older version of Fatdogs): http://oldforum.puppylinux.com (Fatdog posts are in the Puppy Projects section)
We'll see you there in the new forum!
Comments - Edit - Delete
Goodbye to a friend - John de Murga
John de Murga (aka John Murga) was the owner of the official Puppy Linux forum, formerly located in http://murga-linux.com/puppy/index.phpIf you use Fatdog64, you will know that we hosted our support threads on the same forum.
But my involvement with the Puppy Linux forum dates back well before then. I started browsing the forum when I was still a Windows user back in 2006; and joined in 2007 when I was converted into Puppy Linux user - and had been a regular on the forum ever since. Until the forum went down in early July 2020.
---
But in all my years being John's forum, I don't really know John Murga. Not personally. Never met him. Never spoke with him. Not even knowing how he looked like - well, not until a few days before I wrote this. Not through my fault though, I only know bits and pieces that he himself chose to share, for he himself claimed to be 'the man of mystery'.
He and his work, however, left an indelible mark on my life. It was such a simple work. Setting up an online forum. What was so difficult about it? Get a web-hosting site. Click a button, and you've got a forum and up running. Easy. Everyone can do it, right?
No. Not every one can do what John did.
The forum he set up was the forum where I spent lots of my past life on. Where I got help when I started and when I stumbled. Where I, eventually, helped others as I became proficient enough. Where I, unexpectedly, met and became friends with people in the real world.
All of these because he had the tenacity to keep the forum going for 15+ years; and let the forum to govern itself instead of strict policing found in many others. He even had the magnanimity to allow people to post links of competing projects and forums. This was one of the factors why so many people stayed on and the forum grew to become the melting pot of folks from all walks of life, sharing the same interest - Puppy Linux, and Linux in general.
So it was great shock and sadness that I learn of his passing recently in May 2020. I was even more stunned to know that he had left a young family - a daughter and twins who were born shortly after his passing. I have young children myself, I could not even begin to contemplate how it must have felt, to be left so soon, and so suddenly.
Nevertheless, I would like offer a prayer of hope for whom that John's family, that John had left a legacy that not many people could manage to do: to change the life of thousands, if not then ten thousands. John had left many friends that he himself probably didn't know. I was one of them, one among the thousands. For us, he is a hero. In every sense of the word.
Goodbye, John, and all blessings in your next journey.
You will be missed, but not forgotten.
---
Eulogy from 01micko, the current steward of Puppy Linux
Eulogy from Barry Kauler, the creator of Puppy Linux
Comments - Edit - Delete
Fatdog64 updates - behind the scenes
Fatdog64 was last released on 23 Jan. And although we have been quiet, it doesn't mean that things stopped. Things are still going in the background, bug fixing, adding features, etc.Here are a few major things that has been going on, in no particular order.
1. Puppy Linux forum user ICPUG has indicated possible issues with Fatdog savefile residing on an NTFS partition, especially partition shared with Windows. This is due to the way Fatdog uses ntfs-3g: it runs with full permission control enabled. It makes NTFS behaves like POSIX filesystem and we can use it for save directory (not only save file), but on the other hand it makes Windows complain about not-granted access each time the partition needs to be accessed from Windows.
Based on this feedback, we have added "ntfsnoperm" boot parameter to disable that permission control. When this parameter is used, ntfs-3g is run without permission: it would behave just like FAT filesystem (all files and directories are owned by a given uid/gid specified at mount time). It would not be possible to use a save directory on NTFS, but at least the permission wouldn't be touched and Windows will stop complaining.
2. The same ICPUG also found an old bug related to the above: where the user "spot" would be unable to access a Downloads folder that has been relocated to a save partition if the save partition is NTFS. This has been fixed.
3. Fatdog has long supported btrfs. The kernel is built to support btrfs and we have the complete btrfs-progs included; and nominally we support having save directory on btrfs. But this does not actually work; because "aufs" - the unifying filesystem layer that we use to make the magic happen, does not support btrfs the way it supports other filesystem. For the techie: the problem is that aufs cannot have its xino file inside btrfs. It has to be somewhere else.
Thanks for our team member SFR (his Puppy Forum name), this was brought to attention and he shared a fix too. The fix was tested, worked on, and finally merged: now, save directory will work seamlessly on btrfs-formatted partition.
4. The original motivation for doing (3) above was actually to attempt to use a compressed filesystem. This has been a particular interest for me for years. The last time I explored this (a couple of years ago), btrfs wasn't mature enough and there was no other native writable filesystem with compression support. Sure, there were native filesystem with compression support - but they were all readonly (squashfs being the most popular one). Sure, a couple of FUSE filesystems support compression too (in fact it's their number one feature), but they were not actively maintained and being done in userspace, it was slow. I never got enough motivation to implement them properly with Fatdog.
Btrfs finally changed this equation. By now it should be considered mature enough (although probably not as hyped as before), and it supports compression as well. In fact, it supports three different compression algorithms: lzo, lz4 and zstd.
So as we fixed btrfs save directory support, we grafted compression support too. If the kernel parameter "btrfscompress" is specified, the compression support for btrfs will be enabled, using the algorithm specified on that parameter. This has been tested to work wonderfully and using zstd, the compression rate is about 30% on average.
5. We have an update on the in-house "screencaster" application (a program record the video of the display). It now has the ability to take repeated screen capture, better quality video, among other things.
6. Step, another member of Fatdog team, has also revamped the "Samba Shares" application. It has been re-factored and is heavily tested to work across different Samba servers (different Windows versions, etc).
7. We have also fixed a long-standing bug when running Fatdog with RAM layer operation, which can cause inconsistencies if the system is heavily used (a lot of filesystem access) when the "merge down" process happens. SFR found this to be irritating enough to find a solution that works better; and this got incorporated into Fatdog.
An additional feature was also added: the ability to remove whiteouts if they don't hide any files on the lower SFS layers, but not activated by default due to possible conflict with multisession operation (we haven't tested the interaction yet).
8. We also include the "updater script" which re-enable VLC's ability to play youtube video directly, without having to update VLC itself. The script is called update-vlc-playlist-luac.sh. SFR found this script.
And of course, many other smaller bugfixes and feature addition, as well as adding and updating packages on the repo as well.
So when will we have a new release? Well, it will be released when it is released.
Comments - Edit - Delete
Fatdog64 810 Final is released
Maintenance release, basically a bug-fixed version of 810 Beta.Release Notes and Forum announcements
Can't believe that Fatdog is coming to its twelfth year.
Download locations: ibiblio.org, aarnet, uoc.gr, nluug.nl.
Comments - Edit - Delete
CSVfix patches for regex and exec
CSVfix is a tool for manipulating CSV files. Along with the usual column re-ordering and filtering, CSVfix offers a powerful per-cell data transformation using a simple expression language, as well as regular-expression for string matching and editing. And if this is not enough, CSVfix can execute external process - for every cell that needs to be processed. And oh, it's available for Windows too!I find this tool to be very handy in what I need to do, so when I encountered a bug in its regex processing (for its "edit" command), I immediately checked if there is any updates to this tool. Unfortunately, its development seems to have ceased in 2015; and no other people seem to have picked up the development (I did find some forks, but they were all older copies from when it was still hosted in google code).
So I set out to figure out about the problem and hopefully rectify it. I found that the problem was in its regex library, which was a home-grown library (apparently adapted from an algorithm book). It is 2020 as of this time of writing, and C++ now comes with its own STL regex library (std::regex). I decided to rip off the custom regex lib and replace it with the STL regex instead, while keeping the rest of the class interface identical, therefore no other part of the code needed to be changed. This instantly fixed the problem, and as a bonus, now we can use ECMAScript-compatible regex instead of just the basic regex.
Later, I found out that the "exec" command also had a bug (the flag "-ix" did not work properly), so I traced this and fixed it too.
Oh, and during the process, I tried to run its testsuite - and while most of them passed, some did fail. Mainly because of CRLF/LF inconsistencies, so I changed those the test data to use LF. It is also a warning that this tool only works with platform "newline" - CRLF in Windows, and LF in Linux - so if files were to be exchanged between platforms, they must be properly translated first before use.
Here are the individual patches.
- regex patch
- exec patch
- test-case patch
They apply on top of the commit 93804d4 from 2015-02, which was the latest when I wrote this. They are licensed in the same way as the original CSVfix is licensed.
If CSVfix is not powerful enough for you, there are other similar tools:
1. miller is a tool in very similar spirit with CSVfix, but it is (much) more sophisticated. Its "data transformation language" looks more expressive than the one in CSVfix. If you have a problem you cannot solve with CSVfix, miller will probably help you. As a bonus, it is still in active development - that means bugs will be squashed. It is written in C, you will need to compile it if it is not in your package repository. (Fatdog, naturally, has it in its repository).
2. csvkit is a collection of tools that more or less perform the same functions as CSVfix. It supports direct conversion to/from Excel files, importing/exporting into databases (sqlite and postgresql as documented, perhaps others too), as well as running direct SQL queries from CSV files (and databases too). It is written in Python3 so you can install it using pip3. Fatdog has this in its repository too (so you can install it using package manager instead of pip3).
3. rbql basically enables you to run SQL-like on CSV files; but its power is its ability to run python (or javascript, depending on which backend you choose) code for every cell. Fatdog has it in its repository too, although you can just use pip to install it, if you don't run Fatdog.
Comments - Edit - Delete
Fatdog64 801 is Released
It is a rolled-up updates, containing bugfixes and minor feature updates.Release Notes and Forum announcements
Get it from the usual locations:
Primary site - ibiblio.org (US)
nluug.nl - European mirror
aarnet.edu - Australian mirror
uoc.gr - European mirror
ISO Builder: Get it from here: http://distro.ibiblio.org/fatdog/iso/builder/ and choose the builder dated 2019.05 and the package list for 801.
Enjoy.
Comments - Edit - Delete
The Road to Fatdog64 800 Alpha
Today, the first of Fatdog64 800 series is released to the wild - the Alpha release. Despite being labelled "alpha", this release has been tested for a few months and has been used day-to-day in real production machines (in varying degrees) for about two months by all of us, in the team.It has not been a smooth ride all along. Living in the "bleeding edge" means that you really need to prepare to bleed (that's why we don't update the base on every release - it would be downright impossible). Latest packages don't always build, and when they do, they don't always run, and they do, they don't always run stably, and when they do, they don't always work, and when they do, they don't always work correctly, and when they do, they don't always provide good performance, and so on, and so on - you get the picture.
But we have finally arrived. It may not be perfect, and it never will, but for us - it is good enough for day-to-day usage; so the decision to release.
This blog post documents a few of the stumbling blocks that we passed through on our way to the Alpha release. By sharing the knowledge, I hope that others on the same journey can avoid them.
Warning: the information that comes after this is going to be very technical. Don't worry if you don't understand it - just use whatever that you do.
The particular bug that took as considerable time (weeks) to solve was this: it's a bug that causes the desktop to unpredictable, involutary exit to the console.
And it turns out, this problem has __multiple__ underlying causes. Each cause ends up with the X server exiting (sometimes gracefully and sometimes not) so that we're dropped back the console.
Sometimes there crash messages in Xorg.0.log, sometimes don't. Sometimes the exit is immediate (X goes down and we're back in the console), sometimes it's gradual (applications start to fail one by one, before X itself finally gives up its ghost and dies).
Bug #1: fontconfig doesn't like its cache to be tampered with.
This bug happens only when running with the RAM layer (savefile=ram:device:xxx, in Puppy's parlance this is pupmode=13). When anything that uses fontconfig starts (e.g X, gtk2, etc), fontconfig will be initialised and its first action is to scan all the font directories and makes its cache.
When we run using the RAM layer, these caches are stored in the RAM layer, and eventually will be "merged-down" to the actual savelayer (copying the files from the RAM layer to the savelayer, and then removing the copy on the RAM layer, and then refreshing the aufs layered filesystem so that the copy on the RAM layer gets re-surfaced on the root of the filesystem).
This has worked for the longest time, but we found out that in Fatdog 800 this isn't the case. Even the process of copying the files from RAM layer to savelayer (not even deleting them) triggered a cascade of failure in fontconfig, which eventually resulted a crash in all higher-level libraries and applications that uses this.
Fixes #1: We still don't really know what changed - this could be a change in the kernel, fontconfig itself, glibc, or others (remember, in 800, with a new base, **all things were updated** so we can't easily isolate one component from another), but once we know what triggered the collapse, we worked around it by making sure that fontconfig caches are not touched during the merging process.
Bug #2: Xorg server crash on radeon-based systems when DRI3 and glamor is enabled (the default settings).
This has been a long running bug due to Mesa (open-source OpenGL 3D library) changing its infrastructure to support newer, more powerful radeon cards, changing both the acceleration API (DRI2 to DRI3), acceleration methods (EXA to glamor), memory management (GEM, GBM, etc).
But the bug isn't in Mesa alone. Eventually Mesa needs to interface with the video driver, so co-operation with xf86-video-ati driver is needed.
And then there is the kernel too, the radeon DRM driver from the kernel.
There are multiple components and every component is a potential source of failure (which in this case they all contributed to the problem one way or another).
To make it worse, the problem is completely unpredictable. The system can run hours without a glitch before the desktop crashed, or it can crash in next hour after booting. It brought a completely un-reliable experience.
There is no good solution for this because every component keeps changing, so updating one component could very well break another. All we can do is watch bugzillas, forums, mailing lists, and listen to possible solutions.
Fixes #2: I think in this case, we got lucky. Most of the bugs were fixed in mesa 18.2.3, and the final bug was fixed in xf86-video-ati git-master, one commit after 18.1.0. We're going to stick with this combination for a while!
Bug #3: After an unpredictable amount of time, Xorg server will crash (due to failed assert), giving up messages similar to this:
[xcb] Unknown sequence number while processing queue
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
pidgin: xcb_io.c:259: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed.
There is one unanswered report here: https://bugzilla.redhat.com/show_bug.cgi?id=1582478
This initially happened on ROX-Filer when it was worked heavily, so naturally we thought the problem was with ROX-Filer. But then it started to happen elsewhere (other GTK applications), so it could be anywhere in gtk2, glib, glibc, aufs kernel module, or even the kernel itself.
We scoured for solutions for similar problems, and we got some solution like this: https://forum.freecadweb.org/viewtopic.php?t=28054
But they don't work and they don't make sense. FreeCAD is a 3D-heavy application, so disabling DRI3 and (in some others links, disabling 3D hardware-acceleration) doesn't make sense for applications like ROX-Filer which is purely 2D and doesn't make use of any acceleration at all.
This was especially difficult to pinpoint because it happened randomly; and it was one of the thing I would have filed into "unsolved" files, were it not for SFR. SFR found a way to reproduce this problem reliably (by clicking the "refresh" button on ROX-Filer a few hundred times - and he even provided a automation script so we don't have to buy a new mouse after every experiment
).
Once we can reproduce this, re-building the libraries with debug symbols and running them under "gdb" quickly pointed out that the problem is within libxcb - a library at the bottom of the Xorg stack.
Bisecting on libxcb, we found that the problem is caused by a particular code commit that tries to "fix" another bug when dealing with Vulkan drivers:
https://cgit.freedesktop.org/xcb/libxcb/commit/?id=fad81b63422105f9345215ab2716c4b804ec7986
But the way it was fix is, in my opinion, is incorrect (it was reading stuff when it shouldn't - this kind of thing should be protected with a mutex or we'll end up with a race).
Fixes #3: So we reverted this commit and poof! - the problem disappears. I tested clicking the button to about 16,000 times and no more crash.
This was the last bug we squashed.
So there. One symptom, three underlying problems, that can be triggered at different semi-random times, resulting in a totally different error messages and behaviour, confusing all of us.
We falsely declared victory after the first and the second were squashed - only to be humiliated when the crash happened again, in a slightly different way. By the time we squashed the last bug, we were wary enough __not__ to declare it fixed until a few days later and it was finally confirmed that we've finally made it through.
All in all, we spent more than a month to solve all of them. Now that we've past through them, I hope others can avoid the same mistake.
Meanwhile, enjoy Fatdog64 800 Alpha!
Forum announcement here: http://www.murga-linux.com/puppy/viewtopic.php?t=114719
Release Notes here: http://distro.ibiblio.org/fatdog/web/800a.html
Comments - Edit - Delete
