Hey folks, thought I'd share this little bit of info I discovered. For the longest time I've been using pigz (a gzip program which takes advantage of multicore capability) to compress my database backups. Out of curiosity (and that my backups keep getting larger) I wanted to see if using the xz format was possible. The xz program provided by xz-utils on my system isn't multithreaded (from the man page):
-T threads, --threads=threads
Specify the number of worker threads to use. The actual number
of threads can be less than threads if using more threads would
exceed the memory usage limit.
Multithreaded compression and decompression are not implemented
yet, so this option has no effect for now.
As of writing (2010-09-27), it hasn't been decided if threads
will be used by default on multicore systems once support for
threading has been implemented. Comments are welcome. The com-
plicating factor is that using many threads will increase the
memory usage dramatically. Note that if multithreading will be
the default, it will probably be done so that single-threaded
and multithreaded modes produce the same output, so compression
ratio won't be significantly affected if threading will be
enabled by default.
This stinks because the server running the backups has 8 physical cores and no shortage of memory. I found out that 7z and 7za from p7zip create xz archives using multiple threads by default. I did some experiments on a dual-core system (Intel Core2 Duo E6420) with various compression programs and levels using a 2908748kb (2.7GB) database dump from my moodle server. The 7z command-line options are a little weird, but workable. Let me break down one of the commands I used: 7za a dummy -txz -mx3 -si -so <db.sql >7z3_db.sql.xz
| Command Segment | Explanation |
|---|---|
| 7za | 7za is the 7-zip binary. The difference between 7z and 7za is 7za doesn't load any plugins (only uses native algorithms). For our purposes there really is no difference. |
| a | Add files to archive |
| dummy | Necessary when compressing to stdout with the -so option. |
| -txz | Sets the type of archive to xz. |
| -mx3 | Sets the compression level to 3 (valid range is 0-9). |
| -si | Compres |
I have been learning in my class about how multithreading libraries work under the hood, and one thing we talked about was the switch invariant. I understand how it works for the most part. We can disable interrupts and use guard variables to make sure that only one process is ever accessing the critical section of a locks lock function or similarly related functions. However, one thing that we didn't really dive into which I don't understand is how the switch invariant is maintained if a processor switches from the kernel code to a thread running user code.
https://riscv.org/2019/02/adacore-enhances-security-critical-firmware-with-nvidia/
So I know consoles like the early MicroGenius, Dendy and HiTek clones used aftermarket chips and replicated the entire boards to clone the NES.
What I'm curious about is when the tech required to completely clone the console onto a single chip became the standard? It had to have been some time in the 90s yeah? anyone know any more about this?
https://boingboing.net/2016/06/15/intel-x86-processors-ship-with.html
> Recent Intel x86 processors implement a secret, powerful control mechanism that runs on a separate chip that no one is allowed to audit or examine. When these are eventually compromised, they'll expose all affected systems to nearly unkillable, undetectable rootkit attacks.
> ...
> The Intel Management Engine (ME) is a subsystem composed of a special 32-bit ARC microprocessor that's physically located inside the chipset. It is an extra general purpose computer running a firmware blob...
> ...
> On some chipsets, the firmware running on the ME implements a system called Intel's Active Management Technology (AMT). This is entirely transparent to the operating system, which means that this extra computer can do its job regardless of which operating system is installed and running on the main CPU.
> ...
> ME [is] a huge security loophole, and it has been called a very powerful rootkit mechanism. Once a system is compromised by a rootkit, attackers can gain administration access and undetectably attack the computer.
> ...
> There is no way for the x86 firmware or operating system to disable ME permanently. Intel keeps most details about ME absolutely secret. There is absolutely no way for the main CPU to tell if the ME on a system has been compromised, and no way to "heal" a compromised ME. There is also no way to know if malicious entities have been able to compromise ME and infect systems.
> A large portion of ME's security model is "security through obscurity", a practice that many researchers view as the worst type of security. If ME's secrets are compromised (and they will eventually be compromised by either researchers or malicious entities), then the entire ME security model will crumble, exposing every recent Intel system to the worst rootkits imaginable.
