9/5/21

apt update issue: Problem parsing dependency

If you encounter this problem when doing 'apt update'

E: Problem parsing dependency 21 of libc6-dev-s390x-cross:all=2.31-0ubuntu7cross1
E: Error occurred while processing libc6-dev-s390x-cross (NewVersion2)
E: Problem with MergeList /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_focal_main_binary-amd64_Packages
E: The package lists or status file could not be parsed or opened.


Do these:

sudo rm -fr /var/cache/apt/archives/*

sudo dpkg --clear-avail

sudo rm -fr /var/lib/apt/lists/*

sudo apt update

5/10/20

Healthy lifestyle and life expectancy

This is a summary of the Harvard study:
Healthy lifestyle and life expectancy free of cancer, cardiovascular disease, and type 2 diabetes: prospective cohort study https://www.bmj.com/content/368/bmj.l6669
In 3 sentences: 
At age 50, life expectancy free of cancer, cardiovascular disease, and diabetes was 23.7 (95% confidence interval 22.6 to 24.7), 26.4 (25.2 to 27.4), 29.1 (28.0 to 30.0), 31.8 (30.8 to 32.8), and 34.4 (33.1 to 35.5) years among women who adopted zero, one, two, three, and four or five low risk lifestyle factors, respectively. 
Life expectancy free of cancer, cardiovascular disease, and diabetes at age 50 was 23.5 (22.3 to 24.7), 24.8 (23.5 to 26.0), 26.7 (25.3 to 27.9), 28.4 (26.9 to 29.7), and 31.1 (29.5 to 32.5) years among men who adopted zero, one, two, three, and four or five low risk lifestyle factors, respectively. 
The percentage of life expectancy free of cancer, cardiovascular disease, and diabetes from total life expectancies was 74.8%, 77.6%, 80.1%, 82.2%, and 83.6% among women (75.3%, 75.8%, 76.8%, 77.9%, and 79.0% among men) who adopted zero, one, two, three, and four or five low risk lifestyle factors, respectively.
The 5 low risk lifestyle factors are:
1. high Alternate Healthy Eating Index - higher the better
2. no smoking - more smoking the worse
3. moderate activity - more vigorous the better
4. alcohol - none only better than heavy drinking, best is moderate drinking, surprise!
5. low BMI - lower the better until too low
I'm a little disappointed with these findings.  If you do everything right, you're only expected to live 31.1-23.5=7.6 years longer and only 4.2% more free of cancer, cvd, and diabetes.

Keep drinking!

4/10/20

Extracting video from blob url

First press F12, then select Network tab. reload page, then play video.  You should see a file called playlist.m3u8.  Right click it then copy address.

On a command line(Win, Mac, Linux), type youtube-dl followed by pasting the m3u8 link, then enter.

Of course you should already have youtube-dl installed.

4/9/20

Disable ubuntu update manager, timer from command line


On command lines:
sudo systemctl disable apt-daily.timer
sudo systemctl disable apt-daily-upgrade.timer
then edit 2 files:
sudo vi /etc/apt/apt.conf.d/10periodic
sudo vi /etc/apt/apt.conf.d/20auto-upgrades
In both files, change "1" to "0".

4/4/20

Creating Time Lapsed video from dashcam videos

Time lapsed video are usually create from photos.  Here's how to create one from dashcam videos.
First move all videos into a single folder, then list files and redirect output into a text file.  On linux, it's `ls > files.txt`, on Windows, it's `dir /B > files.txt`.

Then edit this file.  This file should list videos in the order of timestamp, if not, adjust them manually. Remove any files that's not video or any videos that you don't want to be included in the final video.  Put 'file ' at the beginning of every line.  (In Vim, you can do this by :%s/^/file /g)

Then create a raw video by select one frame in every 30 frames from all the videos in the file:

ffmpeg -f concat -i files.txt -vf "select=not(mod(n\,30)),setpts=N/(FRAME_RATE*TB)" -vcodec rawvideo -pix_fmt yuv420p -an raw.yuv
Because this is uncompressed raw video, its size will be huge, probably tens of GigBytes. You probably don't want to play it on your computer, let alone uploaded to youtube.  To make it playable, we need to compress it to mp4 format:

ffmpeg -f rawvideo -pix_fmt yuv420p -s:v 1920x1080 -i raw.yuv -vcodec libx264 video.mp4
 You can see an example of the time lapsed video here:

9/3/19

Tornado vs Starlette in 2019

For the last 10 years I have been using mainly  Tornado as my web framework of choice.  And I mostly use it synchronously.  Only when dealing with uploading files did i use some asynchronous tornado features.   Most of my apps have database backends.  With database involved, it's not worth it to convert my code to asynchronous mode.  Tornado has been worked well for me I have no intention of stopping using it in the near future.  But I do want to keep my eyes open for new tool that enables me to build fast app fast.

In the last few years, there are several new async web frameworks.  One of them is Starlette, I decided to evaluate it.  The first step is to compare it basic performance with Tornado.

Starlette, using uvicorn, claims to be one of the fastest python web framework.  I copied the codes directly from its website:
from starlette.responses import PlainTextResponse

async def app(scope, receive, send):
    assert scope['type'] == 'http'
    response = PlainTextResponse('Hello, world!')
    await response(scope, receive, send)
I run it with "uvicorn example:app", then measure rps using 'ab':
ab -n 1000 -c 10 127.0.0.1:8000/
 The result is on average 4200 requests per second.

I then run basic Tornado app, again with code copied directly from tornado website:
import tornado.ioloop
import tornado.web

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world")

def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
    ])

if __name__ == "__main__":
    app = make_app()
    app.listen(8888)
    tornado.ioloop.IOLoop.current().start()
 Using the same measurement, the ab result is a average 1900 requests/second.

So it looks like Starlette/uvicorn is more than 2 times faster than Tornado.   However, the code on Tornado website is actually not the best for production use.  Just replace "app.listen(8888)" the second to last line with following:
server = HTTPServer(app)
server.bind(8888)
server.start(0)
(and put "from tornado.httpserver import HTTPServer" at the beginning of the file),  its performance increases to 4700 requests/second, actually faster than Starlette.

The code change to Tornado make the app start 4 processes instead of 1.  Of course we can also do the similar for the Starlette app:
uvicorn --workers 4 example:app
The result now is 7500 requests/second, surpassing multi-process Tornado again.

These are superficial results that don't mean much.  But it did make me appreciate the works that got into Tornado that keep it performant in these years.

Result Summary:
  • Tornado (default): 1900 rps
  • Startlette/uvicorn (default): 4200 rps
  • Tornado (4 workers): 4700 rps
  • Starlette/uvicorn (4 workers): 7500 rps

12/19/17

DrJava Font Size Problem

I'm helping my son learning Java. One of the first things to do when learning Java is picking an IDE. I like DrJava for its simplicity and small size. However, when I first run DrJava, its font size is way too small on my 4K monitor.

There are 2 ways to fix this - the easy way and the hard way.

The easy way first.
Click Edit:Preferences, then click "Display Options", change "Look and Feel" to the one that ends with "Plastic3DLookandFeel".  Do this on Windows only, for Mac and Linux, default "Look and Feel" is fine.
Click "Font" under "Display Options", change all fonts to double the original size:
Press "OK", then close DrJava then open it again.  Now the fonts should be big enough to read.

(The changes you make are actually saved in a configuration file .drjava in your home directory,  you can edit the file directly for changes, but it's not recommended)

Now the hard way.
For the "easy way", we just changed DrJava's default configuration preferences.  The hard way is to compile a DrJava program with these changes already made in the source code so we don't have to change preferences.

Clone the DrJava repository on github: https://github.com/DrJavaAtRice/drjava

Try to compile it first:
cd drjava/drjava
ant jar

You must have ant already installed.  If it's compiled OK, try run it:
java -jar drjava.jar
(You can just double click drjava.jar file too)

Open src/edu/rice/cs/drjava/config/OptionConstants.java for editing:
Change "Monaco-12" to "Monaco-24"; change "Monospaced-12" to "Monospaced-24";
chanage "dialog-10" to "dialog-20"; change "dialog-12" to "dialog-24".

Save the file then recompile (ant jar).  Now DrJava is "pre-configured" with big fonts.

You might say this "hard way" is pointless.  Why on earth would anyone want to do this? I agree.  But just maybe one wants to provide a "pre-configured" copy of DrJava to his students.  I just show one way to do it.  Another way is to write a .drjava configuration file upon installation.  This is way more complex. Besides, I'm using a standalone jar file, there's no installation to speak of.


7/18/15

Bugzilla "Cannot determine local time zone" problem

I had this problem today and was pulling my hairs out trying to find a solution.
Below is the only fix that worked for me:
In 'lib/DateTime/TimeZone/Local.pm', find 'sub TimeZone {', right after it, put this next line:
return DateTime::TimeZone->new(name => 'Asia/Shanghai');
(Replace 'Asia/Shanghai' with your own timezone.)
Basically all the rest of this sub has no effect now.

Reload the error page, if it show another error says: "List::Util version 1.31 required...", open 'lib/List/AllUtils.pm', goto line 10, or find the line:
use List::Util 1.31 ();
Just delete '1.31'.

After that, it should work.  I'm using Bugzilla 4.4.9.

6/5/15

Cloning HDD to a smaller SSD for a Windows 8 Laptop

My win8 Dell Laptop has a 500GB HDD, I bought a 480GB SSD for hard disk replacement.  Unfortunately, the 500GB HDD's partitions span the whole disk, the exact layout copying wasn't possible.

I formatted the SSD to a GPT disk, then use various cloning softwares to do the cloning.  I tried Macrium Reflect Free, EaseUS Partition Master, Minitool Partition Wizard. None of them get me to Windows startup screen.

The last one I used is Minitool Partition Wizard. During boot, it said can't find OS.  So I tried changing different bios settings out of desperation and hoped one would work.  It did.

The setting is "disable virtualization".  I don't know why it worked, it doesn't make any sense, I'm just glad it did.

So if you have problem migrating HDD to SSD, this might be something you can try.

5/31/15

Re-sizing a Virtualbox disk image in Linux

One of my VirtualBox virtual machine had its disk full, I thought a simple resize should fix the problem. And I went:
vboxmanage modifyhd u140464-02-disk1.vmdk --resize 30000
0%...
Progress state: VBOX_E_NOT_SUPPORTED
VBoxManage: error: Resize hard disk operation for this format is not implemented yet!
What?!
Turned out "modifyhd" only support .vdi, not .vmdk. OK I will just use .vdi then:
vboxmanage clonehd u140464-02-disk1.vmdk u140464-02-disk1.vdi --format vdi
vboxmanage modifyhd u140464-02-disk1.vdi --resize 30000
Then detach vmdk disk from the VM and attach the vdi.
But after I boot it up, the disk was still full. The disk size wasn't changed. After some googling I found I need to install gparted and use it to grow the partition, and gparted has to be run at boot. Too much hassle.  All I want was some additional disk space. So I did it my way.
(Inside the VM)
sudo fdisk /dev/sda
then enter p, it showed:
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      499711      248832   83  Linux
/dev/sda2          501758    33552383    16525313    5  Extended
/dev/sda5          501760    33552383    16525312   8e  Linux LVM
Enter n to create a new partition, press enter 2 times to accept default partition size. For some reason, there was a stray segment, so I have to enter n again to create a new partition. 2 partitions were created, after a p, my partition table looked like this:
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048      499711      248832   83  Linux
/dev/sda2          501758    33552383    16525313    5  Extended
/dev/sda3          499712      501757        1023   83  Linux
/dev/sda4        33552384    61439999    13943808   83  Linux
/dev/sda5          501760    33552383    16525312   8e  Linux LVM
sda3 is useless, so I deleted it by a 'd' followed by a '3'. sda4 is the one I will use. Enter 'w' to write to disk. Then reboot the VM.

After reboot, do a:
sudo mkfs.ext4 /dev/sda4
My /opt was not used, so I just mount sda4 as /opt
sudo mount /dev/sda4 /opt
Add "/dev/sda4 /opt ext4 defaults 0 2" to the end of /etc/fstab so it was mounted automatically at boot.

In the end, I should have just add another virtual disk without bothering resizing the current one.