samadhiweb

smalltalk programming for the web



Ubuntu LVM root filesystem full

20 August 2017

I had carelessly allowed the root filesystem on one of my Ubuntu servers to become full. Here's how I recovered from it.

Setup: Ubuntu server 16.04, entire disk for LVM, all disk space assigned to one volume group (VG), root (4GB) and swap (1.5GB) in their own logical volumes (LV), with the remaining disk space allocated to LXD containers using LVM thin provisioning.

The root filesystem, from here on referred to as root if the context is clear, had become full because I hadn't been purging old Linux kernels. Being full, the filesystem was then unable to complete "apt-get autoremove". I needed to add some space to root, which I took from swap.

Following commands were performed online with the server in multi-user mode. First, turn off swap.

# swapoff -a

Then, reduce the size of swap LV by 1GB.

# lvresize -L-1G /dev/srv-vg/swap_1

Add the 1GB taken to root; the command grows the root LV and resizes the filesystem within the LV.

# lvextend -L+1G /dev/srv-vg/root -r -v

"df -k" shows that root now has an additional 1GB and is no longer full. Fix any broken apt dependencies and purge the old Linux kernels.

# apt-get -f install
# apt-get autoremove

"df -k" now shows that root has even more free space.

Now I set about to give the 1GB back to swap. While it is possible to grow root's LV and filesystem online, shrinking root needs to be done offline by booting from live DVD into single user mode at the server console. (The server runs virtually somewhere "in the cloud". Console access is via VNC. I use the Linux client Vinagre. Obligatory Smalltalk content: Vinagre is also the client I use to connect to RFBServer running in the Pharo image that runs this blog.)

For below commands, I'm at single user mode, booted from live DVD, with all server filesystems unmounted. First, check the root filesystem.

# e2fsck -f /dev/srv-vg/root

Remember the root filesystem is contained within the root LV. Now resize the filesystem. Here "4G" refers to the absolute size.

# resize2fs /dev/srv-vg/root 4G

Next, resize the root LV.

# lvresize -L-1G /dev/srv-vg/root

Check root filesystem again.

# e2fsck -fy /dev/srv-vg/root

Now, give 1GB back to swap.

# lvresize -L+1G /dev/srv-vg/swap_1

Reboot into multi-user mode, turn swap back on, and look around.

This incident happened to the server that is running this blog. That you are now reading this blog post means that the above procedure succeeded. :-P

LibreSSL and Pharo/Squeak SSL plugin

22 October 2015

According to its documentation, on Unix, the Pharo VM's SSL plugin, libSqueakSSL.so, links into OpenSSL libraries dynamically. On my 64bit Ubuntu Trusty machine, OpenSSL is provided by the libssl1.0.0:i386 package.

$ ldd libSqueakSSL.so
    linux-gate.so.1 =>  (0xf77a9000)
    libssl.so.1.0.0 => /lib/i386-linux-gnu/libssl.so.1.0.0 (0xf7727000)
    libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf7579000)
    libcrypto.so.1.0.0 -> /lib/i386-linux-gnu/libcrypto.so.1.0.0 (0xf73cb000)
    /lib/ld-linux.so.2 (0xf77a9000)
    libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xf73c6000)

(By the way, the SSH2 plugin libssh2.so.1.0.1 requires libcrypto too.)

According to packages.ubuntu.com, Trusty's libssl1.0.0 is built from openssl_1.0.1f.orig.tar.gz plus successive upstream patches.

From the OpenBSD developers, LibreSSL is "a version of the TLS/crypto stack forked from OpenSSL in 2014, with goals of modernizing the codebase, improving security, and applying best practice development processes." LibreSSL also comes with libtls, "a new TLS library, designed to make it easier to write foolproof applications".

Let's see how we go about linking libSqueakSSL.so with LibreSSL.

First, download and unpack LibreSSL. Modify the configure script at lines 2287 and 2289 so that LIBCRYPTO_VERSION and LIBSSL_VERSION both say 1:0:0 instead of 35:0:0. Then build LibreSSL:

$ CFLAGS=-m32 LDFLAGS=-m32 ./configure --disable-asm
$ make

I'm building on a 64bit OS, hence "-m32". Without "--disable-asm", the build fails. To get the assembler version, which is recommended for serious usage, either set up a 32bit build environment or muck around with autoconf/configure. I suspect the former is easier. :-)

The output files are $SRC/crypto/.libs/libcrypto.so.1.0.0 and $SRC/ssl/.libs/libssl.so.1.0.0. The shared object files have the "1.0.0" suffix because I modified configure above. Alternatively, I could've played around with autoconf, or built the shared objects with the "35.0.0" suffix and sym/hard-link them for the "1.0.0" versions. TIMTOWTDI.

Next, remove the OpenSSL package:

$ sudo apt-get remove libssl1.0.0:i386
$ ldd libSqueakSSL.so
    linux-gate.so.1 =>  (0xf7718000)
    libssl.so.1.0.0 => not found
    libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf7540000)
    /lib/ld-linux.so.2 (0xf7718000)

Finally, put the LibreSSL shared object files into the right place. Where this right place is depends on your environment. TIMTOWTDI. I choose to put them in the Pharo VM directory with its other plugins, and arrange to start Pharo with LD_LIBRARY_PATH set appropriately. Going by the output of ldd again, the following is required:

$ ln libcrypto.so.1.0.0 libcrypto.so.1

After which:

$ ldd libSqueakSSL.so
    linux-gate.so.1 =>  (0xf7709000)
    libssl.so.1.0.0 => /pkg/pharovm/libssl.so.1.0.0 (0xf7696000)
    libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf74c7000)
    libcrypto.so.1 -> /pkg/pharovm/libcrypto.so.1 (0xf72b8000)
    /lib/ld-linux.so.2 (0xf770a000)

Launch the Pharo 4.0 image and run the Zodiac tests. All tests should pass. Well, except testGetPharoVersion, which looks for a file that apparently no longer exists.

Incidentally, Squeak 5.0-All-in-One's SSL plugin appears to have linked its crypto/SSL libraries in statically, so the only way to upgrade is to build a new plugin.

$ ldd SqueakSSL
    linux-gate.so.1 =>  (0xf7736000)
    libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf7310000)
    /lib/ld-linux.so.2 (0xf7737000)

Pharo Smalltalk on Ubuntu 14.04 64bit

20 April 2014

I run Debian 6 64bit on one of my servers. To run the 32bit Pharo/Cog VM, 32bit libraries are needed and are installed thusly:

# apt-get install ia32-libs

Trying out the newly released Ubuntu 14.04, I learn that ia32-libs is considered a hack and was deprecated in 2013 versions of Ubuntu. I guess the idea is to encourage people to know what they require more precisely.

To get the current Pharo-VM-linux-stable.zip to work on Ubuntu 14.04 64bit, install the following packages:

# apt-get lib32gcc1 
# apt-get libssl0.9.8:i386

On libssl, that is, OpenSSL, Ubuntu also ships 1.0.1f-1ubuntu2, which patches the Heartbleed bug, so you may install that instead.

Deployment War Stories

28 October 2012

I used to run Squeak and subsequently Pharo on FreeBSD. I had the Smalltalk RFB server running within the image, and connected to the image via COTVNC (Chicken of the VNC), an OSX client. I ran Squeak/Pharo in a FreeBSD jail, and doing it this way kept the jail minimalist and tidy. No X stuff was installed.

Then I ran into some difficulty with FFI on CogVM on FreeBSD. (That's for another post.) And I was running FreeBSD 7, which EOLed. So I switched to Linux. Chose Debian 6 after trying out several distros. Similar setup as on FreeBSD: Minimalist LXC container, CogVM, RFB server in the same Pharo image that ran on FreeBSD.

Interestingly, neither COTVNC nor several other VNC clients (both within X on Linux and from OSX, including Java and native ones) worked in this setup: As far as I determined, the VNC connections were all failing because of incompatible encodings, and I tested quite a few combinations.

I wasn't interested in debugging too much, so I did the next simplest thing: Installed an Xvnc environment. So, instead of running an RFB server within Pharo, I had Pharo running within an X environment that was also a VNC server. And then all the VNC clients I was testing with worked. Not as tidy as I wanted, because now the LXC container had a whole bunch of X cruft, but hey, it was working.

But I really prefer FreeBSD. So after getting my production Linux server up and running, I went to investigate my FreeBSD CogVM FFI issue. And it turned out to be a process error on my part. I think. But that's for another post.

A note on my (fairly typical) setup: My development machine is an MBP. My development and pre-production FreeBSD and Linux servers all exist as VirtualBox or VMWare virtual machines on my MBP. The development servers have X, while the pre-production servers are meant to provide the canonical configuration for the production servers, so I run them without X, as much as possible. On FreeBSD, I have never had to run X in production. On Linux, I have the aforementioned Xvnc setup.

Anyhow, CogVM FFI FreeBSD was working again, in my development server running X. Next I tested it in my X-less pre-production server. When I invoked CogVM with "-headless", it said "cannot locate vm-display-X11" or words to that effect, even though said dynamic library was right there together with vm-display-null. I reckon that vm-display-X11 failed to locate an X environment, hence failed to load, hence the error message. Meaning, instead of -headless, I had to run -nodisplay.

So, CogVM ran -nodisplay, and COTVNC connected to it fine. Opened a workspace, typed some FFI-using code, selected the text with the mouse, pressed ESC. What's this? CogVM or Pharo crashed! PharoDebugLog recorded "SubclassResponsibility: NonInteractiveUIManager had the subclass responsibility to implement #newMenuIn:for:".

Hmmm. This didn't happen in Pharo 1.0 and 1.1, IIRC. Seems Pharo's random refactorings (heh, just kidding) changed how -nodisplay was managed between those versions and 1.4, the version I was testing.

After browsing a bit, looks like Pharo's graphical environment is managed by MorphicUIManager, but when invoked -nodisplay, even with a running RFB server, Pharo's graphical environment is managed by NonInteractiveUIManager.

One possibility to get this working is to install a minimal X environment, to run CogVM+Pharo -headless (with MorphicUIManager) instead of -nodisplay (with NonInteractiveUIManager).

But that isn't the simplest thing that could possibly work. What is? Copy & paste, of course. Specifically, I copied/pasted MorphicUIManager>>newMenuIn:for: into NonInteractiveUIManager. Ok, now pressing ESC brings up the context menu, "Do it" does, and output in the Transcript shows the expected FFI-related output.

Next I run my application's startup code in a workspace, one step of which is sending the "explore" message to the application's main instance, thus giving me a handle to said instance should I need to manipulate it, which is kinda the point of having VNC access into the image in the first place.

What's this? Crashed again! PharoDebugLog informs me "MessageNotUnderstood: NonInteractiveUIManager>>explorer:for:withLabel:" and "An attempt to use interactive tools detected, while in non-interactive mode".

Not so simple then...

(To be continued.)